Applying machine learning to text segmentation for information retrieval

Information Retrieval

Volumn 6, Issue 3-4, 2003, Pages 333-362

  Huang, Xiangji ^a   Peng, Fuchun ^a   Schuurmans, Dale ^a   Cercone, Nick ^a   Robertson, Stephen E ^b,c  

^a UNIVERSITY OF WATERLOO   (Canada)

^b MICROSOFT RESEARCH   (United Kingdom)

^c CITY UNIVERSITY   (United Kingdom)

Author keywords

Chinese information retrieval; EM algorithm; Machine learning; Word segmentation

Indexed keywords

EID: 3843085056     PISSN: 13864564     EISSN: None     Source Type: Journal    
DOI: 10.1023/a:1026028229881     Document Type: Article

References (40)

1. Beaulieu M, Gatford M, Huang X, Robertson S, Walker S and Williams P (1997) Okapi at TREC-5. In: Harman DK, Ed., Proceedings of TREC-5, pp. 143-166.
2. Bell T, Cleary J and Witten I (1990) Text Compression. Prentice Hall.
3. Brent M and Tao X (2001) Chinese text segmentation with MBDP-1: Making the most of training corpora. In: Proceedings of ACL2001, France.
4. Buckley C, Singhal A and Mitra M (1997) Using query zoning and correlation within SMART: TREC-5. In: Proceedings of TREC-5, pp. 105-118.
5. Buckley C, Walz J, Mitra M and Cardie C (1998) Using clustering and superConcepts within SMART: TREC-6. In: Proceedings of TREC-6, pp. 107-124.
6. Chang J-S and Su K-Y (1997) An unsupervised iterative method for Chinese new Lexicon extraction. International Journal of Computational Linguistics & Chinese Language Processing.
7. Chen A, He J, Xu L, Gey FC and Meggs J (1997) Chinese text retrieval without using a dictionary. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp. 42-49.
8. Chien L-F, Huang T-I and Chien M-C (1997) Pat-tree-based keyword extraction for Chinese information retrieval. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp. 50-58.
9. Cleary J and Witten I (1984) Data compression using adaptive coding and partial string matching. IEEE Trans on Communications, 32(4):396-402.
10. Dahan D and Brent M (1999) On the discovery of novel word-like units from utterances: An artificial-language study with implications for native-language acquisition. Journal of Experimental Psychology: General, 128:165-185.
11. Dempster A, Laird N and Rubin D (1977) Maximum-likelihood from incomplete data via the EM algorithm. J. Royal Statist. Soc. Ser., B(39).
12. Foo S and Li H (2001) Chinese word segmentation accuracy and its effects on information retrieval. TEXT Technology.
13. Fung P (1998) Extracting key terms from Chinese and Japnese text. International Journal on Computer Processing of Oriental Language, Special Issue on Information Retrieval on Oriental Languages, 99-121.
14. Ge X, Pratt W and Smyth P (1999) Discovering Chinese words from unsegmented text. In: Proceedings of the 22th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 271-272.
15. Gey F, Chen A, He J, Xu L and Meggs J (1996) Term importance, boolean conjunct training, negative terms, and foreign language retrieval: Probabilistic algorithms at TREC-5. In: Proceedings of the Fifth Text REtrieval Conference, pp. 181-190. NIST special publication.
16. Hockenmaier J and Brew C (1998) Error driven segmentation of Chinese. Communications of COLIPS, 8(1):69-84.
17. Hofmann T (1999) Probabilistic latent semantic indexing. In: Proceedings of the 22th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM.
18. Huang X and Roberton S (1998) Okapi Chinese text retrieval experiments at TREC-6. In: Proceedings of TREC-6, pp. 137-142.
19. Huang X and Robertson S (2000) A probabilistic approach to Chinese information retrieval: Theory and experiments. In: Proceedings of the BCS-IRSG 2000: the 22nd Annual Colloquium on Information Retrieval Research, Cambridge, England.
20. Jin W (1992) Chinese segmentation and its disambiguation. In: MCCS-92-227, Computing Research Laboratory, New Mexico State University, Las Cruces, New Mexico.
21. Kwok KL (1997) Comparing representations in Chinese information retrieval. In: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp. 34-41.
22. Kwok KL (1999) Employing multiple representations for Chinese information retrieval. Journal of the American Society for Information Science (JASIS), 50(8):709-723.
23. Kwok KL (2000) Improving English and Chinese ad-hoc retrieval: A tipster text phase 3 project report. Information Retrieval, 3(4):313-338.
24. Kwok KL and Grunfeld L (1996) TREC-5 English and Chinese retrieval experiments using PIRCS. In: Proceedings of TREC-5.
25. Manning C and Schütze H (1999). Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, Massachusetts.
26. Nie JY, Gao J, Zhang J, and Zhou M (2000) On the use of words and n-grams for chinese information retrieval. In: Proceedings of the Fifth International Workshop on Information Retrieval with Asian Languages (IRAL), Hong kong.
27. Nie JY and Ren F (1999) Chinese information retrieval: Using characters or words? Information Processing and Management, 35:443-462.
28. Nie JY, Ren X and Brisebois M (1996) On Chinese text retrieval. In: Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM, pp. 225-233.
29. Palmer D and Burger J (1997) Chinese word segmentation and information retrieval. In: AAAI Spring Symposium on Cross-Language Text and Speech Retrieval, Electronic Working Notes.
30. Peng F and Schuurmans D (2001) Self-supervised Chinese word segmentation. In: Hoffman F et al., Eds., Advances in Intelligent Data Analysis, Proceedings of the Fourth International Conference (IDA-01), LNCS 2189, Springer-Verlag, Berlin, Heidelberg, Cascais, Portugal, pp. 238-247.
31. Ponte J and Croft W (1996) Useg: A retargetable word segmentation procedure for information retrieval. In: Symposium on Document Analysis and Information Retrival 96 (SDAIR).
32. Rabiner L (1989) A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of IEEE, 77(2).
33. Robertson SE and Walker S (1994) Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In: Proceedings of the 17th Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, pp. 232-241.
34. Sparck-Jones K (1979) Search relevance weighting given little relevance information. Journal of Documentation, 35(1).
35. Sparck-Jones K (1991) The role of artificial intelligence in information retrieval. Journal of the American Society for Information Science, 42(8):558-565.
36. Sproat R and Shih C (1990) A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Language, 4:336-351.
37. Teahan WJ, Wen Y, McNab R and Witten IH (2001) A compression-based algorithm for Chinese word segmentation. Computational Linguistics, 26(3):375-393.
38. Voorhees E and Harman D (1998) Overview of the Sixth Text REtrieval Conference (TREC-6). In: Proceedings of the Sixth Text REtrieval Conference (TREC-6). NIST Special Publication.
39. Wilkinson R (1998) Chinese document retrieval at TREC-6. In: Proceedings of TREC-6.
40. Wu Z and Tseng G (1993) Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science, 44(9):532-542.