A compression-based algorithm for Chinese word segmentation

Computational Linguistics

Volumn 26, Issue 3, 2000, Pages 374-393

  Teahan, W J ^a   McNab, Rodger ^b   Wen, Yingying ^b   Witten, Ian H ^b  

^a ROBERT GORDON UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF WAIKATO   (New Zealand)

Author keywords

[No Author keywords available]

Indexed keywords

CHINESE LANGUAGE; CHINESE WORD SEGMENTATION; DELIMITERS; FULL-TEXT SEARCH; GENERAL METHOD; KEYPHRASE EXTRACTION; LANGUAGE MODEL; TEXT COMPRESSIONS;

COMPUTATIONAL LINGUISTICS;

EID: 0001277731     PISSN: 08912017     EISSN: None     Source Type: Journal    
DOI: 10.1162/089120100561746     Document Type: Article

References (27)

1. Anderson, John B. and Seshadri Mohan. 1984. Sequential coding algorithms: A survey and cost analysis. IEEE Transactions on Communications, 32(2):169-176.
2. Bell, Timothy C., John G. Cleary, and Ian H. Witten. 1990. Text Compression. Prentice Hall, Englewood Cliffs, NJ.
3. Brill, Eric. 1995. Transformation-based error-driven learning and natural processing: A case study in part-of-speech tagging. Computational Linguistics, 21(4):543-565.
4. Broglio, John, Jamie P. Callan, and W. Bruce Croft. 1994. Inquery system overview. In Proceedings TIPSTER Text Program (Phase I). Morgan Kaufmann, San Francisco, CA, pages 47-67.
5. Broglio, John, Jamie P. Callan, and W. Bruce Croft. 1996. Technical issues in building an information system for Chinese. CIIR Technical Report IR-86, Center for Intelligent Information Retrieval, University of Massachusetts, Amherst.
6. Cheng, Kwok-Shing, Gilbert H. Young, and Kam-Fai Wong. 1999. A study on word-based and integral-bit Chinese text compression algorithms. Journal of the American Society for Information Science, 50(3):218-228.
7. Cleary, John G. and Ian H. Witten. 1984. Data compression using adaptive coding and partial string matching. IEEE Transactions on Communications, 32(4):396-402.
8. Cormen, Thomas H., Charles E. Leiserson, and Ronald L. Rivest. 1990. Introduction to Algorithms. MIT Press, Cambridge, MA.
9. Dai, Yubin, Christopher S. G. Khoo, and Teck Ee Loh. 1999. A new statistical formula for Chinese text segmentation incorporating contextual information. In Proceedings of ACM SIGIR99, pages 82-89.
10. Frakes, William B. 1992. Stemming algorithms. In William B. Frakes and Ricardo Baeza-Yates, editors, Information Retrieval: Data Structures and Algorithms. Prentice Hall, Englewood Cliffs, NJ, pages 131-160.
11. Frank, E., Gordon W. Paynter, Ian H. Witten, Carl Gutwin, and Craig G. Nevill-Manning. 1999. Domain-specific keyphrase extraction. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 668-673. Stockholm, Sweden. Morgan Kaufmann, San Francisco, CA.
12. Hockenmaier, Julia and Chris Brew. 1998. Error driven segmentation of Chinese. Communications of COLIPS, volume 1, number 1, pages 69-84.
13. Howard, Paul Glor. 1993. The Design and Analysis of Efficient Lossless Data Compression Systems. Ph.D thesis, Brown University, Providence, RI.
14. Klavans, Judith L. and Philip Resnik, editors. 1997. The Balancing Act: Combining Symbolic and Statistical Approaches to Language. MIT Press, Cambridge, MA.
15. Kucera, Henry and W. Nelson Francis. 1967. Computational Analysis of Present-Day American English. Brown University Press, Providence, RI.
16. Liu, Yongquan. 1987. New advances in computers and natural language processing in China. Information Science, 8:64-70.
17. Lee, Kin Hong, Mau Kit Michael Ng, and Qin Lu. 1999. Text segmentation for Chinese spell checking. Journal of the American Society for Information Science, 50(9):751-759.
18. Palmer, David. 1997. A trainable rule-based algorithm for word segmentation. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL '97), Madrid, 1997.
19. Ponte, Jay M. and W. Bruce Croft. 1996. USeg: A retargetable word segmentation procedure for information retrieval. Presented at the Symposium on Document Analysis and Information Retrieval '96 (SDAIR). UMass Technical Report TR96-2, University of Massachusetts, Amherst, MA.
20. Rabiner, Lawrence R. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. In Proceedings of the IEEE, 77(2):257-286.
21. Sproat, Richard, Chilin Shih, William Gail, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics, 22(3):377-404.
22. Teahan, W. J. 1998. Modelling English Text. Ph.D thesis, University of Waikato, NZ.
23. Teahan, W. J., S. Inglis, John G. Cleary, and G. Holmes. 1998. Correcting English text using PPM models. In J. A. Storer and J. H. Reif, editors, Proceeding Data Compression Conference, pages 289-298, Los Alamitos, CA. IEEE Computer Society Press.
24. Witten, Ian H., Alistar Moffat, and Timothy C. Bell. 1999. Managing gigabytes: compressing and indexing documents and images. Second edition. Morgan Kaufmann, San Francisco, CA.
25. Wu, Dekai. 1998. A position statement on Chinese segmentation. Presented at the Chinese Language Processing Workshop, University of Pennsylvania, PA.
26. Wu, Dekai and Pascale Fung. 1994. Improving Chinese tokenization with linguistic filters on statistical lexical acquisition. In ANLP-94, Fourth Conference on Applied Natural Language Processing, pages 180-181, Stuttgart, October 94.
27. Wu, Zimin and Gwyneth Tseng. 1993. Chinese text segmentation for text retrieval achievements and problems. JASIS, 44(9)532-542.