Improving Chinese word segmentation with description length gain

Proceedings of the 2007 International Conference on Artificial Intelligence, ICAI 2007

Volumn 2, Issue , 2007, Pages 846-851

  Kit, Chunyu ^a   Zhao, Hai ^a  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

Chinese word segmentation; Conditional random fields; Description length gain

Indexed keywords

CHINESE WORD SEGMENTATION; CONDITIONAL RANDOM FIELD; DESCRIPTION LENGTH GAIN; LEXICAL INFORMATION; SEGMENTATION PERFORMANCE; STATE-OF-THE-ART PERFORMANCE;

IMAGE SEGMENTATION; RANDOM PROCESSES;

ARTIFICIAL INTELLIGENCE;

EID: 77958111978     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (31)

1. M. R. Brent. An efficient, probabilistically sound algorithm for segmentation and word discovery. Machine Learning, 34:71-105, February 1999.
2. B. Carpenter. Character language models for Chinese word segmentation and named entity recognition. In Proceedings of the Fifth SIGH AN Workshop on Chinese Language Processing, pages 169-172. Sydney, Australia, July 2006. Association for Computational Linguistics.
3. L.-F. Chien. PAT-trec-based keyword extraction for Chinese information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 50-58, Philadelphia, 1997. (Pubitemid 127720304)
4. T. M. Cover and J. A. Thomas. Elements of Information Theory. John Wiley and Sons, Inc., New York, 1991.
5. X. Ge, W. Pratt, and P. Smyth. Discovering Chinese words from unsegmented text. In SIGIR '99: Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 271-272, Berkeley, CA, USA, August 15-19, 1999. ACM.
6. S. Goldwater, T. L. Griffiths, and M. Johnson. Contextual dependencies in unsupervised word segmentation. In COLING-ACL 2006, pages 673-670, Sidney, Australia, 2006.
7. C. Kit. Unsupervised Lexical Learning as Inductive Inference. PhD thesis, University of Sheffield, 2000.
8. C. Kit and X. Liu. An example-based Chinese word segmentation system for CWSB-2. In Proceedings of the Fourth SIGH AN Workshop on Chinese Language Processing, pages 146-149, Jeju Island, Korea, 2005.
9. C. Kit and Y. Wilks. Unsupervised learning of word boundary with description length gain. In M. Osborne and E. T. K. Sang, editors, CoNLL-99, pages 1-6, Bergen, Norway, 1999.
10. J. D. Lafferty, A. McCallum, and F. C. N. Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In IGML'Ol: Proceedings of the 18th International Conference on Machine Learning, pages 282-289, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc.
11. G.-A. Levow. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sidney, Australia, 2006.
12. J. K. Low, H. T. Ng, and W. Guo. A maximum entropy approach to Chinese word segmentation. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 161-164, Jeju Island, Korea, 2005.
13. F. Peng, F. Feng, and A. McCallum. Chinese segmentation and new word detection using conditional random fields. In COLING 2004, pages 562-568, Geneva, Switzerland, 2004.
14. F. Peng and D. Schuurmans. Self-supervised Chinese word segmentation. In The Forth International Symposium on Intelligent Data Analysis, pages 238-247, Lisbon, Portugal, September 2001. (Pubitemid 33348503)
15. J. Rissanen. Modelling by shortest data description. Automatica, 14:465-471, 1978.
16. J. Rissanen. Stochastic Complexity in Statistical Inquiry. World Scientific, N.J., 1989.
17. B. Rosenfeld, R. Feldman, and M. Fresko. A systematic cross-comparison of sequence classifiers. In Proceedings of the Sixth SIAM International Conference on Data Mining (SDM06), pages 563-567, Bethesda, Maryland, 2006.
18. C. E. Shannon. A mathematical theory of communication. The Bell System Technical Journal, 27:379-423, 623-656, July, October 1948.
19. R. Sproat and T. Emerson. The first international Chinese word segmentation bakeoff. In The Second SIGHAN Workshop on Chinese Language Processing, pages 133-143, Sapporo, Japan, 2003.
20. R. Sproat and C. Shih. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4(4):336-351, 1990.
21. M. Sun, D. Shen, and B. K. Tsou. Chinese word segmentation without using lexicon and hand-crafted training data. In COLING-ACL'98, volume 2, pages 1265-1271, Montreal, Quebec, Canada, 1998.
22. R. T.-H. Tsai, H.-C. Hung, C.-L. Sung, H.-J. Dai, and W.-L. Hsu. On closed task of Chinese word segmentation: An improved CRF model coupled with character clustering and automatically generated template matching. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117, Sidney, Australia, 2006.
23. H. Tseng, P. Chang, G. Andrew, D. Jurafsky, and C. Manning. A conditional random field word seg-menter for SIGHAN bakeoff 2005. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 168-171, Jeju Island, Korea, 2005.
24. X. Wang, X. Lin, D. Yu, H. Tian, and X. Wu. Chinese word segmentation with maximum entropy and n-gram language model. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 138-141, Sidney, Australia, 2006.
25. N. Xue. Chinese word segmentation as character tagging. Computational Linguistics and Chinese Language Processing, 8(1):29-48, 2003.
26. M. Yamamoto and K. W. Church. Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics, 27(1):1-30, 2001.
27. J. Zhang, J. Gao, and M. Zhou. Extraction of Chinese compound words - an experimental study on a very large corpus. In Proceedings of the Second Chinese Language Processing Workshop, pages 132-139, Hong Kong, China, 2000.
28. M. Zhang, G.-D. Zhou, L.-P. Yang, and D.-H. Ji. Chinese word segmentation and named entity recognition based on a context-dependent mutual information independence model. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 154-157, Sidney, Australia, 2006.
29. H. Zhao, C.-N. Huang, and M. Li. An improved Chinese word segmentation system with conditional random field. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 162-165, Sidney, Australia, 2006.
30. H. Zhao, C.-N. Huang, M. Li, and B.-L. Lu. Effective tag set selection in Chinese word segmentation via conditional random field modeling. In Proceedings of PACLIC-20, pages 87-94, Wuhan, China, 2006.
31. M.-H. Zhu, Y.-L. Wang, Z.-X. Wang, H.-Z. Wang, and J.-B. Zhu. Designing special post-processing rules for SVM-based Chinese word segmentation. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 217-220, Sidney, Australia, 2006.