Chinese word segmentation and named entity recognition: A pragmatic approach

Computational Linguistics

Volumn 31, Issue 4, 2005, Pages 531-574

  Gao, Jianfeng ^a   Wu, Andi ^b   Li, Mu ^a   Huang, Chang Ning ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b GrapeCity Inc

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; IMAGE SEGMENTATION; WORD PROCESSING;

APPLICATION-SPECIFIC STANDARDS; CHINESE LANGUAGE PROCESSING; CHINESE WORD SEGMENTATION; DIFFERENT GRANULARITIES; MATHEMATICAL FRAMEWORKS; NAMED ENTITY RECOGNITION; NATURAL LANGUAGE PROCESSING APPLICATIONS; STATE-OF-THE-ART PERFORMANCE;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 33646401779     PISSN: 08912017     EISSN: 15309312     Source Type: Journal    
DOI: 10.1162/089120105775299177     Document Type: Article

References (68)

1. Aho, Alfred V., Ravi Sethi, and Jeffrey D. Ullman. 1986. Compilers: Principles, Techniques, and Tools. Addison-Wesley, Boston.
2. Baayen, R. Harald. 1989. A Corpus-based Approach to Morphological Productivity: Statistical Analysis and Psycholinguistic Interpretation. Ph.D. thesis, Free University, Amsterdam.
3. Berger, Adam, Stephen Della Pietra, and Vincent J. Della Pietra. 1996. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39-71.
4. Brill, Eric. 1995. Transformation-based error-driven learning and natural language processing: A case study in Part-of-Speech tagging. Computational Linguistics, 21(4)543-565.
5. Chang, Eric, Jianlai Zhou, Yu Shi, and Chao Huang. 2001. Speech lab in a box: A Mandarin speech toolbox to Jumpstart speech related research. In Eurospeech'2001, pages 2799-2782, Aalborg, Denmark.
6. Chang, Jing-Shin and Keh-Yih Su. 1997. An unsupervised iterative method for Chinese new lexicon extraction. International Journal of Computational Linguistics and Chinese Language Processing, 2(2):97-148.
7. Chen, Aitao. 2003. Chinese word segmentation using minimal linguistic knowledge. In The Second SIGHAN Workshop on Chinese Language Processing. July 11-12, Sapporo, Japan.
8. Chen, Keh-Jiann and Ming-Hong Bai. 1998. Unknown word detection for Chinese by a corpus-based learning method. International Journal of Computational Linguistics and Chinese Language Processing, 3(1):27-44.
9. Chen, Stanley F. and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech and Language, 13:359-394.
10. Cheng, Kowk-Shing, Gilbert H. Yong, and Kam-Fai Wong. 1999. A study on word-based and integral-bit Chinese text compression algorithms. JASIS, 50(3):218-228.
11. Chien, Lee-Feng. 1997. PAT-tree-based keyword extraction for Chinese information retrieval. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, New York.
12. Chu, Min, Hu Peng, Yong Zhao, Zhengyu Niu, and Eric Chang. 2003. Microsoft Mulan - a bilingual TTS system. In Proceedings of the 2003 International Conference on Acoustics, Speech and Signal Processing, vol. 1, pages 264-267.
13. Collins, Michael and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in Neural Information Processing Systems (NIPS 14), pages 625-632, MIT Press.
14. Collins, Michael. 2002. Discriminative training methods for Hidden Markov Models: theory and experiments with the perception algorithm. In Proceedings of the 2002 Conference on Empirical Method in Natural Language Processing, pages 1-8.
15. Collins, Michael. 2004. Parameter estimation for statistical parsing models: theory and practice of distribution-free methods. Chapter 1 in Harry Bunt, John Carroll and Giorgio Satta, editors, New Developments in Parsing Technology, Kluwer.
16. Dai, Yubin, Christopher S. G. Khoo, and Tech Ee Loh. 1999. A new statistical formula for Chinese word segmentation incorporating contextual information. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 82-89.
17. Della Pietra, Stephen, Vincent Della Pietra, and John Lafferty. 1997. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19, 380-393.
18. Duda, Richard O., Peter E. Hart, and David G. Stork. 2001. Pattern Classification. John Wiley & Sons, Inc.
19. Dudewicz, Edward J. and Satya N. Mishra. 1988. Modern Mathematical Statistics. Wiley Series in Probability and Mathematical Statistics, John Wiley & Sons, Inc.
20. Gao, Jianfeng and Kai-Fu Lee. 2000. Distribution based pruning of backoff language models. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, October 3-6, Hong Kong, pages 579-586.
21. Gao, Jianfeng, Joshua Goodman, and Jiangbo Miao. 2001. The use of clustering techniques for language modeling-application to Asian language. Computational Linguistics and Chinese Language Processing, 6(1):27-60.
22. Gao, Jianfeng, Joshua Goodman, Mingjing Li, and Kai-Fu Lee. 2002. Toward a unified approach to statistical language modeling for Chinese. ACM Transactions on Asian Language Information Processing, 1(1):3-33.
23. Gao, Jianfeng, Mu Li, and Chang-Ning Huang. 2003. Improved source-channel model for Chinese word segmentation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, July 7-12, Sapporo, Japan, pages 272-280.
24. Gao, Jianfeng, Andi Wu, Mu Li, Chang-Ning Huang, Hongqiao Li, Xinsong Xia, and Haowei Qin. 2004. Adaptive Chinese word segmentation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, July 21-26, Barcelona, pages 462-469.
25. Gao, Jianfeng, Hao Yu, Peng Xu, and Wei Yuan. 2005. Minimum sample risk methods for language modeling. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing.
26. Frakes, William B. and Ricardo Baeza-Yates (eds.). 1992. Information Retrieval. Prentice Hall, Englewood Cliffs, NJ.
27. Fung, Pascale and Dekai Wu. 1994. Statistical augmentation of a Chinese machinereadable dictionary. In Proceeding of the Second Annual Workshop on Very Large Corpora, Kyoto, pages 180-181.
28. Katz, S. M. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE ASSP 35(3):400-401.
29. Katz, S. M. 1996. Distribution of content words and phrases in text and language modeling. Natural Language Engineering, 1996(2):15-59.
30. Kneser, Reinhard and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceeding of the IEEE International Conference on Acoustics, Speech, and Signal Processing, pages 181-184.
31. Krauth, W. and M. Mèzard. 1987. Learning algorithm with optimal stability in neural networks. Journal of Physics A, 20:745-752.
32. Hockenmaier, Julia and Chris Brew. 1998. Error driven segmentation of Chinese. In Communications of COLIPS, 8(1):69-84.
33. Huang, Chu-Ren, Keh-Jiann Chen, Chang Lili, and Feng-Yi Chen. 1997. Segmentation standard for Chinese natural language processing. International Journal of Computational Linguistics and Chinese Language Processing, 2(2):47-62.
34. Joachims, Thorsten. 2002. Learning to Classify Text Using Support Vector Machines. Kluwer.
35. Li, Hongqiao, Chang-Ning Huang, Jianfeng Gao, and Xiaozhong Fan. 2004. The use of SVM for Chinese new word identification. In Proceedings of the First International Joint Conference on Natural Language Processing, Sanya City, Hainan Island, China, March 22-24, pages 497-504.
36. Li, Mu, Jianfeng Gao, Chang-Ning Huang, and Jianfeng Li. 2003. Unsupervised training for overlapping ambiguity resolution in Chinese word segmentation. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, July 11-12, Sapporo, Japan, pages 1-7.
37. Liang, Nanyuan. 1987- CDWS (A written Chinese automatic segmentation system). In (Journal of Chinese Information Processing), 2:44-52.
38. Lin, Ming-Yu, Tung-Hui Chiang, and Keh-Yi Su. 1993. A preliminary study on unknown word problem in Chinese word segmentation. In Proceedings of the ROC Computational Linguistics Conference VI, 119-141.
39. Manning, Christopher D. and Hinrich Schutze. 1999. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA.
40. Mitchell, Tom M. 1997. Machine Learning. McGraw-Hill, New York.
41. Nie, Jian-Yun, Wanying Jin, and Mareie-Louise Hannan. 1994. A hybrid approach to unknown word detection and segmentation of Chinese. In Proceedings of the International Conference on Chinese Computing, pages 326-335, Singapore.
42. Och, Franz. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, July 7-12, Sapporo, Japan, pages 160-167.
43. Packard, Jerome. 2000. The Morphology of Chinese: A Linguistics and Cognitive Approach. Cambridge University Press, Cambridge.
44. Palmer, David D. 1997. A trainable rule-based algorithm for word segmentation. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, Madrid, pages 321-328.
45. Peng, Fuchun, Fangfang Feng, and Andrew McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In COLING 2004, pages 562-568.
46. Roche, Emmanuel and Yves Schabes. 1997. Finite-State Language Processing. MIT Press, Cambridge, MA.
47. Rosenfeld, Roland, Stanley F. Chen, and Xiaojin Zhu. 2001. Whole sentence exponential language models: A vehicle for linguistic statistical integration. Computer Speech and Language, 15(1)55-73.
48. Sproat, Richard and Tom Emerson. 2003. The first international Chinese word segmentation bakeoff. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, July 11-12, Sapporo, Japan.
49. Sproat, Richard and Chilin Shih. 2002. Corpus-based methods in Chinese morphology and phonology. In Proceedings of the 2002 International Conference on Computational Linguistics.
50. Sproat, Richard, Alan W. Black, Stanley Chen, Shankar Kumar, Mari Ostendorf, and Christoper Richards. 2001. Normalization of non-standard words. Computer Speech and Languages, 15:287-333.
51. Sproat, Richard, Chilin Shih, William Gale, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics, 22(3):377-404.
52. Sproat, Richard. 1992. Morphology and Computation. MIT Press, Cambridge, MA.
53. Sproat, Richard and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4:336-351.
54. Su, Keh-Yih and Chin-Hui Lee. 1994. Speech recognition using weighted HMM and subspace projection approaches. IEEE Trans., Speech and Audio Processing, 2(1):69-79.
55. Sun, Jian, Jianfeng Gao, Lei Zhang, Ming Zhou, and Chang-Ning Huang. 2002. Chinese named entity identification using class-based language model. In Proceedings of the 2002 International Conference on Computationl Linguistics, pages 967-973.
56. Sun, Jian, Ming Zhou, and Jianfeng Gao. 2003. Chinese named entity identification using class-based language model. International Journal of Computational Linguistics and Chinese Language Processing, 8(2):1-28.
57. Sun, Maosong. and Benjamin K. Tsou. 2001. (A review and evaluation on automatic segmentation of Chinese). (Contemporary Linguistics), 3(1):22-32.
58. Sun, Maosong, Dayang Shen, and Benjamin K. Tsou. 1998. Chinese word segmentation without using lexicon and hand-crafted training data. In Proceedings of the 17th International Conference on Computational Linguistics, Montreal, Quebec, Canada, pages 1265-1271.
59. Sun, Maosong and Zhengping Zuo. 1998. Overlapping ambiguity in Chinese text. (In Chinese) Quantitative and Computational Studies on the Chinese Language, HK, pages 323-338.
60. Teahan, W. J., Yingying Wen, Rodger McNab, and Ian Witten. 2000. A compression-based algorithm for Chinese word segmentation. Computational Linguistics, 26(3):375-393.
61. Uchimoto, Kiyotaka, Satoshi Sekine, and Hitoshi Isahara. 2001. The unknown word problem: a morphological analysis of Japanese using maximum entropy aided by a dictionary. In Proceedings of the 2001 Conference on EMNLP, pages 91-99.
62. Vapnik, Vladimir N. 1998. Statistical Learning Theory. John Wiley & Sons, Inc. Wu, Andi and Zixin Jiang. 2000.
63. Statistically-enhanced new word identification in a rule-based Chinese system. In Proceedings of the Second ACL Chinese Processing Workshop, pages 41-66.
64. Wu, Andi. 2003. Customizable segmentation of morphologically derived words in Chinese. In International Journal of Computational Linguistics and Chinese Language Processing, 8(1):1-27.
65. Wu, Zimin and Gwyneth Tseng. 1993. Chinese text segmentation for text retrieval achievements and problems. JASIS, 44(9): 532-542.
66. Xia, Fei. 1999. Segmentation guideline, Chinese Treebank Project. Technical report, University of Pennsylvania.
67. Xue, Nianwen. 2003. Chinese word segmentation as character tagging. International Journal of Computational Linguistics and Chinese Language Processing, 8(1):29-48.
68. Zhu, Xiaodan, Mu Li, Jianfeng Gao, and Chang-Ning Huang. Single character Chinese named entity recognition. In Proceedings of the Second SIGHAN Workshop on Chinese Language Proceeding, July 11-12, Sapporo, Japan, pages 125-132.