Extract Chinese Unknown Words from a Large-scale Corpus Using Morphological and Distributional Evidences

IJCNLP 2011 - Proceedings of the 5th International Joint Conference on Natural Language Processing

Volumn , Issue , 2011, Pages 837-845

  Zhang, Kaixu ^a   Wang, Ruining ^a   Xue, Ping ^b   Sun, Maosong ^a  

^a TSINGHUA UNIVERSITY   (China)

^b BOEING COMPANY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; NATURAL LANGUAGE PROCESSING SYSTEMS;

ACCESSOR VARIETIES; CHINESE CHARACTERS; CHINESE UNKNOWN WORDS; CHINESE WEB; CHINESE WORD SEGMENTATION; LARGE-SCALES; ON-THE-FLY; PERFORMANCE; SEGMENTATION MODELS; WEB CORPORA;

EXTRACTION;

EID: 85065661385     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (22)

1. J. Aoe, K. Morimoto, and T. Sato. 1992. An efficient implementation of trie structures. Software: Practice and Experience, 22(9):695-721, September.
2. J. S Chang and K. Y Su. 1997. An unsupervised iterative method for chinese new lexicon extraction. International Journal of Computational Linguistics & Chinese Language Processing, 1(1):101-157.
3. K. J. Chen and W. Y. Ma. 2002. Unknown word extraction for chinese documents. In Proceedings of the 19th international conference on Computational linguistics-Volume 1, pages 1-7. Association for Computational Linguistics.
4. M. Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, page 1-8.
5. T. Emerson. 2005. The second international chinese word segmentation bakeoff. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 123-133. Jeju Island, Korea.
6. H. Feng, K. Chen, X. Deng, and W. Zheng. 2004a. Accessor variety criteria for chinese word extraction. Computational Linguistics, 30(1):75-93.
7. H. Feng, K. Chen, C. Kit, and X. Deng. 2004b. Unsupervised segmentation of chinese corpus using accessor variety. Natural Language Processing-IJCNLP 2004, pages 694-703.
8. C. M Hong, C. M Chen, and C. Y Chiu. 2009. Automatic extraction of new words based on google news corpora for supporting lexicon-based chinese word segmentation systems. Expert Systems with Applications, 36(2):3641-3651.
9. W. Jiang, L. Huang, and Q. Liu. 2009. Automatic adaptation of annotation standards: Chinese word segmentation and POS tagging ╟a case study. In Proceedings of the 47th ACL, page 522-530, Suntec, Singapore, August. Association for Computational Linguistics.
10. Z. Jin and K. Tanaka-Ishii. 2006. Unsupervised segmentation of chinese text by use of branching entropy. In Proceedings of the COLING/ACL on Main conference poster sessions, pages 428-435. Association for Computational Linguistics.
11. Z. Li and M. Sun. 2009. Punctuation as implicit annotations for chinese word segmentation. Computational Linguistics, 35(4):505-512.
12. G. O.H.C Ling, M. Asahara, and Y. Matsumoto. 2003. Chinese unknown word identification using character-based tagging and chunking. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics-Volume 2, page 197-200.
13. Y. Liu, B. Wang, F. Ding, and S. Xu. 2008. Information retrieval oriented word segmentation based on character associative strength ranking. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1061-1069. Association for Computational Linguistics.
14. W. Y. Ma and K. J. Chen. 2003. A bottom-up merging algorithm for chinese unknown word extraction. In Proceedings of the second SIGHAN workshop on Chinese language processing-Volume 17, pages 31-38. Association for Computational Linguistics.
15. A. McCallum and W. Li. 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003-Volume 4, page 188-191.
16. H. T. Ng and J. K. Low. 2004. Chinese part-of-speech tagging: one-at-a-time or all-at-once? word-based or character-based. In Proc of EMNLP.
17. F. Peng, F. Feng, and A. McCallum. 2004. Chinese segmentation and new word detection using conditional random fields. In Proceedings of the 20th international conference on Computational Linguistics, page 562. Association for Computational Linguistics.
18. M. Sun, D. Shen, and B. K Tsou. 1998. Chinese word segmentation without using lexicon and hand-crafted training data. In Proceedings of the 17th international conference on Computational linguistics-Volume 2, pages 1265-1271. Association for Computational Linguistics Morristown, NJ, USA.
19. H. Tseng, P. Chang, G. Andrew, D. Jurafsky, and C. Manning. 2005. A conditional random field word segmenter for sighan bakeoff 2005. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 168-171. Jeju Island, Korea.
20. N. Xue. 2003. Chinese word segmentation as character tagging. Computational Linguistics and Chinese Language Processing, 8(1):29-48.
21. K. Zhang, M. Sun, and P. Xue. 2010. A local generative model for chinese word segmentation. Information Retrieval Technology, pages 420-431.
22. H. Zhao and C. Kit. 2008. An empirical comparison of goodness measures for unsupervised chinese word segmentation with a unified framework. In The Third International Joint Conference on Natural Language Processing (IJCNLP-2008), Hyderabad, India.