A unified character-based tagging framework for chinese word segmentation

ACM Transactions on Asian Language Information Processing

Volumn 9, Issue 2, 2010, Pages

  Zhao, Hai ^a   Huang, Chang Ning ^b   Li, Mu ^b   Lu, Bao Liang ^a  

^a SHANGHAI JIAO TONG UNIVERSITY   (China)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Assistant segmenter; Character based tagging method; Chinese word segmentation; Conditional random field; Tag set selection

Indexed keywords

ACTIVE AREA; CHINESE LANGUAGE; CHINESE WORD SEGMENTATION; CONDITIONAL RANDOM FIELD; ENSEMBLE LEARNING; FEATURE TEMPLATE; LINGUISTIC RESOURCES; SEGMENTER; STATE-OF-THE-ART PERFORMANCE; TAGGING PROBLEM; TRAINING CORPUS;

IMAGE SEGMENTATION; LEARNING ALGORITHMS; LINGUISTICS; SOFTWARE AGENTS;

FEATURE EXTRACTION;

EID: 77953745128     PISSN: 15300226     EISSN: 15583430     Source Type: Journal    
DOI: 10.1145/1781134.1781135     Document Type: Article

References (50)

1. ASAHARA, M., GOH, C. L., WANG, X., AND MATSUMOTO, Y., 2003. Combining segmenter and chunker for Chinese word segmentation. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 144-147.
2. BRILL, E., 1995. Transformation-based error-driven learning and natural language processing: A case study in part-of-speech tagging. Comput. Linguist. 21, 4, 543-565.
3. CHEN, A., 2003. Chinese word segmentation using minimal linguistic knowledge. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 148-151.
4. COHN, T., SMITH, A., AND OSBORNE, M., 2005. Scaling conditional random fields using errorcorrecting codes. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05). 10-17.
5. EMERSON, T., 2005. The second international Chinese word segmentation bakeoff. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN'05). 123-133.
6. FAN, C.-K. AND TSAI, W.-H, 1988. Automatic word identification in Chinese sentences by the relaxation technique. Comput. Proc. Chinese Oriental Lang. 4, 1, 33-56.
7. FU, G.-H. AND WANG, X.-L, 1999. Unsupervised Chinese word segmentation and unknown word identification. In Proceedings of the 5th Natural Language Processing Pacific Rim Symposium (NLPRS'99). 32-37.
8. GAO, J., LI, M., WU, A., AND HUANG, C.-N., 2005. Chinese word segmentation and named entity recognition: A pragmatic approach. Comput. Linguist. 31, 4, 531-574.
9. GAO, J., WU, A., LI, M., HUANG, C.-N., LI, H., XIA, X., AND QIN, H., 2004. Adaptive Chinese word segmentation. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL'04). 462-469.
10. GOH, C.-L., ASAHARA, M., AND MATSUMOTO, Y., 2005. Chinese word segmentatin by classification of characters. Comput. Linguist. Chinese Lang. Proc. 10, 3, 381-396.
11. GRINSTEAD, C. AND SNELL, J. L., 1997. Introduction to Probability. American Mathematical Society, Providence, RI.
12. HOCKENMAIER, J. AND BREW, C., 1998. Error driven segmentation of Chinese. Comm. COLIPS 8, 1, 69-84.
13. KUNCHEVA, L. I. AND WHITAKER, C. J., 2003. Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy. Mach. Learn. 51, 2, 181-207.
14. LAFFERTY, J. D., MCCALLUM, A., AND PEREIRA, F. C. N., 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning (ICML'01). 282-289.
15. LAU, T. P. AND KING, I., 2005. Two-phase lmr-rc tagging for Chinese word segmentation. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN'05). 183-186.
16. LEVOW, G.-A., 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 108-117.
17. LI, M., GAO, J., HUANG, C.-N., AND LI, J., 2003. Unsupervised training for overlapping ambiguity resolution in Chinese word segmentation. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 1-7.
18. LI, S., 2005. Chinese word segmentation in ICT-NLP. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN'05). 187-188.
19. LIANG, P., 2005. Semi-supervised learning for natural language. M.S. thesis, Massachusetts Institute of Technology.
20. LOW, J. K., NG, H. T., AND GUO, W., 2005. A maximum entropy approach to Chinese word segmentation. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN'05). 161-164.
21. LUO, X., SUN, M., AND TSOU, B. K., 2002. Covering ambiguity resolution in Chinese word segmentation based on contextual information. In Proceedings of the 19th International Conference on Computational Linguistics (COLING'02). 1-7.
22. MALOUF, R., 2002. A comparison of algorithms for maximum entropy parameter estimation. In Proceedings of the Conference on Natural Language Learning (CoNLL'02). 49-55.
23. NG, H. T. AND LOW, J. K., 2004. Chinese part-of-speech tagging: One-at-a-time or all-at-once? Word-based or character-based? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP'04). 277-284.
24. NIE, J.-Y., JIN, W., AND HANNAN, M.-L., 1994. A hybrid approach to unknown word detection and segmentation of Chinese. In Proceedings of the International Conference on Chinese Computing (ICCC'94). 326-335.
25. PACKARD, J., 2000. The Morphology of Chinese: A Linguistics and Cognitive Approach. Cambridge University Press, Cambridge, UK.
26. PALMER, D. D., 1997. A trainable rule-based algorithm for word segmentation. In Proceedings of the Association for Computational Linguistics (ACL'97). 321-328.
27. PENG, F., FENG, F., AND MCCALLUM, A., 2004. Chinese segmentation and new word detection using conditional random fields. In Proceedings of the 19th International Conference on Computational Linguistics (COLING'04). 562-568.
28. RATNAPARKHI, A., 1996. A maximum entropy part-of-speech tagger. In Proceedings of the Empirical Method in Natural Language Processing Conference (EMNLP'96). 133-142.
29. ROSENFELD, B., FELDMAN, R., AND FRESKO, M., 2006. A systematic cross-comparison of sequence classifiers. In Proceedings of the SIAM International Conference on Data Mining (SDM'06). 563-567.
30. SHA, F. AND PEREIRA, F., 2003. Shallow parsing with conditional random fields. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (ACL-HLT'03). 134-141.
31. SPROAT, R. AND EMERSON, T., 2003. The first international Chinese word segmentation bakeoff. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 133-143.
32. SPROAT, R. AND SHIH, C., 1990. A statistical method for finding word boundaries in Chinese text. Comput. Proc. Chinese Oriental Lang. 4, 4, 336-351.
33. SPROAT, R. AND SHIH, C., 2002. Corpus-based methods in Chinese morphology and phonology. In Proceedings of the 19th International Conference on Computational Linguistics (COLING'02).
34. SPROAT, R., SHIH, C., GALE, W., AND CHANG, N., 1996. A stochastic finite-state wordsegmentation algorithm for Chinese. Comput. Linguist. 22, 3, 377-404.
35. SUN, C., HUANG, C.-N., AND GUAN, Y., 2006. Combinative ambiguity string detection and resolution based on annotated corpus. In Proceedings of the 3rd Student Workshop on Computational Linguistics (SWCL'06).
36. SUN, M., SHEN, D., AND TSOU, B. K., 1998. Chinese word segmentation without using lexicon and hand-crafted training data. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics (COLING-ACL'98). 1265-1271.
37. SUN, M. AND TSOU, B. K., 2001. A review and evaluation on automatic segmentation of Chinese (in Chinese). Contemporary Linguist. 3, 1, 22-32.
38. TSAI, J.-L., 2006. BMM-based Chinese word segmentor with word support model for the SIGHAN bakeoff 2006. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 130-133.
39. TSAI, R. T.-H., HUNG, H.-C., SUNG, C.-L., DAI, H.-J., AND HSU, W.-L, 2006. On closed task of Chinese word segmentation: An improved CRF model coupled with character clustering and automatically generated template matching. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 108-117.
40. TSENG, H., CHANG, P., ANDREW, G., JURAFSKY, D., AND MANNING, C., 2005. A conditional random field word segmenter for SIGHAN bakeoff 2005. In Proceedings of the 4th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 168-171.
41. WANG, X., LIN, X., YU, D., TIAN, H., AND WU, X., 2006. Chinese word segmentation with maximum entropy and N-gram language model. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 138-141.
42. WU, A. AND JIANG, Z., 2000. Statistically-enhanced new word identification in a rule-based Chinese system. In Proceedings of the 2nd Chinese Processing Workshop (ACL'00). 46-51.
43. XUE, N., 2003. Chinese word segmentation as character tagging. Comput. Linguist. Chinese Lang. Proc. 8, 1, 29-48.
44. XUE, N. AND SHEN, L., 2003. Chinese word segmentation as LMR tagging. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 176-179.
45. YUAN, Y., 1997. Statistics based approaches towards Chinese language processing. Ph.D. thesis, National University of Singapore.
46. ZHANG, H.-P. AND LIU, Q., 2003. Chinese lexical analysis using hierarchical hidden markov model. In Proceedings of the 2nd SIGHAN Workshop on Chinese Language Processing (SIGHAN'03). 63-70.
47. ZHANG, M., ZHOU, G.-D., YANG, L.-P., AND JI, D.-H., 2006. Chinese word segmentation and named entity recognition based on a context-dependent mutual information independence model. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 154-157.
48. ZHANG, R., KIKUI, G., AND SUMITA, E., 2006. Subword-based tagging by conditional random fields for Chinese word segmentation. In Proceedings of the Human Language Technology Conference/North American Chapter of the Association for Computational Linguistics (HLT/NAACL'06). 193-196.
49. ZHOU, G. D., 2005. A chunking strategy towards unknown word detection in Chinese word segmentation. In Proceeding of the 2nd International Joint Conference on Natural Language Processing (IJCNLP'05). R. Dale, K.-F. Wong, J. Su, and O. Y. Kwong, Eds. Lecture Notes in Computer Science, vol.3651, 530-541.
50. ZHU, M.-H., WANG, Y.-L., WANG, Z.-X., WANG, H.-Z., AND ZHU, J.-B, 2006. Designing special post-processing rules for SVM-based Chinese word segmentation. In Proceedings of the 5th SIGHAN Workshop on Chinese Language Processing (SIGHAN'06). 217-220.