Integrating unsupervised and supervised word segmentation: The role of goodness measures

Information Sciences

Volumn 181, Issue 1, 2011, Pages 163-183

  Zhao, Hai ^a,b   Kit, Chunyu ^a  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^b SHANGHAI JIAO TONG UNIVERSITY   (China)

Author keywords

Accessor variety; Boundary entropy; Character tagging; Chinese word segmentation; Conditional random fields; Description length gain; Unknown word detection; Unsupervised segmentation

Indexed keywords

ACCESSOR VARIETY; BOUNDARY ENTROPY; CHARACTER TAGGING; CHINESE WORD SEGMENTATION; CONDITIONAL RANDOM FIELD; DESCRIPTION LENGTH GAIN; UNKNOWN WORD DETECTION; UNSUPERVISED SEGMENTATION;

COMPUTATIONAL LINGUISTICS; ENTROPY; SUPPORT VECTOR MACHINES;

IMAGE SEGMENTATION;

EID: 77958083019     PISSN: 00200255     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ins.2010.09.008     Document Type: Article

References (60)

1. R.K. Ando, L. Lee, Mostly-unsupervised statistical segmentation of Japanese: Applications to Kanji, in: Proceedings of the First Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL 2000), Seattle, Washington, 2000, pp. 241-248.
2. B. Carpenter, Character language models for Chinese word segmentation and named entity recognition, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 169-172.
3. J.-S. Chang, and K.-Y. Su An unsupervised iterative method for Chinese new lexicon extraction International Journal of Computational Linguistics and Chinese Language Processing (CLCLP) 2 2 1997 97 148
4. L.-F. Chien, PAT-tree-based keyword extraction for Chinese information retrieval, in: Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'97), Philadelphia, 1997, pp. 50-58.
5. T. Emerson, The second international Chinese word segmentation bakeoff, in: Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing (SIGHAN-4), Jeju Island, Korea, 2005, pp. 123-133.
6. H. Feng, K. Chen, X. Deng, and W. Zheng Accessor variety criteria for Chinese word extraction Computational Linguistics 30 1 2004 75 93
7. H. Feng, K. Chen, C. Kit, and X. Deng Unsupervised segmentation of Chinese corpus using accessor variety K.-Y. Su, J. Tsujii, J.H. Lee, O.Y. Kwong, Natural Language Processing - IJCNLP 2004 LNAI vol. 3248 2005 Springer 694 703
8. G.-H. Fu, X.-L. Wang, Unsupervised Chinese word segmentation and unknown word identification, in: The Fifth Natural Language Processing Pacific Rim Symposium 1999 (NLPRS'99), Closing the Millennium, Beijing, China, 1999, pp. 32-37.
9. G.-H. Fu, C. Kit, and J.J. Webster Chinese word segmentation as morpheme-based lexical chunking Information Sciences 178 9 2008 2282 2296
10. X. Ge, W. Pratt, and P. Smyth Discovering Chinese words from unsegmented text Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'99) 1999 ACM Berkeley, CA 271 272
11. G. Grefenstette Tokenisation H. van Halteren, Syntactic Wordclass Tagging 1999 Kluwer Dordrecht 117 133
12. C. Grinstead, and J.L. Snell Introduction to Probability 1997 American Mathematical Society Providence, RI
13. Z.S. Harris From phoneme to morpheme Language 31 2 1955 90 222
14. Z.S. Harris Morpheme boundaries within words Papers in Structural and Transformational Linguistics 1970 Reidel Dordrecht, Holland 68 77
15. C.-N. Huang, and H. Zhao Chinese word segmentation: A decade review Journal of Chinese Information Processing 21 3 2007 8 20
16. J.H. Huang, and D. Powers Chinese word segmentation based on contextual entropy D.H. Ji, K.-T. Lua, Proceedings of the 17th Pacific Asian Conference on Language, Information and Computation (PACLIC 17) 2003 COLIPS Publication Sentosa, Singapore 152 158
17. A.J. Jacobs, Y.W. Wong, Maximum entropy word segmentation of Chinese text, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 108-117.
18. F. Jiao, S. Wang, C.-H. Lee, R. Greiner, D. Schuurmans, Semi-supervised conditional random fields for improved sequence segmentation and labeling, in: COLING/ACL-2006, Sydney, Australia, 2006, pp. 209-216.
19. Z. Jin, K. Tanaka-Ishii, Unsupervised segmentation of Chinese text by use of branching entropy, in: COLING/ACL 2006, Sidney, Australia, 2006, pp. 428-435.
20. C. Kit, Unsupervised lexical learning as inductive inference, Ph.D. Thesis, University of Sheffield, 2000.
21. C. Kit, Y. Wilks, Unsupervised learning of word boundary with description length gain, in: Osborne, M., Sang, E.T.K. (Eds.), Computational Natural Language Learning (CoNLL-99), Bergen, Norway, 1999, pp. 1-6.
22. C. Kit, H. Zhao, Improving Chinese word segmentation with description length gain, in: The 2007 International Conference on Artificial Intelligence (ICAI-2007), Las Vegas, Nevada, USA, 2007, pp. 846-851.
23. J.D. Lafferty, and A. McCallum Conditional random fields: Probabilistic models for segmenting and labeling sequence data Proceedings of the Eighteenth International Conference on Machine Learning (ICML'01) 2001 Morgan Kaufmann San Francisco, CA, USA 282 289
24. G.-A. Levow, The third international Chinese language processing bakeoff: Word segmentation and named entity recognition, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 108-117.
25. W. Liu, H. Li, Y. Dong, N. He, H. Luo, H. Wang, France Telecom R& D Beijing word segmenter for SIGHAN bakeoff 2006, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 108-117.
26. J.K. Low, H.T. Ng, W. Guo, A maximum entropy approach to Chinese word segmentation, in: Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Jeju Island, Korea, 2005, pp. 161-164.
27. X. Lü, L. Zhang, and J. Hu Statistical substring reduction in linear time K.-Y. Su, J. Tsujii, J.H. Lee, O.Y. Kwong, Natural Language Processing - IJCNLP 2004 LNAI vol. 3248 2005 Springer 320 327
28. K.-T. Lua, and K.-W. Gan An application of information theory in Chinese word segmentation Computer Processing of Chinese and Oriental Languages 8 1 1994 115 123
29. A. Mikheev Text segmentation R. Mitkov, The Oxford Handbook of Computational Linguistics 2003 Oxford University Press 201 218
30. D.D. Palmer Tokenisation and sentence segmentation R. Dale, H. Moisl, H. Somers, Handbook of Natural Language Processing 2000 Marcel Dekker New York 11 36
31. F. Peng, F. Feng, A. McCallum, Chinese segmentation and new word detection using conditional random fields, in: Proceedings of the 20th International Conference on Computational Linguistics (COLING 2004), Geneva, Switzerland, pp. 562-568.
32. F. Peng, X. Huang, D. Schuurmans, N. Cercone, S. Robertson, Using self-supervised word segmentation in Chinese information retrieval, in: Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR'01), Tampere, Finland, 2001, pp. 349-350.
33. F. Peng, D. Schuurmans, Self-supervised Chinese word segmentation, in: The Fourth International Symposium on Intelligent Data Analysis (IDA-2001), Lisbon, Portugal, 2001, pp. 238-247.
34. J.M. Ponte, W.B. Croft, USeg: A retargetable word segmentation procedure for information retrieval, Presented at the Symposium on Document Analysis and Information Retrieval'96 (SDAIR),Technical Report TR96-2, University of Massachusetts, Amherst, MA, 1996.
35. B. Rosenfeld, R. Feldman, M. Fresko, A systematic cross-comparison of sequence classifiers, in: SDM 2006, Bethesda, Maryland, pp. 563-567.
36. C.E. Shannon A mathematical theory of communication The Bell System Technical Journal 27 1948 379 423 623-656
37. R. Sproat, T. Emerson, The first international Chinese word segmentation bakeoff, in: The Second SIGHAN Workshop on Chinese Language Processing (SIGHAN-2), Sapporo, Japan, 2003, pp. 133-143.
38. R. Sproat, and C. Shih A statistical method for finding word boundaries in Chinese text Computer Processing of Chinese and Oriental Languages 4 4 1990 336 351
39. M. Sun, D. Shen, B.K. Tsou, Chinese word segmentation without using lexicon and hand-crafted training data, in: COLING-ACL'98, vol. 2, Montreal, Quebec, Canada, 1998, pp. 1265-1271.
40. M. Sun, M. Xiao, and B.K. Tsou Chinese word segmentation without using dictionary based on unsupervised learning strategy Chinese Journal of Computers 27 6 2004 736 742
41. J. Suzuki, A. Fujino, H. Isozaki, Semi-supervised structured output learning based on a hybrid generative and discriminative approach, in: Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL'07), Prague, Czech, 2007, pp. 791-800.
42. W.J. Teahan, Y. Wen, R. McNab, and I.H. Witten A compression-based algorithm for Chinese word segmentation Computational Linguistics 26 3 2000 375 393
43. R.T.-H. Tsai, H.-C. Hung, C.-L. Sung, H.-J.Dai, W.-L. Hsu, On closed task of Chinese word segmentation: An improved CRF model coupled with character clustering and automatically generated template matching, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 108-117.
44. C.-H. Tung, and H.-J. Lee Identification of unknown words from corpus International Journal of Computer Processing of Chinese and Oriental Languages 8 Suppl. 1995 131 146
45. X. Wang, X. Lin, D. Yu, H. Tian, X. Wu, Chinese word segmentation with maximum entropy and N-gram language model, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 138-141.
46. Z. Wang, C. Huang, J. Zhu, The character-based CRF segmenter of MSRA & NEU for the 4th Bakeoff, in: Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing (SIGHAN-6), Hyderabad, India, 2008, pp.98-101.
47. J.J. Webster, C. Kit, Tokenization as the initial phase in nlp, in: Proceedings of the 14th International Conference on Computational Linguistics (COLING-92), vol. IV, Nantes, France, 1992, pp. 1106-1110.
48. Y. Xiong, J. Zhu, H. Huang, and H. Xu Minimum tag error for discriminative training of conditional random fields Information Sciences 179 1-2 2009 169 179
49. N. Xue Chinese word segmentation as character tagging International Journal of Computational Linguistics and Chinese Language Processing 8 1 2003 29 48
50. J. Zhang, J. Gao, M. Zhou, Extraction of Chinese compound words - An experimental study on a very large corpus, in: Proceedings of the Second Chinese Language Processing Workshop, Hong Kong, China, 2000, pp. 132-139.
51. M., Zhang, G.-D. Zhou, L.-P. Yang, D.-H. Ji, Chinese word segmentation and named entity recognition based on a context-dependent mutual information independence model, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 154-157.
52. H. Zhao, Huang, C.-N., M. Li, An improved Chinese word segmentation system with conditional random field, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 162-165.
53. H. Zhao, C.-N. Huang, Li, M., Lu, B.-L., Effective tag set selection in Chinese word segmentation via conditional random field modeling, in: Proceedings of the 20th Pacific Asian Conference on Language, Information and Computation (PACLIC 20), Wuhan, China, 2006, pp. 87-94.
54. H. Zhao, C. Kit, Incorporating global information into supervised learning for Chinese word segmentation, in: Proceedings of the 10th Conference of the Pacific Association for Computational Linguistics (PACLING 2007), Melbourne, Australia, 2007, pp. 66-74.
55. H. Zhao, C. Kit, Unsupervised segmentation helps supervised learning of character tagging for word segmentation and named entity recognition, in: The Sixth SIGHAN Workshop on Chinese Language Processing (SIGHAN-6), Hyderabad, India, 2008, pp. 106-111.
56. H. Zhao, C. Kit, An empirical comparison of goodness measures for unsupervised Chinese word segmentation with a unified framework, in: The Third International Joint Conference on Natural Language Processing (IJCNLP-2008), vol. 1, Hyderabad, India, 2008, pp. 9-16.
57. H. Zhao, and C. Kit Exploiting unlabeled text with different unsupervised segmentation criteria for Chinese word segmentation The 9th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2008), Haifa, Israel, 2008 Research in Computing Science 33 2008 93 104
58. H. Zhao, and C. Kit Scaling conditional random fields by one-against-the-other decomposition Journal of Computer Science and Technology 23 4 2008 612 619
59. H. Zhao, C. Kit, A simple and efficient model pruning method for conditional random fields, in: Proceedings of the 22nd International Conference on the Computer Processing of Oriental Languages (ICCPOL-2009), Hong Kong, China, 2009, pp. 149-159.
60. M.-H. Zhu, Y.-L. Wang, Z.-X. Wang, H.-Z. Wang, J.-B. Zhu, Designing special post-processing rules for SVM-based Chinese word segmentation, in: Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing (SIGHAN-5), Sydney, Australia, 2006, pp. 217-220.