Critical Tokenization and its Properties

Computational Linguistics

Volumn 23, Issue 4, 1997, Pages 569-596

  Guo, Jin ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

Author keywords

[No Author keywords available]

Indexed keywords

SET THEORY;

AMBIGUITY RESOLUTION; CHARACTER STRINGS; FORWARD-AND-BACKWARD; MATHEMATICAL DESCRIPTIONS; MATHEMATICAL FOUNDATIONS; MAXIMUM MATCHINGS; OVERLAPPING AMBIGUITIES; PARTIALLY ORDERED SET;

MAXIMUM PRINCIPLE;

EID: 0345877442     PISSN: 08912017     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (58)

1. Aho, Alfred V., R. Sethi, and Jeffrey D. Ullman. 1986. Compilers, Principles, Techniques, and Tools. Addison-Wesley Publishing Co.
2. Aho, Alfred V. and Jeffrey D. Ullman. 1972. The Theory of Parsing, Translation, and Compiling, Volume 1: Parsing. Prentice-Hall, Inc.
3. Allen, James. 1995. Natural Language Understanding, 2nd edition. The Benjamin/Cummings Publishing Co.
4. Bai, Shuanhu. 1995. An integrated model of Chinese word segmentation and part of speech tagging. In Liwei Chen and Qi Yuan, editors, Advances and Applications on Computational Linguistics. Tsinghua University Press, pages 56-61.
5. Chen, Keh-Jiann and Shing-Huan Liu. 1992. Word identification for Mandarin Chinese sentences. In Proceedings of the 14th International Conference on Computational Linguistics (COLING'92), pages 101-107. Nantes, France.
6. Chiang, Tung-Hui, Jing-Shin Chang, Ming-Yu Lin, and Keh-Yih Su. 1992. Statistical models for word segmentation and unknown word resolution. In Proceedings of the 5th R.O.C. Computational Linguistics Conference (ROCLING V), pages 121-146, Taiwan.
7. Fan, C-K. and W-H. Tsai. 1988. Automatic word identification in Chinese sentences by the relaxation technique. Computer Processing of Chinese and Oriental Languages 4(1):33-56.
8. Gan, Kok-Wee. 1995. Integrating Word Boundary Disambiguation with Sentence Understanding. Ph.D. dissertation, Department of Computer Science and Information Systems, National University of Singapore.
9. Gan, Kok-Wee, Martha Palmer, and Kim-Teng Lua. 1996. A statistically emergent approach for language processing: Application to modeling context effects in ambiguous Chinese word boundary perception. Computational Linguistics 22(4)531-553.
10. Garside, Roger, Geoffrey Leech, and Geoffrey Sampson, editors. 1987. The Computational Analysis of English: A Corpus-based Approach. Longman, London.
11. Gazdar, G., E. Klein, G. Pullum, and I. Sag. 1985. Generalized Phrase Structure Grammar. Harvard University Press.
12. Grosz, B. J., K. S. Jones, and B. L. Webber, editors. 1986. Readings in Natural Language Processing. M. Kaufmann Publishers.
13. Guo, Jin. 1993. Statistical language modeling and some experimental results on Chinese syllables to words transcription. Journal of Chinese Information Processing 7(1):18-27.
14. Guo, Jin. 1997. Chinese Language Modeling for Speech Recognition. Ph.D. dissertation, Institute of Systems Science, National University of Singapore.
15. He, Kekang, Hui Xu, and Bo Sun. 1991. Design principles of an expert system for automatic word segmentation of written Chinese texts. Journal of Chinese Information Processing 5(2):1-14.
16. Huang, Xiangxi. 1989. A produce-test approach to automatic segmentation of written Chinese. Journal of Chinese Information Processing 3(4):42-49.
17. Huang, Changning and Ying Xia, editors. 1996. Essays on Language Information Processing. Tsinghua University Press, Beijing.
18. Hudson, R. A. 1984. Word Grammar. Basil Blackwell.
19. Jie, Chunyu. 1989. A systematic approach model for methods of Chinese automatic word segmentation and their evaluation. In Proceedings of the Chinese Computing Conference, pages 71-78, Beijing.
20. Jie, Chunyu, Yuan Liu, and Nanyuan Liang. 1991a. On the methods of Chinese automatic segmentation. Journal of Chinese Information Processing 3(1):1-9.
21. Jie, Chunyu, Yuan Liu, and Nanyuan Liang. 1991b. The design and implementation of the CASS practical automatic Chinese word segmentation system. Journal of Chinese Information Processing 5(4):27-34.
22. Jin, Wanying and Lu Chen. 1995. Identifying unknown words in Chinese corpora. In Proceedings of the 1995 Natural Language Processing Pacific Rim Symposium (NLPRS'95), pages 234-239, Seoul.
23. Kolman, Bernard and Robert C. Busby. 1987. Discrete Mathematical Structures for Computer Science, 2nd edition. Prentice-Hall, Inc.
24. Lai, T. B. Y., S. C. Lun, C. F. Sun, and M. S. Sun. 1992. A tagging-based first-order Markov model approach to automatic word identification for Chinese sentences. In Proceedings of the 1992 International Conference on Computer Processing of Chinese and Oriental Languages, pages 17-23.
25. Li, Wen-Jie, H-H. Pan, M. Zhou, K-F. Wong, and V. Lum. 1995. Corpus-based maximum-length Chinese noun phrase extraction. In Proceedings of the 1995 Natural Language Processing Pacific Rim Symposium (NLPRS'95), pages 246-251, Seoul.
26. Liang, Nanyuan. 1986. On computer automatic word segmentation of written Chinese. Journal of Chinese Information Processing 1(1).
27. Liang, Nanyuan. 1987. CDWS - A written Chinese automatic word segmentation system. Journal of Chinese Information Processing 1(2):44-52.
28. Liang, Nanyuan. 1990. The knowledge of Chinese words segmentation. Journal of Chinese Information Processing 4(2):29-33.
29. Liu, Yongquan. 1986a. Language Modernization and Computer. Wuhan University Press, Wuhan, China.
30. Liu, Yongquan. 1986b. On dictionary. Journal of Chinese Information Processing 1(1).
31. Liu, Yongquan. 1988. Word re-examination. Journal of Chinese Information Processing 2(2):47-50.
32. Liu, Yuan, Qiang Tan, and Xukun Shen. 1994. Contemporary Chinese Language Word Segmentation Specification for Information Processing and Automatic Word Segmentation Methods. Tsinghua University Press, Beijing.
33. Lua, Kim Teng. 1990. From character to word - An application of information theory. Computer Processing of Chinese and Oriental Languages 4(4):304-313.
34. Lua, Kim Teng. 1994. Application of information theory binding in word segmentation. Computer Processing of Chinese and Oriental Languages 8(1):115-124.
35. Lua, Kim Teng. 1995. Experiments on the use of bigram mutual information in Chinese natural language processing. In Proceedings of the 1995 International Conference on Computer Processing of Oriental Languages (ICCPOL-95), pages 306-313, Hawaii.
36. Ma, Yan. 1996. The study and realization of an evaluation-based automatic segmentation system. In Changning Huang and Ying Xia, editors, Essays in Language Information Processing. Tsinghua University Press, Beijing, pages 2-36.
37. Nie, Jieyun, Jin Wanying, and M-L. Hannan. 1994. A hybrid approach to unknown word detection and segmentation of Chinese. In Proceedings of the International Conference on Chinese Computing 1994 (ICCC-94), pages 326-335, Singapore.
38. Pachunke, T., O. Mertineit, K. Wothke, and R. Schmidt. 1992. Broad coverage automatic morphological segmentation of German words. In Proceedings of the 14th International Conference on Computational Linguistics (COLING'92), pages 1218-1222, Nantes, France.
39. Seo, J. and R. F. Simmons. 1989. Syntactic graphs: A representation for the union of all ambiguous parse trees. Computational Linguistics 15(1):19-32.
40. Sproat, Richard and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages 4(4):336-349.
41. Sproat, Richard, Chilin Shih, William Gale, and Nancy Chang. 1996. A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics 22(3):377-404.
42. Sun, Maosong and Changning Huang. 1996. Word segmentation and part-of-speech tagging for unrestricted Chinese texts. Tutorial given at the 1996 International Conference on Chinese Computing (ICCC-96), Singapore.
43. Sun, Maosong and Benjemin T'sou. 1995. Ambiguity resolution in Chinese word segmentation. In Proceedings of the 10th Pacific Asia Conference on Language, Information and Computation (PACLIC-95), pages 121-126, Hong Kong.
44. Tung, C. H. and H. J. Lee. 1994. Identification of unknown words from corpus. Computer Processing of Chinese and Oriental Languages 8(Supplement):131-146.
45. Wang, Yongcheng, Haiju Su, and Yan Mo. 1990. Automatic processing Chinese word. Journal of Chinese Information Processing 4(4):1-11.
46. Wang, Xiaolong. 1989. Word Separating and Mutual Translation of Syllable and Character Strings. Ph.D. dissertation, Department of Computer Science and Engineering, Harbin Institute of Technology, Harbin, China.
47. Wang, Xiaolong, Kaizhu Wang, and Xiaohua Bai. 1991. Separating syllables and characters into words in natural language understanding. Journal of Chinese Information Processing 5(3):48-58.
48. Webster, Jonathan J. and Chunyu Kit. 1992. Tokenization as the initial phase in NLP. In Proceedings of the 14th International Conference on Computational Linguistics (COLING'92), pages 1,106-1,110, Nantes, France.
49. Wong, K-F., H-H. Pan, B-T. Low, C-H. Cheng, V. Lum, and S-S. Lam. 1995. A tool for computer-assisted open response analysis. In Proceedings of the 1995 International Conference on Computer Processing of Oriental Languages, pages 191-198, Hawaii.
50. Wong, K. F., V. Y. Lum, C-Y. Leung, C-H. Leung, W-K. Kan, and L-C. Chan. 1994. A parallel approach for identifying word boundaries in Chinese text. IPOC Technical Report, /CHIRP/WP/SE/022, Department of Systems Engineering, Chinese University of Hong Kong.
51. Wu, Horng Jyh, Jin Guo, Ho Chung Lui, and Hwee Boon Low. 1994. Corpus-based speech and language research in the Institute of Systems Science. In Proceedings of the International Symposium on Speech, Image Processing and Neural Networks (ISPIPNN'94), pages 142-145, Hong Kong.
52. Wu, Ming-Wen and Keh-Yih Su. 1993. Corpus-based automatic compound extraction with mutual information and relative frequency count. In Proceedings of R.O.C. Computational Linguistics Conference (ROCLING) VI, pages 207-216, Taiwan.
53. Yao, Tian-Shun, Gui-Ping Zhang, and Ying-Ming Wu. 1990. A rule-based Chinese automatic segmentation system. Journal of Chinese Information Processing 4(1):37-43.
54. Yeh, C-L. and H-J. Lee. 1991. Rule-based word identification for Mandarin Chinese sentences - A unification approach. Computer Processing of Chinese and Oriental Languages 5(2):97-118.
55. Yosiyuki, K., T. Takenobu, and T. Hozumi. 1992. Analysis of Japanese compound nouns using collocation information. In Proceedings of the 14th International Conference on Computational Linguistics (COLING'92), pages 865-869, Nantes, France.
56. Yun, B-H., H. Lee, and H-C. Rim. 1995. Analysis of Korean compound nouns using statistical information. In Proceedings of the 1995 International Conference on Computer Processing of Oriental Languages (ICCPOL-95), pages 76-79, Honolulu, Hawaii.
57. Zhang, Jun-Sheng, Zhi-Da Chen, and Shun-De Chen. 1991. A method of word identification for Chinese by constraint satisfaction and statistical optimization techniques. In Proceedings of R.O.C. Computational Linguistics Conference (ROCLING) IV, pages 147-165, Taiwan.
58. Zhang, Jun-Sheng, Shun-De Chen, S. J. Ker, Y. Chan, and J. S. Liu. 1994. A multi-corpus approach to recognition of proper names in Chinese texts. Computer Processing of Chinese and Oriental Languages 8(1):73-86.