A Stochastic Finite-State Word-Segmentation Algorithm for Chinese

Computational Linguistics

Volumn 22, Issue 3, 1996, Pages 377-404

  Sproat, Richard ^a,c   Gale, William ^a,c   Shih, Chilin ^a,c   Chang, Nancy ^b,c  

^a LUCENT TECHNOLOGIES   (United States)

^b UNIVERSITY OF CAMBRIDGE   (United Kingdom)

^c NONE

Author keywords

[No Author keywords available]

Indexed keywords

IMAGE SEGMENTATION; SPEECH SYNTHESIS; STOCHASTIC SYSTEMS;

ASIAN LANGUAGES; BOUNDARY INFORMATION; FINITE-STATE MODELS; LEXICAL INFORMATION; MODEL SEGMENT; SYSTEM'S PERFORMANCE; WEIGHTED FINITE-STATE TRANSDUCERS; WORD SEGMENTATION;

STOCHASTIC MODELS;

EID: 0001076101     PISSN: 08912017     EISSN: None     Source Type: Journal    
DOI: None     Document Type: Article

References (50)

1. Antworth, Evan. 1990. PC-KIMMO: A Two-Level Processor for Morphological Analysis. Occasional Publications in Academic Computing, 16. Summer Institute of Linguistics, Dallas, TX.
2. Baayen, Harald. 1989. A Corpus-Based Approach to Morphological Productivity: Statistical Analysis and Psycholinguistic Interpretation. Ph.D. thesis, Free University, Amsterdam.
3. Becker, Richard, John Chambers, and Allan Wilks. 1988. The New S Language. Wadsworth and Brooks, Pacific Grove.
4. Chang, Chao-Huang and Cheng-Der Chen. 1993. A study on integrating Chinese word segmentation and part-of-speech tagging. Communications of the Chinese and Oriental Languages Information Processing Society, 3(2):69-77.
5. Chang, Jyun-Shen, C.-D. Chen, and Shun-De Chen. 1991. Xianzhishi manzu ji jilu zuijiahua de zhongwen duanci fangfa [Chinese word segmentation through constraint satisfaction and statistical optimization]. In Proceedings of ROCLING IV, pages 147-165, Taipei. ROCLING.
6. Chang, Jyun-Shen, Shun-De Chen, Ying Zheng, Xian-Zhong Liu, and Shu-Jin Ke. 1992. Large-corpus-based methods for Chinese personal name recognition. Journal of Chinese Information Processing, 6(3):7-15.
7. Chao, Yuen-Ren. 1968. A Grammar of Spoken Chinese. University of California Press, Berkeley, CA.
8. Chen, Keh-Jiann and Shing-Huan Liu. 1992. Word identification for Mandarin Chinese sentences. In Proceedings of COLING-92, pages 101-107. COLING.
9. Church, Kenneth and William Gale. 1991. A comparison of the enhanced Good-Turing and deleted estimation methods for estimating probabilities of English bigrams. Computer Speech and Language, 5(1):19-54.
10. Church, Kenneth and Patrick Hanks. 1989. Word association norms, mutual information and lexicography. In 27th Annual Meeting of the Association for Computational Linguistics, pages 76-83, Morristown, NJ. Association for Computational Linguistics.
11. DeFrancis, John. 1984. The Chinese Language. University of Hawaii Press, Honolulu.
12. Fan, C.-K. and W.-H. Tsai. 1988. Automatic word identification in Chinese sentences by the relaxation technique. Computer Processing of Chinese and Oriental Languages, 4:33-56.
13. Fung, Pascale and Dekai Wu. 1994. Statistical augmentation of a Chinese machine-readable dictionary. In WVLC-94, Second Annual Workshop on Very Large Corpora.
14. Gan, Kok-Wee. 1994. Integrating Word Boundary Disambiguation with Sentence Understanding. Ph.D. thesis, National University of Singapore.
15. Gu, Ping and Yuhang Mao. 1994. Hanyu zidong fenci de jinlin pipei suanfa ji qi zai QHFY hanying jiqi fanyi xitong zhong de shixian [The adjacent matching algorithm of Chinese automatic word segmentation and its implementation in the QHFY Chinese-English system]. In International Conference on Chinese Computing, Singapore.
16. Hirschberg, Julia. 1993. Pitch accent in context: Predicting intonational prominence from text. Artificial Intelligence, 63:305-340.
17. Huang, Chu-Ren, Kathleen Ahrens, and Keh-jiann Chen. 1993. A data-driven approach to psychological reality of the mental lexicon: Two studies on Chinese corpus linguistics. Presented at the conference on Language and its Psychobiological Bases, December.
18. Kaplan, Ronald and Martin Kay. 1994. Regular models of phonological rule systems. Computational Linguistics, 20:331-378.
19. Karttunen, Lauri, Ronald Kaplan, and Annie Zaenen. 1992. Two-level morphology with composition. In COLING-92, pages 141-148. COLING.
20. Koskenniemi, Kimmo. 1983. Two-Level Morphology: A General Computational Model for Word-Form Recognition and Production. Ph.D. thesis, University of Helsinki, Helsinki.
21. Kupiec, Julian. 1992. Robust part-of-speech tagging using a hidden Markov model. Computer Speech and Language. Submitted.
22. Li, B.Y., S. Lin, C.F. Sun, and M.S. Sun. 1991. Yi zhong zhuyao shiyong yuliaoku biaoji jinxing qiyi jiaozheng de zuida pipei hanyu zidong fenci suanfa sheji [A maximum-matching word segmentation algorithm using corpus tags for disambiguation]. In ROCLlNC IV, pages 135-146, Taipei. ROCLING.
23. Li, Charles and Sandra Thompson. 1981. Mandarin Chinese: A Functional Reference Grammar. University of California Press, Berkeley, CA.
24. Liang, Nanyuan. 1986. Shumian hanyu zidong fenci xitong-CDWS [A written Chinese automatic segmentation system-CDWS]. Journal of Chinese Information Processing, 1(1):44-52.
25. Lin, Ming-Yu, Tung-Hui Chiang, and Keh-Yi Su. 1993. A preliminary study on unknown word problem in Chinese word segmentation. In ROCLlNG 6, pages 119-141. ROCLING.
26. Mohri, Mehryar. 1993. Analyse et représentation par automates de structures syntaxiques composées. Ph.D. thesis, University of Paris 7, Paris.
27. Mohri, Mehryar. 1995. Minimization algorithms for sequential transducers. Theoretical Computer Science. Submitted.
28. Nagata, Masaaki. 1994. A stochastic Japanese morphological analyzer using a forward-DP backward A* N-best search algorithm. In Proceedings of COLING-94, pages 201-207. COLING.
29. Nie, Jian-Yun, Wanying Jin, and Marie-Louise Hannan. 1994. A hybrid approach to unknown word detection and segmentation of Chinese. In International Conference on Chinese Computing, Singapore.
30. Peng, Z.-Y. and J-S. Chang. 1993. Zhongwen cihui qiyi zhi yanjiu - duanci yu cixing biaoshi [Research on Chinese lexical ambiguity - segmentation and part-of-speech tagging]. In ROCLING 6, pages 173-193. ROCLING.
31. Pereira, Fernando, Michael Riley, and Richard Sproat. 1994. Weighted rational transductions and their application to human language processing. In ARPA Workshop on Human Language Technology, pages 249-254. Advanced Research Projects Agency, March 8-11.
32. PRCNSC, 1994. Contemporary Chinese Language Word Segmentation Specification for Information Processing. People's Republic of China National Standards Committee. In Chinese.
33. ROCLING. 1993. Jisuan yuyanxue tongxun [Computational linguistics communications]. Newsletter of the Republic of China Computational Linguistics Society (ROCLING), April. In Chinese.
34. Shih, Chilin. 1986. The Prosodic Domain of Tone Sandhi in Chinese. Ph.D. thesis, UCSD, La Jolla, CA.
35. Sproat, Richard. 1992. Morphology and Computation. MIT Press, Cambridge, MA.
36. Sproat, Richard. 1994. English noun-phrase accent prediction for text-to-speech. Computer Speech and Language, 8:79-94.
37. Sproat, Richard. 1995. A finite-state architecture for tokenization and grapheme-to-phoneme conversion for multilingual text analysis. In Susan Armstrong and Evelyne Tzoukermann, editors, Proceedings of the EACL SIGDAT Workshop, pages 65-72, Dublin, Ireland. Association for Computational Linguistics.
38. Sproat, Richard and Chilin Shih. 1990. A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4:336-351.
39. Sproat, Richard and Chilin Shih. 1995. A corpus-based analysis of Mandarin nominal root compounds. Journal of East Asian Linguistics, 4(1):1-23.
40. Tang, Ting-Chih. 1988. Hanyu Cifa Jufa Lunji [Studies on Chinese Morphology and Syntax], volume 2. Student Book Company, Taipei. In Chinese.
41. Tang, Ting-Chih. 1989. Hanyu Cifa Jufa Xuji [Studies on Chinese Morphology and Syntax: 2], volume 2. Student Book Company, Taipei. In Chinese.
42. Torgerson, Warren. 1958. Theory and Methods of Scaling. Wiley, New York.
43. Tzoukermann, Evelyne and Mark Liberman. 1990. A finite-state morphological processor for Spanish. In COLING-90, Volume 3, pages 3: 277-286. COLING.
44. Wang, Liang-Jyh, Wei-Chuan Li, and Chao-Huang Chang. 1992. Recognizing unregistered names for Mandarin word identification. In Proceedings of COLING-92, pages 1239-1243. COLING.
45. Wang, Michelle and Julia Hirschberg. 1992. Automatic classification of intonational phrase boundaries. Computer Speech and Language, 6:175-196.
46. Wang, Yongheng, Haiju Su, and Yan Mo. 1990. Automatic processing of Chinese words. Journal of Chinese Information Processing, 4(4):1-11.
47. Wieger, L. 1965. Chinese Characters. Dover, New York. Republication of second edition, published 1927 by Catholic Mission Press.
48. Wu, Dekai and Pascale Fung. 1994. Improving Chinese tokenization with linguistic filters on statistical lexical acquisition. In Proceedings of the Fourth Conference on Applied Natural Language Processing, pages 180-181, Stuttgart, October.
49. Wu, Zimin and Gwyneth Tseng. 1993. Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science, 44(9):532-542.
50. Yeh, Ching-long and Hsi-Jian Lee. 1991. Rule-based word identification for Mandarin Chinese sentences - a unification approach. Computer Processing of Chinese and Oriental Languages, 5(2):97-118.