Mostly-unsupervised statistical segmentation of Japanese kanji sequences

Natural Language Engineering

Volumn 9, Issue 2, 2003, Pages 127-149

  Ando, Rie Kubota ^a   Lee, Lillian ^b  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b Department of Computer Science and School of Operations Research and Information Engineering   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; INFORMATION RETRIEVAL; OPTICAL CHARACTER RECOGNITION; STATISTICAL METHODS; TEXT PROCESSING;

STATISTICAL SEGMENTATION; WORD SEGMENTATION;

WORD PROCESSING;

EID: 0037600401     PISSN: 13513249     EISSN: None     Source Type: Journal    
DOI: 10.1017/S1351324902002954     Document Type: Article

References (45)

1. Brent, M. R. and Tao, X. (2001) Chinese text segmentation with MBDP-I: Making the most of training corpora, Proc. ACL.
2. Church, K. W. and Hanks, P. (1990) Word association norms, mutual information, and lexicography. Computational Linguistics 16(1): 22-29.
3. Cover, T. M. and Thomas, J. A. (1991) Elements of Information Theory, Wiley-Interscience, New York.
4. Dai, Y., Loh, T. E. and Khoo, C. S. G. (1999) A new statistical formula for Chinese text segmentation incorporating contextual information. Proc. 22nd SIGIR, pp. 82-89.
5. Fuchi, T. and Takagi, S. (1998) Japanese morphological analyzer using word co-occurrence - JTAG. Proc. COLING-ACL '98, pp. 409-413.
6. Fung, P. (1998) Extracting key terms from Chinese and Japanese texts. Computer Processing of Oriental Languages 12(1).
7. Ge, X., Pratt, W. and Smyth, P. (1999) Discovering Chinese words from unsegmented text. Proc. 22nd SIGIR, pp. 271-272 (Poster abstract.)
8. Goldsmith, J. (1990) Unsupervised learning of the morphology of a natural language. Computational Linguistics 27(2): pp. 153-198.
9. Grishman, R., Macleod, C. and Sterling, J. (1992) Evaluating parsing strategies using standardized parse files. Proc. 3rd ANLP, pp. 156-161.
10. Ito, A. and Kohda, K. (1995) Language modeling by string pattern N-gram for Japanese speech recognition. Proc. ICASSP.
11. Kashioka, H., Kawata, Y., Kinjo, Y., Finch, A. and Black, E. W. (1998) Use of mutual information based character clusters in dictionary-less morphological analysis of Japanese. Proc. COLING-ACL '98, pp. 658-662.
12. Kurohashi, S. and Nagao, M. (1999) Japanese morphological analysis system JUMAN version 3.61 manual. In Japanese.
13. Lua, K.-T. (1995) Experiments on the use of bigram mutual information for Chinese natural language processing. Int. Conf. Computer Processing of Oriental Languages (ICCPOL), pp. 23-25.
14. Lua, K.-T. and Gan, K.-W. (1994) An application of information theory in Chinese word segmentation. Computer Processing of Chinese and Oriental Languages 8(1): 115-123.
15. Magerman, D. M. and Marcus, M. P. (1990) Parsing a natural language using information statistics. Proc. AAAI, pp. 984-989.
16. Manber, U. and Myers, G. (1993) Suffix arrays: A new method for on-line string searches. SIAM J. Comput. 22(5): 935-948.
17. Matsukawa, T., Miller, S. and Weischedel, R. (1993) Example-based correction of word segmentation and part of speech labelling. Proc. Human Language Technologies Workshop (HLT), pp. 227-32.
18. Matsumoto, Y., Kitauchi, A., Yamashita, T., Hirano, Y., Imaichi, O., and Imamura, T. (1997) Japanese morphological analysis system ChaSen manual. Technical Report NAIST-IS-TR97007, Nara Institute of Science and Technology. In Japanese.
19. Matsumoto, Y. and Nagao, M. (1994) Improvements of Japanese morphological analyzer JUMAN. Proc. Int. Workshop on Sharable Natural Language Resources, pp. 22-28.
20. Mori, S. and Nagao, M. (1998) Unknown word extraction from corpora using n-gram statistics. J. Infor. Process. Soc. Japan 39(7): 2093-2100. In Japanese.
21. Nagao, M. and Mori, S. (1994) A new method of N-gram statistics for large number of n and automatic extraction of words and phrases from large text data of Japanese. Proc. 15th COLING, pp. 611-615.
22. * n-best search algorithm. Proc. 15th COLING, pp. 201-207.
23. Nagata, M. (1996a) Automatic extraction of new words from Japanese texts using generalized forward-backward search. Proc. Conf. Empirical Methods in Natural Language Processing. pp. 48-59.
24. Nagata, M. (1996b) Context-based spelling correction for Japanese OCR. Proc. 16th COLING, pp. 806-811.
25. Nagata, M. (1997) A self-organizing Japanese word segmenter using heuristic word identification and re-estimation. Proc. 5th Workshop on Very Large Corpora, pp. 203-215.
26. Ogawa, Y. and Matsuda, T. (1999) Overlapping statistical segmentation for effective indexing of Japanese text. Infor. Process. & Manage. 35, 463-480.
27. Palmer, D. (1997) A trainable rule-based algorithm for word segmentation. Proc. 35th ACL/8th EACL, pp. 321-328.
28. Papageorgiou, C. P. (1994) Japanese word segmentation by hidden Markov model. Proc. Human Language Technologies Workshop (HLT), pp. 283-288.
29. Peng, F. and Schuurmans, D. (2001) Self-supervised Chinese word segmentation. Advances in Intelligent Data Analysis (Proc. IDA-01), pp. 238-247.
30. Saffran, J. R. (2001) The use of predictive dependencies in language learning. J. Memory & Language 44: 493-515.
31. Saffran, J. R., Aslin, R. N. and Newport, E. L. (1996) Statistical learning by 8-month-old infants. Science 274(5294): 1926-1928.
32. Sproat, R. and Shih, C. (1990) A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages 4: 336-351.
33. Sproat, R., Shih, C., Gale, W. and Chang, N. (1996) A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics 22(3): 377-404.
34. Sun, M., Shen, D. and Tsou, B. K. (1998) Chinese word segmentation without using lexicon and hand-crafted training data. Proc. COLING-ACL, pp. 1265-1271.
35. Takeda, K. and Fujisaki, T. (1987) Automatic decomposition of kanji compound words using stochastic estimation. J. Infor. Process. Soc. Japan 28(9): 952-961, In Japanese.
36. Takeuchi, K. and Matsumoto, Y. (1995) HMM parameter learning for Japanese morphological analyzer. Proc. 10th Pacific Asia Conference on Language, Information and Computation (PACLING), pp. 163-172.
37. Teahan, W. J., Wen, Y., McNab, R. and Witten, I. H. (2000) A compression-based algorithm for Chinese word segmentation. Computational Linguistics 26(3): 275-393.
38. Teller, V. and Batchelder, E. O. (1994) A probabilistic algorithm for segmenting non-kanji Japanese strings. Proc. 12th AAAI, pp. 742-747.
39. Tomokiyo, L. M. and Ries, K. (1997) What makes a word: learning base units in Japanese for speech recognition. Proc. ACL Special Interest Group in Natural Language Learning (CoNLL), pp. 60-69.
40. Wu, D. (1998) A position statement on Chinese segmentation. http://www.cs.ust.hk/~dekai/papers/segmentation.html (Presented at the Chinese Language Processing Workshop, University of Pennsylvania.)
41. Wu, M.-W. and Su, K.-Y. (1993) Corpus-based automatic compound extraction with mutual information and relative frequency count. Proc. R.O.C. Computational Linguistics Conference VI, pp. 207-216.
42. Wu, Z. and Tseng, G. (1993) Chinese text segmentation for text retrieval: Achievements and problems. J. Am. Soc. Infor. Sci. 44(9): 532-542.
43. Yamamoto, M. (1996) A re-estimation method for stochastic language modeling from ambiguous observations. Proc. 4th Workshop on Very Large Corpora, pp. 155-167.
44. Yamamoto, M. and Church, K. W. (2001) Using suffix arrays to compute term frequency and document frequency for all substrings in a corpus. Computational Linguistics 27(1): 1-30.
45. Yamron, J., Baker, J., Bamberg, P., Chevalier, H., Dietzel, T., Elder, J., Kampmann, F., Mandel, M., Manganaro, L., Margolis, T. and Steele, E. (1993) LINGSTAT: An interactive machine-aided translation system. Proc. Human Language Technologies Workshop (HLT). pp. 191-195.