Mining web data for Chinese segmentation

Journal of the American Society for Information Science and Technology

Volumn 58, Issue 12, 2007, Pages 1820-1837

  Wang, Fu Lee ^a   Yang, Christopher C ^b  

^a CITY UNIVERSITY OF HONG KONG   (Hong Kong)

^b CHINESE UNIVERSITY OF HONG KONG   (Hong Kong)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; IMAGE SEGMENTATION; INDEXING (OF INFORMATION); INFORMATION RETRIEVAL SYSTEMS; LINGUISTICS; WEBSITES;

CHINESE DOCUMENTS; SEGMENTATION ALGORITHM; TRAINING DATA;

DATA MINING;

EID: 35348893799     PISSN: 15322882     EISSN: 15322890     Source Type: Journal    
DOI: 10.1002/asi.20629     Document Type: Article

References (61)

1. Agirre, E., Ansa, O., Hovy, E., & Martinez, D. (2002). Enriching very large Ontologies using the WWW. In Proceedings of the First Workshop on Ontology Learning (OL'2000), Berlin, Germany, August 2000, CEUR Workshop Proceedings Series, Vol. 31, Sun SITE Central Europe (CEUR), Aachen, German, (pp. 73-77). Held in conjunction with the 14th European Conference on Artificial Intelligence ECAI'2000, Berlin, Germany.
2. Baidu. (2006). Retrieved from http://www.baidu.com
3. Banko, M., Brill, E., Dumais, S., & Lin, J. (2002). AskMSR: Question answering using the Worldwide Web. In Proceedings of 2002 AAAI Spring Symposium on Mining Answers from Texts and Knowledge Bases, Palo Alto, California, USA, March 2002, American Association for Artificial Intelligence, Menlo Park, California (pp. 7-8).
4. Brants, T., & Franz, A. (2006). Web 1T 5-Gram Version 1, Linguistic Data Consortium, Philadelphia. Retrieved from http://www.ldc.upenn.edu/Catalog/ CatalogEntry.jsp?catalogId=LDC2006T13
5. Brin, S., & Page, L., The anatomy of a large-scale hypertextual Web search engine. In Proceedings of the Seventh International World Wide Web Conference (WWW7), Brisbane, Australia, April 1998 (pp. 107-117). Amsterdam, Netherland: Elsevier Science B.V.
6. Cambridge University Library. (2006). Chinese classification system. Retrieved from http://www.lib.cam.ac.uk/mulu/class.html
7. Chang, J.S., & Su, K.Y. (1997). An unsupervised iterative method for Chinese new lexicon extraction. International Journal of Computational Linguistics and Chinese Language Processing, 2(2), 97-148.
8. Chen, K.J., & Bai, M.H. (1998). Unknown word detection for Chinese by a corpus-based learning method. International Journal of Computational Linguistics and Chinese Language Processing, 3(1), 27-44.
9. Chen, K.J., & Ma, W.Y. (2002). Unknown word extraction for Chinese documents. In Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002), Taipei, Taiwan, August 2002, Association for Computational Linguistics, Morristown, New Jersy, USA, (vol. 1, pp. 1-7).
10. Church, K., & Hanks, P. (1989). Word association norms, mutual information, and lexicography. In Proceedings of 27th Annual Meeting of the Association for Computational Linguistics (ACL-89), Vancouver, British Columbia, Canada, June 1989, Association for Computational Linguistics, Morristown, New Jersey, USA, (pp. 76-83).
11. Cimiano, P., Handschuh, S., Staab, S. (2004), Towards the self-annotating Web. In Proceedings of the 13th World Wide Web Conference (WWW2004), New York, USA, May 2004 (pp. 462-471). New York, USA: ACM Press.
12. Cormen, T.H., Leiserson, C.E., & Rivest, R.L. (1989). Introduction to Algorithms. Cambridge, MA: The MIT Press.
13. Edmundson, H. (1968). New method in automatic extraction. Journal of the ACM, 16(2), 264-285.
14. Etzioni, O., Cafarella, M., Downey, D., Kok, S., Popescu, A.M., Shaked, T., Soderland, S., Weld, D.S., Yates, A. (2004). Web-scale information extraction in KnowItAll (preliminary results). In Proceedings of the 13th World Wide Web Conference (WWW2004), New York, USA, May 2004 (pp. 100-109). New York, USA: ACM Press.
15. Gan, K.W. (1995). Integrating word boundary identification with sentence understanding. Unpublished doctoral dissertation, National University of Singapore.
16. Ge, X.P., Wanda, P., Padhraic, S. (1999). Discovering Chinese words from unsegmented text. In Proceedings of the 22nd Annual Inernational ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '99) Berkeley, California, USA, ACM Press, New York, USA, (pp. 271-272).
17. Goh, C.L., Asahara, M., & Matsumoto, Y. (2005) Chinese word segmentation by classification of characters. Computational Linguistics and Chinese Language Processing, 10(3), 381-396.
18. Google (2005). Retrieved from http://www.google.com
19. Hawking, D. (2006). Web search engines: Part 2. Computer, 39(8), 88-90.
20. Huang, C.H., Chen, K.J., Chang, L.L., & Chen, F.Y. (1997). Segmentation standard for Chinese natural language processing. International Journal of Computational Linguistics and Chinese Language Processing, 2(2), 47-62.
21. Keller, F., Maria, L., & Olga, O. (2002). Using the Web to overcome data sparseness. In Jan Hajic and Yuji Matsumoto (Eds.). In Proceedings of the ACL-02 Conference on Empirical Methods in Natural Language Processing, Philadelphia, Pennsylvania USA, July 2002, Association for Computational Linguistics, Morristown, New Jersey, USA, (Vol. 10, pp. 230-237)
22. Kit, C., Pan, H., Chen, H. (2002). Learning case-based knowledge for disambiguating Chinese word segmentation: a preliminary study. In Proceedings of the first SIGHAN Workshop on Chinese Language Processing, Taipei, Taiwan, September 2002, Association for Computational Linguistics, Morristown, New Jersey, USA, (Vol. 18, pp. 1-7).
23. Kucera, H., & Francis, W. (1967). Computational analysis of presentday American English. Providence, RI: Brown University Press.
24. Kwok C.C.T., Etzioni, O., & Weld, D.S. (2001). Scaling question answering to the Web. In Proceedings of the 10th International World-Wide Web Conference (WWW10), Hong Kong, May 2001 (pp. 150-161). New York, USA: ACM Press.
25. Language Information Sciences Research Center. (2006). The most common Chinese new words in 2004. Hong Kong: City University of Hong Kong.
26. Leung, C.H., & Kan, W.K. (1996). A statistical learning approach to improving the accuracy of Chinese word segmentation. Literary and Linguistic Computing, 11(2), 87-92.
27. Li, H.Q., Huang, CN., Gao, J.F., & Fan, X.Z. (2004). The use of SVM for Chinese new word identification. In Proceedings of the First International Joint Conference on Natural Language Processing (IJCNLP-04), Sanya City, Hainan Island, China, March 2004, Lecture Notes in Computer Science Series, Vol. 3248, Springer, 2005, Springer, Berlin, Germany, (pp. 723-732).
28. Li, M., Gao, J., Huang, C., & Li, J. (2003). Unsupervised training for over-lapping ambiguity resolution in Chinese word segmentation. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan, July 2003, Association for Computational Linguistics, Morristown, New Jersey, USA. (vol. 17, pp, 1-7).
29. Library of Congress. (2004). New Chinese romanization guidelines. Retrieved fromhttp://www.loc.gov/catdir/cpso/romanization/chinese.pdf
30. Luhn, H. (1958). Automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2), 159-165.
31. Lunde, K. (1998). CJKV information processing: Chinese, Japanese, Korean & Vietnamese computing. New York: O'Reilly.
32. Ma, W.Y., & Chen, K.J. (2003) A bottom-up merging algorithm for Chinese unknown word extraction. In Annual Meeting of the ACL, Proceedings of the Second SIGHAN workshop on Chinese language processing, Sapporo, Japan, 31-38.
33. Mihalcea, R., & Moldovan, D. (1999). A method for word sense disambiguation of unrestricted text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL-99), College Park, Maryland, USA, June 1999, Association for Computational Linguistics, Morristown, New Jersey, USA, (pp. 152-158).
34. Moukdad, H., & Cui, H. (2005). How do search engines handle Chinese queries? Webology, 2(3), Article 17. Retrieved from http://www.Webology.ir/2005/ v2n3/a17.html
35. National Library of Australia. (2002). Kinetica Chinese Japanese Korean (CJK) service report. Retrieved from http://www.nla.gov.au/librariesaustralia/ cjk/events/cjkreport2002.html
36. National People's Congress. (1958). Han yu pin yin fang an, Beijing Author.
37. Nie, J.Y., Jin, W., Hannaan, M.L. (1994). A hybrid approach to unknown word detection and segmentation of Chinese. In Proceedings of International Conference on Chinese Computing (ICCC-94), Singapore, June 1994, National University of Singapore, Singapore, (pp. 326-335).
38. Pan, W., Yip, P.C., & Han, Y.S. (1993). Studies in Chinese word-formation. Taipei: Student Book Company.
39. Radev, D.R., Qi, H., Zheng, Z., Blair-Goldensohn, S., Zhang, Z., Fan, W., Prager, J. (2001). Mining the Web for answers to natural language questions. In Proceedings of the tenth International Conference on Information and Knowledge Management (CIKM 2001), Atlanta, Georgia, USA, November 2001 (pp. 143-150). New York, USA: ACM Press.
40. Rijsbergen, C.V. (1979). Information retrieval (2nd ed.). Glascow: Department of Computer Science, University of Glasgow.
41. Salton, G., & Buckley, C. (1988). Term-weighting approaches in automatic text retrieval. Information Processing and Management, 24, 513-523.
42. Sampson, G. (1989). How fully does a machine-usable dictionary cover English text? Literary and Linguistic Computing, 4, 29-35.
43. Soanes, C., Stevenson, A. (2005). Oxford Dictionary of English, Oxford, UK: Oxford University Press.
44. Sproat, R., Emerson, T. (2003). The first international Chinese word segmentation bakeoff. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan, July 2003, Association for Computational Linguistics, Morristown, New Jersey, USA. (vol. 17, pp. 133-143).
45. Sproat, R., & Shih, C. (1990). A statistical method for finding word boundaries in Chinese text. Computer Processing of Chinese and Oriental Languages, 4, 336-351.
46. Sproat, R., Shih, C. (2001). Corpus-based methods in Chinese morphology and phonology. 2001 Linguistic Society of America Institute (LSA 2001), University of California - Santa Barbara, California, USA. Retrieved from, http://compling.ai.uiuc.edu/rws/newindex/notes.pdf
47. Sproat, R., Shih C, Gale, W., Chang, N. (1996). A Stochastic Finite-State Word-Segmentation Algorithm for Chinese, Computational Linguistics, 22(3), 377-404.
48. State Bureau of Technology Supervision. (1992). Contemporary Chinese languages word-segmentation specification for information processing. Beijing: Author.
49. Sun, M.S., Shen, D.Y., T'sou, B.K. (1998). Chinese word segmentation without using lexicon and hand-crafted training data. In Proceedings of the 17th International Conference on Computational Linguistics (COLING 1998), Montreal, Quebec, Canada, August 1998, Association for Computational Linguistics, Morristown, New Jersey, USA, (vol. 2, pp. 1265-1271).
50. Teahan, W.J., Wen, Y.Y., McNad, R., & Witten I. (2000). A compression-based algorithm for Chinese word segmentation. Computational Linguistics, 26(3), 375-393.
51. Tsai, J.L., Sung, C.L., Hsu W.L. (2003) Chinese word auto-confirmation agent. In Proceedings of Fifth Conference on Computational Linguistics and Speech Processing (ROCLING XV), Hsinchu, Taiwan, September 2003, Association for Computational Linguistics and Chinese Language Processing, Taipei, Taiwan, (pp. 175-192).
52. Wang, Y.H., Su, H.J., & Mo, Y. (1990). Automatic processing of Chinese words. Journal of Chinese Information Processing. 4(4), 1-11.
53. Wettler, M., Rapp, R., (1993) Computation of word associations based on the co-occurrences of words in large corpora. In Proceedings of the First Workshop on Very Large Corpora (WVLC-1), Columbus, Ohio, June 1993, Association for Computational Linguistics, Morristown, New Jersey, USA, (pp. 84-93).
54. Wu, Z., & Tseng, G. (1993). Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science, 44(9), 532-542.
55. Xia, F. (1999). Segmentation guideline. Chinese Treebank Project. Technical Report, University of Pennsylvania. Retrieved from http://morph/ldc.upenn.edu/ctb/
56. Yang, C. C., & Li, K. W. (2003). Segmenting Chinese unknown words by heuristic method. In Proceedings of the International Conference on Asia Digital Libraries, Malaysia, December 2003, Lecture Notes in Computer Science, vol. 2911, Springer, Berlin, Germany, (pp. 510-520).
57. Yang, C. C., & Li, K. W. (2004), Error analysis of Chinese text segmentation using statistical approach. In Proceedings of ACM/IEEE Joint Conference on Digital Libraries, Tucson, Arizona, USA, June 2004 (pp. 256-257), New York, USA: ACM Press.
58. Yang, C.C., & Li, K.W. (2005). A heuristic method based on a statistical approach for Chinese text segmentation. Journal of the American Society for Information Science and Technology, 56(13), 1438-1447.
59. Yang, C.C., Luk, J., Yung, J., & Yen, J. (2000). Combination and boundary detection approach for Chinese indexing. Journal of the American Society for Information Science, Special Topic Issue on Digital Libraries, 51(4), 340-351.
60. Yang, J., Senellart, J., Zajac, R. (2003). SYSTRAN'S Chinese word segmentation. In Proceedings of the Second SIGHAN Workshop on Chinese Language Processing, Sapporo, Japan, July 2003, Association for Computational Linguistics, Morristown, New Jersey, USA. (vol. 17, pp. 180-183).
61. Zipf, G.K. (1949). Human behavior and the principle of least effort. Cambridge, Massachusetts, USA: Addison-Wesley.