Text preprocessing

Handbook of Natural Language Processing, Second Edition

Volumn , Issue , 2010, Pages 9-30

  Palmer, David D ^a  

^a Advanced Methods Group   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

NATURAL LANGUAGE PROCESSING SYSTEMS;

CORPUS SIZE; HUMAN LANGUAGE; LINGUISTIC ANALYSIS; NATURAL LANGUAGE TEXT; NATURAL LANGUAGES; TEXT CORPORA; TEXT PREPROCESSING; WRITING SYSTEMS;

LINGUISTICS;

EID: 84893263751     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (45)

1. Aberdeen, J., J. Burger, D. Day, L. Hirschman, P. Robinson, and M. Vilain (1995). MITRE: Description of the Alembic system used for MUC-6. In Proceedings of the Sixth Message Understanding Conference (MUC-6), Columbia, MD.
2. Aho, A. V., R. Sethi, and J. D. Ullman (1986). Compilers, Principles, Techniques, and Tools. Reading, MA: Addison-Wesley Publishing Company.
3. Ando, R. K. and L. Lee (2003). Mostly-unsupervised statistical segmentation of Japanese Kanji sequences.Journal of Natural Language Engineering 9, 127-149.
4. Aroonmanakun, W. (2002). Collocation and THai word segmentation. In Proceedings of SNLPCOCOSDA2002, Bangkok, Thailand.
5. Baroni, M., F. Chantree, A. Kilgarriff, and S. Sharoff (2008). Cleaneval: A competition for cleaning web pages. In Proceedings of the Sixth Language Resources and Evaluation Conference (LREC 2008), Marrakech, Morocco.
6. Bayer, S., J. Aberdeen, J. Burger, L. Hirschman, D. Palmer, and M. Vilain (1998). Theoretical and computational linguistics: Toward a mutual understanding. In J. Lawler and H. A. Dry (Eds.), Using Computers in Linguistics. London, U.K.: Routledge.
7. Breiman, L., J. H. Friedman, R. Olshen, and C. J. Stone (1984). Classification and Regression Trees. Belmont, CA: Wadsworth International Group.
8. Chang, P.-C., M. Galley, and C. D. Manning (2008). Optimizing Chinese word segmentation for machine translation performance. In Proceedings of the Third Workshop on Statistical Machine Translation, Columbus, OH, pp. 224-232
9. Comrie, B., S. Matthews, and M. Polinsky (1996). The Atlas of Languages. London, U.K.: Quarto Inc.
10. Crystal, D. (1987). The Cambridge Encyclopedia of Language. Cambridge, U.K.: Cambridge University Press.
11. Daniels, P. T. and W. Bright (1996). The World’s Writing Systems. New York: Oxford University Press.
12. Francis, W. N. and H. Kucera (1982). Frequency Analysis of English Usage. New York: Houghton Mifflin Co.
13. Fung, P. and D. Wu (1994). Statistical augmentation of a Chinese machine-readable dictionary. In Proceedings of Second Workshop on Very Large Corpora (WVLC-94), Kyoto, Japan.
14. Gao, J., M. Li, A. Wu, and C.-N. Huang (2005). Chinese word segmentation and named entity recognition: A pragmatic approach. Computational Linguistics 31(4), 531-574.
15. Grefenstette, G. and P. Tapanainen (1994). What is a word, What is a sentence? Problems of Tokenization. In The 3rd International Conference on Computational Lexicography (COMPLEX 1994), Budapest, Hungary.
16. Hankamer, J. (1986). Finite state morphology and left to right phonology. In Proceedings of the Fifth West Coast Conference on Formal Linguistics, Stanford, CA.
17. Hockenmaier, J. and C. Brew (1998). Error driven segmentation of Chinese. Communications of COLIPS 8(1), 69-84.
18. Kawtrakul, A., C. Thumkanon, T. Jamjanya, P. Muangyunnan, K. Poolwan, and Y. Inagaki (1996). Agradual refinement model for a robust Thai morphological analyzer. In Proceedings of COLING96, Copenhagen, Denmark.
19. Kiss, T. and J. Strunk (2006). Unsupervised multilingual sentence boundary detection. Computational Linguistics 32(4), 485-525.
20. Liberman, M. Y. and K. W. Church (1992). Text analysis and word pronunciation in text-to-speech synthesis. In S. Furui and M. M. Sondhi (Eds.), Advances in Speech Signal Processing, pp. 791-831. New York: Marcel Dekker, Inc.
21. Ma, Y. and A. Way (2009). Bilingually motivated domain-adapted word segmentation for statistical machine translation. In Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), Athens, Greece, pp. 549-557
22. Martin, J., H. Johnson, B. Farley, and A. Maclachlan (2003). Aligning and using an english-inuktitut parallel corpus. In Proceedings of the HLT-NAACL 2003 Workshop on Building and Using Parallel Texts Data Driven: Machine Translation and Beyond, Edmonton, Canada, pp. 115-118
23. Matsumoto, Y. and M. Nagao (1994). Improvements of Japanese morphological analyzer JUMAN. In Proceedings of the International Workshop on Sharable Natural Language Resources, Nara, Japan.
24. Matsumoto, Y., A. Kitauchi, T. Yamashita, Y. Hirano, O. Imaichi, and T. Imamura (1997). Japanese morphological analysis system ChaSen manual. Technical Report NAIST-IS-TR97007, Nara Institute of Science and Technology, Nara, Japan (in Japanese).
25. Meknavin, S., P. Charoenpornsawat, and B. Kijsirikul (1997). Feature-based Thai word segmentation. In Proceedings of the Natural Language Processing Pacific Rim Symposium 1997 (NLPRS97), Phuket, Thailand.
26. Mikheev, A. (2002). Periods, capitalized words, etc. Computational Linguistics 28(3), 289-318.
27. Müller, H., V. Amerl, and G. Natalis (1980). Worterkennungsverfahren als Grundlage einer Universalmethode zur automatischen Segmentierung von Texten in Sätze. Ein Verfahren zur maschinellen Satzgrenzenbestimmung im Englischen. Sprache und Datenverarbeitung 1.
28. Nagata, M. (1994). Astochastic Japanese morphological analyzer using a Forward-DP backward A* n-best search algorithm. In Proceedings of COLING94, Kyoto, Japan.
29. Nicol, G. T. (1993). Flex-The Lexical Scanner Generator. Cambridge, MA:The Free Software Foundation.
30. Nunberg, G. (1990). The Linguistics of Punctuation. C.S.L.I. Lecture Notes, Number 18. Stanford, CA: Center for the Study of Language and Information.
31. Palmer, D. D. (1997). A trainable rule-based algorithm for word segmentation. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics (ACL97), Madrid, Spain.
32. Palmer, D. D. and M. A. Hearst (1997). Adaptive multilingual sentence boundary disambiguation. Computational Linguistics 23(2), 241-67.
33. Park, Y. and R. J. Byrd (2001). Hybrid text mining for finding abbreviations and their definitions. In Proceedings of the 2001 Conference on Empirical Methods in Natural Language Processing, Pittsburgh, PA.
34. Reynar, J. C. and A. Ratnaparkhi (1997). A maximum entropy approach to identifying sentence boundaries. In Proceedings of the Fifth ACL Conference on Applied Natural Language Processing, Washington, DC.
35. Riley, M. D. (1989). Some applications of tree-based modelling to speech and language indexing. In Proceedings of the DARPA Speech and Natural Language Workshop, San Mateo, CA, pp. 339-352. Morgan Kaufmann.
36. Sampson, G. R. (1995). English for the Computer. Oxford, U.K.: Oxford University Press.
37. Sproat, R. and T. Emerson (2003). The first international Chinese word segmentation bakeoff. In Proceedings of the Second SigHan Workshop on Chinese Language Processing, Sapporo, Japan.
38. Sproat, R. and C. Shih (2001). Corpus-based methods in Chinese morphology and phonology. Technical Report, Linguistic Society of America Summer Institute, Santa Barbara, CA.
39. Sproat, R. W., C. Shih, W. Gale, and N. Chang (1996). A stochastic finite-state word-segmentation algorithm for Chinese. Computational Linguistics 22(3), 377-404.
40. Teahan, W.J., Y. Wen, R. McNab, and I. H. Witten (2000). A compression-based algorithm for Chinese word segmentation. Computational Linguistics 26(3), 375-393.
41. Unicode Consortium (2006). The Unicode Standard, Version 5.0. Boston, MA: Addison-Wesley.
42. Wu, A. (2003). Customizable segmentation of morphologically derived words in Chinese. International Journal of Computational Linguistics and Chinese Language Processing 8(1), 1-27.
43. Wu, D. and P. Fung (1994). Improving Chinese tokenization with linguistic filters on statistical lexical acquisition. In Proceedings of the Fourth ACL Conference on Applied Natural Language Processing, Stuttgart, Germany.
44. Wu, Z. and G. Tseng (1993). Chinese text segmentation for text retrieval: Achievements and problems. Journal of the American Society for Information Science 44(9), 532-542.
45. Zhang, R., K. Yasuda, and E. Sumita (2008). Improved statistical machine translation by multiple Chinese word segmentation. In Proceedings of the Third Workshop on Statistical Machine Translation, Columbus, OH, pp. 216-223