Lexical normalization for social media text

ACM Transactions on Intelligent Systems and Technology

Volumn 4, Issue 1, 2013, Pages

  Han, Bo ^a,b   Cook, Paul ^a   Baldwin, Timothy ^a,b  

^a CSIRO   (Australia)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Lexical normalization; Microblog; Short text message; Text analysis; Text preprocessing

Indexed keywords

LEXICAL NORMALIZATION; MICRO-BLOG; SHORT TEXT MESSAGES; TEXT ANALYSIS; TEXT PREPROCESSING;

TELEPHONE SYSTEMS;

SOCIAL NETWORKING (ONLINE);

EID: 84873657672     PISSN: 21576904     EISSN: 21576912     Source Type: Journal    
DOI: 10.1145/2414425.2414430     Document Type: Article

References (56)

1. AW, A., ZHANG, M., XIAO, J., AND SU, J. 2006. A phrase-based statistical model for SMS text normalization. In Proceedings of the COLING/ACL Main Conference Poster Sessions. 33-40.
2. BALDWIN, T. AND LUI, M. 2010. Language identification: The long and the short of the matter. In Proceedings of the Human Language Technologies Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT'10). 229-237.
3. BENSON, E., HAGHIGHI, A., AND BARZILAY, R. 2011. Event discovery in social media feeds. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 389-398.
4. BRANTS, T. AND FRANZ, A. 2006. Web 1T 5-gram Version 1.
5. BRILL, E. AND MOORE, R. C. 2000. An improved error model for noisy channel spelling correction. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. 286-293.
6. BURTON, K., JAVA, A., AND SOBOROFF, I. 2009. The ICWSM 2009 spinn3r dataset. In Proceedings of the 3rd Annual Conference on Weblogs and Social Media.
7. CHOUDHURY, M., SARAF, R., JAIN, V., MUKHERJEE, A., SARKAR, S., AND BASU, A. 2007. Investigation and modeling of the structure of texting language. Int. J. Document Anal. Recogn. 10, 157-174.
8. CONTRACTOR, D., FARUQUIE, T. A., AND SUBRAMANIAM, L. V. 2010. Unsupervised cleansing of noisy text. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters. 189-196.
9. COOK, P. AND STEVENSON, S. 2009. An unsupervised model for text message normalization. In Proceedings of the Workshop on Computational Approaches to Linguistic Creativity (CALC'09) 71-78.
10. DAVID GRAFF, C. C. 2003. English Gigaword. Linguistic Data Consortium. http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2003T05.
11. DE MARNEFFE, M., MACCARTNEY, B., AND MANNING, C. D. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC'06).
12. FAN, R.-E., CHANG, K.-W., HSIEH, C.-J., WANG, X.-R., AND LIN, C.-J. 2008. LIBLINEAR: A library for large linear classification. J. Mach. Learn. Res. 9, 1871-1874.
13. FOSTER, J., Ç ETINOGLU, O., WAGNER, J., ROUX, J. L., HOGAN, S., NIVRE, J., HOGAN, D., AND VAN GENABITH, J. 2011. #hardtoparse: POS tagging and parsing the twitterverse. In Analyzing Microtext: Papers from the 2011 AAAI Workshop, 20-25.
14. GIMPEL, K., SCHNEIDER, N., O'CONNOR, B., DAS, D., MILLS, D., EISENSTEIN, J., HEILMAN, M., YOGATAMA, D., FLANIGAN, J., AND SMITH, N. A. 2011. Part-of-Speech tagging for twitter: Annotation, features, and experiments. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 42-47.
15. GONZÁLEZ-IBÁÑEZ, R., MURESAN, S., AND WACHOLDER, N. 2011. Identifying sarcasm in twitter: A closer look. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies: Short Papers. Vol. 2, 581-586.
16. GOUWS, S., HOVY, D., AND METZLER, D. 2011. Unsupervised mining of lexical variants from noisy text. In Proceedings of the Ist Workshop on Unsupervised Learning in NLP. 82-90.
17. HAN, B. AND BALDWIN, T. 2011. Lexical normalisation of short text messages: Makn sens a #twitter. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 368-378.
18. HAN, B., COOK, P., AND BALDWIN, T. 2012. Automatically constructing a normalisation dictionary for microblogs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Natural Language Learning (EMNLP-CoNLL'12). To appear.
19. HOW, Y. AND KAN, M.-Y. 2005. Optimizing predictive text entry for short message service on mobile phones. In Proceedings of the Inernational Conference on Human Computer Interfaces International (HCII 05).
20. IZUMI, E., UCHIMOTO, K., SAIGA, T., SUPNITHI, T., AND ISAHARA, H. 2003. Automatic error detection in the Japanese learners' english spoken data. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics. Vol. 2, 145-148.
21. JARVELIN, K. AND KEKALAINEN, J. 2002. Cumulated gain-based evaluation of IR techniques. ACM Trans. Inf. Syst. 20, 4.
22. JIANG, L., YU, M., ZHOU, M., LIU, X., AND ZHAO, T. 2011. Target-Dependent twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics. 151-160.
23. KAUFMANN, J. AND KALITA, J. 2010. Syntactic normalization of twitter messages. In Proceedings of the International Conference on Natural Language Processing.
24. KLEIN, D. AND MANNING, C. D. 2003. Fast exact inference with a factored model for natural language parsing. In Proceedings of the Conference on Advances in Neural Information Processing Systems 15 (NIPS'02). 3-10.
25. KOEHN, P., HOANG, H., BIRCH, A., CALLISON-BURCH, C., FEDERICO, M., BERTOLDI, N., COWAN, B., SHEN, W., MORAN, C., ZENS, R., DYER, C., BOJAR, O., CONSTANTIN, A., AND HERBST, E. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions. 177-180.
26. KULLBACK, S. AND LEIBLER, R. A. 1951. On information and sufficiency. Ann. Math. Statist. 22, 49-86.
27. LAFFERTY, J. D., MCCALLUM, A., AND PEREIRA, F. C. N. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the 18th International Conference on Machine Learning. 282-289.
28. LEVENSHTEIN, V. I. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10, 707-710.
29. LI, M., ZHANG, Y., ZHU, M., AND ZHOU, M. 2006. Exploring distributional similarity based models for query spelling correction. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics. 1025-1032.
30. LIN, D. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 17th International Conference on Computational Linguistics. 768-774.
31. LIN, J. 1991. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory 37, 1, 145-151.
32. LIU, F., WENG, F., AND JIANG, X. 2012. A broad-coverage normalization system for social media language. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL'12).
33. LIU, F., WENG, F., WANG, B., AND LIU, Y. 2011a. Insertion, deletion, or substitution? Normalizing text messages without pre-categorization nor supervision. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 71-76.
34. LIU, X., ZHANG, S., WEI, F., AND ZHOU, M. 2011b. Recognizing named entities in tweets. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. 359-367.
35. LODHI, H., SAUNDERS, C., SHAWE-TAYLOR, J., CRISTIANINI, N., AND WATKINS, C. 2002. Text classification using string kernels. J. Mach. Learn. Res. 2, 419-444.
36. LUI, M. AND BALDWIN, T. 2011. Cross-Domain feature selection for language identification. In Proceedings of 5th International Joint Conference on Natural Language Processing. 553-561.
37. O'CONNOR, B., KRIEGER, M., AND AHN, D. 2010. TweetMotif: Exploratory search and topic summarization for twitter. In Proceedings of the 4th International Conference on Weblogs and Social Media (ICWSM'10). 384-385.
38. PAPINENI, K., ROUKOS, S., WARD, T., AND ZHU, W.-J. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. 311-318.
39. PETERSON, J. L. 1980. Computer programs for detecting and correcting spelling errors. Comm. ACM 23, 676-687.
40. PHILIPS, L. 2000. The double metaphone search algorithm. C/C++ Users J. 18, 38-43.
41. RABINER, L. R. 1989. A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77, 2, 257-286.
42. RITTER, A., CHERRY, C., AND DOLAN, B. 2010. Unsupervised modeling of twitter conversations. In Proceedings of the Human Language Technologies Annual Conference of the North American Chapter of the Association for Computational Linguistics (HLT'10). 172-180.
43. RITTER, A., CLARK, S., MAUSAM, AND ETZIONI, O. 2011. Named entity recognition in tweets: An experimental study. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. 1524-1534.
44. SAKAKI, T., OKAZAKI, M., AND MATSUO, Y. 2010. Earthquake shakes twitter users: Real-Time event detection by social sensors. In Proceedings of the 19th International Conference on World Wide Web. 851-860.
45. SHANNON, C. E. 1948. A mathematical theory of communication. Bell Syst. Tech. J. 27, 379-423, 623-656.
46. SPROAT, R., BLACK, A. W., CHEN, S., KUMAR, S., OSTENDORF, M., AND RICHARDS, C. 2001. Normalization of nonstandard words. Comput. Speech Lang. 15, 3, 287-333.
47. STOLCKE, A. 2002. Srilm-an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing. 901-904.
48. SUN, G., CONG, G., LIU, X., LIN, C.-Y., AND ZHOU, M. 2007. Mining sequential patterns and tree patterns to detect erroneous sentences. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics. 81-88.
49. THURLOW, C. 2003. Generation txt? The sociolinguistics of young people's text-messaging. Discourse Anal. Online 1, 1.
50. TOUTANOVA, K., KLEIN, D., MANNING, C. D., AND SINGER, Y. 2003. Feature-Rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. Vol. 1, (NAACL'03). Association for Computational Linguistics, 173-180.
51. TOUTANOVA, K. AND MOORE, R. C. 2002. Pronunciation modeling for improved spelling correction. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL'02). 144-151.
52. TWITTER. 2011. 200 million tweets per day. http://techcrunch.com/2012/12/ 18/twitter-passes-200m-monthly-active-users-a-42-increase-over-9-months//
53. WENG, J. AND LEE, B.-S. 2011. Event detection in Twitter. In Proceedings of the 5th International AAAI Conference on Weblogs and Social Media.
54. WONG, W., LIU, W., AND BENNAMOUN, M. 2006. Integrated scoring for spelling error correction, abbreviation expansion and case restoration in dirty text. In Proceedings of the 5th Australasian Conference on Data Mining and Analytics. 83-89.
55. XUE, Z., YIN, D., AND DAVISON, B. D. 2011. Normalizing microtext. In Proceedings of the AAAI-11 Workshop on Analyzing Microtext. 74-79.
56. YEH, A. 2000. More accurate tests for the statistical significance of result differences. In Proceedings of the 18th Conference on Computational Linguistics. Vol. 2. 947-953.