Rewriting the orthography of SMS messages

Natural Language Engineering

Volumn 16, Issue 2, 2010, Pages 133-159

  Yvon, François ^a  

^a Laboratoire Interdisciplinaire des Sciences du Numérique   (France)

Author keywords

[No Author keywords available]

Indexed keywords

AUTOMATIC NORMALIZATION; MEDIATED INTERACTION; PHRASE-BASED STATISTICAL MACHINE TRANSLATION; SMS MESSAGES; SYSTEM-BASED; WEIGHTED FINITE-STATE TRANSDUCERS; WRITTEN TEXTS;

INFORMATION THEORY; SPEECH TRANSMISSION;

LINGUISTICS;

EID: 77953580312     PISSN: 13513249     EISSN: 14698110     Source Type: Journal    
DOI: 10.1017/S1351324909990258     Document Type: Article

References (48)

1. Allauzen, C., Mohri, M., and Roark, B. 2005. The design principles and algorithms of a weighted grammar library. International Journal of Foundations of Computer Science 16(3): 403-421
2. Anis, J. 2001. Parlez-vous textoi Guide des nouveaux langages du ŕeseau. Paris, France: É ditions du Cherche Midi.
3. Aw, A., Zhang, M., Xiao, J., and Su, J. 2006. A phrase-based statistical model for SMS text normalization. In N. Calzolari, C. Cardie, and P. Isabelle (eds.) Proceedings of COLING/Association for Computational Linguistics, pp. 33-40. Sydney, Australia: the Association for Computational Linguistics.
4. Bazzi, I., and Glass, J. 2000. Modelling OOV words for robust speech recognition. In Proceedings of the Internatinal Conference on Spoken Language Processing (ICSLP), pp. 401-404 Beijing, China: International Speech Communication Association (ISCA).
5. Beaufort, R., Roekhaut, S., and Fairon, C. 2008. D́efinition d'un syst̀eme d'alignement SMS/franciais standard ̀a l'aide d'un filtre de composition. In S. Heiden, and B. Pincemin (eds.) Actes des Jourńees Internationales de l'Analyse des Donńees Textuelles (JADT), pp. 55-166. Lyon: Presses Universitaires de Lyon.
6. B́echet, F. 2001. LIA PHON: Un syst̀eme complet de phońetisation de textes. Traitement Automatique des Langues 42(1): 47-67.
7. Boula de Mareüil, P., d'Alessandro, C., Yvon, F., Auberǵ e, V., Vaissìere, J., and Amelot, A. 2000. A French phonetic lexicon with variants for speech and language processing. In Proceedings of the 2nd Language Resources Engineering Conference (LREC), vol.I, pp. 273-276, Athens, Greece.
8. Brill, E., and Moore, R. C. 2000. An improved error model for noisy channel spelling correction. In C.-N. Huang, and K. Vijay-Shankar, K. (eds.) Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pp. 286-293 Hong Kong: the Association for Computational Linguistics.
9. Brown, P. F., Cocke, J., Pietra, S. D., Pietra, V. J. D., Jelinek, F., Lafferty, J. D., Mercer, R. L., and Roossin, P. S. 1990. A statistical approach to machine translation. Computational Linguistics 16(2): 79-85.
10. Chen, S. F., and Goodman, J. T. 1996. An empirical study of smoothing techniques for language modeling. In A. Joshi, and M. Palmer (eds.) Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pp. 310-318 Santa Cruz, NM: the Association for Computational Linguistics.
11. Choudhury, M., Saraf, R., Jain, V., Sarkar, S., and Basu, A. 2007. Investigation and modeling of the structure of texting language. In C. Knoblock, D. Lopresti, S. Roy, and L. Venkata Subramaniam (eds.) Proceedings of the IJCAI Workshop on 'Analytics for Noisy Unstructured Text Data', pp. 63-70. Hyderabad, India: International Association for Pattern Recognition.
12. Church, K. W., and Gale, W. 1991. Probability scoring for spelling correction. Statistics and Computing 1: 91-103.
13. Clark, A. 2003. Pre-processing very noisy text. In Proceedings of Workshop on Shallow Processing of Large Corpora, Lancaster, UK.
14. Crystal, D. 2001. Language and the Internet. Cambridge, UK: Cambridge University Press.
15. Divay, M., and Vitale, A. J. 1997. Algorithm for grapheme-to-phoneme translation for French and English: applications. Computational Linguistics 23(4): 495-524.
16. Eisner, J. 2002. Parameter estimation for probabilistic finite-state transducers. In E. Charniak, and D. Lin (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 1-8. Philadelphia, PA: the Association for Computational Linguistics.
17. Fairon, C., Klein, J. R., and Paumier, S. 2006. Le langage SMS. Louvain, Belgium: UCL Presses.
18. Fairon, C., and Paumier, S. 2006. A translated corpus of 30,000 French SMS. In Proceedings of LREC 2006, Genoa, Italy.
19. Gillick, L., and Cox, S. 1989. Some statistical issues in the comparison of speech recognition algorithm. In Proceedings of the IEEE Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 532-535, Glasgow, UK.
20. Golding, A. R., and Roth, D. 1999. A winnow-based approach to context-sensitive spelling correction. Machine Learning 34: 107-130
21. Golding, A. R., and Schabes, Y. 1996. Combining trigram-based and feature-based methods for context-sensitive spelling correction. In A. Joshi, and M. Palmer (eds.) the Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pp. 71-78 Santa Cruz, CA: the Association for Computational Linguistics.
22. Guimier de Neef, E., Debeurme, A., and Park, J. 2007. TILT correcteur de SMS: ́evaluation et bilan quantitatif. In Actes de la Conf́erence sur le Traitement Automatique des Langues (TALN'07), pp. 123-132, Toulouse, France.
23. Jansche, M. 2003. Inference of string mappings for language technology. PhD thesis, Ohio State University.
24. Jelinek, F. 1990. Self-organized language modeling for speech recognition. In Waibel, A., and Lee, K.-F. (eds.), Readings in Speech Recognition, pp. 450-506, San Mateo, CA: Morgan- Kaufman.
25. Kaplan, R.M., and Kay,M. 1994. Regular models of phonological rule systems. Computational Linguistics 20(3): 331-378 (First appeared as a paper presented to the Winter Meeting of the Linguistic Society of America, New York, 1981.).
26. Kobus, C., Yvon, F., and Damnati, G. 2008a. Normalizing SMS: are two metaphors better than one? In D. Scott, and H. Uszkoreit (eds.) Proceedings of the 22nd International Conference on Computational Linguistics (COLING 2008), pp. 441-448 Manchester, UK: the Association for Computational Linguistics.
27. Kobus, C., Yvon, F., and Damnati, G. 2008b. Transcrire les SMS comme on reconnâit la parole. In Actes de la Conf́erence sur le Traitement Automatique des Langues (TALN'08), pp. 128-138, Avignon, France.
28. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. 2007. Moses: open source toolkit for statistical machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Demonstration Session, Prague, Czech Republic.
29. Koehn, P., Och, F. J., and Marcu, D. 2003. Statistical phrase-based translation. In M. Hearst, and M. Ostendorf (eds.) Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistic, pp. 127-133 Edmondton, AB, Canada: the Association for Computational Linguistics.
30. Kukich, K. 1992. Techniques for automatically correcting words in text. Computing Surveys 24(4): 377-439.
31. Lita, L. V., Ittycheriah, A., Roukos, S., and Kambhatla, N. 2003. tRuEcasIng. In E. W. Hinrichs, and D. Roth (eds.) Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pp. 152-159 Sapporo, Japan: the Association for Computational Linguistics.
32. Mikheev, A. 2000. Document centered approach to text normalization. In SIGIR '00: Proceedings of the 23rd annual International ACM SIGIR conference on Research and development in information retrieval, pp. 136-143 New York: Association for Computing Machinery.
33. Mitton, R. 1996. Spellchecking by computer. Journal of the Simplified Spelling Society 20(1): 4-11.
34. Mohri, M. 1997a. On the use of sequential transducers in natural language processing. In Roche, E., and Schabes, Y. (eds.), Finite State Natural Language Processing, Cambridge, MA: MIT Press.
35. Mohri, M. 1997b. Transducers in language and speech. Computational Linguistics 23(2): 269-311.
36. Mohri, M., Pereira, F., and Riley, M. 2000. The design principles of a weighted finite-state transducer library. Theoretical Computer Science (231): 17-32.
37. Mohri, M., and Sproat, R. W. 1996. An efficient compiler for weighted rewrite rules. In A. Joshi, and M. Palmer (eds.) Proceedings of the annual Meeting of the Association for Computational Linguistics, pp. 231-238 Santa Cruz, CA: the Association for Computational Linguistics.
38. Och, F. J. 2003. Minimum error rate training in statistical machine translation. In E. W. Hinrichs, and D. Roth (eds.) Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pp. 160-167 Sapporo, Japan: the Association for Computational Linguistics.
39. Och, F. J., and Ney, H. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics 29(1): 19-51.
40. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. 2001. Bleu: a method for automatic evaluation of machine translation. Technical Report RC22176 (W0109-022), IBM Research Division, Thomas J. Watson Research Center.
41. Roche, E., and Schabes, Y. 1997. Introduction to finite-state devices in natural language processing. In Roche, E., and Schabes, Y. (eds.), Finite State Natural Language Processing, pp. 1-66, Cambridge, MA: MIT Press.
42. Simard, M., and Deslauriers, A. 2001. Real-time automatic insertion of accents in French text. Journal of Natural Language Engineering 7(2): 143-165
43. Sproat, R., Black, A., Chen, S., Kumar, S., Ostendorf,M., and Richards, C. 2001. Normalization of non-standard words. Computer Speech and Language 15(3): 287-333.
44. Stolcke, A. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Langage Processing (ICSLP), vol.2, pp. 901-904, Denver, CO.
45. Toutanova, K., and Moore, R. 2002. Pronunciation modeling for improved spelling correction. In E. Charniak, and D. Lin (eds.) Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 144-151 Philadelphia, PA: the Association for Computational Linguistics.
46. V́eronis, J., and Guimier de Neef, E. 2006. Le traitement des nouvelles formes de communication ́ecrite. In Sabah, G. (ed.), Compŕehension automatique des langues et interaction, pp. 227-248 Paris: Herm̀es Science.
47. Yvon, F., de Mareüil, P. B., d'Alessandro, C., Auberǵ e, V., Bagein, M., Bailly, G., B́echet, F., Foukia, S., Goldman, J.-P., Keller, E., O'Shaughnessy, D., Pagel, V., Sannier, F., V́eronis, J., and Zellner, B. 1998. Objective evaluation of grapheme-to-phoneme conversion for text-tospeech synthesis in French. Computer, Speech and Language 12(4): 393-410.
48. Zhu, C., Tang, J., Li, H., Ng, H. T., and Zhao, T. 2007. A unified tagging approach to text normalization. In A. Zaenen, and A. van den Bosch (eds.) Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 688-695 Prague, Czech Republic: the Association for Computational Linguistics.