Language identification of short text segments with n-gram models

Proceedings of the 7th International Conference on Language Resources and Evaluation, LREC 2010

Volumn , Issue , 2010, Pages 3423-3430

  Vatanen, Tommi ^a   Väyrynen, Jaakko J ^a   Virpioja, Sami ^a  

^a AALTO UNIVERSITY   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

CLASSIFIERS; NATURAL LANGUAGE PROCESSING SYSTEMS;

IDENTIFICATION ACCURACY; LANGUAGE IDENTIFICATION; LANGUAGE MODEL; N-GRAM MODELING; NAIVE BAYES CLASSIFIERS; RANKING METHODS; SMOOTHING TECHNIQUES; STATE OF THE ART;

COMPUTATIONAL LINGUISTICS;

EID: 84910031762     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (27)

1. Kenneth Beesley. 1988. Language identifier: A computer program for automatic natural-language identification of on-line text. In Proceedings of the 29th ATA Annual Conference, pages 47-54.
2. Gerrit Reinier Botha and Etienne Barnard. 2007. Factors that affect the accuracy of text-based language identification. In Proceedings of PRASA 2007, pages 7-10.
3. William B. Cavnar and John M. Trenkle. 1994. N-gram-based text categorization. In Proceedings of SDAIR'94, pages 161-175.
4. Stanley F. Chen and Joshua Goodman. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4):359-393.
5. Ron Cole, Joseph Mariani, Hans Uszkoreit, Giovanni Varile, Annie Zaenen, Victor Zue, and Antonio Zampolli, editors. 1997. Survey of the State of the Art in Human Language Technology. Cambridge University Press. Web Edition in PDF.
6. Joaquim Ferreira da Silva and Gabriel Pereira Lopes. 2006. Identification of document language is not yet a completely solved problem. In Proceedings of CIMCA'06, pages 212-219.
7. Marc Damashek. 1995. Gauging similarity with n-grams: Language-independent categorization of text. Science, 267(5199):843-849.
8. Jacques Duchateau, Kris Demuynck, and Patrick Wambacq. 2002. Confidence scoring based on backward language models. In Proceedings of ICASSP 2002, Volume 1, pages 221-224.
9. Ted Dunning. 1994. Statistical identification of language. Technical Report MCCS-94-273, Computing Research Lab, New Mexico State University.
10. I. J. Good. 1953. The population frequencies of species and the estimation of population parameters. Biometrika, 40:237-264.
11. Joshua Goodman and Jianfeng Gao. 2000. Language model size reduction by pruning and clustering. In Proceedings of ICSLP, pages 16-20.
12. Joshua Goodman. 2001. A bit of progress in language modeling, extended version. Technical Report MSR-TR-2001-72, Microsoft Research.
13. Harald Hammarström. 2007. A fine-grained model for language identification. In Proceedings of iNEWS-07 Workshop at SIGIR 2007, pages 14-20.
14. Arthur S. House and Edward P. Neuburg. 1977. Toward automatic identification of the language of an utterance. I. Preliminary methodological considerations. Journal of the Acoustical Society of America, 62(3):708-713.
15. Slava M. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, 35:400-401.
16. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of ICASSP-95, pages 181-184.
17. Canasai Kruengkrai, Prapass Srichaivattana, Virach Sornlertlamvanich, and Hitoshi Isahara. 2005. Language identification based on string kernels. In Proceedings of ISCIT-2005, pages 896-899.
18. Paul McNamee. 2005. Language identification: a solved problem suitable for undergraduate instruction. Journal of Computing Sciences in Colleges, 20(3):94-101.
19. Hermann Ney, Ute Essen, and Reinhard Kneser. 1994. On structuring probabilistic dependence in stochastic language modelling. Computer Speech and Language, 8(1):1-38.
20. Radim Řehůřek and Milan Kolkus. 2009. Language identification on the web: Extending the dictionary method. In Proceedings of CICLing 2009, pages 357-368.
21. Penelope Sibun and Jeffrey C. Reynar. 1996. Language identification: Examining the issues. In Proceedings of SDAIR'96, pages 125-135.
22. Vesa Siivola, Teemu Hirsimäki, and Sami Virpioja. 2007. On growing and pruning Kneser-Ney smoothed n-gram models. IEEE Transactions on Audio, Speech & Language Processing, 15(5):1617-1624.
23. Andreas Stolcke. 1998. Entropy-based pruning of backoff language models. In Proceedings of DARPA Broadcast News Transcription and Understanding Workshop, pages 270-274.
24. Andreas Stolcke. 2002. SRILM - an extensible language modeling toolkit. In Proceedings of ICSLP, pages 901-904. http://www.speech.sri.com/projects/srilm/.
25. William John Teahan. 2000. Text classification and segmentation using minimum cross-entropy. In Proceedings of RIAO'00, pages 943-961.
26. Érika Valencia and François Yvon. 1997. Identification de l'origine linguistique: le cas des noms propres. In Actes des Journes Scientifiques et Techniques du Réseau Francil (JST97), pages 473-480.
27. Chengxiang Zhai and John Lafferty. 2001. A study of smoothing methods for language models applied to Ad Hoc information retrieval. In Proceedings of SIGIR'01, pages 334-342.