langid.py: An off-the-shelf language identification tool

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2012, Pages 25-30

  Lui, Marco ^a   Baldwin, Timothy ^a  

^a UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

DESIGN AND IMPLEMENTATIONS; DOCUMENT DATASETS; EMPIRICAL COMPARISON; END-USERS; HIGH-ACCURACY; IDENTIFICATION TOOLS; LANGUAGE IDENTIFICATION; MICRO-BLOG; TRAINING DATA;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 85118481535     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (18)

1. Alfred V. Aho and Margaret J. Corasick. 1975. Efficient string matching: an aid to bibliographic search. Communications of the ACM, 18(6):333-340, June.
2. Timothy Baldwin and Marco Lui. 2010. Language identification: The long and the short of the matter. In Proceedings of NAACL HLT 2010, pages 229-237, Los Angeles, USA.
3. Jamie Callan and Mark Hoy, 2009. ClueWeb09 Dataset. Available at http://boston.lti.cs.cmu.edu/Data/clueweb09/.
4. Simon Carter, Wouter Weerkamp, and Manos Tsagkias. to appear. Microblog language identification: Overcoming the limitations of short, unedited and idiomatic text. Language Resources and Evaluation Journal.
5. William B. Cavnar and John M. Trenkle. 1994. N-gram-based text categorization. In Proceedings of the Third Symposium on Document Analysis and Information Retrieval, Las Vegas, USA.
6. Hakan Ceylan and Yookyung Kim. 2009. Language identification of search engine queries. In Proceedings of ACL2009, pages 1066-1074, Singapore.
7. George Forman. 2003. An Extensive Empirical Study of Feature Selection Metrics for Text Classification. Journal of Machine Learning Research, 3(7-8):1289-1305, October.
8. Rayid Ghani, Rosie Jones, and Dunja Mladenic. 2004. Building Minority Language Corpora by Learning to Generate Web Search Queries. Knowledge and Information Systems, 7(1):56-83, February.
9. Harald Hammarstrom. 2007. A Fine-Grained Model for Language Identication. In Proceedings of iNEWS07, pages 14-20.
10. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. MT summit, 11.
11. Marco Lui and Timothy Baldwin. 2011. Cross-domain feature selection for language identification. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 553-561, Chiang Mai, Thailand.
12. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for Naive Bayes text classification. In Proceedings of the AAAI-98 Workshop on Learning for Text Categorization, Madison, USA.
13. J.R. Quinlan. 1986. Induction of Decision Trees. Machine Learning, 1(1):81-106, October.
14. Penelope Sibun and Jeffrey C. Reynar. 1996. Language determination: Examining the issues. In Proceedings of the 5th Annual Symposium on Document Analysis and Information Retrieval, pages 125-135, Las Vegas, USA.
15. Jörg Tiedemann. 2009. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces. Recent Advances in Natural Language Processing, V:237-248.
16. Erik Tromp and Mykola Pechenizkiy. 2011. Graph-Based N-gram Language Identification on Short Texts. In Proceedings of Benelearn 2011, pages 27-35, The Hague, Netherlands.
17. Tommi Vatanen, Jaakko J. Vayrynen, and Sami Virpioja. 2010. Language identification of short text segments with n-gram models. In Proceedings of LREC 2010, pages 3423-3430.
18. Yiming Yang and Jan O. Pedersen. 1997. A comparative study on feature selection in text categorization. In Proceedings of ICML 97.