Language identification: How to distinguish similar languages?

Proceedings of the International Conference on Information Technology Interfaces, ITI

Volumn , Issue , 2007, Pages 541-546

  Ljubešić, Nikola ^a   Mikelić, Nives ^a   Boras, Damir ^a  

^a UNIVERSITY OF ZAGREB   (Croatia)

Author keywords

Croatian language; Forbidden words method; Most frequent words method; Second order Markov model; Web corpus; Written language identification

Indexed keywords

INFORMATION RETRIEVAL SYSTEMS; INFORMATION TECHNOLOGY; LINGUISTICS; MARKOV PROCESSES; TECHNOLOGY;

CROATIAN LANGUAGE; FORBIDDEN WORDS METHOD; INTERNATIONAL CONFERENCES; LANGUAGE IDENTIFICATION; MARKOV MODELLING; MOST FREQUENT WORDS METHOD; SECOND ORDERS; SECOND-ORDER MARKOV MODEL; WEB-CORPUS; WRITTEN LANGUAGE IDENTIFICATION;

QUERY LANGUAGES;

EID: 48349136970     PISSN: 13301012     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ITI.2007.4283829     Document Type: Conference Paper

References (26)

1. Aslam JA, Frost M. An information-theoretic measure for document similarity. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), 2003.
2. Basis Techis Rosette Language Identifier. http://www.basistech.com/ language-identification/ [07/01/2006]
3. Batchelder EO. A learning experience: Training an artificial neural network to discriminate languages. Technical Report, 1992.
4. Beesley KR. Language identifier: A computer program for automatic natural language identification on on-line text. In Proceedings of the 29th Annual Conference of the American Translators Association, 1988. p.47-54.
5. Cavnar WB, Trenkle JM. N-gram-based text categorization. In Proceedings of SDAIR-94, the 3rd Annual Symposium on Document Analysis and Information Retrieval. Las Vegas, Nevada, USA, 1994. p. 161-175.
6. Damashek M. Gauging similarity with n-grams: language independent categorization of text. Science 1995; 267(5199):843-848.
7. Dunning T. Statistical identification of language. Technical Report MCCS. New Mexico State University, 1994. p.94-273.
8. Elworthy, D. Language Identification With Confidence Limits. In CoRR: Computation and Language Journal, 1999.
9. Henrich P. Language identification for the automatic grapheme-to-phoneme conversion of foreign words in a german text-to-speech system. In Proceedings of Eurospeech 1989, European Speech Communication and Technology, 1989. p. 220-223.
10. Ingle N. A language identification table. In The Incorporated Linguist 1976, 15(4).
11. Johnson, S. Solving the problem of language recognition. Technical report. School of Computer Studies, University of Leeds, 1993.
12. Kruengkrai C, Srichaivattana P, Sornlertlamvanich V, Isahara H. Language identification based on string kernels. In Proceedings of the 5th International Symposium on Communications and Information Technologies (ISCIT), 2005.
13. Kulikowski S. Using short words: a language identification algorithm. Unpublished technical report, 1991.
14. Lodhi H, Shawe-Taylor J, Cristianini N, Watkins CJCH. Text classification using string kernels. Journal of Machine Learning Research, 2:419-444.
15. McNamee P, Mayfield J. Character N-gram Tokenization for European Language Text Retrieval. Information Retrieval 2004; 7:73-97.
16. Mustonen S. Multiple discriminant analysis in linguistic problems. In Statistical Methods in Linguistics. Skriptor Fack, Stockholm, 1965;(4).
17. Newman P. Foreign language identification - first step in the translation process. In K. Kummer (editor), Proceedings of the 28th Annual Conference of the American Translators Association, 1987. p.509-516.
18. Schmitt JC. Trigram-based method of language identification, October 1991. U.S. Patent number: 5062143.
19. Sibun, P. and Reynar, J. C. Language Determination: Examining the Issues. In Proceedings of the 5th Annual Symposium on Document Analysis and Information Retrieval, pp. 125-135, Las Vegas, Nevada, 1996.
20. Souter et al. Natural Language Identification Using Corpus-Based Models. Hermes J. Linguistics 1994; 13:183-203.
21. TextCat Language Guesser Demo. http://www.let.rug.nl/ vannoord/TextCat/Demo/ [07/01/2006]
22. Teytaud O, Jalam R. Kernel-based text categorization. In Proceedings of the International Joint Conference on Neural Networks (IJCNN), 2001.
23. Ueda Y, Nakagawa S. Prediction for phoneme/syllable/word-category and identification of language using HMM. In Proceedings of the 1990 International Conference on Spoken Language Processing; November 1990; Kobe, Japan. Volume 2, p.1209-1212.
24. News portals: http://www.net.hr, http://www.b92.net, http://novice.siol. net [4/26/2007]
25. Xerox Language Identifier. http://www.xrce.xerox.com/competencies/ content-analysis/tools/guesser-ISO-8859-1.en.html [07/01/2006]
26. Ziegler DV. The automatic identification of languages using linguistic recognition signals. State University of New York at Buffalo, Buffalo, NY, 1992.