The textcat package for n-gram based text categorization in R

Journal of Statistical Software

Volumn 52, Issue 6, 2013, Pages 1-17

  Hornik, Kurt ^a   Mair, Patrick ^a   Rauch, Johannes ^a   Geiger, Wilhelm ^a   Buchta, Christian ^a   Feinerer, Ingo ^b  

^a VIENNA UNIVERSITY OF ECONOMICS AND BUSINESS   (Austria)

^b VIENNA UNIVERSITY OF TECHNOLOGY   (Austria)

Author keywords

Language identification; N grams; R; Text categorization; Text mining; Textcat

Indexed keywords

EID: 84873900391     PISSN: 15487660     EISSN: None     Source Type: Journal    
DOI: 10.18637/jss.v052.i06     Document Type: Article

References (33)

1. Ahmed B, Cha SH, Tappert C (2004). "Language Identification from Text Using n-Gram Based Cumulative Frequency Addition." In Proceedings of Student/Faculty Research Day, CSIS, Pace University, May 7th, 2004. URL http://www.csis.pace.edu/~ctappert/srd2004/paper12.pdf.
2. Apache (2010). "The Apache SpamAssassin Project." URL http://spamassassin.apache.org/.
3. Armstrong-Warwick S, Thompson HS, McKelvie D, Petitpierre D (1994). "Data in Your Language: The ECI Multilingual Corpus 1." In Proceedings of the International Work- shop on Sharable Natural Language Resources, pp. 97-106. Nara, Japan. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.44.950.
4. Baayen RH (2008). Analyzing Linguistic Data: A Practical Introduction to Statistics Using R. Cambridge University Press, Cambridge. URL http://www.ualberta.ca/~baayen/publications/baayenCUPstats.pdf.
5. Batchelder EO (1992). "A Learning Experience: Training an Artificial Neural Network to Discriminate Languages." Technical report, City University of New York.
6. Beesley KR (1988). "Language Identifier: A Computer Program for Automatic Natural- Language Identification of On-Line Text." In Languages at Crossroads: Proceedings of the 29th Annual Conference of the American Translators Association, pp. 47-54. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.22.9868.
7. Buchta C, Hornik K, Feinerer I, Meyer D (2012). tau: Text Analysis Utilities. R package version 0.0-15, URL http://CRAN.R-project.org/package=tau.
8. Cavnar WB, Trenkle JM (1994). "n-Gram-Based Text Categorization." In Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pp. 161-175. Las Vegas, US. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.53.9367.
9. Dunning T (1994). "Statistical Identification of Language." Technical Report MCCS 94-273, Computing Research Lab (CRL), New Mexico State University. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.48.1958.
10. Egghe L (2000). "The Distribution of N-Grams." Scientometrics, 47(2), 237-252.
11. Feinerer I, Hornik K, Meyer D (2008)."Text Mining Infrastructure in R."Journal of Statistical Software, 25(5), 1-54. URL http://www.jstatsoft.org/v25/i05/.
12. Grefenstette G (1995). "Comparing Two Language Identification Schemes." In Proceedings of the 3rd International Conference on Statistical Analysis of Textual Data (JADT 1995), volume II, pp. 263-268. Rome. URL http://www.xrce.xerox.com/content/download/20524/147563/file/Gref---Comparing-two-language-identification-schemes.pdf.
13. Ha LQ, Hanna P, Ming J, Smith FJ (2009). "Extending Zipf's Law to n-Grams for Large Corpora." Artificial Intelligence Review, 32(1-4), 101-113.
14. Henrich P (1989). "Language Identification for the Automatic Grapheme-to-Phoneme Conversion of Foreign Words in a German Text-to-Speech System." In EUROSPEECH-1989 (First European Conference on Speech Communication and Technology), pp. 2220-2223. URL http://www.isca-speech.org/archive/eurospeech_1989/e89_2220.html.
15. Hornik K, Rauch J, Buchta C, Feinerer I (2013). textcat: n-Gram Based Text Categorization. R package version 1.0-0, URL http://CRAN.R-project.org/package=textcat.
16. Ingle NC (1976)."A Language Identification Table."The Incorporated Linguist, 15(4), 98-101.
17. Keesan C (1987). "Identification of Written Slavic Languages." In K Kummer (ed.), Proceedings of the 28th Annual Conference of the American Translators Association, pp. 517-528.
18. Khreisat L (2009). "A Machine Learning Approach for Arabic Text Classification Using n-Gram Frequency Statistics." Journal of Informetrics, 3(1), 72-77. doi:10.1016/j.joi. 2008.11.005.
19. Ljubešić N, Mikelić N, Boras D (2007). "Language Identification: How to Distinguish Similar Languages." In V Lužar-Stifter, VH Dobrić (eds.), Proceedings of the 29th International Conference on Information Technology Interfaces, pp. 541-546. SRCE University Com-puting Centre, Zagreb. URL http://www.nljubesic.net/main/publications_files/ljubesic07-language.pdf.
20. Murray IA (2002). "Probabilistic Language Modelling." Technical report, Cavendish Labo-ratory, Cambridge, The Inference Group. URL http://www.inference.phy.cam.ac.uk/is/papers/langreport.pdf.
21. Mustonen S (1965). "Multiple Discriminant Analysis in Linguistic Problems." Statistical Methods in Linguistics, 4, 37-44. Stockholm.
22. Newman P (1987). "Foreign Language Identification: First Step in the Translation Process." In K Kummer (ed.), Proceedings of the 28th Annual Conference of the American Translators Association, pp. 509-516.
23. R Core Team (2012). R: A Language and Environment for Statistical Computing. R Foun-dation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, URL http://www.R-project.org/.
24. Scannell KP (2007)."The Crúbadán Project: Corpus building for under-resourced languages." In C Fairon, H Naets, A Kilgarriff, GM de Schryver (eds.), Building and Exploring Web Corpora: Proceedings of the 3rd Web as Corpus Workshop, volume 4 of Cahiers du Cental, pp. 5-15. Presses universitaires de Louvain, Louvain-la-Neuve, Belgium. URL http://borel.slu.edu/pub/wac3.pdf.
25. Sibun P, Reynar JC (1996). "Language Identification: Examining the Issues." In 5th Sympo-sium on Document Analysis and Information Retrieval, pp. 125-135. Las Vegas, Nevada, U.S.A. URL http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.52.4524.
26. Singh AK (2006). "Study of Some Distance Measures for Language and Encoding Identification." In Proceedings in the Workshop on Linguistic Distances, Sydney, July 2006, pp. 63-72. URL http://acl.ldc.upenn.edu/W/W06/W06-1109.pdf.
27. Souter C, Churcher G, Hayes J, Hughes J, Johnson S (1994)."Natural Language Identification Using Corpus-Based Models." Hermes Journal of Linguistics, 13, 183-203. URL http://download2.hermes.asb.dk/archive/FreeH/H13_15.pdf.
28. van Noord G (1997). "TextCat." URL http://odur.let.rug.nl/~vannoord/TextCat.
29. Wikipedia (2013a). "n-Gram - Wikipedia, The Free Encyclopedia." URL http://en.wikipedia.org/wiki/N-gram, accessed 2013-01-15.
30. Wikipedia (2013b). "Serbo-Croatian - Wikipedia, The Free Encyclopedia." URL http://en.wikipedia.org/wiki/Serbo-Croatian, accessed 2013-01-15.
31. Wikipedia (2013c). "XPath - Wikipedia, The Free Encyclopedia." URL http://en.wikipedia.org/wiki/XPath, accessed 2013-01-15.
32. WiseGuys (2003). "libTextCat: Lightweight Text Categorization." URL http://software.wise-guys.nl/libtextcat/.
33. Zipf GK (1949). Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley, Reading, MA.