A Language and Character Set Determination Method Based on N-gram Statistics

ACM Transactions on Asian Language Information Processing

Volumn 1, Issue 3, 2002, Pages 269-278

  Suzuki, Izumi ^a   Mikami, Yoshiki ^a   Ohsato, Ario ^a   Chubachi, Yoshihide ^b  

^a NAGAOKA UNIVERSITY OF TECHNOLOGY   (Japan)

^b Numeric Co Ltd   (Japan)

Author keywords

Algorithms; character set; corpus based analysis; Languages; local language site; N gram; natural languages; Text categorization; Unicode

Indexed keywords

EID: 80155181779     PISSN: 15300226     EISSN: 15583430     Source Type: Journal    
DOI: 10.1145/772755.772759     Document Type: Article

References (6)

1. Cavnar, W. and Trenkle, J. 1994. N-gram based text categorization. In Proceedings of the 3rd Annual Symposium on Document Analysis and Information Retrieval (Las Vegas, NV, 1994), 161-175
2. Frank, E., Chui, C. and Witten, I. H. 2000. Text categorization using compression models. In Proceedings of the IEEE Data Compression Conference (Snowbird, UT, March 2000). IEEE Computer Society, 276-288
3. Manning, C. D. and Schulze, H. 1999. Foundations of Statistical Natural Language Processing. The MIT Press
4. Martino, M. J. et al. 2000. Natural language determination using correlation between common words. U.S. Patent No. 6,023,670
5. Martino, M. J. et al. 2001. Natural language determination using partial words. U.S. Patent No. 6, 216, 102
6. Suzuki, I., Ohsato, A., and Mikami, Y. 2001. A language and character set recognition method based on the immune system. In Proceedings of the 2nd International Symposium on Advanced Intelligent Systems (Korea, Aug. 2001), 292-296