Web page language identification based on urls

Proceedings of the VLDB Endowment

Volumn 1, Issue 1, 2008, Pages 176-187

  Baykan, Eda ^a   Henzinger, Monika ^a   Weber, Ingmar ^a  

^a EPFL   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION RETRIEVAL; LEARNING ALGORITHMS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; SEARCH ENGINES; WEB CRAWLER;

F MEASURE; LANGUAGE IDENTIFICATION; RANDOM SAMPLE; TOP LEVEL DOMAINS; WEB PAGE LANGUAGES; WEB SEARCHES;

WEBSITES;

EID: 78149404935     PISSN: None     EISSN: 21508097     Source Type: Conference Proceeding    
DOI: 10.14778/1453856.1453880     Document Type: Article

References (16)

1. E. Baykan, S. Castelberg, and M. Henzinger. A comparison of techniques for sampling web pages. In IIWeb 2006 Workshop on Information Integration on the Web in conjunction with WWW 2006, 2006.
2. W. B. Cavnar and J. M. Trenkle. N-gram-based text categorization. In Symposium on Document Analysis and Information Retrieval (SDAIR), pages 161-175, 1994.
3. T. Dunning. Statistical identification of language. 1994. MCCS-94-273.
4. G. Grefenstette. Comparing two language identification schemes. In International Conference on Statistical Analysis of Textual Data (JADT), pages 263-268, 1995.
5. T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning. Springer, 2001.
6. K. Hayati. Language identification on the world wide web. Masters project, University of California, Santa Cruz, 2004.
7. N. C. Ingle. A language identification table. The Incorporated Linguist, 15(4), 1976.
8. M.-Y. Kan. Web page classification without the web page. In International World Wide Web Conference (WWW), pages 262-263, 2004.
9. B. Martins and M. J. Silva. Language identification in web pages. In Symposium on Applied Computing (SAC), pages 764-768, 2005.
10. A. K. McCallum. Bow: A toolkit for statistical language modeling, text retrieval, classification and clustering. http://www.cs.cmu.edu/~mccallum/bow, 1996.
11. K. Nigam, J. Lafferty, and A. MacCallum. Using maximum entropy for text classification. Workshop on Machine Learning for Information Filtering, pages 61-67, 1999.
12. P. Sibun and J. C. Reynar. Language identification: Examining the issues. In Symposium on Document Analysis and Information Retrieval (SDAIR), pages 125-135, 1996.
13. K. Somboonviwat, M. Kitsuregawa, and T. Tamura. Simulation study of language specific web crawling. In ICDE Workshops, page 1254, 2005.
14. W. Teahan and D. Harper. Using compression-based language models for text categorization. In Workshop on Language Modeling and Information Retrieval, 2001.
15. V. B. Vega and S. Bressan. Continuous-learning weighted trigram approach for indonesian language distinction: A preliminary study. In International Conference on Computer Processing of Oriental Languages (ICCPOL), 2001.
16. R. Yan. Matlabarsenal. http://www.informedia.cs.cmu.edu/yanrong/MATLABArsenal/MATLABArsenal.htm, 2006.