A modular open-source focused crawler for mining monolingual and bilingual corpora from the web

Proceedings of the Annual Meeting of the Association for Computational Linguistics

Volumn , Issue , 2013, Pages 43-51

  Papavassiliou, Vassilis ^a   Prokopidis, Prokopis ^a   Thurmair, Gregor ^b  

^a INSTITUTE FOR LANGUAGE AND SPEECH PROCESSING   (Greece)

^b Linguatec   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONAL LINGUISTICS; INFORMATION RETRIEVAL SYSTEMS; TEXT PROCESSING;

AUTOMATIC ACQUISITION; BILINGUAL CORPORA; DEDUPLICATION; DOMAIN SPECIFIC; FOCUSED CRAWLER; MAIN MODULE; MODULARS; NORMALISATION; OPEN-SOURCE; TEXT CLASSIFICATION;

CLASSIFICATION (OF INFORMATION);

EID: 85121796154     PISSN: 0736587X     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (25)

1. Rehan Akbani, Stephen Kwek, and Nathalie Japkowicz. 2004. Applying support vector machines to imbalanced datasets. In In Proceedings of the 15th European Conference on Machine Learning (ECML, pages 39-50.
2. José João Almeida and Alberto Simões. 2010. Automatic parallel corpora and bilingual terminology extraction from parallel websites. In 3rd Workshop on Building and Using Comparable Corpora . Ethem Alpaydin. 2010. Introduction to Machine Learning. The MIT Press, 2nd edition.
3. Luciano Barbosa, Vivek Kumar Rangarajan Sridhar, Mahsa Yarmohammadi, and Srinivas Bangalore. 2012. Harvesting parallel text in multiple languages with limited supervision. In COLING, pages 201-214.
4. Marco Baroni, Adam Kilgarriff, Jan Pomikálek, and Pavel Rychlý. 2006. WebBootCaT: Instant Domain-Specific Corpora to Support Human Translators. In Proceedings of the 11th Annual Conference of EAMT, pages 47-252, Norway.
5. Marco Baroni, Francis Chantree, Adam Kilgarriff, and Serge Sharoff. 2008. Cleaneval: A competition for cleaning web pages. In LREC'08.
6. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The WaCky Wide Web: A collection of very large linguistically processed webcrawled corpora. Language Resources and Evaluation, 43(3):209-226.
7. Adrian W. Bowman and Adelchi Azzalini. 1997. Applied smoothing techniques for data analysis: The kernel approach with S-Plus illustrations, volume 18. Oxford University Press.
8. Jisong Chen, Rowena Chau, and Chung-Hsing Yeh. 2004. Discovering parallel text from the World Wide Web. In Proceedings of ACSW Frontiers '04, volume 32, pages 157-161, Darlinghurst, Australia.
9. Alain Désilets, Benoit Farley, Marta Stojanovic, and Geneviève Patenaude. 2008. WeBiText: Building Large Heterogeneous Translation Memories from Parallel Web Content. In Proceedings of Translating and the Computer (30), London, UK.
10. Miquel Esplà-Gomis and Mikel L. Forcada. 2010. Combining Content-Based and URL-Based Heuristics to Harvest Aligned Bitexts from Multilingual Sites with Bitextor. The Prague Bulletin of Mathemathical Lingustics, 93:77-86.
11. Gumwon Hong, Chi-Ho Li, Ming Zhou, and Hae-Chang Rim. 2010. An empirical study on web mining of parallel data. In Proceedings of the 23rd COLING, pages 474-482.
12. Adam Kilgarriff and Gregory Grefenstette. 2003. Introduction to the special issue on the web as corpus. Computational Linguistics, 29(3):333-348.
13. Jian-Yun Nie, Michel Simard, Pierre Isabelle, and Richard Durand. 1999. Cross-language information retrieval based on parallel texts and automatic mining of parallel texts from the Web. In Proceedings of the 22nd annual international ACM SIGIR conference on research and development in information retrieval, pages 74-81, New York.
14. Christopher Olston and Marc Najork. 2010. Web crawling. Found. Trends Inf. Retr., 4(3):175-246.
15. Pavel Pecina, Antonio Toral, Vassilis Papavassiliou, Prokopis Prokopidis, and Josef van Genabith. 2012.
16. Domain adaptation of statistical machine translation using web-crawled resources: A case study. In Proceedings of the 16th Annual Conference of EAMT, pages 145-152, Trento, Italy.
17. Xiaoguang Qi and Brian D. Davison. 2009. Web page classification: Features and algorithms. ACM Computing Surveys, 41:11-31.
18. Philip Resnik and Noah A. Smith. 2003. The Web as a parallel corpus. Computational Linguistics, 29:349-380.
19. Lei Shi, Cheng Niu, Ming Zhou, and Jianfeng Gao. 2006. A dom tree alignment model for mining parallel data from the web. In COLING/ACL-2006, pages 489-496.
20. Miroslav Spousta, Michal Marek, and Pavel Pecina. 2008. Victor: The Web-Page Cleaning Tool. In Proceedings of the 4th Web as Corpus Workshop - Can we beat Google?, pages 12-17, Marrakech.
21. Padmini Srinivasan, Filippo Menczer, and Gautam Pant. 2005. A General Evaluation Framework for Topical Crawlers. Information Retrieval, 8:417-447.
22. Martin Theobald, Jonathan Siddharth, and Andreas Paepcke. 2008. Spotsigs: Robust and efficient near duplicate detection in large web collections. In Proceedings of the 31st annual international ACM SIGIR conference on research and development in information retrieval, pages 563-570.
23. Masao Utiyama, Daisuke Kawahara, Keiji Yasuda, and Eiichiro Sumita. 2009. Mining parallel texts from mixed-language web pages. In MT Summit.
24. Torsten Zesch, Christof Müller, and Iryna Gurevych. 2008. Extracting lexical semantic knowledge from wikipedia and wiktionary. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech.
25. Ying Zhang, Ke Wu, Jianfeng Gao, and Phil Vines. 2006. Automatic Acquisition of Chinese-English Parallel Corpus from the Web. In Proceedings of the 28th European Conference on Information Retrieval, pages 420-431.