Building a 70 billion word corpus of English from ClueWeb

Proceedings of the 8th International Conference on Language Resources and Evaluation, LREC 2012

Volumn , Issue , 2012, Pages 502-506

  Pomikalek, Jan ^a,b   Jakubicek, Milos ^a,b   Rychly, Pavel ^a,b  

^a MASARYK UNIVERSITY   (Czech Republic)

^b LEXICAL COMPUTING LTD   (United Kingdom)

Author keywords

Clueweb; Corpus; Encoding; English; Word sketch

Indexed keywords

ENCODING (SYMBOLS); INDEXING (MATERIALS WORKING); QUERY LANGUAGES;

CLUEWEB; CORPUS; ENGLISH; LANGUAGE RESOURCES; MANAGEMENT SYSTEMS; NEAR- DUPLICATES; PRE-PROCESSING STEP; WORD SKETCH;

DATA HANDLING;

EID: 84907013032     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (11)

1. Marco Baroni and Silvia Bernardini. 2004. BootCaT: Bootstrapping Corpora and Terms from the Web. In Proceedings of LREC, volume 4.
2. Andrei Broder. 2000. Identifying and Filtering Near-Duplicate Documents. In Combinatorial Pattern Matching, pages 1-10. Springer.
3. Moses S. Charikar. 2002. Similarity Estimation Techniques from Rounding Algorithms. In Proceedings of the Thirty-Fourth Annual ACM Symposium on Theory of Computing, pages 380-388. ACM.
4. Miloš Jakubí?cek, Pavel Rychlý, Adam Kilgarriff, and Diana McCarthy. 2010. Fast Syntactic Searching in Very Large Corpora for Many Languages. In PACLIC 24 Proceedings of the 24th Pacific Asia Conference on Language, Information and Computation, pages 741-747, Tokyo.
5. A. Kilgarriff, P. Rychlý, P. Smrž, and D. Tugwell. 2004. The Sketch Engine. In Proceedings of the Eleventh EURALEX International Congress, pages 105-116, Lorient, France. Universite de Bretagne-Sud.
6. Adam Kilgarriff, Siva Reddy, Jan Pomikálek, and Avinesh PVS. 2010. A corpus factory for many languages. In LREC Workshop on Web Services and Processing Pipelines. Valetta, Malta: ELRA.
7. Jan Pomikálek. 2011. Removing Boilerplate and Duplicate Content from Web Corpora. Phd thesis, Masaryk University.
8. Pavel Rychlý. 2000. Korpusové manažery a jejich efektivní implementace. PhD Thesis, Masaryk University, Brno.
9. Pavel Rychlý. 2007. Manatee/Bonito-A Modular Corpus Manager. In 1st Workshop on Recent Advances in Slavonic Natural Language Processing, pages 65-70, Brno.
10. Helmut Schmid. 1994. Probabilistic Part-of-Speech Tagging Using Decision Trees. In Proceedings of International Conference on New Methods in Language Processing, volume 12, pages 44-49. Manchester, UK.
11. George Kingsley Zipf. 1949. Human Behavior and the Principle of Least Effort. Addison-Wesley Press.