The evolution of a crawling strategy for an academic document search engine: Whitelists and blacklists

Proceedings of the 4th Annual ACM Web Science Conference, WebSci'12

Volumn volume, Issue , 2012, Pages 340-343

  Wu, Jian ^a   Teregowda, Pradeep ^a   Ramírez, Juan Pablo Fernández ^a   Mitra, Prasenjit ^a   Zheng, Shuyi ^b   Giles, C Lee ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

^b FACEBOOK   (United States)

Author keywords

Information retrieval; Search engine; Web crawling

Indexed keywords

INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; METADATA; WEBSITES;

CRAWLING STRATEGY; DOCUMENT SEARCH; EXTRACTING INFORMATION; INFORMATION EXTRACTION; VERTICAL SEARCH ENGINE; WEB CRAWLING;

SEARCH ENGINES;

EID: 84869071720     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2380718.2380762     Document Type: Conference Paper

References (5)

1. Baeza-Yates, R. A., and Castillo, C. Crawling the infinite web. J. Web Eng. 6, 1 (2007), 49-72.
2. Cho, J., Garcia-Molina, H., and Page, L. Efficient crawling through url ordering. Computer Networks 30, 1-7 (1998), 161-172.
3. Cho, J., and Schonfeld, U. Rankmass crawler: A crawler with high pagerank coverage guarantee. In VLDB (2007), 375-386.
4. Olston, C., and Najork, M. Web crawling. Foundations and Trends in Information Retrieval (2010), 175-246.
5. Zheng, S., Dmitriev, P., and Giles, C. L. Graph-based seed selection for web-scale crawlers. In CIKM (2009), 1967-1970.