User browsing behavior-driven web crawling

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2011, Pages 87-92

  Liu, Minghai ^a,b   Cai, Rui ^a   Zhang, Ming ^b   Zhang, Lei ^a  

^a MICROSOFT RESEARCH ASIA   (China)

^b PEKING UNIVERSITY   (China)

Author keywords

URL pattern discovery; web crawling; web log mining

Indexed keywords

BREADTH-FIRST; IMPORTANCE MEASURE; LINK STRUCTURE; LOG MINING; PAGERANK; PATTERN DISCOVERY; USER INTERESTS; WEB CRAWLERS; WEB CRAWLING; WEB LOG MINING;

KNOWLEDGE MANAGEMENT;

BEHAVIORAL RESEARCH;

EID: 83055181892     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063576.2063593     Document Type: Conference Paper

References (22)

1. Uniform resource identifier: Generic syntax. RFC 3986.
2. S. Abiteboul, M. Preda, and G. Cobena. Adaptive on-line page importance computation. In Proc. WWW, pages 280-290, 2003.
3. R. Baeza-Yates, C. Castillo, M. Marin, and A. Rodriguez. Crawling a country: better strategies than breadth-first for web page ordering. In Proc. WWW, pages 864-872, 2005.
4. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer Networks, 30(1-7):107-117, 1998.
5. R. Cai, J.-M. Yang, W. Lai, Y. Wang, and L. Zhang. iRobot: an intelligent crawler for web forums. In Proc. WWW, pages 447-456, 2008.
6. S. Chakrabarti, M. van den Berg, and B. Dom. Focused crawling: a new approach to topic-specific web resource discovery. Computer Networks, 31(11-16):1623-1640, 1999.
7. J. Cho, H. Garcia-Molina, and L. Page. Efficient crawling through URL ordering. Computer Networks.
8. J. Cho and U. Schonfeld. RankMass crawler: a crawler with high personalized PageRank coverage guarantee. In Proc. VLDB, pages 375-386, 2007.
9. D. Fetterly, N. Craswell, and V. Vinay. The impact of crawl policy on web search effectiveness. In Proc. SIGIR, pages 580-587, 2009.
10. P. Grünwald. A tutorial introduction to the minimum description length principle. Advances in Minimum Description Length: Theory and Applications. MIT Press, 2005.
11. J. Kleinberg. Authoritative sources in a hyperlinked environment. Journal of the ACM, 46(5):604-632, 1999.
12. T. Lei, R. Cai, J.-M. Yang, Y. Ke, X. Fan, and L. Zhang. A pattern tree-based approach to learning URL normalization rules. In Proc. WWW, pages 611-620, 2010.
13. F. Menczer, G. Pant, and P. Srinivasan. Topical web crawlers: evaluating adaptive algorithms. ACM Trans. Internet Techn., 4(4):378-419, 2004. (Pubitemid 40009828)
14. C. Olston and M. Najork. Web crawling. Foundations and Trends® in Information Retrieval, 4(3):175-246, 2010.
15. L. Page, S. Brin, R. Motwani, and T. Winograd. The PageRank citation ranking: bringing order to the Web. Technical report, Stanford University, 1999.
16. S. Pandey and C. Olston. User-centric web crawling. In Proc. WWW, pages 401-411, 2005.
17. S. Pandey and C. Olston. Crawl ordering by search impact. In Proc. WSDM, pages 3-14, 2008.
18. G. Pant and P. Srinivasan. Learning to crawl: comparing classification schemes. ACM Trans. Inf. Syst.
19. B. Pinkerton. Finding what people want: experiences with the web crawler. In Proc. WWW, pages 3-14, 1994.
20. J. Rissanen. Stochastic complexity and modeling. The Annals of Statistics, 14(3):1080-1100, 1986.
21. M. L. A. Vidal, A. S. da Silva, E. S. de Moura, and J. M. B. Caval-canti. Structure-driven crawler generation by example. In Proc. SIGIR, pages 292-299, 2006.
22. Y. Wang, J.-M. Yang, W. Lai, R. Cai, L. Zhang, and W.-Y. Ma. Exploring traversal strategy for efficient web forum crawling. In Proc. SIGIR, pages 459-466, 2008.