iRobot: An intelligent crawler for web forums

Proceeding of the 17th International Conference on World Wide Web 2008, WWW'08

Volumn , Issue , 2008, Pages 447-456

  Cai, Rui ^a   Yang, Jiang Ming ^a   Lai, Wei ^a   Wang, Yida ^a   Zhang, Lei ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Forum crawler; Sitemap construction; Traversal path selection

Indexed keywords

DATA MINING; DECISION SUPPORT SYSTEMS; INFORMATION MANAGEMENT; INTERNET; KNOWLEDGE BASED SYSTEMS; SEARCH ENGINES;

FORUM CRAWLER; INTELLIGENT CRAWLERS; INTERNET USERS; LINK STRUCTURES; NETWORK BANDWIDTHS; RANDOMLY SAMPLES; SITEMAP CONSTRUCTION; TRAVERSAL PATH SELECTION; WEB APPLICATIONS; WEB DATA MININGS; WEB FORUMS;

WORLD WIDE WEB;

EID: 57349112859     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1367497.1367558     Document Type: Conference Paper

References (28)

1. Internet Forum Software. http://en.wikipedia.org/wiki/category:internet- forum-software.
2. RFC 1738 - Uniform Resource Locators (URL). http://www.faqs.org/rfcs/ rfcl738.html.
3. The Sitemaps Protocol, http://sitemaps.org/protoeol.php.
4. R. A. Baeza-Yates and B. A. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley, 1999.
5. th WWW, pages 864-872, Chiba, Japan, May 2005.
6. th WWW, pages 111-120, Banff, Alberta, Canada, May 2007.
7. O. Brandman, Junghoo Cho, H. Garcia-Molina, and N. Shivakumar. Crawler-friendly Web servers. ACM SIGMETRICS Performance Evaluation Review, 28(2):9-14, Sept. 2000.
8. S. Brin and L. Page. The anatomy of a large-scale hypertextual Web search engine. Computer Networks, 30(1-7):107-117, 1998.
9. th WWW, pages 1157-1166, Santa Clara, California, USA, Apr. 1997.
10. D. Cai, S. Yu, J.-R. Wen, and W.-Y. Ma. VIPS: a vision based page segmentation algorithm. Microsoft Technical Report, MSR-TR-2003-79, 2003.
11. S. Chakrabarti, M. van den Berg, and B. Dom. Focused crawling: a new approach to topic-specific Web resource discovery. Computer Networks, 31(11-16):1623-1640, 1999.
12. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. MIT Press, Sept. 2001.
13. V. Crescenzia, P. Merialdoa, and P. Missier. Clustering Web pages based on their structure. Data & Knowledge Engineering, 54(3):279-299, Sept. 2005.
14. th SCG, pages 253-262, NY, USA, Jun. 2004.
15. th KDD, pages 419-428, Aug. 2005.
16. Y. Guo, K. Li, K. Zhang, and G. Zhang. Board forum crawling: a Web crawling method for Web forum. In Proc. 2006 IEEE/WIC/ACM Int. Conf. Web Intelligence, pages 745-748, Hong Kong, Dec. 2006.
17. th SIGIR, pages 284-291, Seattle, Washington, USA, Aug. 2006.
18. J. P. Lage, A. S. da Silva, P. B. Golgher, and A. H. F. Laender. Automatic generation of agents for collecting hidden Web pages for data extraction. Data & Knowledge Engineering, 49(2):177-196, May 2004.
19. th WWW, pages 141-150, Banff, Alberta, Canada, May 2007.
20. th WWW, pages 401-411, Chiba, Japan, May 2005.
21. th VLDB, pages 129-138, Sept. 2001.
22. th WWW, pages 502-511, NY, USA, May 2004.
23. L. Rosenfeld and P. Morville. Information Architecture for the World Wide Web. O'Reilly, Feb. 1998.
24. R. Song, H. Liu, J.-R. Wen, W.-Y. Ma. Learning important models for Web page blocks based on layout and content analysis. ACMSIGKDD Explorations Newsletter, 6(2): 14-23, Dec. 2004.
25. th SIGIR, pages 292-299, Seattle, USA, Aug. 2006.
26. Y. Zhai and B. Liu. Structured data extraction from the Web based on partial tree alignment. IEEE Trans. Knowl. Data E ng., 18(12): 1614-1628, Dec. 2006.
27. th WWW, pages 221-230, Banff, Canada, May 2007.
28. th KDD, pages 894-902, San Jose, CA, USA, Aug. 2007.