Learnable topic-specific web crawler

Journal of Network and Computer Applications

Volumn 28, Issue 2, 2005, Pages 97-114

  Rungsawang, Arnon ^a   Angkawattanawit, Niran ^a  

^a KASETSART UNIVERSITY   (Thailand)

Author keywords

Learnable web crawler; Topic specific web crawler; Web crawler

Indexed keywords

ALGORITHMS; INFORMATION ANALYSIS; KNOWLEDGE ACQUISITION; SEARCH ENGINES; WEBSITES;

KEYWORDS; TOPIC-SPECIFIC WEB CRAWLER; WEB CRAWLER;

WORLD WIDE WEB;

EID: 10844235587     PISSN: 10848045     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jnca.2004.01.001     Document Type: Article

References (28)

1. Aggarwal C, Garawi F, Yu P. Intelligent crawling on the world wide web with arbitrary predicates. In: Proceedings of the 10th International World Wide Web Conference, Hongkong; 2001. p. 96-105.
2. Baeza-Vales R, Ribeiro-Neto B. Modern information retrieval. Reading, MA: Addison-Wesley; 1999.
3. Bharat K, Henzinger M. Improved algorithms for topic distillation in a hyperlinked environment. In: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Melbourne, Australia, August; 1998. p. 104-11.
4. Bra PD, Houben GJ, Kornatzky Y, Post R. Searching for arbitrary information in the WWW: the fish search for mosaic. In: Proceedings of the Second International World Wide Web Conference, Chicago, Illinois, USA, April; 1994. p. 107-17.
5. Brin S, Page L. The anatomy of a large-scale hypertextual web search engine. In: Proceedings of Seventh International World Wide Web Conference, Brisbane, Australia, April; 1998. p. 107-17.
6. Chakrabarti S, Dom B, Gibson D, Kleinberg J, Raghavan P, Rajagopalan S. Automatic resource compilation by analyzing hyperlink structure and associated text. Comput Networks 1998;30(1-7):65-74.
7. Chakrabarti S, van den Berg M, Dom B. Focused crawling: a new approach to topic-specific web resource discovery. Comput Networks 1999;31(11-16):1623-40.
8. Cho J, Garcia-Molina H, Page L. Efficient crawling through url ordering. Comput Networks 1998;30(1-7):161-72.
9. Dean J, Henzinger MR. Finding related pages in the world wide web. In: Proceedings of the Eighth International World Wide Web Conference, Toronto, Canada, May; 1999. p. 389-401.
10. Degeratu M, Pant G, Menczer F. Latency-dependent fitness in evolutionary multithreaded web agents. In: Proceedings of GECCO Workshop on Evolutionary Computation and Multi-Agent Systems, San Francisco, California, USA, July; 2001. p. 313-6.
11. Diligenti M, Coetzee FM, Lawrence S, Giles CL, Gori M. Focused crawling using context graphs. In: Proceedings of the 26th International Conference on Very Large Data Bases, Cairo, Egypt, September; 2000. p. 527-34.
12. Hersovici M, Jacovi M, Maarek YS, Pelleg D, Shtalhaim M, Ur S. The shark-search algorithm-an application: tailored web site mapping. In: Proceedings of the Seventh International World Wide Web Conference, Brisbane, Australia, April; 1998. p. 317-26.
13. Heydon A, Najork M. Mercator: a scalable, extensible web crawler. World Wide Web: Internet Web Inf Syst 1999;2(4):219-29.
14. Inktomi Press Release, Web surpasses one billion documents. Technical report. Inktomi Corporation; 2000.
15. Kleinberg J. Authoritative sources in a hyperlinked environment. J ACM 1999;46(5):604-32.
16. Li L, Shang Y, Zhang W. Improvement of hits-based algorithms on web documents. In: Proceedings of 11th International World Wide Web Conference, Honolulu, Hawaii, USA, May; 2002. p. 527-35.
17. Menczer F, Belew RK. Artificial life applied to adaptive information agents. In: Proceedings of the AAAI Spring Symposium on Information Gathering from Heterogeneous, Distributed Environments, California, USA, March; 1995.
18. Menczer F, Belew RK. Adaptive retrieval agents: internalizing local context and scaling up to the web. Mach Learn 2000;39(2/3):203-42.
19. Menczer F, Monge AE. Scalable web search by adaptive online agents: an infospider case study. In: Intelligent information agents: agent-based information discovery and management on the Internet, Berlin: Springer; 1999. p. 323-47.
20. Menczer F, Pant G, Ruiz ME. Evaluating topic-driven web crawlers. In: Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New Orleans, Louisiana, USA, September; 2001. p. 241-9.
21. Menczer F, Pant G, Srinivasan P. Topical web crawlers: evaluating adaptive algorithms. ACM Trans Internet Technol 2003:5:1-38.
22. Mukherjea S. WTMS: a system for collecting and analyzing topic-specific web information. Comput Networks 2000a;3(1-6):457-71.
23. Mukherjea S. Organizing topic-specific web information. In: Proceedings of the 11th ACM Conference on Hypertext and Hypermedia, San Antonio, Texas, USA, May; 2000b. p. 133-41.
24. Murray BH, Moore A. Sizing the internet. A white paper. Cyveillance, Inc.; 2000.
25. Page L, Brin S, Motwani R, Winograd T. The pagerank citation ranking: bringing order to the web. Technical report. Stanford University; 1997.
26. Pant G, Menczer F. Myspiders: evolve your own intelligent web crawlers. Autonomous Agents Multi-Agent Syst 2002;5(2):221-9.
27. Pant G, Srinivasan P. Exploration versus exploitation in topic driven crawlers. In: Proceedings of WWW-02 Workshop on Web Dynamics; 2002.
28. Rennie J, McCallum A. Efficient web spidering with reinforcement learning. In: Proceedings of the 16th International Conference on Machine Learning, Bled, Slovenia, June; 1999. P. 335-43.