An investigation of web crawler behavior: Characterization and metrics

Computer Communications

Volumn 28, Issue 8, 2005, Pages 880-897

  Dikaiakos, Marios D ^a   Stassopoulou, Athena ^b   Papageorgiou, Loizos ^a  

^a UNIVERSITY OF CYPRUS   (Cyprus)

^b Intercollege ^*  (Cyprus)

Author keywords

Crawlers; Web characterization

Indexed keywords

HTTP; JAVA PROGRAMMING LANGUAGE; LOCAL AREA NETWORKS; RESOURCE ALLOCATION; SEARCH ENGINES; SERVERS;

WEB CHARACTERIZATION; WEB CRAWLERS; WEB SERVERS; WEB TRAFFIC;

WORLD WIDE WEB;

EID: 17644390582     PISSN: 01403664     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.comcom.2005.01.003     Document Type: Article

References (29)

1. http://citeseer.nj.nec.com (last accessed Oct. 2003).
2. http://www.altavista.com.
3. http://www.fastsearch.com.
4. http://www.google.com.
5. http://www.inktomi.com.
6. V. Almeida, D. Menasce, R. Riedi, F. Peligrinelli, R. Fonseca, W. Meira Jr, Analyzing Web Robots and Their Impact on Caching. In Proceedings of the Sixth International Workshop on Web Caching and Content Distribution, June 2001, pp. 299-310.
7. A. Arasu, J. Cho, H. Garcia-Molina, A. Paepcke, and S. Raghavan Searching the Web ACM Transactions on Internet Technology 1 1 2001 2 43
8. M. Arlitt, T. Jin, Workload Characterization of the 1998 World Cup Web Site. Technical Report HPL-1999-35R1, Hewlett-Packard Laboratories, September 1999.
9. M. Arlitt, C.L. Williamson, Web Server Workload Characterization: The Search for Invariants. In Proceedings of the 1996 Sigmetrics Conference on Measurement and Modeling of Computer Systems, ACM, May 1996, pp. 126-137.
10. P. Barford, A. Bestavros, A. Bradley, and M. Crovella Changes in Web client access patterns: Characteristics and caching implications World Wide Web (special issue on Characterization and Performance Evaluation) 2 1999 15 28
11. P. Barford, M. Crovella, Generating Representative Web Workloads for Network and Server Performance Evaluation. In Proceedings of the 1998 Sigmetrics Conference on Measurement and Modeling of Computer Systems, ACM, 1998, pp. 151-160.
12. L. Breslau, P. Cao, L. Fan, G. Phillips, S. Shenker, Web Caching and Zipf-like Distributions: Evidence and Implications. In Proceedings of the 1999 IEEE Infocom Conference, IEEE, March 1999, pp. 126-137.
13. S. Brin, and L. Page The anatomy of a large-scale hypertextual (web) search engine Computer Networks and ISDN Systems 30 1-7 1998 107 117
14. S. Chakrabarti, M. van den Berg, B. Dom, Focused Crawling: A New Approach to Topic-Specific Web Resource Discovery. In Proceedings of the 8th World Wide Web Conference, Elsevier, Toronto, 1999, pp. 545-562.
15. M. Crovella Performance characteristics of the world-wide web C. Lindemann G. Haring M. Reiser Performance Evaluation: Origins and Directions 1999 Springer 219 233
16. M.E. Crovella, A. Bestavros, Self-Similarity in World Wide Web Traffic. Evidence and Possible Causes. In Proceedings of the 1996 Sigmetrics Conference on Measurement and Modeling of Computer Systems, ACM, 1996, pp. 160-169.
17. B. Davison, and D. Brian Davison's Web Caching and Content Delivery Resources 2002 http://www.webcaching.com
18. M. Dikaiakos, A. Stassopoulou, L. Papageorgiou, Characterizing Crawler Behavior from Web Server Access Logs. Technical Report TR-2002-4, Department of Computer Science, University of Cyprus, November 2002.
19. M.D. Dikaiakos, and D. Zeinalipour-Yazti A distributed middleware infrastructure for personalized services Computer Communications 27 15 2004 1464 1480
20. A. Feldmann Characteristics of TCP connection arrivals K. Park W. Willinger Self-Similar Network Traffic and Performance Evaluation 2000 Wiley
21. S. Gribble, E. Brewer, System Design Issues for Internet Middleware Services: Deductions from a Large Client Trace. In Proceedings of the 1997 Usenix Symposium on Internet Technologies and Systems. (USITS-97), 1997.
22. B. Krishnamurthy, and J. Rexford Web protocols and practice 2001 Addison-Wesley
23. M. Levene, and A. Poulovassilis Web dynamics Software Focus 2 2 2001 60 67
24. G. Paliouras, C. Papatheodorou, V. Karkaletsis, C. Spyropoulos, Clustering the Users of Large Web Sites into Communities. In Proceedings of the International Conference on Maching Learning (ICML), 2000, pp. 719-726.
25. M. Perkowitz, and O. Etzioni Towards adaptive Web sites: Conceptual framework and case study Artificial Intelligence 118 2000 245 275
26. ACM SIGCOMM. The Internet Traffic Archive. http://ita.ee.lbl.gov.
27. P.-N. Tan, and V. Kumar Discovery of web Robot sessions based on their navigational patterns Data Mining and Knowledge Discovery 6 1 2002 9 35
28. S. Ye, G. Lu, X. Li, Workload-aware Web Crawling and Server Workload Detection. In Network Research Workshop, 18th Asian Pacific Advanced Network Meeting (APAN 2004), July 2004. www.cn.apan.net/cairns/NRW/ (last accessed, Dec. 2004).
29. D. Zeinalipour-Yazti, and M. Dikaiakos Design and implementation of a distributed crawler and filtering processor A. Halevy A. Gal Proceedings of the Fifth International Workshop on Next Generation Information Technologies and Systems (NGITS 2002), volume 2382 of Lecture Notes in Computer Science 2002 Springer 58 74