Optimal crawling strategies for web search engines

Proceedings of the 11th International Conference on World Wide Web, WWW '02

Volumn , Issue , 2002, Pages 136-147

  Wolf, J L ^a   Squillante, M S ^a   Yu, P S ^a   Sethuraman, J ^b   Ozsen, L ^c  

^a IBM T J WATSON RESEARCH CENTER   (United States)

^b Och Spine at New York Presbyterian Hospitals   (United States)

^c NORTHWESTERN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTATIONALLY EFFICIENT; CRAWLING STRATEGY; DYNAMIC WEBSITE; NETWORK-FLOW THEORY; PROBABILITY THEORY; RESOURCE ALLOCATION PROBLEM; SIMULATION EXPERIMENTS; STOCHASTIC FRAMEWORK; UPDATE PATTERNS; WEB ENVIRONMENT; WEB PAGE; WEB SEARCH ENGINES;

INFORMATION RETRIEVAL; OPTIMIZATION; PROBLEM SOLVING; SEARCH ENGINES;

WEBSITES;

EID: 77953055403     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/511446.511465     Document Type: Conference Paper

References (29)

1. R. Ahuja, T. Magnanti and J. Orlin, Network Flows. Prentice Hall, 1993.
2. A. Arasu, J. Cho, H. Garcia-Molina, A. Paepcke and S. Raghavan, "Searching the Web", ACM Transactions on Internet Technology, 1(1), 2001.
3. J. Blazewicz, K. Ecker, G. Schmidt and J. Weglarz, Scheduling in Computer and Manufacturing Systems, Springer-Verlag, 1993.
4. A. Broder, Personal communication.
5. J. Challenger, P. Dantzig, A. Iyengar, M. S. Squillante, and L. Zhang. Efficiently serving dynamic data at highly accessed web sites. Preprint, May 2001.
6. J. Cho and H. Garcia-Molina, "Synchronizing a Database to Improve Freshness", ACM SICMOD Conference, 2000.
7. E. Coffman, Z. Liu and R. Weber, "Optimal Robot Scheduling for Web Search Engines", INRIA Research Report, 1997.
8. F. Douglas, A. Feldmann and B. Krishnamurthy, "Rate of Change and other Metrics: A Live Study of the World Wide Web", USENIX Symposium on Internetworking Technologies and Systems, 1999.
9. B. Fox, "Discrete Optimization via Marginal Analysis", Management Science, 13:210-216, 1966.
10. G. Frederickson and D. Johnson, "The Complexity of Selection and Ranking in X+Y and Matrices with Sorted Columns", Journal of Computer and System Science, 24:197-208, 1982.
11. Z. Galil and N. Megiddo, "A Fast Selection Algorithm and the Problem of Optimum Distribution of Efforts". Journal of the ACM, 26:58-64, 1981.
12. Ibaraki, T., and Katoh, N., "Resource Allocation Problems: Algorithmic Approaches", MIT Press. Cambridge, MA, 1988.
13. International Business Machines Corporation, Optimization Subroutine Library Guide and Reference. IBM, 1995.
14. N. Katoh and T. Ibaraki, "Resource Allocation Problems", in Handbook of Combinatorial Optimization, D-Z. Du and P. Pardalos, editors, Kluwer Academic Press, 2000.
15. D. Knuth, The Art of Computer Programming, vol. 2. Addison Wesley, 1973.
16. A. Iyengar, M. Squillante and L. Zhang, "Analysis and Characterization of Large-Scale Web Server Access Patterns and Performance", World Wide Web. 2:85-100, 1999.
17. S. Lawrence and C. Giles, "Accessibility of Information on the Web", Nature, 400:107-109, 1999.
18. G. Nemhauser and L. Wolsey, Integer and Combinatorial Optimization, J. Wiley, 1988.
19. V. N. Padmanabhan and L. Qiu. "The Content and Access Dynamics of a Busy Web Site: Findings and Implications", ACM SIGCOMM '00 Conference, 2000.
20. M. Pinedo, Scheduling: Theory, Algorithms and Systems, Prentice-Hall, 1995.
21. J. Pitkow and P. Pirolli, "Life, Death and Lawfulness on the Electronic Frontier", CHI Conference on Human Factors in Computing Systems, 1997.
22. W. Press, B. Flannery, S. Teukolsky and W. Vetterling, Numerical Recipes, Cambridge University Press, 1986.
23. S. M. Ross. Stochastic Processes. John Wiley and Sons, Second Edition, 1997.
24. K. Sigman. Stationary Marked Point Processes: An Intuitive Approach. Chapman and Hall, 1995.
25. M. Squillante, D. Yao and L. Zhang, "Web Traffic Modeling and Web Server Performance Analysis". IEEE Conference on Decision and Control, 1999.
26. J. Talim, Z. Liu, P. Nain and E. Coffman, "Optimizing the Number of Robots for Web Search Engines", Telecommunication Systems Journal. 17(1-2):234-243, 2001.
27. C. Wills and M. Mikhailov, "Towards a Better Understanding of Web Resources and Server Responses for Improved Caching", WWW Conference. 1999.
28. R. W. Wolff. Stochastic Modeling and the Theory of Queues. Prentice Hall, 1989.
29. G. Zipf, Human Behavior and the Principle of Least Effort, Addison-Wesley, 1949.