On the feasibility of multi-site web search engines

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2009, Pages 425-434

  Baeza Yates, Ricardo ^a   Gionis, Aristides ^a   Junqueira, Flavio ^a   Plachouras, Vassilis ^a   Telloli, Luca ^a  

^a YAHOO RESEARCH   (Spain)

Author keywords

Cost model; Distributed systems; Query processing; Web search

Indexed keywords

ALL DISTRIBUTED; CENTRALIZED ARCHITECTURE; CENTRALIZED SYSTEMS; COMMUNICATION COST; COMPUTATIONAL COSTS; COMPUTE RESOURCES; COST MODELS; DATA LOCALITY; DISTRIBUTED ENVIRONMENTS; DISTRIBUTED SEARCH ENGINES; DISTRIBUTED SYSTEMS; DISTRIBUTED WEB SEARCH; DOCUMENT COLLECTION; ENGINEERING TASKS; MULTI-SITE; PROCESSING ALGORITHMS; REAL COSTS; RESEARCH QUESTIONS; TOTAL COSTS; WEB SEARCH ENGINES; WEB SEARCHES;

COSTS; ENGINES; INFORMATION RETRIEVAL; KNOWLEDGE MANAGEMENT; QUERY PROCESSING; SEARCH ENGINES; SECURITY OF DATA;

WORLD WIDE WEB;

EID: 74549206193     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1645953.1646009     Document Type: Conference Paper

References (38)

1. C. S. Badue et al. Analyzing imbalance among homogeneous index servers in a web search system. Inf. Process. Management., 43(3):592-608, 2007.
2. R. Baeza-Yates, C. Castillo, F. Junqueira, V. Plachouras, and F. Silvestri. Challenges in distributed information retrieval (invited paper). In ICDE, pages 6-20, 2007.
3. R. Baeza-Yates, A. Gionis, F. Junqueira, V. Murdock, V. Plachouras, and F. Silvestri. The impact of caching on search engines. In SIGIR, pages 183-190, 2007.
4. L. A. Barroso, J. Dean, and U. Hölzle. Web search for a planet: The google cluster architecture. IEEE Micro, 23(2):22-28, 2003.
5. L. A. Barroso and U. Hölzle. The Datacenter as a Computer - an introduction to the design of warehouse-scale machines. Morgan and Claypool Publishers, May 2009.
6. M. Bender, S. Michel, P. Triantafillou, and G. Weikum. Design Alternatives for Large-Scale Web Search: Alexander was Great, Aeneas a Pioneer, and Anakin has the Force. In 1st LSDS-IR Workshop, pages 16-22, 2007.
7. E. A. Brewer. Lessons from giant-scale services. IEEE Internet Computing, 5(4):46-55, 2001.
8. S. Brin and L. Page. The anatomy of a large-scale hypertextual web search engine. Computer Networks and ISDN Systems, 30(1-7):107-117, April 1998.
9. J. Callan. Distributed information retrieval. In W. B. Croft, editor, Advances in Information Retrieval. Recent Research from the Center for Intelligent Information Retrieval, chapter 5, pages 127-150. Kluwer Academic Publishers, 2000.
10. J. P. Callan, Z. Lu, and W. B. Croft. Searching distributed collections with inference networks. In SIGIR, pages 21-28, 1995.
11. B. B. Cambazoglu, V. Plachouras, and R. Baeza-Yates. Quantifying performance and quality gains in distributed web search engines. In SIGIR, pages 411-418, 2009.
12. N. Cardwell, S. Savage, and T. Anderson. Modeling TCP latency. In INFOCOM, pages 1742-1751, 2000.
13. J. Cho and H. Garcia-Molina. Parallel crawlers. In WWW, pages 124-135, 2002.
14. K. Church, A. Greenberg, and J. Hamilton. On delivering embarrassingly distributed cloud services. In HotNets, 2008.
15. N. Craswell, F. Crimmins, D. Hawking, and A. Moffat. Performance and cost tradeoffs in web search. In ADC, pages 161-169, 2004.
16. EuroStat. European prices for industrial electricity, 2007-8.
17. J. Exposto, J. Macedo, A. Pina, A. Alves, and J. Rufino. Geographical partition for distributed web crawling. In Workshop on Geographic Information Retrieval, pages 55-60, 2005.
18. T. Fagni, R. Perego, F. Silvestri, and S. Orlando. Boosting the performance of web search engines: Caching and prefetching query results by exploiting historical usage data. ACM Trans. Inf. Syst., 24(1):51-78, 2006.
19. X. Fan, W.-D. Weber, and L. A. Barroso. Power provisioning for a warehouse-sized computer. In ISCA, pages 13-23, 2007.
20. S. Garcia, H. E. Williams, and A. Cannane. Access-ordered indexes. In ACS, pages 7-14, 2004.
21. A. Heydon and M. Najork. Mercator: A scalable, extensible web crawler. World Wide Web, 2(4):219-229, 1999.
22. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR, pages 154-161, 2005.
23. H.-T. Lee, D. Leonard, X. Wang, and D. Loguinov. IRLbot: scaling to 6 billion pages and beyond. In WWW, pages 427-436, 2008.
24. J. Li, B. T. Loo, J. Hellerstein, F. Kaashoek, D. R. Karger, and R. Morris. On the feasibility of peer-to-peer web indexing and search. In 2nd International Workshop on Peer-to-Peer Systems (IPTPS), Feb. 2003.
25. S. Melink, S. Raghavan, B. Yang, and H. Garcia-Molina. Building a distributed full-text index for the web. ACM Trans. Inf. Syst., 19(3):217-241, 2001.
26. W. Meng, C. Yu, and K.-L. Liu. Building efficient and effective metasearch engines. ACM Comput. Surv., 34(1):48-89, 2002.
27. A. Moffat, W. Webber, and J. Zobel. Load balancing for term-distributed parallel retrieval. In SIGIR, pages 348-355, 2006.
28. A. Moffat, W. Webber, J. Zobel, and R. Baeza-Yates. A pipelined architecture for distributed text query evaluation. Inf. Retr., 10(3):205-231, 2007.
29. A. Ntoulas and J. Cho. Pruning policies for two-tiered inverted index with correctness guarantee. In SIGIR, pages 191-198, 2007.
30. OfCom. The international communications market 2008, 2008. Chapter 2, International comparative pricing.
31. S. Orlando, R. Perego, and F. Silvestri. Design of a parallel and distributed web search engine. In ParCo, pages 197-204, 2001.
32. K. M. Risvik, Y. Aasheim, and M. Lidal. Multi-tier architecture for web search engines. In LAWEB, pages 132-143, 2003.
33. P. C. Saraiva, E. S. de Moura, N. Ziviani, W. Meira, R. Fonseca, and B. Riberio-Neto. Rank-preserving two-level caching for scalable search engines. In SIGIR, pages 51-58, 2001.
34. V. Shkapenyuk and T. Suel. Design and implementation of a high-performance distributed web crawler. In ICDE, 2002.
35. L. Si and J. Callan. Relevant document distribution estimation method for resource selection. In SIGIR, pages 298-305, 2003.
36. A. Tomasic and H. Garcia-Molina. Query processing and inverted indices in shared-nothing text document information retrieval systems. The VLDB Journal, 2(3):243-276, 1993.
37. Uptime Institute. Tier classifications define site infrastructure performance. www.upsite.com/file-downloads/PDF/Tier-Classification.pdf, 2006.
38. J. Zhang and T. Suel. Efficient query evaluation on large textual collections in a peer-to-peer environment. In 5th IEEE International Conference on Peer-to-Peer Computing, pages 225-233, 2005.