Tuning the capacity of search engines: Load-driven routing and incremental caching to reduce and balance the load

ACM Transactions on Information Systems

Volumn 28, Issue 2, 2010, Pages

  Puppin, Diego ^a,c   Silvestri, Fabrizio ^a   Perego, Raffaele ^a   Baeza Yates, Ricardo ^b  

^a CNR   (Italy)

^b YAHOO RESEARCH   (Spain)

^c GOOGLE INC   (United States)

Author keywords

Collection selection; Distributed IR; Incremental caching; Web search engines

Indexed keywords

CACHING TECHNIQUE; COMPUTATIONAL LOADS; COMPUTING LOAD; COMPUTING POWER; DISTRIBUTED IR; DOCUMENT MODEL; HIGH QUALITY; INCREMENTAL CACHING; QUERY COSTS; SYSTEM CONFIGURATIONS;

ARCHITECTURE; BALANCING; INFORMATION RETRIEVAL; SEARCH ENGINES; USER INTERFACES;

WORLD WIDE WEB;

EID: 77950595883     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/1740592.1740593     Document Type: Article

References (48)

1. BADUE, C. S., BAEZA-YATES, R., RIBEIRO-NETO, B., ZIVIANI, A., AND ZIVIANI, N. 2007. Analyzing imbalance among homogeneous index servers in a Web search system. Inform. Process. Manage. 43, 3, 592-608. (Pubitemid 46164824)
2. BAEZA-YATES, R., CASTILLO, C., JUNQUEIRA, F., PLACHOURAS, V., AND SILVESTRI, F. 2007a. Challenges in distributed information retrieval (invited paper). In Proceedings of International Conference on Data Engineering (ICDE). IEEE CS Press.
3. BAEZA-YATES, R., GIONIS, A., JUNQUEIRA, F., MURDOCK, V., PLACHOURAS, V., AND SILVESTRI, F. 2007b. The impact of caching on search engines. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, New York, NY, 183-190. (Pubitemid 350164960)
4. BAEZA-YATES, R., GIONIS, A., JUNQUEIRA, F. P., MURDOCK, V., PLACHOURAS, V., AND SILVESTRI, F. 2008. Design trade-offs for search engine caching. ACM Trans. Web 2, 4, 1-28.
5. BARROSO, L., DEAN, J., AND HÖLZE, U. 2003. Web search for a planet: The Google cluster architecture. IEEE Micro 22, 2.
6. BEITZEL, S. M., JENSEN, E. C., CHOWDHURY, A., GROSSMAN, D., AND FRIEDER, O. 2004. Hourly analysis of a very large topically categorized Web query log. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, New York, NY, 321-328.
7. BOLDI, P. AND VIGNA, S. 2004. The webgraph framework I: compression techniques. In Proceedings of the 13th International Conference on World Wide Web (WWW). ACM Press, New York, NY, 595-602. (Pubitemid 40752793)
8. BRIN, S. AND PAGE, L. 1998. The anatomy of a large-scale hypertextual Web search engine. In Proceedings of the Seventh International Conference on World Wide Web (WWW). Elsevier Science Publishers B. V., Amsterdam, The Netherlands, 107-117.
9. BRODER, A. Z., GLASSMAN, S. C., MANASSE, M. S., AND ZWEIG, G. 1997. Syntactic clustering of the Web. In Selected Papers from the Sixth International Conference on World Wide Web. Elsevier Science Publishers Ltd., Amsterdam, The Netherlands, 1157-1166.
10. CALLAN, J., LU, Z., AND CROFT, W. 1995. Searching distributed collections with inference networks. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. E. A. Fox, P. Ingwersen, and R. Fidel, Eds. ACM Press, 21-28.
11. CHIERICHETTI, F., PANCONESI, A., RAGHAVAN, P., SOZIO, M., TIBERI, A., AND UPFAL, E. 2007. Finding near neighbors through cluster pruning. In Proceedings of the Twenty-Sixth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS). ACM, New York, NY, 103-112. (Pubitemid 47620885)
12. CHOWDHURY, A., FRIEDER, O., GROSSMAN, D., AND MCCABE, M. C. 2002. Collection statistics for fast duplicate document detection. ACM Trans. Inform. Syst. 20, 2, 171-191. (Pubitemid 44642301)
13. DEAN, J. AND GHEMAWAT, S. 2004. Mapreduce: simplified data processing on large clusters. In Proceedings of the 6th Conference Symposium on Operating Systems Design and Implementation (OSDI). USENIX Association, Berkeley, CA, 10-10.
14. DHILLON, I. S., MALLELA, S., AND MODHA, D. S. 2003. Information-theoretic co-clustering. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD). 89-98.
15. FAGNI, T., PEREGO, R., SILVESTRI, F., AND ORLANDO, S. 2006. Boosting the performance of Web search engines: Caching and prefetching query results by exploiting historical usage data. ACM Trans. Inform. Syst. 24, 1, 51-78.
16. FRIEDER, O. AND SIEGELMANN, H. T. 1991. On the allocation of documents in multiprocessor information retrieval systems. In Proceedings of the 14th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM Press, New York, NY, 230-239.
17. Google. 2007. Google begins move to universal search. http://www.google.com/intl/en/press/pressrel/universalsearch-20070516.html.
18. GRAVANO, L. AND GARCIA-MOLINA, H. 1995. Generalizing GlOSS to vector-space databases and broker hierarchies. In Proceedings of the 21th International Conference on Very Large Data Bases (VLDB). Morgan Kaufmann Publishers Inc., San Francisco, CA, 78-89.
19. GRAVANO, L., GARCIA-MOLINA, H., AND TOMASIC, A. 1994. Precision and recall of GlOSS estimators for database discovery. Techn. note number STAN-CS-TN-94-10, Stanford University.
20. HOAD, T. C. AND ZOBEL, J. 2003. Methods for identifying versioned and plagiarized documents. J. Amer. Soc. Inform. Sci. Tech. 54, 3, 203-215.
21. JAIN, A. AND DUBES, R. 1988. Algorithms for Clustering Data. Prentice Hall.
22. JANSEN, B. AND SPINK, A. 2006. How are we searching the World Wide Web? A comparison of nine search engine transaction logs. Inform Proc. and Management 42, 248-263. (Pubitemid 41119088)
23. JANSEN, B. J., SPINK, A., BATEMAN, J., AND SARACEVIC, T. 1998. Real life information retrieval: a study of user queries on the Web. SIGIR Forum 32, 1, 5-17. (Pubitemid 128619000)
24. KAREDLA, R., LOVE, J. S., AND WHERRY, B. G. 1994. Caching strategies to improve disk system performance. Computer 27, 3, 38-46.
25. LARKEY, L. S., CONNELL, M. E., AND CALLAN, J. 2000. Collection selection and results merging with topically organized U. S. patents and TREC data. In Proceedings of the 9th International Conference on Information and Knowledge Management (CIKM). ACM Press, New York, NY, 282-289.
26. LEMPEL, R. AND MORAN, S. 2003. Predictive caching and prefetching of query results in search engines. In Proceedings of the 12th International Conference on World Wide Web (WWW). ACM, New York, NY, 19-28.
27. LIU, X. AND CROFT, W. B. 2004. Cluster-based retrieval using language models. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM Press, New York, NY, 186-193.
28. MARKATOS, E. P. 2001. On caching search engine query results. Comput. Comm. 24, 2, 137-143. (Pubitemid 32092795)
29. MOFFAT, A. AND ZOBEL, J. 1994. Information retrieval systems for large document collections. In Proceedings of the Text REtrieval Conference. 85-94.
30. PAPADIMITRIOU, S. AND SUN, J. 2008. Disco: Distributed co-clustering with map-reduce. In Proceedings of the IEEE International Conference on Data Mining (ICDM).
31. Pew Internet and American Life Project. 2005. Search engine use shoots up in the past year and edges towards email as the primary internet application. http://www.pewinternet.org/pdfs/PIP-SearchData-1105.pdf.
32. POBLETE, B. AND BAEZA-YATES, R. 2008. Query-sets: using implicit feedback and query patterns to organize Web documents. In Proceedings of the 17th International Conference on World Wide Web (WWW). ACM, New York, NY, USA, 41-50.
33. PUPPIN, D. 2008. Collection selection... now, with more documents! In Proceedings of the 2nd International Conference on Scalable Information Systems (InfoScale). ICST, Brussels, Belgium.
34. PUPPIN, D. AND SILVESTRI, F. 2006. The query-vector document model. In Proceedings of the 15th ACM International Conference on Information and Knowledge Management (CIKM). ACM, New York, NY, 880-881. (Pubitemid 47203675)
35. PUPPIN, D., SILVESTRI, F., AND LAFORENZA, D. 2006. Query-driven document partitioning and collection selection (invited paper). In Proceedings of the 1st International Conference on Scalable Information Systems (InfoScale). ACM, New York, NY, USA, 34.
36. PUPPIN, D., SILVESTRI, F., PEREGO, R., AND BAEZA-YATES, R. 2007. Load-balancing and caching for collection selection architectures. In Proceedings of the 2nd International Conference on Scalable Information Systems (InfoScale). ICST, Brussels, Belgium, 1-10.
37. RAGHAVAN, V. V. AND SEVER, H. 1995. On the reuse of past optimal queries. In SIGIR'95: Proceedings of the 18th annual international ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM, New York, NY, 344-350.
38. RANDALL, K. H., STATA, R., WIENER, J. L., AND WICKREMESINGHE, R. G. 2002. The link database: Fast access to graphs of the Web. In Proceedings of the Data Compression Conference (DCC). IEEE Computer Society, Los Alamitos, CA, 122.
39. ROBERTSON, S. E. AND WALKER, S. 1994. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Springer-Verlag, Berlin, Germany, 232-241.
40. SILVERSTEIN, C., HENZINGER, M., MARAIS, H., AND MORICZ, M. 1999. Analysis of a very large Web search engine query log. In ACM SIGIR Forum. 6-12.
41. SILVESTRI, F. 2007. Sorting out the document identifier assignment problem. In Proceedings of the European Conference on IR Research (ECIR). G. Amati, C. Carpineto, and G. Romano, Eds. Lecture Notes in Computer Science, vol. 4425. Springer, 101-112. (Pubitemid 350259543)
42. TOMASIC, A., GRAVANO, L., LUE, C., SCHWARZ, P., AND HAAS, L. 1997. Data structures for efficient broker implementation. ACM Trans. Inform. Syst. 15, 3, 223-253. (Pubitemid 127685705)
43. VAN RIJSBERGEN, C. 1979. Information Retrieval. Butterworths.
44. WEBBER, W., MOFFAT, A., ZOBEL, J., AND BAEZA-YATES, R. 2006. A pipelined architecture for distributed text query evaluation. Inform. Retrieval. 10, 3.
45. XIE, Y. AND O'HALLARON, D. R. 2002. Locality in search engine queries and its implications for caching. In Proceedings of the 21st Annual Joint Conference of the IEEE Computer and Communications Societies (INFOCOM).
46. XU, J. AND CALLAN, J. 1998. Effective retrieval with distributed collections. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR).
47. XU, J. AND CROFT, W. B. 1999. Cluster-based language models for distributed retrieval. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR). 254-261.
48. YUWONO, B. AND LEE, D. L. 1997. Server ranking for distributed text retrieval systems on the Internet. In Proceedings of the 5th International Conference on Database Systems for Advanced Applications (DASFAA). World Scientific Press, 41-50.