Robust result merging using sample-based score estimates

ACM Transactions on Information Systems

Volumn 27, Issue 3, 2009, Pages

  Shokouhi, Milad ^a   Zobel, Justin ^b  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Distributed information retrieval; Result fusion; Result merging; Uncooperative collections

Indexed keywords

DISTRIBUTED INFORMATION RETRIEVAL; GLOBAL SCORE; HIDDEN WEB; METASEARCH; RESULT MERGING; RETRIEVAL MODELS; SIMILARITY SCORES; UNCOOPERATIVE ENVIRONMENTS;

CONTENT BASED RETRIEVAL; INFORMATION RETRIEVAL; INFORMATION SERVICES; RAPID THERMAL ANNEALING;

MERGING;

EID: 72449148083     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/1508850.1508852     Document Type: Article

References (78)

1. ABBACI, F., SAVOY, J., AND BEIGBEDER, M. 2002. A methodology for collection selection in heterogeneous contexts. In Proceedings of the International Conference on Information Technology: Coding and Computing. IEEE Computer Society Press, Los Alamitos, CA, 529.
2. ASLAM, J., PAVL U, V., AND SAVELL, R. 2003. A unified model for metasearch, pooling, and system evaluation. In Proceedings of the 12th International Conference on Information and Knowledge Management (New Orleans, LA). 484-491.
3. ASLAM, J. A. AND MONTAGUE, M. 2001. Models for metasearch. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New Orleans, LA). 276-284.
4. AVRAHAMI, T., YAU, L., SI, L., AND CALLAN, J. 2006. The FedLemur: Federated search in the real world. J. Amer. Soc. Inform. Sci. Tech. 57, 3, 347-358.
5. AZZOPARDI, L., BAILLIE, M., AND CRESTANI, F. 2006. Adaptive query-based sampling for distributed IR. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, WA). 605-606.
6. BAILLIE, M., AZZOPARDI, L., AND CRESTANI, F. 2006. Adaptive query-based sampling of distributed collections. In SPIRE String Processing and Information Retrieval Symposium. Springer, Glasgow, U.K. 316-328.
7. BAR-YOSSEF, Z. AND GUREVICH, M. 2006. Random sampling from a search engine's index. In Proceedings of the 15th International Conference on the World Wide Web (Edinburgh, U.K.). 367-376.
8. BERNSTEIN, Y., SHOKOUHI, M., AND ZOBEL, J. 2006. Compact features for detection of near-duplicates in distributed retrieval. In SPIRE String Processing and Information Retrieval Symposium. Springer, Glasgow, U.K. 110-121.
9. BHARAT, K. AND BRODER, A. 1998. A technique for measuring the relative size and overlap of public web search engines. Comput. Netw. ISDN Syst. 30, 1-7, 379-388.
10. CALLAN, J. 2000. Distributed information retrieval. Advances in Information Retrieval. Kluwer, Norwell, MA, Chapter 5, 127-150.
11. CALLAN, J. AND CONNELL, M. 2001. Query-based sampling of text databases. ACM Trans. Inform. Syst. 19, 2, 97-130.
12. CALLAN, J., CONNELL, M., AND DU, A. 1999. Automatic discovery of language models for text databases. In Proceedings of the ACM SIGMOD International Conference on Management of Data (Philadelphia, PA). 479-490.
13. CALLAN, J., CROFT, B., AND BROGLIO, J. 1997. TREC and TIPSTER experiments with INQUERY. In Readings in Information Retrieval. Morgan Kaufmann, San Francisco, CA, 436-439.
14. CALLAN, J., LU, Z., AND CROFT, W. B. 1995. Searching distributed collections with inference networks. In Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, WA). 21-28.
15. CAVERLEE, J., LIU, L., AND BAE, J. 2006. Distributed query sampling: A quality-conscious approach. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, WA). 340-347.
16. CRASWELL, N., BAILEY, P., AND HAWKING, D. 2000. Server selection on the World Wide Web. In Proceedings of the Fifth ACM Conference on Digital Libraries (San Antonio) TX. 37-46.
17. CRASWELL, N., HAWKING, D., AND THISTLEWAITE, P. 1999. Merging results from isolated search engines. In Proceedings of the 10th Australasian Database Conference. Springer-Verlag, Auckland, New Zealand, 189-200.
18. CROFT, B. 2000. Combining approaches to information retrieval. Advances in Information Retrieval. Kluwer, Norwell, MA, Chapter 1, 1-36.
19. DREILINGER, D. AND HOWE, A. 1997. Experiences with selecting search engines using metasearch. ACM Trans. Inform. Sys. 15, 3, 195-222.
20. D'SOUZA, D. AND THOM, J. 1999. Collection selection using n-term indexing. In Proceedings of the Second International Symposium on Cooperative Database Systems for Advanced Applications (CODAS'99). Springer, Wollongong, Australia, 52-63.
21. D'SOUZA, D., THOM, J., AND ZOBEL, J. 2004. Collection selection for managed distributed document databases. Inform. Process. Manage. 40, 3, 527-546.
22. FOX, E. AND SHAW, J. 1993. Combination of multiple searches. In Proceedings of the Second Text REtrieval Conference. NIST Special Publication. National Institute of Science and Technology, Gaithersburg, MD, 243-252.
23. FRENCH, J., POWELL, A., CALLAN, J., VILES, C., EMMITT, T., PREY, K., AND MOU, Y. 1999. Comparing the performance of database selection algorithms. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Berkeley, CA). 238-245
24. GARCIA, S., WILLIAMS, H., AND CANNANE, A. 2004. Access-ordered indexes. In Proceedings of the 27th Australasian Computer Science Conference (Darlinghurst, Australia). 7-14.
25. GLOVER, E., LAWRENCE, S., BIRMINGHAM, W., AND GILES, C. 1999. Architecture of a metasearch engine that supports user information needs. In Proceedings of the 8th ACM CIKM Conference on Information and Knowledge Management (Kansas City, MO). 210-216.
26. GRAVANO, L., CHANG, C. K., GARCIA-MOLINA, H., AND PAEPCKE, A. 1997. STARTS: Stanford proposal for Internet meta-searching. In Proceedings of the ACM SIGMOD International Conference on Management of Data (Tucson, AZ). 207-218.
27. GRAVANO, L. AND GARCIA-MOLINA, H. 1995. Generalizing GlOSS to vector-space databases and broker hierarchies. In Proceedings of the 21st International Conference on Very Large Data Bases (San Francisco, CA). 78-89.
28. GRAVANO, L., GARCIA-MOLINA, H., AND TOMASIC, A. 1994a. The effectiveness of GlOSS for the text database discovery problem. In Proceedings of the ACM SIGMOD International Conference on Management of Data (Minneapolis, MN). 126-137.
29. GRAVANO, L., GARCIA-MOLINA, H., AND TOMASIC, A. 1994b. Precision and recall of GlOSS estimators for database discovery. In Proceedings of the Third International Conference on Parallel and Distributed Information Systems (Washington, DC). 103-106.
30. GRAVANO, L., GARCIA-MOLINA, H., AND TOMASIC, A. 1999. GlOSS: Text-source discovery over the Internet. ACM Trans. Database Sys. 24, 2, 229-264.
31. GRAVANO, L., IPEIROTIS, P., AND SAHAMI, M. 2003. Qprober: A system for automatic classification of hidden-Web databases. ACM Trans. Inform. Sys. 21, 1, 1-41.
32. GROSS, J. 2003. Linear Regression. Springer, Berlin, Germany.
33. HEDLEY, Y., YOUNAS, M., JAMES, A., AND SANDERSON, M. 2004a. A two-phase sampling technique for information extraction from hidden Web databases. In Proceedings of the 6th Annual ACM International Workshop on Web Information and Data Management (Washington, DC). 1-8.
34. HEDLEY, Y., YOUNAS, M., JAMES, A., AND SANDERSON, M. 2004b. A two-phase sampling technique to improve the accuracy of text similarities in the categorisation of hidden Web databases. In Proceedings of the International Conference on Web Informations Systems. Springer, Brisbane, Australia, 516-527.
35. IPEIROTIS, P. AND GRAVANO, L. 2004. When one sample is not enough: Improving text database selection using shrinkage. In Proceedings of the ACM SIGMOD International Conference on Management of Data (Paris, France). 767-778.
36. JOACHIMS, T., GRANKA, L., PAN, B., HEMBROOKE, H., AND GAY, G. 2005. Accurately interpreting click-through data as implicit feedback. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Salvador, Brazil). 154-161.
37. KIRSCH, T. 2003. Document retrieval over networks wherein ranking and relevance scores are computed at the client for multiple database documents. U.S. Patent 5,659,732.
38. LAFFERTY, J. AND ZHAI, C. 2001. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New Orleans, LA). 111-119. (Pubitemid 32951063)
39. LARKEY, L., CONNELL, M., AND CALLAN, J. 2000. Collection selection and results merging with topically organized U.S. patents and TREC data. InProceedings of the Ninth International Conference on Information and Knowledge Management (McLean, VA). 282-289.
40. LAWRENCE, S. AND GILES, C. 1998. Inquirus, the NECi meta search engine. In Proceedings of the 7th International Conference on the World Wide Web. Elsevier Science Publishers B. V., Brisbane, Australia, 95-105.
41. LEE, J. 1997. Analyses of multiple evidence combination. In Proceedings of the 20th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Philadelphia, PA). 267-276.
42. LILLIS, D., TOOLAN, F., COLLIER, R., AND DUNNION, J. 2006. ProbFuse: A probabilistic approach to data fusion. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, WA). 139-146.
43. LU, J. AND CALLAN, J. 2002. Pruning long documents for distributed information retrieval. In Proceedings of the 11th ACM CIKM International Conference on Information and Knowledge Management (McLean, VA). 332-339.
44. MANMATHA, R., RATH, T., AND FENG, F. 2001. Modeling score distributions for combining the outputs of search engines. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New Orleans, LA). 267-275.
45. NG, K. 1998. An investigation of the conditions for effective data fusion in information retrieval. Ph.D. dissertation. Rutgers University, New Brunswick, NJ.
46. NOTTELMANN, H. AND FUHR, N. 2003. Evaluating different methods of estimating retrieval quality for resource selection. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Toronto, Ont., Canada). 290-297.
47. OGILVIE, P. AND CALLAN, J. 2001. The effectiveness of query expansion for distributed information retrieval. In Proceedings of the 10th ACM CIKM International Conference on Information and Knowledge Management (Atlanta, GA). 183-190.
48. OZTEKIN, B., KARYPIS, G., AND KUMAR, V. 2002. Expert agreement and content based reranking in a meta search environment using Mearf. In Proceedings of the 11th International Conference on the World Wide Web (Honolulu, HI). 333-344.
49. PALTOGLOU, G., SALAMPASIS,M., AND SATRATZEMI,M. 2007. Results merging algorithmusingmultiple regression models. In Proceedings of the Euorpean Conference on Information Retrieval. Springer, Rome, Italy, 173-184.
50. PORTER, M. 1997. Analgorithm for suffix stripping. InReadings in Information Retrieval. Morgan Kaufmann, San Francisco, CA, 313-316.
51. POWELL, A. L. AND FRENCH, J. 2003. Comparing the performance of collection selection algorithms. ACM Trans. Inform. Sys. 21, 4, 412-456.
52. RASOLOFO, Y., ABBACI, F., AND SAVOY, J. 2001. Approaches to collection selection and results merging for distributed information retrieval. In Proceedings of the 10th ACM CIKM International Conference on Information and Knowledge Management (Atlanta, GA). 191-198.
53. RASOLOFO, Y., HAWKING, D., AND SAVOY, J. 2003. Result merging strategies for a current news metasearcher. Inform. Process. Manage. 39, 4, 581-609.
54. SELBERG, E. AND ETZIONI, O. 1995. Multi-service search and comparison using the metacrawler. In Proceedings of the 4th International Conference on the World Wide Web. Oreilly, Boston, MA.
55. SELBERG, E. AND ETZIONI, O. 1997. The MetaCrawler architecture for resource aggregation on the web. IEEE Expert 12, 1, 8-14.
56. SHOKOUHI, M. 2007. Central-rank-based collection selection in uncooperative distributed information retrieval. In Proceedings of the Euorpean Conference on Information Retrieval. Springer, Rome, Italy, 160-172.
57. SHOKOUHI, M., SCHOLER, F., AND ZOBEL, J. 2006a. Sample sizes for query probing in uncooperative distributed information retrieval. In Proceedings of the 8th Asia Pacific Web Conference. Springer (Harbin, China). 63-75.
58. SHOKOUHI, M. AND ZOBEL, J. 2007. Federated text retrieval from uncooperative overlapped collections. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands). 495-502.
59. SHOKOUHI, M., ZOBEL, J., AND BERNSTEIN, Y. 2007. Distributed text retrieval from overlapping collections. In Proceedings of the 18th Australasian Database Conference. CRPIT, vol.63. ACS, Ballarat, Australia, 141-150.
60. SHOKOUHI, M., ZOBEL, J., SCHOLER, F., AND TAHAGHOGHI, S. 2006b. Capturing collection size for distributed non-cooperative retrieval. In Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Seattle, WA). 316-323.
61. SI, L. AND CALLAN, J. 2002. Using sampled data and regression to merge search engine results. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Tampere, Finland). 19-26.
62. SI, L. AND CALLAN, J. 2003a. The effect of database size distribution on resource selection algorithms. In Proeedings of the SIGIR 2003 Workshop on Distributed Information Retrieval (Toronto, Ont., Canada). 31-42.
63. SI, L. AND CALLAN, J. 2003b. Relevant document distribution estimation method for resource selection. In Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Toronto, Ont., Canada). 298-305.
64. SI, L. AND CALLAN, J. 2003c. A semisupervised learning method to merge search engine results. ACM Trans. Inform. Sys. 21, 4, 457-491.
65. SI, L. AND CALLAN, J. 2004. Unified utility maximization framework for resource selection. In Proceedings of the 13th ACM CIKM Conference on Information and Knowledge Management (Washington, DC). 32-41.
66. SI, L. AND CALLAN, J. 2005. Modeling search engine effectiveness for federated search. In Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Salvador, Brazil). 83-90.
67. SI, L., JIN, R., CALLAN, J., AND OGILVIE, P. 2002. A language modeling framework for resource selection and results merging. In Proceedings of the 11th ACM CIKM International Conference on Information and Knowledge Management (McLean, VA). 391-397.
68. THOMAS, P. AND HAWKING, D. 2007. Evaluating sampling methods for uncooperative collections. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands). 503-510.
69. VOGT, C. 1999. Adaptive combination of evidence for information retrieval. Ph.D. dissertation. University of California, San Diego, La Jolla, CA.
70. VOGT, C. AND COTTRELL, G. 1999. Fusion via a linear cmbination of scores. Inform. Retr. 1, 3, 151-173.
71. WANG, Y. AND DEWITT, D. 2004. Computing PageRank in a distributed internet search engine system. In Proceedings of the 30th International Conference on Very Large Data Bases (Toronto, Ont., Canada). 420-431.
72. WU, S. AND MCCLEAN, S. 2006. Performance prediction of data fusion for information retrieval. Inform. Process. Manage. 42, 4, 899-915.
73. WU, S. AND MCCLEAN, S. 2007. Result merging methods in distributed information retrieval with overlapping databases. Inform. Retr. 10, 3, 297-319.
74. XU, J. AND CALLAN, J. 1998. Effective retrieval with distributed collections. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Melbourne, Australia). 112-120.
75. XU, J. AND CROFT, B. 1999. Cluster-based language models for distributed retrieval. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Berkeley, CA). 254-261.
76. XU, J., WU, S., AND LI, X. 2007. Estimating collection size with logistic regression. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Amsterdam, The Netherlands). 789-790.
77. ZHAI, C. 2001. Notes on the lemur TFIDF model. School of Computer Science. Carnegie Mellon University, Pittsburgh, PA. unpublished report. www.cs.cmu.edu/lemur/1.1/tfidf.ps.
78. ZOBEL, J. 1997. Collection selection via lexicon inspection. In Proceedings of the Australian Document Computing Symposium (Melbourne, Australia). 74-80.