Predicting query performance by query-drift estimation

ACM Transactions on Information Systems

Volumn 30, Issue 2, 2012, Pages

  Shtok, Anna ^a   Kurland, Oren ^a   Carmel, David ^b   Raiber, Fiana ^a   Markovits, Gad ^a  

^a TECHNION ISRAEL INSTITUTE OF TECHNOLOGY   (Israel)

^b IBM RESEARCH   (United States)

Author keywords

Query drift; Query performance prediction; Score distribution

Indexed keywords

EMPIRICAL EVALUATIONS; PREDICTION METHODS; PREDICTION QUALITY; QUERY DRIFT; QUERY PERFORMANCE; RETRIEVAL METHODS; RETRIEVAL MODELS; SCORE DISTRIBUTION; STANDARD DEVIATION;

INFORMATION RETRIEVAL; STATISTICS;

FORECASTING;

EID: 84863752368     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/2180868.2180873     Document Type: Article

References (67)

1. ABDUL-JALEEL, N., ALLAN, J., CROFT, W. B., DIAZ, F., LARKEY, L., LI, X., SMUCKER, M. D., AND WADE, C. 2004. UMASS at trec 2004-Novelty and hard. In Proceedings of the Text Retrieval Conference (TREC-13).
2. AMATI, G., CARPINETO, C., AND ROMANO, G. 2004. Query difficulty, robustness and selective application of query expansion. In Proceedings of the European Conference on IR Research (ECIR'04). 127-137.
3. ARAMPATZIS, A. AND ROBERTSON, S. 2011. Modeling score distributions in information retrieval. Inf. Re-triev. 14, 1, 26-46.
4. ARAMPATZIS, A., KAMPS, J., AND ROBERTSON, S. 2009. Where to stop reading a ranked list? Threshold optimization using truncated score distributions. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 524-531.
5. ASLAM, J. A. AND PAVLU, V. 2007. Query hardness estimation using Jensen-Shannon divergence among multiple scoring functions. In Proceedings of the European Conference on IR Research (ECIR'07). 198-209.
6. BENDERSKY, M., CROFT, W. B., AND DIAO, Y. 2011. Quality-Biased ranking of Web documents. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM'11). 95-104.
7. BERNSTEIN, Y., BILLERBECK, B., GARCIA, S., LESTER, N., SCHOLER, F., AND ZOBEL, J. 2005. RMIT university at trec 2006: Terabyte and robust track. In Proceedings of the Text Retrieval Conference (TREC-14).
8. BUCKLEY, C. 2004. Why current IR engines fail. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. Poster. 584-585.
9. BUCKLEY, C., SALTON, G., ALLAN, J., AND SINGHAL, A. 1994. Automatic query expansion using SMART: TREC3. In Proceedings of the Text Retrieval Conference (TREC-3). 69-80.
10. CARMEL, D. AND YOM-TOV, E. 2010. Estimating the Query Difficulty for Information Retrieval. Synthesis Lectures on Information Concepts, Retrieval, and Services. Morgan & Claypool.
11. CARMEL, D., YOM-TOV, E., DARLOW, A., AND PELLEG, D. 2006. What makes a query difficult? In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 390-397.
12. CLARKE, C. L. A., CRASWELL, N., AND SOBOROFF, I. 2009. Overview of the trec 2009 Web track. In Proceedings of the Text Retrieval Conference (TREC).
13. CORMACK, G. V., SMUCKER, M. D., AND CLARKE, C. L. A. 2010. Efficient and effective spam filtering and re-ranking for large Web datasets. CoRR abs/1004.5168.
14. CROFT, W. B. AND LAFFERTY, J. 2003. Language Modeling for Information Retrieval. Information Retrieval Book Series, Number 13. Kluwer.
15. CRONEN-TOWNSEND, S., ZHOU, Y., AND CROFT, W. B. 2002. Predicting query performance. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 299-306.
16. CRONEN-TOWNSEND, S., ZHOU, Y., AND CROFT, W. B. 2004. A language modeling framework for selective query expansion. Tech. rep. IR-338, Center for Intelligent Information Retrieval, University of Massachusetts.
17. CRONEN-TOWNSEND, S., ZHOU, Y., AND CROFT, W. B. 2006. Precision prediction based on ranked list coherence. Inf. Retriev. 9, 6, 723-755.
18. CUMMINS, R., JOSE, J. M., AND O'RIORDAN, C. 2011a. Improved query performance prediction using standard deviation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 1089-1090.
19. CUMMINS, R., LALMAS, M., O'RIORDAN, C., AND JOSE, J. M. 2011b. Navigating the user query space. In Proceedings of the International Symposium on String Processing and Information Retrieval (SPIRE'11). 380-385.
20. DAI, K., KANOULAS, E., PAVLU, V., AND ASLAM, J. A. 2011. Variational bayes for modeling score distributions. Inf. Retriev. 14, 1, 47-67.
21. DIAZ, F. 2007. Performance prediction using spatial autocorrelation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 583-590.
22. FANG, H. AND ZHAI, C. 2005. An exploration of axiomatic approaches to information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 480-487.
23. FANG, H., TAO, T., AND ZHAI, C. 2004. A formal study of information retrieval heuristics. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 49-56.
24. FUHR, N. 1992. Probabilistic models in information retrieval. Comput. J. 35, 3, 243-255.
25. HARMAN, D. 1992. Relevance feedback revisited. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 1-10.
26. HARMAN, D. AND BUCKLEY, C. 2004. The NRRC reliable information access (ria) workshop. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 528-529.
27. HAUFF, C., HIEMSTRA, D., AND DE JONG, F. 2008a. A survey of preretrieval query performance predictors. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'08). 1419-1420.
28. HAUFF, C., MURDOCK, V., AND BAEZA-YATES, R. 2008b. Improved query difficulty prediction for the Web. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'08). 439-448.
29. HAUFF, C., KELLY, D., AND AZZOPARDI, L. 2010. A comparison of user and system query performance predictions. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'10). 979-988.
30. HE, B. AND OUNIS, I. 2004. Inferring query performance using pre-retrieval predictors. In Proceedings of the International Symposium on String Processing and Information Retrieval (SPIRE'04). 43-54.
31. KANOULAS, E., DAI, K., PAVLU, V., AND ASLAM, J. A. 2010. Score distribution models: Assumptions, intuition, and robustness to score manipulation. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 242-249.
32. LAFFERTY, J. D. AND ZHAI, C. 2001. Document language models, query models, and risk minimization for information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 111-119.
33. LAVRENKO, V. AND CROFT, W. B. 2001. Relevance-based language models. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 120-127.
34. LIN, J., METZLER, D., ELSAYED, T., AND WANG, L. 2010. Of ivory and smurfs: Loxodontan mapreduce experiments for Web search. In Proceedings of the Text Retrieval Conference (TREC).
35. LIU, X. AND CROFT, W. B. 2008. Evaluating text representations for retrieval of the best group of documents. In Proceedings of the European Conference on IR Research (ECIR'08). 454-462.
36. LV, Y. AND ZHAI, C. 2011. When documents are very long, bm25 fails! In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 1103-1104.
37. MANMATHA, R., RATH, T. M., AND FENG, F. 2001. Modeling score distributions for combining the outputs of search engines. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 267-275.
38. METZLER, D. AND CROFT, W. B. 2005. A Markov random field model for term dependencies. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 472-479.
39. METZLER, D. AND CROFT, W. B. 2007. Linear feature-based models for information retrieval. Inf. Retriev. 10, 3, 257-274.
40. MITRA, M., SINGHAL, A., AND BUCKLEY, C. 1998. Improving automatic query expansion. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 206-214.
41. MOTHE, J. AND TANGUY, L. 2005. Linguistic features to predict query difficulty. In ACM SIGIR'05 Workshop on Predicting Query Difficulty-Methods and Applications.
42. PÉ REZ-IGLESIAS, J. AND ARAUJO, L. 2009. Ranking list dispersion as a query performance predictor. In Proceedings of the 2nd International Conference on Theory of Information Retrieval (ICTIR'09). 371-374.
43. PÉ REZ-IGLESIAS, J. AND ARAUJO, L. 2010. Standard deviation as a query hardness estimator. In Proceedings of the International Symposium on String Processing and Information Retrieval (SPIRE'10). 207-212.
44. PONTE, J. M. AND CROFT, W. B. 1998. A language modeling approach to information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 275-281.
45. RAIBER, F. AND KURLAND, O. 2010. On identifying representative relevant documents. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'10). 99-108.
46. ROBERTSON, S. 2007. On score distributions and relevance. In Proceedings of the European Conference on IR Research (ECIR'07). 40-51.
47. ROBERTSON, S. E., WALKER, S., JONES, S., HANCOCK-BEAULIEU, M., AND GATFORD, M. 1994. Okapi at trec-3. In Proceedings of the Text Retrieval Conference (TREC).
48. ROCCHIO, J. J. 1971. Relevance feedback in information retrieval. In The SMART Retrieval System: Experiments in Automatic Document Processing, G. Salton Ed., Prentice Hall, 313-323.
49. SALTON, J., WONG, A., AND YANG, C. S. 1975. A vector space model for automatic indexing. Comm. ACM 18, 11, 613-620.
50. SCHOLER, F. AND GARCIA, S. 2009. A case for improved evaluation of query difficulty prediction. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 640-641.
51. SCHOLER, F., WILLIAMS, H. E., AND TURPIN, A. 2004. Query association surrogates for Web search. J. A m. Soc. Inf. Sci. Technol. 55, 7, 637-650.
52. SEO, J. AND CROFT, W. B. 2010. Geometric representations for multiple documents. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 251-258.
53. SHTOK, A., KURLAND, O., AND CARMEL, D. 2009. Predicting query performance by query-drift estimation. In Proceedings of the International Conference on Theory of Information Retrieval (ICTIR'09). 305-312.
54. SHTOK, A., KURLAND, O., AND CARMEL, D. 2010. Using statistical decision theory and relevance models for query performance prediction. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 259-266.
55. SONG, F. AND CROFT, W. B. 1999. A general language model for information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval (Poster abstract). 279-280.
56. TERRA, E. L. AND WARREN, R. 2005. Poison pills: Harmful relevant documents in feedback. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'05). 319-320.
57. TOMLINSON, S. 2004. Robust, Web and terabyte retrieval with hummingbird search server at trec 2004. In Proceedings of the Text Retrieval Conference (TREC-13).
58. TURTLE, H. R. AND CROFT, W. B. 1990. Inference networks for document retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 1-24.
59. VINAY, V., COX, I. J., MILIC-FRAYLING, N., AND WOOD, K. R. 2006. On ranking the effectiveness of searches. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 398-404.
60. VOORHEES, E. M. 2004. Overview of the trec 2004 robust retrieval track. In Proceedings of the Text Retrieval Conference (TREC-13).
61. YOM-TOV, E., FINE, S., CARMEL, D., AND DARLOW, A. 2005. Learning to estimate query difficulty: Including applications to missing content detection and distributed information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 512-519.
62. ZHAI, C. AND LAFFERTY, J. D. 2001a. Model-Based feedback in the language modeling approach to information retrieval. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'01). 403-410.
63. ZHAI, C. AND LAFFERTY, J. D. 2001b. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 334-342.
64. ZHAO, Y., SCHOLER, F., AND TSEGAY, Y. 2008. Effective preretrieval query performance prediction using similarity and variability evidence. In Proceedings of the European Conference on IR Research (ECIR'08). 52-64.
65. ZHOU, Y. 2007. Retrieval performance prediction and document quality. Ph.D. thesis, University of Massachusetts Amherst.
66. ZHOU, Y. AND CROFT, W. B. 2006. Ranking robustness: A novel framework to predict query performance. In Proceedings of the International Conference on Information and Knowledge Management (CIKM'06). 567-574.
67. ZHOU, Y. AND CROFT, W. B. 2007. Query performance prediction in Web search environments. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. 543-550.