Large-scale validation and analysis of interleaved search evaluation

ACM Transactions on Information Systems

Volumn 30, Issue 1, 2012, Pages

  Chapelle, Olivier ^a   Joachims, Thorsten ^b   Radlinski, Filip ^c   Yue, Yisong ^d  

^a YAHOO RESEARCH   (United States)

^b Department of Obstetrics Gynecology   (United States)

^c MICROSOFT   (United States)

^d CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

Clicks; Interleaving; Judgments; Online evaluation; Search engine; Sensitivity

Indexed keywords

CLICKS; INTERLEAVING; JUDGMENTS; ON-LINE EVALUATION; SENSITIVITY;

INFORMATION RETRIEVAL SYSTEMS; SEARCH ENGINES;

INFORMATION RETRIEVAL;

EID: 84859070397     PISSN: 10468188     EISSN: 15582868     Source Type: Journal    
DOI: 10.1145/2094072.2094078     Document Type: Article

References (65)

1. AGICHTEIN, E., BRILL, E., DUMAIS, S., AND RAGNO, R. 2006. Learning user interaction models for prediction web search results preferences. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 3-10.
2. AGRAWAL, R.,HALVERSON, A., KENTHAPADI, K.,MISHRA, N., AND TSAPARAS, P. 2009. Generating labels from clicks. In Proceedings of the 2nd ACM International Conference on Web Search and Data Mining. 172-181.
3. ALI, K. AND CHANG, C. 2006. On the relationship between click-rate and relevance for search engines. In Proceedings of the Conference on Data-Mining and Information Engineering. 213-222. (Pubitemid 350108049)
4. ALLAN, J., ASLAM, J. A., CARTERETTE, B., PAVLU, V., AND KANOULAS, E. 2008. Million query track 2008 overview. In Proceedings of TREC.
5. ASLAM, J. A.,PAVLU, V., AND YILMAZ, E. 2005. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In Proceedings of the ICML Workshop on Learning with Partially Classified Training Data.
6. BECKER, H.,MEEK, C., AND CHICKERING, D. M. 2007. Modeling contextual factors of click rates. In Proceedings of the National Conference on Artificial Intelligence (AAAI). AAAI Press, 1310-1315. (Pubitemid 350149748)
7. BOYAN, J., FREITAG, D., AND JOACHIMS, T. 1996. A machine learning architecture for optimizing web search engines. In Proceedings of the AAAI Workshop on Internet Based Information Systems. 1-8.
8. BUCKLEY, C. AND VOORHEES, E. M. 2004. Retrieval evaluation with incomplete information. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 25-32.
9. BUSCHER, G.,DENGEL, A., AND VAN ELST, L. 2008. Eyemovements as implicit relevance feedback. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), M. Czerwinski, A. M. Lund, and D. S. Tan, Eds. ACM, 2991-2996.
10. CARTERETTE, B.,ALLAN, J., AND SITARAMAN, R. 2006. Minimal test collections for retrieval evaluation. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 268-275.
11. CARTERETTE, B.,BENNETT, P. N.,CHICKERING, D.M., AND DUMAIS, S. T. 2008. Here or there: Preference judgements for relevance. In Proceedings of the European Conference on Information Retrieval (ECIR). 16-27. (Pubitemid 351499055)
12. CARTERETTE, B. AND JONES, R. 2007. Evaluating search engines bymodeling the relationship between relevance and clicks. In Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS). 217-224.
13. CHAPELLE, O.,METLZER, D., ZHANG, Y., AND GRINSPAN, P. 2009. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM'09). ACM, New York, NY, 621-630.
14. CHAPELLE, O. AND ZHANG, Y. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th International Conference on World Wide Web. ACM, New York, NY, 1-10.
15. CLARKE, C. L. A., KOLLA, M., CORMACK, G. V., VECHTOMOVA, O., ASHKAN, A., BÜTTCHER, S., AND MACKINNON, I. 2008. Novelty and diversity in information retrieval evaluation. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 659-666.
16. CLAYPOOL, M., LE, P., WASEDA, M., AND BROWN, D. 2001. Implicit interest indicators. In Proceedings of the International Conference on Intelligent User Interfaces (IUI). 33-40. (Pubitemid 32470940)
17. CLEVERDON, C. W.,MILLS, J., AND KEEN, E. M. 1966. Factors Determining the Performance of Indexing Systems. Cranfield: College of Aeronautics.
18. CRASWELL, N.,ZOETER, O.,TAYLOR,M. J., AND RAMSEY, B. 2008. An experimental comparison of click position-bias models. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM). M. Najork, A. Z. Broder, and S. Chakrabarti, Eds., ACM, 87-94. (Pubitemid 351588782)
19. DUPRET, G.,MURDOCK, V., AND PIWOWARSKI, B. 2007.Web search engine evaluation using clickthrough data and a user model. In Proceedings of the WWW Workshop on Query Log Analysis.
20. EFRON, B. AND TIBSHIRANI, R. 1993. An Introduction to the Bootstrap. Chapman & Hall/CRC.
21. FEIGE, U., RAGHAVAN, P., PELEG, D., AND UPFAL, E. 1994. Computing with noisy information. SIAM J. Comput. 23, 5, 1001-1018.
22. FOX, S.,KARNAWAT, K.,MYDLAND, M.,DUMAIS, S., AND WHITE, T. 2005. Evaluating implicit measures to improve web search. ACM Trans. Inf. Syst. 23, 2, 147-168. (Pubitemid 40786322)
23. HE, J., ZHAI, C., AND LI, X. 2009. Evaluation of methods for relative comparison of retrieval systems based on clickthroughs. In Proceedings of the 18th ACM Conference on Information and Knowledge Management (CIKM'09). ACM, New York, NY, 2029-2032.
24. HOFMANN, K., WHITESON, S., AND DE RIJKE, M. 2011. A probabilistic method for inferring preferences from clicks. In Proceedings of the ACM Conference on Information and Knowledge Management (CIKM).
25. HUFFMAN, S. B. AND HOCHSTER, M. 2007. How well does result relevance predict session satisfaction? In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 567-574. (Pubitemid 350165005)
26. JOACHIMS, T. 2002. Optimizing Search Engines Using Clickthrough Data. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD). ACM, New York, NY, 132-142.
27. JOACHIMS, T. 2003. Evaluating Retrieval Performance using Clickthrough Data. In Text Mining, J. Franke, G. Nakhaeizadeh, and I. Renz, Eds., Physica/Springer Verlag, 79-96.
28. JOACHIMS, T.,FREITAG, D., ANDMITCHELL, T. 1997.WebWatcher: a tour guide for theWorldWideWeb. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Vol. 1. Morgan Kaufmann, 770-777.
29. JOACHIMS, T., GRANKA, L., PAN, B., HEMBROOKE, H., RADLINSKI, F., AND GAY, G. 2007. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Syst. 25, 2, Article 7.
30. KANTOR, P. 1988. National, language-specific evaluation sites for retrieval systems and interfaces. In Proceedings of the International Conference on Computer-Assisted Information Retrieval (RIAO). 139-147.
31. KELLY, D. 2005. Implicit feedback: Using behavior to infer relevance. In New Directions in Cognitive Information Retrieval, 169-186.
32. KELLY, D. AND TEEVAN, J. 2003. Implicit feedback for inferring user preference: A bibliography. Proceedings of the ACM SIGIR Forum 37, 2, 18-28.
33. KOZIELECKI, J. 1981. Psychological Decision Theory. Kluwer.
34. KULKARNI, A., TEEVAN, J., SVORE, K., AND DUMAIS, S. 2011. Understanding temporal query dynamics. In Proceedings of the ACM International Conference on Web Search and Data Mining (WSDM). 167-176.
35. LAMING, D. 1986. Sensory Analysis. Academic Press, London.
36. LIEBERMAN, H. 1995. Letizia: An agent that assists Web browsing. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Morgan Kaufmann, 924-929.
37. LIU, Y., FU, Y., ZHANG, M., MA, S., AND RU, L. 2007. Automatic search engine performance evaluation with click-through data analysis. In Proceedings of the International World Wide Web Conference (WWW). 1133-1134. (Pubitemid 47582348)
38. MANNING, C. D., RAGHAVAN, P., AND SCHUETZE, H. 2008. Introduction to Information Retrieval. Cambridge University Press.
39. MOE, K. K., JENSEN, J. M., AND LARSEN, B. 2007. A qualitative look at eye-tracking for implicit relevance feedback. In Proceedings of the Workshop on Context-Based Information Retrieval, B.-L. Doan, J. M. Jose, and M. Melucci, Eds., CEUR Workshop Proceedings, vol. 326, CEUR-WS.org.
40. MOOD, A., GRAYBILL, F., AND BOES, D. 1974. Introduction to the Theory of Statistics 3rd Ed. McGraw-Hill, New York, NY.
41. MORITA, M. AND SHINODA, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 272-281.
42. OARD, D. W. AND KIM, J. 2001. Modeling information content using observable behavior. In Proceedings of the Annual Meeting of the American Society for Information Science and Technology. 28-45.
43. RADLINSKI, F., BENNETT, P. N., CARTERETTE, B., AND JOACHIMS, T. 2009. Sigir workshop report: Redundancy, diversity and interdependent document relevance. SIGIR Forum 43, 2, 46-52.
44. RADLINSKI, F. AND CRASWELL, N. 2010. Comparing the sensitivity of information retrieval metrics. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 667-674.
45. RADLINSKI, F. AND JOACHIMS, T. 2005. Query chains: Learning to rank from implicit feedback. In Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD). 239-248. (Pubitemid 43218285)
46. RADLINSKI, F. AND JOACHIMS, T. 2006. Minimally invasive randomization for collecting unbiased preferences from clickthrough logs. In Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence (AAAI). 1406-1412. (Pubitemid 44705471)
47. RADLINSKI, F., KURUP, M., AND JOACHIMS, T. 2008. How does clickthrough data reflect retrieval quality. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (CIKM'08). ACM, New York, NY, 43-52.
48. RADLINSKI, F.,KURUP,M., AND JOACHIMS, T. 2010a. Evaluating search engine relevance with click-based metrics. In Preference Learning, J. Fuernkranz and E. Huellermeier, Eds., Springer, 337-362.
49. RADLINSKI, F., SZUMMER, M., AND CRASWELL, N. 2010b. Inferring query intent from reformulations and clicks. In Proceedings of the International World Wide Web Conference (WWW). 1171-1172.
50. SALOJÄRVI, J., PUOLAMÄKI, K., AND KASKI, S. 2005. Implicit relevance feedback from eye movements. In Proceedings of the International Conference on Artificial Neural Networks (ICANN). W. Duch, J. Kacprzyk, E. Oja, and S. Zadrozny, Eds., Springer, 513-518.
51. SANDERSON, M. AND ZOBEL, J. 2005. Information retrieval system evaluation: Effort, sensitivity and reliability. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 162-169.
52. SHAO, J. AND TU, D. 1995. The Jackknife and Bootstrap. Springer.
53. SOBOROFF, I., NICHOLAS, C., AND CAHAN, P. 2001. Ranking retrieval systems without relevance judgments. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 66-73. (Pubitemid 32951057)
54. TEEVAN, J., DUMAIS, S., AND HORVITZ, E. 2007. The potential value of personalizing search. In Proceedings of SIGIR. 756-757.
55. TURPIN, A. AND SCHOLER, F. 2006. User performance versus precision measures for simple search tasks. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 11-18.
56. VOORHEES, E. AND HARMAN, D. 2005. TREC: Experiment and Evaluation in Information Retrieval. MIT press.
57. VOORHEES, E. M. AND BUCKLEY, C. 2002. The effect of topic set size on retrieval experiment error. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 316-323.
58. WANG, K., WALKER, T., AND ZHENG, Z. 2009. PSkip: estimating relevance ranking quality from web search clickthrough data. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD'09). ACM, New York, NY, 1355-1364.
59. WHITE, R., RUTHVEN, I., JOSE, J., AND VAN RIJSBERGEN, C. 2005. Evaluating implicit feedback models using searcher simulations. ACM Trans. Inf. Syst. 23, 3, 325-361. (Pubitemid 43906379)
60. WHITE, R., RUTHVEN, I., AND JOSE, J. M. 2002. The use of implicit evidence for relevance feedback in web retrieval. In Proceedings of the European Conference on Information Retrieval (ECIR). F. Crestani, M. Girolami, and C. J. van Rijsbergen, Eds., Lecture Notes in Computer Science, vol. 2291, Springer, 93-109.
61. YUE, Y., BRODER, J.,KLEINBERG, R., AND JOACHIMS, T. 2009. The K-armed Dueling Bandits Problem. In Proceedings of the Annual Conference on Learning Theory (COLT).
62. YUE, Y., GAO, Y., CHAPELLE, O., ZHANG, Y., AND JOACHIMS, T. 2010. Learning more powerful test statistics for click-based retrieval evaluation. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, NY, 507-514.
63. YUE, Y. AND JOACHIMS, T. 2009. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the International Conference on Machine Learning (ICML). 1201-1208.
64. YUE, Y. AND JOACHIMS, T. 2011. Beat the mean bandit. In Proceedings of the International Conference on Machine Learning (ICML). 241-248.
65. YUE, Y., PATEL, R., AND ROEHRIG, H. 2010. Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. In Proceedings of the International World Wide Web Conference (WWW). 1011-1018.