Evaluating search engine relevance with click-based metrics

Preference Learning

Volumn , Issue , 2011, Pages 337-361

  Radlinski, Filip ^a   Kurup, Madhu ^b   Joachims, Thorsten ^c  

^a MICROSOFT RESEARCH   (United Kingdom)

^b AMAZON   (United States)

^c Department of Obstetrics Gynecology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84859045219     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-642-14125-6_16     Document Type: Chapter

References (31)

1. E. Agichtein, E. Brill, S. Dumais, R. Ragno, Learning user interaction models for prediction web search results preferences, in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2006), pp. 3-10
2. R. Agrawal, A. Halverson, K. Kenthapadi, N. Mishra, P. Tsaparas, Generating labels from clicks, in Proceedings of ACM International Conference on Web Search and Data Mining (WSDM) (2009), pp. 172-181
3. K. Ali, C. Chang, On the relationship between click-rate and relevance for search engines, in Proceedings of Data Mining and Information Engineering (2006)
4. J.A. Aslam, V. Pavlu, E. Yilmaz, A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments, in ICML Workshop on Learning with Partially Classified Training Data (2005)
5. J. Boyan, D. Freitag, T. Joachims, A machine learning architecture for optimizing web search engines, in AAAI Workshop on Internet Based Information Systems (1996)
6. C. Buckley, E.M. Voorhees, Retrieval evaluation with incomplete information, in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2004), pp. 25-32
7. B. Carterette, J. Allan, R. Sitaraman, Minimal test collections for retrieval evaluation, in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2006), pp. 268-275
8. B. Carterette, P.N. Bennett, D.M. Chickering, S.T. Dumais, Here or there: Preference judgements for relevance, in Proceedings of the European Conference on Information Retrieval (ECIR) (2008), pp. 16-27
9. B. Carterette, R. Jones, Evaluating search engines by modeling the relationship between relevance and clicks, in Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS) (2007), pp. 217-224
10. K. Crammer, Y. Singer, Pranking with ranking, in Proceedings of the International Conference on Advances in Neural Information Processing Systems (NIPS) (2001), pp. 641-647
11. G. Dupret, V. Murdock, B. Piwowarski, Web search engine evaluation using clickthrough data and a user model, in WWW Workshop on Query Log Analysis (2007)
12. S. Fox, K. Karnawat, M. Mydland, S. Dumais, T. White, Evaluating implicit measures to improve web search, ACM Trans. Inf. Sci. (TOIS) 23(2), 147-168 (2005)
13. S.B. Huffman, M. Hochster, How well does result relevance predict session satisfaction? in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2007), pp. 567-573
14. T. Joachims, Evaluating retrieval performance using clickthrough data, in Text Mining, ed. by J. Franke, G. Nakhaeizadeh, I. Renz (Physica Verlag, 2003)
15. T. Joachims, Optimizing search engines using clickthrough data, in Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD) (2002), pp. 132-142
16. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, G. Gay, Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Trans. Inf. Sci. (TOIS) 25(2) (2007), Article 7
17. D. Kelly, J. Teevan, Implicit feedback for inferring user preference: A bibliography. ACM SIGIR Forum 37(2), 18-28 (2003)
18. J. Kozielecki, Psychological Decision Theory (Kluwer, 1981)
19. D. Laming, Sensory Analysis (Academic, 1986)
20. Y. Liu, Y. Fu, M. Zhang, S. Ma, L. Ru, Automatic search engine performance evaluation with click-through data analysis, in Proceedings of the International World Wide Web Conference (WWW) (2007)
21. C.D. Manning, P. Raghavan, H. Schuetze, Introduction to Information Retrieval (Cambridge University Press, 2008)
22. F. Radlinski, T. Joachims, Query chains: Learning to rank from implicit feedback, in Proceedings of the ACM International Conference on Knowledge Discovery and Data Mining (KDD) (2005)
23. F. Radlinski, M. Kurup, T. Joachims, How does clickthrough data reflect retrieval quality, in Proceedings of the ACM Conference on Information and Knowledge Management (CIKM) (2008), pp. 43-52
24. S. Rajaram, A. Garg, Z.S. Zhou, T.S. Huang, Classification approach towards ranking and sorting problems, in Lecture Notes in Artificial Intelligence (2003), pp. 301-312
25. J. Reid, A task-oriented non-interactive evaluation methodology for information retrieval systems. Inf. Retr. 2, 115-129 (2000)
26. I. Soboroff, C. Nicholas, P. Cahan, Ranking retrieval systems without relevance judgments, in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2001), pp. 66-73
27. A. Spink, D. Wolfram, M. Bernard, J. Jansen, T. Saracevic, Searching to web: The public and their queries. J. Am. Soc. Inf. Sci. Technol. 52(3), 226-234 (2001)
28. A. Turpin, F. Scholer, User performance versus precision measures for simple search tasks, in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR) (2006), pp. 11-18
29. E.M. Voorhees, D.K. Harman (eds.), TREC: Experiment and Evaluation in Information Retrieval (MIT, 2005)
30. Y. Yue, T. Joachims, Interatively optimizing information systems as a dueling bandits problem, in NIPS 2008 Workshop on Beyond Search: Compuitations Intelligence for the Web (2008)
31. Y. Yue, T. Joachims, Interatively optimizing information retrieval systems as a dueling bandits problem, in Proceedings of the International Conference on Machine Learning (ICML) (2009)