How does clickthrough data reflect retrieval quality?

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2008, Pages 43-52

  Radlinski, Filip ^a   Kurup, Madhu ^a   Joachims, Thorsten ^a  

^a Department of Computer Science and School of Operations Research and Information Engineering   (United States)

Author keywords

Clickthrough data; Expert judgments; Implicit feedback; Retrieval evaluation

Indexed keywords

CLICKTHROUGH DATA; E-PRINTS; EVALUATION RESULTS; EXPERIMENT DESIGN; EXPERT JUDGMENTS; IMPLICIT FEEDBACK; OPERATIONAL SEARCH; PAIRED COMPARISON; QUERY REFORMULATION; RETRIEVAL EVALUATION; RETRIEVAL QUALITY; SAMPLE SIZES; SENSORY ANALYSIS; USER BEHAVIORS; USER-CENTERED;

IMAGE RETRIEVAL; KNOWLEDGE MANAGEMENT; SEARCH ENGINES;

BEHAVIORAL RESEARCH;

EID: 67650085898     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1458082.1458092     Document Type: Conference Paper

References (23)

1. E. Agichtein, E. Brill, S. Dumais, and R. Ragno. Learning user interaction models for prediction web search results preferences. In Proc. of SIGIR, 2006.
2. K. Ali and C. Chang. On the relationship between click-rate and relevance for search engines. In Proc. of Data-Mining and Information Engineering, 2006.
3. J. A. Aslam, V. Pavlu, and E. Yilmaz. A sampling technique for efficiently estimating measures of query retrieval performance using incomplete judgments. In ICML Workshop on Learning with Partially Classified Training Data, 2005.
4. J. Boyan, D. Freitag, and T. Joachims. A machine learning architecture for optimizing web search engines. In AAAI Workshop on Internet Based Information Systems, 1996.
5. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proc. of SIGIR, 2004.
6. B. Carterette, J. Allan, and R. Sitaraman. Minimal test collections for retrieval evaluation. In Proc. of SIGIR, 2006.
7. B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there: Preference judgements for relevance. In Proc. of ECIR, 2008.
8. B. Carterette and R. Jones. Evaluating search engines by modeling the relationship between relevance and clicks. In Proc. of NIPS, 2007.
9. G. Dupret, V. Murdock, and B. Piwowarski. Web search engine evaluation using clickthrough data and a user model. In WWW Workshop on Query Log Analysis, 2007.
10. S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Transactions on Information Science (TOIS), 23(2):147-168, April 2005.
11. S. B. Human and M. Hochster. How well does result relevance predict session satisfaction? In Proc. of SIGIR, 2007.
12. T. Joachims. Optimizing search engines using clickthrough data. In Proc. of KDD, 2002.
13. T. Joachims. Evaluating retrieval performance using clickthrough data. In J. Franke, G. Nakhaeizadeh, and I. Renz, editors, Text Mining. Physica Verlag, 2003.
14. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Science (TOIS), 25 (2), 2007. Article 7.
15. D. Kelly and J. Teevan. Implicit feedback for inferring user preference: A bibliography. ACM SIGIR Forum, 37(2):18-28, 2003.
16. J. Kozielecki. Psychological Decision Theory. Kluwer, 1981.
17. D. Laming. Sensory Analysis. Academic Press, 1986.
18. Y. Liu, Y. Fu, M. Zhang, S. Ma, and L. Ru. Automatic search engine performance evaluation with click-through data analysis. In Proc. of WWW, 2007.
19. C. D. Manning, P. Raghavan, and H. Schuetze. Introduction to Information Retrieval. Cambridge University Press, 2008.
20. J. Reid. A task-oriented non-interactive evaluation methodology for information retrieval systems. Information Retrieval, 2:115-129, 2000.
21. I. Soboro, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proc. of SIGIR, 2001.
22. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proc. of SIGIR, 2006.
23. E. M. Voorhees and D. K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005