Estimating interleaved comparison outcomes from historical click data

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 1779-1783

  Hofmann, Katja ^a   Whiteson, Shimon ^a   De Rijke, Maarten ^a  

^a UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

a b testing; evaluation; implicit feedback; information retrieval; interleaved comparisons; reusability

Indexed keywords

COMPARISON METHODS; DATA REUSE; EVALUATION; HISTORICAL DATA; IMPLICIT FEEDBACK; INTERLEAVED COMPARISONS; PROBABILISTIC APPROACHES; PROBABILISTIC METHODS; RETRIEVAL EVALUATION;

KNOWLEDGE MANAGEMENT; REUSABILITY;

INFORMATION RETRIEVAL;

EID: 84871103071     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2398516     Document Type: Conference Paper

References (21)

1. B. Carterette. Robust test collections for retrieval evaluation. In SIGIR '07, pages 55-62, 2007.
2. B. Carterette and R. Jones. Evaluating search engines by modeling the relationship between relevance and clicks. In NIPS '07, volume 20, pages 217-224, 2008.
3. G. Dupret and C. Liao. A model to estimate intrinsic document relevance from the clickthrough logs of a web search engine. In WSDM '10, pages 181-190, 2010.
4. G. Dupret, V. Murdoch, and B. Piwowarski. Web search engine evaluation using click-through data and a user model. In Proceedings of the Workshop on Query Log Analysis, 2007.
5. F. Guo, C. Liu, and Y. M. Wang. Efficient multiple-click models in web search. In WSDM '09, pages 124-131, 2009.
6. J. He, C. Zhai, and X. Li. Evaluation of methods for relative comparison of retrieval systems based on clickthroughs. In CIKM '09, pages 2029-2032, 2009.
7. K. Hofmann, B. Huurnink, M. Bron, and M. de Rijke. Comparing click-through data to purchase decisions for retrieval evaluation. In SIGIR '10, pages 761-762, 2010.
8. K. Hofmann, S. Whiteson, and M. de Rijke. A probabilistic method for inferring preferences from clicks. In CIKM '11, pages 249-258, 2011.
9. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4): 422-446, October 2002.
10. S. Ji, K. Zhou, C. Liao, Z. Zheng, G.-R. Xue, O. Chapelle, G. Sun, and H. Zha. Global ranking by exploiting user clicks. In SIGIR '09, pages 35-42, 2009.
11. T. Joachims. Optimizing search engines using clickthrough data. In KDD '02, pages 133-142, 2002.
12. S. Jung, J. L. Herlocker, and J. Webster. Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3):791-807, 2007.
13. J. Kamps, M. Koolen, and A. Trotman. Comparative analysis of clicks and judgments for IR evaluation. In WSCD'09, pages 80-87, 2009.
14. L. Li, W. Chu, J. Langford, and X. Wang. Unbiased offline evaluation of contextual-bandit-based news article recommendation algorithms. In WSDM '11, pages 297-306, 2011.
15. U. Ozertem, R. Jones, and B. Dumoulin. Evaluating new search engine configurations with pre-existing judgments and clicks. In WWW'11, pages 397-406, 2011.
16. D. Precup, R. Sutton, and S. Singh. Eligibility traces for off-policy policy evaluation. In ICML'00, pages 759-766, 2000.
17. F. Radlinski and N. Craswell. Comparing the sensitivity of information retrieval metrics. In SIGIR '10, pages 667-674, 2010.
18. F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In CIKM '08, pages 43-52, 2008.
19. A. M. Strehl, J. Langford, L. Li, and S. M. Kakade. Learning from logged implicit exploration data. In NIPS 2010, pages 2217-2225, 2010.
20. R. S. Sutton and A. G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1998.
21. J. Zhang and J. Kamps. A search log-based approach to evaluation. In ECDL'10, pages 248-260, 2010.