Users versus models: What observation tells us about effectiveness metrics

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2013, Pages 659-668

  Moffat, Alistair ^a   Thomas, Paul ^b   Scholer, Falk ^c  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b CSIRO   (Australia)

^c RMIT UNIVERSITY   (Australia)

Author keywords

Evaluation; Retrieval experiment; System measurement

Indexed keywords

EFFECTIVENESS METRICS; EVALUATION; INFORMATION SEEKING; RELEVANCE JUDGMENT; RELEVANT DOCUMENTS; RETRIEVAL SYSTEMS; SYSTEM MEASUREMENT; TASK PERFORMANCE;

BEHAVIORAL RESEARCH; KNOWLEDGE MANAGEMENT;

SEARCH ENGINES;

EID: 84889574389     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2505515.2507665     Document Type: Conference Paper

References (27)

1. H. Akaike. A new look at the statistical model identification. IEEE Trans. Automatic Control, 19(6):716-723, 1974.
2. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. In Proc. SIGIR, pages 773-774, Amsterdam, The Netherlands, 2007.
3. C. Buckley and E. M. Voorhees. Retrieval system evaluation. In E. M. Voorhees and D. K. Harman, editors, TREC: Experiment and Evaluation in Information Retrieval, chapter 3, pages 53-75. MIT Press, Cambridge, Massachusetts, 2005.
4. K. P. Burnham and D. R. Anderson. Model selection and multimodel inference: A practical information-theoretic approach. Springer, New York, 2nd edition, 2005.
5. B. Carterette. System effectiveness, user models, and user utility: A conceptual framework for investigation. In Proc. SIGIR, pages 903-912, Beijing, China, 2011.
6. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In Proc. CIKM, pages 611-620, Glasgow, Scotland, 2011.
7. O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proc. CIKM, pages 621-630, Hong Kong, China, 2009.
8. O. Chapelle and Y. Zhang. A dynamic Bayesian network click model for web search ranking. In Proc. WWW, pages 1-10, Madrid, Spain, 2009.
9. N. Craswell, O. Zoeter, M. J. Taylor, and B. Ramsey. An experimental comparison of click position-bias models. In Proc. WSDM, pages 87-94, Palo Alto, CA, 2008.
10. G. Dupret. Discounted cumulative gain and user decision models. In Proc. SPIRE, pages 2-13, Pisa, Italy, 2011.
11. G. Dupret and B. Piwowarski. A user behavior model for average precision and its generalization to graded judgments. In Proc. SIGIR, pages 531-538, Geneva, Switzerland, 2010.
12. S. B. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In Proc. SIGIR, pages 567-574, Amsterdam, The Netherlands, 2007.
13. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Information Systems, 20(4):422-446, 2002.
14. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In Proc. SIGIR, pages 154-161, Salvador, Brazil, 2005.
15. T. Jones, D. Hawking, P. Thomas, and R. Sankaranarayana. Relative effect of spam and irrelevant documents on user interaction with search engines. In Proc. CIKM, pages 2113-2116, Glasgow, 2011.
16. A. Moffat. Seven numeric properties of effectiveness metrics. In Proc. AIRS, 2013. To appear.
17. A. Moffat, F. Scholer, and P. Thomas. Models and metrics: IR evaluation as a user process. In Proc. Australasian Document Computing Symp., pages 47-54, Dec. 2012.
18. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Information Systems, 27(1):2:1-2:27, 2008.
19. S. Robertson. A new interpretation of average precision. In Proc. SIGIR, pages 689-690, Singapore, 2008.
20. M. D. Smucker and C. L. A. Clarke. Stochastic simulation of time-biased gain. In Proc. CIKM, pages 2040-2044, Maui, Hawaii, 2012.
21. M. D. Smucker and C. L. A. Clarke. Time-based calibration of effectiveness measures. In Proc. SIGIR, pages 95-104, Portland, Oregon, 2012.
22. P. Thomas, T. Jones, and D. Hawking. What deliberately degrading search quality tells us about discount functions. In Proc. SIGIR, pages 1107-1108, Beijing, 2011.
23. P. Thomas, F. Scholer, and A. Moffat. What users do: The eyes have it. In Proc. AIRS, 2013. To appear.
24. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proc. SIGIR, pages 11-18, Seattle, Washington, 2006.
25. W.-C. Wu, D. Kelly, A. Edwards, and J. Arguello. Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. In Proc. 4th Information Interaction in Context Symp., pages 254-257, Nijmegen, The Netherlands, 2012.
26. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In Proc. CIKM, pages 1561-1564, Toronto, Canada, 2010.
27. Y. Zhang, L. A. F. Park, and A. Moffat. Click-based evidence for decaying weight distributions in search effectiveness metrics. Information Retrieval, 13(1):46-69, 2010.