Models and metrics: IR evaluation as a user process

Proceedings of the 17th Australasian Document Computing Symposium, ADCS 2012

Volumn , Issue , 2012, Pages 47-54

  Moffat, Alistair ^a   Scholer, Falk ^b   Thomas, Paul ^c  

^a UNIVERSITY OF MELBOURNE   (Australia)

^b RMIT UNIVERSITY   (Australia)

^c CSIRO   (Australia)

Author keywords

Evaluation; Retrieval experiment; System measurement

Indexed keywords

EFFECTIVENESS METRICS; EVALUATION; EVALUATION APPROACH; INFORMATION SEEKING; RELEVANCE JUDGMENT; RETRIEVAL SYSTEMS; SEARCH BEHAVIOR; SEARCH TASKS; SYSTEM MEASUREMENT; TASK PERFORMANCE; USER MODELS;

BEHAVIORAL RESEARCH; EXPERIMENTS;

SEARCH ENGINES;

EID: 84871629959     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2407085.2407092     Document Type: Conference Paper

References (16)

1. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. In Proc. SIGIR, pages 773-774, Amsterdam, July 2007.
2. A. Ashkan and C. L. A. Clarke. On the informativeness of cascade and intent-aware effectiveness measures. In Proc. WWW, pages 407-416, Hyderabad, India, Apr. 2011.
3. B. Carterette. System effectiveness, user models, and user utility: A conceptual framework for investigation. In Proc. SIGIR, pages 903-912, Beijing, China, 2011.
4. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In Proc. CIKM, pages 611-620, Glasgow, Scotland, 2011.
5. O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proc. CIKM, pages 621-630, Hong Kong, China, 2009.
6. C. L. A. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proc. WSDM 2011, pages 75-84, Hong Kong, China, 2011.
7. S. B. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In Proc. SIGIR, pages 567-574, Amsterdam, July 2007.
8. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Trans. Information Systems, 20(4):422-446, 2002.
9. E. Kanoulas, B. Carterette, P. D. Clough, and M. Sanderson. Evaluating multi-query sessions. In Proc. SIGIR, pages 1053-1062, Beijing, China, July 2011.
10. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Information Systems, 27(1):2:1-2:27, Dec. 2008.
11. S. Robertson. A new interpretation of average precision. In Proc. SIGIR, pages 689-690, Singapore, July 2008.
12. M. D. Smucker and C. L. A. Clarke. Time-based calibration of effectiveness measures. In Proc. SIGIR, pages 95-104, Portland, Oregon, Aug. 2012.
13. P. Thomas, T. Jones, and D. Hawking. What deliberately degrading search quality tells us about discount functions. In Proc. SIGIR, pages 1107-1108, Beijing, July 2011.
14. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proc. SIGIR, pages 11-18, Seattle, Washington, Aug. 2006.
15. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In Proc. CIKM, pages 1561-1564, Toronto, Canada, 2010.
16. Y. Zhang, L. A. F. Park, and A. Moffat. Click-based evidence for decaying weight distributions in search effectiveness metrics. Information Retrieval, 13(1), Feb. 2010.