The fault, dear researchers, is not in cranfield, but in our metrics, that they are unrealistic

CEUR Workshop Proceedings

Volumn 909, Issue , 2012, Pages 11-12

  Smucker, Mark D ^a   Clarke, Charles L A ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84883064432     PISSN: 16130073     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. L. Azzopardi. Usage based effectiveness measures: monitoring application performance in information retrieval. CIKM, pages 631-640, 2009.
2. L. Azzopardi, K. J̈arvelin, J. Kamps, and M. D. Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum, 44:35-47, January 2011.
3. J. Banks, J. S. Carson II, B. L. Nelson, and D. M. Nicol. Discrete-Event System Simulation. Prentice Hall, 5th edition, 2010.
4. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In CIKM, pages 611-620, 2011.
5. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM, pages 621-630, Hong Kong, 2009.
6. M. D. Dunlop. Time, relevance and interaction modelling for information retrieval. In SIGIR, pp. 206-213. 1997.
7. G. Dupret. Discounted cumulative gain and user decision models. In Proceedings of the 18th international conference on String processing and information retrieval, SPIRE'11, pages 2-13, Berlin, Heidelberg, 2011. Springer-Verlag.
8. D. Harman. Information Retrieval Evaluation. Morgan & Claypool, 2011.
9. W. Hersh, A. Turpin, S. Price, B. Chan, D. Kramer, L. Sacherek, and D. Olson. Do batch and user evaluations give the same results? In SIGIR, pages 17-24. ACM, 2000.
10. M. Y. Ivory and M. A. Hearst. The state of the art in automating usability evaluation of user interfaces. ACM Computing Surveys, 33(4):470-516, 2001.
11. K. J̈arvelin and J. Kek̈al̈ainen. Cumulated gain-based evaluation of IR techniques. TOIS, 20(4):422-446, 2002.
12. D. Kelly. Methods for Evaluating Interactive Information Retrieval Systems with Users, volume 3. Foundations and Trends in Information Retrieval, 2009.
13. H. Keskustalo, K. J̈arvelin, T. Sharma, and M. L. Nielsen. Test collection-based IR evaluation needs extension toward sessions: A case of extremely short queries. In AIRS, pp. 63-74, 2009.
14. R. Khan, D. Mease, and R. Patel. The impact of result abstracts on task completion time. In Workshop on Web Search Result Summarization and Presentation, WWW'09, 2009.
15. J. Lin and M. D. Smucker. How do users find things with PubMed? Towards automatic utility evaluation with user simulations. In SIGIR'08, pages 19-26. ACM, 2008.
16. M. D. Smucker and C. L. A. Clarke. Time-based calibration of effectiveness measures. In SIGIR, 10 pages, 2012.
17. A. Turpin, F. Scholer, K. J̈arvelin, M. Wu, and J. S. Culpepper. Including summaries in system evaluation. In SIGIR'09, pages 508-515. ACM, 2009.
18. E. M. Voorhees. I come not to bury Cranfield, but to praise it. In HCIR'09, pages 13-16, 2009.
19. E. M. Voorhees and D. K. Harman, editors. TREC. MIT Press, 2005.
20. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In CIKM, pages 1561-1564, Toronto, 2010.
21. Y. Zhang, L. A. Park, and A. Moffat. Click-based evidence for decaying weight distributions in search effectiveness metrics. Information Retrieval, 13:46-69, February 2010.