Time-based calibration of effectiveness measures

SIGIR'12 - Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2012, Pages 95-104

  Smucker, Mark D ^a   Clarke, Charles L A ^a  

^a UNIVERSITY OF WATERLOO   (Canada)

Author keywords

information retrieval; search evaluation

Indexed keywords

DOCUMENT LENGTH; EFFECTIVENESS MEASURE; SEARCH EVALUATION; SEARCH PROCESS; SIMPLIFYING ASSUMPTIONS; TEST COLLECTION; USER BEHAVIORS; USER ORIENTED; USER STUDY;

INFORMATION RETRIEVAL;

BEHAVIORAL RESEARCH;

EID: 84866603223     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2348283.2348300     Document Type: Conference Paper

References (43)

1. I. J. Aalbersberg. Incremental relevance feedback. In SIGIR, pp. 11-22. 1992.
2. P. Arvola, J. Kekäläinen, and M. Junkkari. Expected reading effort in focused retrieval evaluation. Information Retrieval, 13:460-484, 2010.
3. L. Azzopardi. The economics in interactive information retrieval. In SIGIR, pp. 15-24, 2011.
4. L. Azzopardi, K. Järvelin, J. Kamps, and M. D. Smucker. Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum, 44:35-47, 2011.
5. A. Broder. On the resemblance and containment of documents. In Compression and Complexity of Sequences, pp. 21-29, 1997.
6. C. Buckley. Looking at limits and tradeoffs: Sabir Research at TREC 2005. In TREC. 2005.
7. B. Carterette. System effectiveness, user models, and user utility: A conceptual framework for investigation. In SIGIR, pp. 903-912, 2011.
8. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In CIKM, pp. 611-620, 2011.
9. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In CIKM, pp. 621-630, 2009.
10. C. L. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In WSDM, pp. 75-84, 2011.
11. W. S. Cooper. On selecting a measure of retrieval effectiveness. JASIS, 24(2):87-100, Mar/Apr 1973.
12. W. S. Cooper. On selecting a measure of retrieval effectiveness: Part II. implementation of the philosophy. JASIS, 24(6):413-424, Nov/Dec 1973.
13. M. D. Dunlop. Time, relevance and interaction modelling for information retrieval. In SIGIR, pp. 206-213. 1997.
14. G. Dupret. Discounted cumulative gain and user decision models. In SPIRE, pp. 2-13. 2011.
15. G. Dupret and B. Piwowarski. A user behavior model for average precision and its generalization to graded judgments. In SIGIR, pp. 531-538, 2010.
16. N. Fuhr. A probability ranking principle for interactive information retrieval. Information Retrieval, 11:251-265, 2008.
17. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM TOIS, 20(4):422-446, 2002.
18. D. Kelly. Methods for Evaluating Interactive Information Retrieval Systems with Users, now Publishers, 2009.
19. H. Keskustalo, K. Järvelin, T. Sharma, and M. L. Nielsen. Test collection-based IR evaluation needs extension toward sessions: A case of extremely short queries. In AIRS, pp. 63-74, 2009.
20. R. Khan, D. Mease, and R. Patel. The impact of result abstracts on task completion time. In Workshop on Web Search Result Summarization and Presentation, WWW'09.
21. J. Lin and M. D. Smucker. How do users find things with PubMed? Towards automatic utility evaluation with user simulations. In SIGIR, pp. 19-26. 2008.
22. L. Lorigo, M. Haridasan, H. Brynjarsdóttir, L. Xia, T. Joachims, G. Gay, L. Granka, F. Pellacini, and B. Pan. Eye tracking and online search: Lessons learned and challenges ahead. JASIS, 59(7):1041-1052, 2008.
23. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM TOIS, 27(1):1-27, 2008.
24. M. A. Najork, H. Zaragoza, and M. J. Taylor. HITS on the Web: How does it compare? In SIGIR, pp. 471-478, 2007.
25. S. Robertson. On GMAP: and other transformations. In CIKM, pp. 78-83, 2006.
26. S. Robertson. A new interpretation of average precision. In SIGIR, pp. 689-690, 2008.
27. S. E. Robertson, E. Kanoulas, and E. Yilmaz. Extending average precision to graded relevance judgments. In SIGIR, pp. 603-610, 2010.
28. S. E. Robertson and S. Walker. Some simple effective approximations to the 2-Poisson model for probabilistic weighted retrieval. In SIGIR, pp. 232-241, 1994.
29. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In SIGIR, pp. 525-532, 2006.
30. T. Sakai and S. Robertson. Modelling a user population for designing information retrieval metrics. In EVIA, pp. 30-41, 2008.
31. A. Singhal, G. Salton, M. Mitra, and C. Buckley. Document length normalization. IPM, 32(5):619-633, 1996.
32. T. Strohman, D. Metzler, H. Turtle, and W. B. Croft. Indri: A language-model based search engine for complex queries (extended version). Tech Report IR-407, CIIR, University of Massachusetts Amherst, 2005.
33. M. D. Smucker. An analysis of user strategies for examining and processing ranked lists of documents. In HCIR, 2011.
34. M. D. Smucker and C. Jethani. Human performance and retrieval precision revisited. In SIGIR, pp. 595-602, 2010.
35. L. T. Su. Evaluation measures for interactive information retrieval. IPM, 28:503-516, March 1992.
36. A. Tombros and M. Sanderson. Advantages of query biased summaries in information retrieval. In SIGIR, pp. 2-10, 1998.
37. A. Turpin, F. Scholer, K. Järvelin, M. Wu, and J. S. Culpepper. Including summaries in system evaluation. In SIGIR, pp. 508-515. 2009.
38. A. H. Turpin and W. Hersh. Why batch and user evaluations do not give the same results. In SIGIR, pp. 225-231, 2001.
39. E. M. Voorhees. Overview of the TREC 2005 Robust Retrieval Track. In TREC, 2005.
40. R. W. White, I. Ruthven, J. M. Jose, and C. J. van Rijsbergen. Evaluating implicit feedback models using searcher simulations. ACM TOIS, 23(3):325-361, 2005.
41. Y. Yang and A. Lad. Modeling expected utility of multi-session information distillation. In ICTIR, pp. 164-175, 2009.
42. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In CIKM, pp. 1561-1564, 2010.
43. Y. Zhang, L. A. Park, and A. Moffat. Click-based evidence for decaying weight distributions in search effectiveness metrics. Information Retrieval, 13:46-69, 2010.