Summaries, ranked retrieval and sessions: A unified framework for information access evaluation

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 473-482

  Sakai, Tetsuya ^a   Dou, Zhicheng ^a  

^a MICROSOFT RESEARCH ASIA   (China)

Author keywords

Diversity; Evaluation; Metrics; Sessions; Test collections

Indexed keywords

DIVERSITY; EVALUATION; METRICS; SESSIONS; TEST COLLECTION;

BEHAVIORAL RESEARCH;

INFORMATION RETRIEVAL;

EID: 84883092343     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484031     Document Type: Conference Paper

References (29)

1. R. Agrawal, G. Sreenivas, A. Halverson, and S. Leong. Diversifying search results. In Proceedings of ACM WSDM 2009, pages 5-14, 2009.
2. J. Allan, B. Carterette, and J. Lewis. When will information retrieval be "good enough,"? In Proceedings of ACM SIGIR 2005, pages 433-440, 2005.
3. P. Arvola, J. Kekäläinen, and M. Junkkari. Expected reading effort in focused retrieval evaluation. Information Retrieval, 13(5):460-484, 2010.
4. L. Azzopardi. Usage based effectiveness measures. In Proceedings of ACM CIKM 2009, pages 631-640, 2009.
5. F. Baskaya, H. Keskustalo, and K. Järvelin. Time drives interaction: Simulating sessions in diverse searching environments. In Proceedings of ACM SIGIR 2012, pages 105-114, 2012.
6. B. Carterette. System effectiveness, user models, and user utility: A conceptual framework for investigation. In Proceedings of ACM SIGIR 2011, pages 903-912, 2011.
7. B. Carterette. Multiple testing in statistical analysis of systems-based information retrieval experiments. ACM TOIS, 30(1), 2012.
8. B. Carterette, E. Kanoulas, and E. Yilmaz. Simulating simple user behavior for system effectiveness evaluation. In Proceedings of ACM CIKM 2011, pages 611-620, 2011.
9. O. Chapelle, S. Ji, C. Liao, E. Velipasaoglu, L. Lai, and S.-L. Wu. Intent-based diversification of web search results: Metrics and algorithms. Information Retrieval, 14(6):572-592, 2011.
10. C. L. A. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of ACM WSDM 2011, pages 75-84, 2011.
11. C. L. A. Clarke, N. Craswell, I. Soboroff, and E. M. Voorhees. Overview of the TREC 2011 web track. In Proceedings of TREC 2011, 2012.
12. G. Dupret and B. Piwowarski. A user behavior model for average precision and its generalization to graded judgments. In Proceedings of ACM SIGIR 2010, pages 531-538, 2010.
13. T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM TOIS, 25(2), 2007.
14. E. Kanoulas, B. Carterette, P. D. Clough, and M. Sanderson. Evaluating multi-query sessions. In Proceedings of ACM SIGIR 2011, pages 1053-1062, 2011.
15. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Proceedings of the ACL 2004 Workshop on Text Summarization Branches Out, 2004.
16. A. Nenkova, R. Passonneau, and K. McKeown. The pyramid method: Incorporating human content selection variation in summarization evaluation. ACM Transactions on Speech and Language Processing, 4(2): Article 4, 2007.
17. S. Rajput, M. Ekstrand-Abueg, V. Pavlu, and J. Aslam. Constructing test collections by inferring document relevance via extracted relevant information. In Proceedings of ACM CIKM 2012, pages 145-154, 2012.
18. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM SIGIR 2006, pages 525-532, 2006.
19. T. Sakai, M. P. Kato, and Y.-I. Song. Click the search button and be happy: Evaluating direct and immediate information access. In Proceedings of ACM CIKM 2011, pages 621-630, 2011.
20. T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. In Proceedings of ACM SIGIR 2011, pages 1043-1042, 2011.
21. T. Sakai and R. Song. Diversified search evaluation: Lessons from the NTCIR-9 INTENT task. Information Retrieval, 2013.
22. M. D. Smucker and C. L. A. Clarke. Modeling user variance in time-biased gain. In Proceedings of HCIR 2012, 2012.
23. M. D. Smucker and C. L. A. Clarke. Time-based calibration of effectiveness measures. In Proceedings of ACM SIGIR 2012, pages 95-104, 2012.
24. A. Turpin, F. Scholer, K. Järvelin, M. Wu, and J. S. Culpepper. Including summaries in system evaluation. In Proceedings of ACM SIGIR 2009, pages 508-515, 2009.
25. E. M. Voorhees. Overview of the TREC 2005 robust retrieval track. In Proceeings of TREC 2005, 2006.
26. Y. Yang and A. Lad. Modeling expected utility of multi-session information distillation. In Proceedings of ICTIR 2009 (LNCS 5766), pages 164-175, 2009.
27. E. Yilmaz, J. Aslam, and S. Robertson. A new rank correlation coefficient for information retrieval. In Proceedings of ACM SIGIR 2008, pages 587-594, 2008.
28. E. Yilmaz, M. Shokouhi, N. Craswell, and S. Robertson. Expected browsing utility for web search evaluation. In Proceedings of ACM CIKM 2010, pages 1561-1564, 2010.
29. K. Zhou, R. Cummins, M. Lalmas, and J. M. Jose. Evaluating aggregated search pages. In Proceedings of ACM SIGIR 2012, pages 115-124, 2012.