Preference based evaluation measures for novelty and diversity

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 413-422

  Chandar, Praveen ^a   Carterette, Ben ^a  

^a University of Delaware   (United States)

Author keywords

Evaluation; Novelty and diversity

Indexed keywords

DOCUMENT RANKING; EVALUATION; EVALUATION FRAMEWORK; EVALUATION MEASURES; EXPECTED UTILITY; NOVELTY AND DIVERSITY; RELEVANCE ASSESSMENTS; USER'S PREFERENCES;

INFORMATION RETRIEVAL; SEARCH ENGINES;

INFORMATION SCIENCE;

EID: 84883094439     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484094     Document Type: Conference Paper

References (28)

1. R. Agrawal, S. Gollapudi, A. Halverson, and S. Ieong. Diversifying search results. Proceedings of WSDM '09, page 5, 2009.
2. J. Arguello, F. Diaz, and J. Callan. Learning to aggregate vertical results into web search results. In Proceedings of CIKM '11, page 201, New York, USA, 2011. ACM Press.
3. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of SIGIR '04, page 25, New York, USA, 2004. ACM Press.
4. J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. Proceedings of SIGIR '98, pages 335-336, 1998.
5. B. Carterette. An analysis of np-completeness in novelty and diversity ranking. Information Retrieval, 14(1):89-106, Dec. 2010.
6. B. Carterette. System effectiveness, user models, and user utility. In Proceedings of SIGIR '11, page 903, New York, USA, 2011. ACM Press.
7. B. Carterette, P. N. Bennett, D. M. Chickering, and T. Susan. Here or there preference judgments for relevance. In Proceedings of ECIR '08, pages 16-27, 2008.
8. P. Chandar and B. Carterette. Using preference judgments for novel document retrieval. Proceedings of SIGIR '12, page 861, 2012.
9. O. Chapelle, S. Ji, C. Liao, E. Velipasaoglu, L. Lai, and S.-L. Wu. Intent-based diversification of web search results: metrics and algorithms. Information Retrieval, 14(6):572-592, May 2011.
10. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. Proceeding of CIKM '09, page 621, 2009.
11. C. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of WSDM '11, pages 75-84. ACM, 2011.
12. C. Clarke, M. Kolla, and O. Vechtomova. An effectiveness measure for ambiguous and underspecified queries. Advances in Information Retrieval Theory, pages 188-199, 2010.
13. C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. Proceedings of SIGIR '08, page 659, 2008.
14. C. L. A. Clarke, N. Craswell, I. Soboroff, and E. M. Voorhees. Overview of the trec 2011 web track. In Proceedings of The Eighteenth Text REtrieval Conference TREC, pages 1-9, Gaithersburg, Maryland, 2011. NIST.
15. T. Leelanupab, G. Zuccon, and J. M. Jose. A query-basis approach to parametrizing novelty-biased cumulative gain. In Proceedings of the Third international conference on Advances in information retrieval theory, ICTIR '11, pages 327-331. Springer-Verlag, 2011.
16. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1):1-27, Dec. 2008.
17. P. Over. Trec-6 interactive track report. In The Sixth Text Retrieval Conference (TREC-6), pages 57-64, 1998.
18. F. Radlinski, M. Kurup, and T. Joachims. How does clickthrough data reflect retrieval quality? In Proceedings of CIKM '08, pages 43-52, New York, USA, 2008. ACM.
19. D. Rafiei, K. Bharat, and A. Shukla. Diversifying web search results. In Proceedings of WWW O10, pages 781-790, New York, USA, Apr. 2010. ACM.
20. M. E. Rorvig. The simple scalability of documents. Journal of the American Society for Information Science, 41(8):590-598, Dec. 1990.
21. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of SIGIR '06, pages 525-532, New York, USA, 2006. ACM.
22. T. Sakai, N. Craswell, R. Song, S. Robertson, Z. Dou, and C. Lin. Simple evaluation metrics for diversified search results. In Proceedings of the 3rd International Workshop on Evaluating Information Access (EVIA), 2010.
23. T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. In Proceedings of SIGIR '11, pages 1043-1052. ACM, 2011.
24. M. Sanderson, M. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proceeding of SIGIR '10, pages 555-562. ACM, 2010.
25. R. L. Santos, C. Macdonald, and I. Ounis. Exploiting query reformulations for web search result diversification. Proceedings of WWW '10, page 881, 2010.
26. E. M. Voorhees and H. T. Dang. Overview of the trec 2005 question answering track. In TREC 2005, 1999.
27. C. Zhai, W. Cohen, and J. Lafferty. Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In Proceedings of SIGIR '03, pages 10-17. ACM, 2003.
28. W. Zheng, X. Wang, H. Fang, and H. Cheng. Coverage-based search result diversification. Information Retrieval, 2011.