Evaluating diversified search results using per-intent graded relevance

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2011, Pages 1043-1052

  Sakai, Tetsuya ^a   Song, Ruihua ^a  

^a MICROSOFT RESEARCH ASIA   (China)

Author keywords

Ambiguity; Diversity; Evaluation; Graded relevance; Test collection

Indexed keywords

AMBIGUITY; DIVERSITY; EVALUATION; GRADED RELEVANCES; TEST COLLECTION;

EID: 80052111133     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2009916.2010055     Document Type: Conference Paper

References (31)

1. R. Agrawal, S. Gollapudi, A. Halverson, and S. Leong. Diversifying search results. In Proceedings of ACM WSDM 2009, pages 5-14, 2009.
2. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In Proceedings of ACM SIGIR 2005, pages 27-34, 2005.
3. C. Brandt, T. Joachims, Y. Yue, and J. Bank. Dynamic ranked retrieval. In Proceedings of WSDM 2011, pages 247-256, 2011.
4. B. Carterette and P. Chandar. Probabilistic models of novel document rankings for faceted topic retrieval. In Proceedings of ACM CIKM 2009, pages 1287-1296, 2009.
5. O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of ACM CIKM 2009, pages 621-630, 2009.
6. C. L. Clarke, N. Craswell, and I. Soboroff. Overview of the TREC 2009 web track. In Proceedings of TREC 2009, 2010.
7. C. L. Clarke, N. Craswell, I. Soboroff, and A. Ashkan. A comparative analysis of cascade measures for novelty and diversity. In Proceedings of ACM WSDM 2011, 2011.
8. C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In Proceedings of ACM SIGIR 2008, pages 659-666, 2009.
9. C. L. Clarke, M. Kolla, and O. Vechtomova. An effectiveness measure for ambiguous and underspecified queries. In Advances in Information Retrieval Theory (ICTIR 2009), LNCS 5766, pages 188-199, 2009.
10. Z. Dou, S. Hu, K. Chen, R. Song, and J.-R. Wen. Multi-dimensional search result diversification. In Proceedings of ACM WSDM 2011, 2011.
11. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002. (Pubitemid 44642296)
12. K. Kishida, K. hua Chen, S. Lee, K. Kuriyama, N. Kando, and H.-H. Chen. Overview of CLIR task at the sixth NTCIR workshop. In Proceedings of NTCIR-6, 2007.
13. A. Moffat and J. Zobel. Rank-biased precision for measurement of retrieval effectiveness. ACM Transactions on Information Systems, 27(1):Article No.2, 2008.
14. F. Radlinski and N. Craswell. Comparing the sensitivity of information retrieval metrics. In Proceedings of ACM SIGIR 2010, pages 667-674, 2010.
15. D. Rafiei, K. Bharat, and A. Shukla. Diversifying web search results. In Proceedings of ACM WWW 2010, pages 781-790, 2010.
16. S. E. Robertson. The probability ranking principle in IR. Journal of Documentation, 33:130-137, 1977.
17. S. E. Robertson, E. Kanoulas, and E. Yilmaz. Extending average precision to graded relevance judgments. In Proceedings of ACM SIGIR 2010, pages 603-610, 2010.
18. T. Sakai. Bootstrap-based comparisons of IR metrics for finding one relevant document. In Proceedings of AIRS 2006 (LNCS 4182), pages 374-389, 2006. (Pubitemid 44812213)
19. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of ACM SIGIR 2006, pages 525-532, 2006.
20. T. Sakai, N. Craswell, R. Song, S. Robertson, Z. Dou, and C.-Y. Lin. Simple evaluation metrics for diversified search results. In Proceedings of EVIA 2010, pages 42-50, 2010.
21. T. Sakai and S. Robertson. Modelling a user population for designing information retrieval metrics. In Proceedings of EVIA 2008, pages 30-41, 2008.
22. M. Sanderson. Test collection based evaluation of information retrieval systems. Foundations and Trends in Information Retrieval, 4:247-375, 2010.
23. M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proceedings of ACM SIGIR 2010, pages 555-562, 2010.
24. R. L. Santos, C. Macdonald, and I. Ounis. Selectively diversifying search results. In Proceedings of ACM CIKM 2010, 2010.
25. I. Soboroff. Test collection diagnosis and treatment. In Proceedings of EVIA 2010, pages 34-41, 2010.
26. R. Song, D. Qi, H. Liu, T. Sakai, J.-Y. Nie, H.-W. Hon, and Y. Yu. Constructing a test collection with multi-intent queries. In Proceedings of EVIA 2010, pages 51-59, 2010.
27. E. Sormunen. Liberal relevance criteria of TREC: Counting on negligible documents? In Proceedings of ACM SIGIR 2002, pages 324-330, 2002.
28. E. M. Voorhees and C. Buckley. The effect of topic set size on retrieval experiment error. In Proceedings of ACM SIGIR 2002, pages 316-323, 2002.
29. W. Webber, A. Moffat, and J. Zobel. The effect of pooling and evaluation depth on metric stability. In Proceedings of EVIA 2010, pages 7-15, 2010.
30. E. Yilmaz, J. A. Aslam, and S. Robertson. A new rank correlation coefficient for information retrieval. In Proceedings of ACM SIGIR 2008, pages 587-594, 2008.
31. C. Zhai, W. W. Cohen, and J. Lafferty. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. In Proceedings of ACM SIGIR 2003, pages 10-17, 2003.