Extending average precision to graded relevance judgments

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 603-610

  Robertson, Stephen E ^a   Kanoulas, Evangelos ^a,b   Yilmaz, Emine ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

Author keywords

Average precision; Effectiveness metrics; Graded relevance; Information retrieval; Learning to rank

Indexed keywords

AVERAGE PRECISION; COMPARATIVE EVALUATIONS; EFFECTIVENESS METRICS; EVALUATION METRICS; GRADED RELEVANCE; INFORMATIVENESS; LEARNING TO RANK; OBJECTIVE FUNCTIONS; PROBABILISTIC INTERPRETATION; RANKING FUNCTIONS; RELEVANCE JUDGMENT; RETRIEVAL EFFECTIVENESS; RETRIEVAL SYSTEMS; TEST METRICS; USER MODELS;

OPTIMIZATION;

INFORMATION RETRIEVAL;

EID: 77956029237     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835550     Document Type: Conference Paper

References (27)

1. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between ir effectiveness measures and user satisfaction. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 773-774, New York, NY, USA, 2007. ACM.
2. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In G. Marchionini, A. Moffat, J. Tait, R. Baeza-Yates, and N. Ziviani, editors, Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 27-34. ACM Press, August 2005.
3. P. Bailey, N. Craswell, A. P. de Vries, I. Soboroff, and P. Thomas. Overview of the trec 2008 enterprise track. In Proceedings of the Seventeenth Text REtrieval Conference (TREC 2008), 2008.
4. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 667-674, New York, NY, USA, 2008. ACM.
5. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05: Proceedings of the 22nd international conference on Machine learning, pages 89-96, New York, NY, USA, 2005. ACM Press.
6. C. J. C. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. In B. Schölkopf, J. C. Platt, T. Hoffman, B. Schölkopf, J. C. Platt, and T. Hoffman, editors, NIPS, pages 193-200. MIT Press, 2006.
7. P. Donmez, K. M. Svore, and C. J. Burges. On the local optimality of lambdarank. In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 460-467, New York, NY, USA, 2009. ACM.
8. K. Järvelin and J. Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 41-48, New York, NY, USA, 2000. ACM Press.
9. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems, 20(4):422-446, 2002.
10. T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay. Accurately interpreting clickthrough data as implicit feedback. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 154-161, New York, NY, USA, 2005. ACM.
11. E. Kanoulas and J. A. Aslam. Empirical justification of the gain and discount function for ndcg. In To appear in CIKM '09: Proceedings of the 18th ACM international conference on Information and knowledge management, 2009.
12. J. Kekäläinen. Binary and graded relevance in ir evaluations: comparison of the effects on ranking of ir systems. Inf. Process. Manage., 41(5):1019-1033, 2005.
13. J. Kekäläinen and K. Järvelin. Using graded relevance assessments in ir evaluation. J. Am. Soc. Inf. Sci. Technol., 53(13):1120-1129, 2002.
14. T. Minka, J. Winn, J. Guiver, and A. Kannan. Infer.net user guide: Tutorials and examples.
15. M. S. Pollock. Measures for the comparison of information retrieval systems. American Documentation, 19(4):387-397, 1968.
16. S. Robertson. A new interpretation of average precision. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 689-690, New York, NY, USA, 2008. ACM.
17. T. Sakai. Ranking the NTCIR Systems Based on Multigrade Relevance, volume 3411/2005 of Lecture Notes in Computer Science. Springer Berlin / Heidelberg, February 2005.
18. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 525-532, New York, NY, USA, 2006. ACM.
19. T. Sakai. On penalising late arrival of relevant documents in information retrieval evaluation with graded relevance. In First International Workshop on Evaluating Information Access (EVIA 2007), pages 32-43, 2007.
20. T. Sakai and S. Robertson. Modelling a user population for designing information retrieval metrics. In The Second International Workshop on Evaluating Information Access (EVIA 2008) (NTCIR-7 workshop) Tokyo, December 2008, 2008.
21. J. X. Tao Qin, Tie-Yan Liu and H. Li. Letor: A benchmark collection for research on learning to rank for information retrieval. Information Retrieval Journal, 2010.
22. M. Taylor, J. Guiver, S. E. Robertson, and T. Minka. Softrank: optimizing non-smooth rank metrics. In WSDM '08: Proceedings of the international conference on Web search and web data mining, pages 77-86, New York, NY, USA, 2008. ACM.
23. E. M. Voorhees. Evaluation by highly relevant documents. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74-82, New York, NY, USA, 2001. ACM.
24. J. Xu and H. Li. Adarank: a boosting algorithm for information retrieval. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 391-398, New York, NY, USA, 2007. ACM.
25. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In P. S. Yu, V. Tsotras, E. Fox, and B. Liu, editors, Proceedings of the Fifteenth ACM International Conference on Information and Knowledge Management, pages 102-111. ACM Press, November 2006.
26. E. Yilmaz and S. Robertson. Deep versus shallow judgments in learning to rank. In SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 662-663, New York, NY, USA, 2009. ACM.
27. Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, USA, 2007. ACM Press.