Empirical justification of the gain and discount function for nDCG

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2009, Pages 611-620

  Kanoulas, Evangelos ^a,b   Aslam, Javed A ^b  

^a UNIVERSITY OF SHEFFIELD   (United Kingdom)

^b Northeastern University   (United States)

Author keywords

Evaluation; Generalizability theory; nDCG

Indexed keywords

DATA SETS; GAIN VALUES; OPTIMAL GAIN; OPTIMIZATION PROCEDURES; RELATIVE VALUE; RELEVANCE JUDGMENT; RETRIEVAL EFFECTIVENESS; VARIANCE DECOMPOSITION;

KNOWLEDGE MANAGEMENT; OPTIMIZATION; SYSTEM STABILITY;

FUNCTION EVALUATION;

EID: 74549195035     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1645953.1646032     Document Type: Conference Paper

References (14)

1. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between ir effectiveness measures and user satisfaction. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 773-774, New York, NY, USA, 2007. ACM.
2. J. A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In SIGIR '05: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 27-34, New York, NY, USA, 2005. ACM.
3. D. Banks, P. Over, and N.-F. Zhang. Blind men and elephants: Six approaches to trec data. Inf. Retr., 1(1-2):7-34, 1999.
4. D. Bodoff and P. Li. Test theory for assessing ir test collection. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 367-374, 2007.
5. R. L. Brennan. Generalizability Theory. Springer-Verlag, New York, 2001.
6. C. Buckley and E. M. Voorhees. Evaluating evaluation measure stability. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 33-40, 2000.
7. C. J. C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender. Learning to rank using gradient descent. In ICML '05: Proceedings of the 22nd international conference on Machine learning, pages 89-96, New York, NY, USA, 2005. ACM.
8. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan. Evaluation over thousands of queries. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 651-658, New York, NY, USA, 2008. ACM.
9. K. Järvelin and J. Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 41-48, New York, NY, USA, 2000. ACM.
10. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422-446, 2002.
11. J. Kekäläinen. Binary and graded relevance in ir evaluations: comparison of the effects on ranking of ir systems. Inf. Process. Manage., 41(5):1019-1033, 2005.
12. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 525-532, New York, NY, USA, 2006. ACM.
13. T. Sakai. On penalising late arrival of relevant documents in information retrieval evaluation with graded relevance. In First International Workshop on Evaluating Information Access (EVIA 2007), pages 32-43, 2007.
14. E. M. Voorhees. Evaluation by highly relevant documents. In SIGIR '01: Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74-82, New York, NY, USA, 2001. ACM.