A theoretical analysis of NDCG ranking measures

Journal of Machine Learning Research

Volumn 30, Issue , 2013, Pages 25-54

  Wang, Yining ^a   Wang, Liwei ^b   Li, Yuanzhi ^a   He, Di ^b   Chen, Wei ^c   Liu, Tie Yan ^c  

^a TSINGHUA UNIVERSITY   (China)

^b Peking University   (China)

^c MICROSOFT RESEARCH ASIA   (China)

Author keywords

Consistent Distinguishability; NDCG; Ranking; Ranking measures

Indexed keywords

ARTIFICIAL INTELLIGENCE; SOFTWARE ENGINEERING;

CENTRAL PROBLEMS; DISTINGUISHABILITY; EMPIRICAL STUDIES; LARGE DATASETS; NDCG; RANKING; RANKING FUNCTIONS; RANKING MEASURES;

INFORMATION RETRIEVAL;

EID: 84898043454     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Conference Paper

References (39)

1. S. Agarwal, T. Graepel, R. Herbrich, S. Har-Peled, and D. Roth. Generalization bounds for the area under an ROC curve. 2004.
2. A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between IR effectiveness measures and user satisfaction. In SIGIR, pages 773-774, 2007.
3. J.A. Aslam, E. Yilmaz, and V. Pavlu. The maximum entropy method for analyzing retrieval measures. In Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, pages 27-34. ACM, 2005.
4. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval, Volume 82. Addison-Wesley New York, 1999.
5. M.F. Balcan, N. Bansal, A. Beygelzimer, D. Coppersmith, J. Langford, and G.B. Sorkin. Robust reductions from ranking to classification. Machine learning, 72(1): 139-153, 2008.
6. P.L. Bartlett, M.I. Jordan, and J.D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473): 138-156, 2006.
7. D. Buffoni, C. Calauzenes, P. Gallinari, and N. Usunier. Learning scoring functions with order-preserving losses and standardized supervision. In Proceedings of the 28th International Conference on Machine Learning, 2011.
8. C.J.C. Burges, R. Ragno, and Q. V. Le. Learning to rank with nonsmooth cost functions. In Advances in Neural Information Processing Systems 19: Proceedings of the 2006 Conference, Volume 19, page 193. The MIT Press, 2007.
9. Clément Calauzènes, Nicolas Usunier, and Patrick Gallinari. On the (non-)existence of convex, calibrated surrogate losses for ranking. In P. Bartlett, F.C.N. Pereira, C.J.C. Burges, L. Bottou, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, pages 197-205. 2012.
10. Z. Cao, T. Qin, T.Y. Liu, M.F. Tsai, and H. Li. Learning to rank: from pairwise approach to listwise approach. In ICML, pages 129-136, 2007.
11. O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 621-630. ACM, 2009.
12. Stéphan J.M. Clémençon and Nicolas Vayatis. Empirical performance maximization for linear rank statistics. In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems 21, pages 305-312. 2009.
13. S. Clemençon, G. Lugosi, and N. Vayatis. Ranking and empirical minimization of U-statistics. The Annals of Statistics, 36(2): 844-874, 2008.
14. D. Cossock and T. Zhang. Statistical analysis of bayes optimal subset ranking. Information Theory, IEEE Transactions on, 54(11): 5140-5154, 2008.
15. K. Crammer and Y. Singer. Pranking with ranking. In Advances in Neural Information Processing Systems, 2002.
16. W.B. Croft, D. Metzler, and T. Strohman. Search engines: Information retrieval in practice. Addison-Wesley, 2010.
17. J. Duchi, L. Mackey, and M.I. Jordan. On the consistency of ranking algorithms. In Proceedings of the 27th International Conference on Machine Learning, pages 327-334, 2010.
18. Y. Freund, R. Iyer, R.E. Schapire, and Y. Singer. An efficient boosting algorithm for combining preferences. The Journal of Machine Learning Research, 4: 933-969, 2003. ISSN 1532-4435.
19. J. Hájek, Z. Šidák, and P.K. Sen. Theory of rank tests. Academic press New York, 1967.
20. R. Herbrich, T. Graepel, and K. Obermayer. Large margin rank boundaries for ordinal regression. Advances in Neural Information Processing Systems, pages 115-132, 1999.
21. K. Järvelin and J. Kekäläinen. IR evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 41-48. ACM, 2000.
22. K. Järvelin and J. Kekäläinen. Cumulated gain-based evaluation of IR techniques. ACM Transactions on Information Systems (TOIS), 20(4): 422-446, 2002.
23. T. Joachims. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 133-142. ACM, 2002.
24. Evangelos Kanoulas and Javed A. Aslam. Empirical justification of the gain and discount function for NDCG. In Proceedings of the 18th ACM conference on Information and knowledge management, pages 611-620. ACM, 2009.
25. M.G. Kendall. A new measure of rank correlation. Biometrika, 30(1/2): 81-93, 1938.
26. R. Nallapati. Discriminative models for information retrieval. In Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 64-71. ACM, 2004.
27. P. Ravikumar, A. Tewari, and E. Yang. On NDCG consistency of listwise ranking methods. In Proceedings of 14th International Conference on Artificial Intelligence and Statistics, AISTATS, 2011.
28. C. Rudin. The p-norm push: A simple convex ranking algorithm that concentrates at the top of the list. The Journal of Machine Learning Research, 10: 2233-2271, 2009.
29. T. Sakai. Evaluating evaluation metrics based on the bootstrap. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 525-532. ACM, 2006.
30. G. Sansone. Orthogonal Functions. Interscience Publishers Inc., New York, 1959.
31. A. Tewari and P.L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research, 8: 1007-1025, 2007.
32. A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 11-18. ACM, 2006.
33. H. Valizadegan, R. Jin, R. Zhang, and J. Mao. Learning to rank by optimizing NDCG measure. Advances in Neural Information Processing Systems, 22: 1883-1891, 2009.
34. E.M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 74-82. ACM, 2001.
35. F. Xia, T.Y. Liu, J. Wang, W. Zhang, and H. Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192-1199. ACM, 2008.
36. Y. Yue, T. Finley, F. Radlinski, and T. Joachims. A support vector method for optimizing average precision. In Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval, pages 271-278. ACM, 2007.
37. T. Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. Annals of Statistics, pages 56-85, 2004a.
38. T. Zhang. Statistical analysis of some multi-category large margin classification methods. The Journal of Machine Learning Research, 5: 1225-1251, 2004b.
39. K. Zhou, H. Zha, Y. Chang, and G.R. Xue. Learning the gain values and discount factors of discounted cumulative gains. IEEE Transactions on Knowledge and Data Engineering, to appear.