On the choice of effectiveness measures for learning to rank

Information Retrieval

Volumn 13, Issue 3, 2010, Pages 271-290

  Yilmaz, Emine ^a   Robertson, Stephen ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Empirical risk minimization; Evaluation; Evaluation metrics; Learning to rank; Training

Indexed keywords

EID: 77953632234     PISSN: 13864564     EISSN: 15737659     Source Type: Journal    
DOI: 10.1007/s10791-009-9116-x     Document Type: Article

References (18)

1. Aslam, J. A., Yilmaz, E., & Pavlu V. (2005). The maximum entropy method for analyzing retrieval measures. In Marchionini, G., Moffat A., Tait, J., Baeza-Yates, R., & Ziviani, N., (Eds.), Proceedings of the 28th annual international ACM SIGIR conference on research and development in information retrieval (pp. 27-34). ACM Press, August 2005.
2. Burges, C. J. C., Ragno, R., & Le Q. V. (2006). Learning to rank with nonsmooth cost functions. In Schölkopf, B., Platt, J. C., & Hoffman, T. (Eds.), NIPS (pp. 193-200). Cambridge, MA: MIT Press.
3. Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., et al. (2005). Learning to rank using gradient descent. In ICML '05: Proceedings of the 22nd international conference on machine learning (pp. 89-96). New York, NY, USA: ACM.
4. Donmez, P., Svore, K., & Burges, C. J. (2008). On the optimality of lambdarank. Technical report, Microsoft Research.
5. Järvelin, K., & Kekäläinen, J. (2000). Ir evaluation methods for retrieving highly relevant documents. In SIGIR '00: Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval (pp. 41-48). New York, NY, USA: ACM.
6. Joachims, T. (2005). A support vector method for multivariate performance measures. In ICML '05: Proceedings of the 22nd international conference on Machine learning (pp 377-384). New York, NY, USA: ACM.
7. Le, Q. V., & Smola. A. J. (2007). Direct optimization of ranking measures. CoRR, abs/0704. 3359.
8. Ling, C. X., Huang, J., & Zhang, H. (2003). Auc: A statistically consistent and more discriminating measure than accuracy. In IJCAI '03: Proceedings of the 18th international conference on artificial intelligence (pp. 329-341).
9. Liu, T.-Y., & He Y. (2008). Are algorithms directly optimizing ir measures really direct? Technical report, Microsoft Research.
10. Robertson, S. (2008). A new interpretation of average precision. In SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval (pp. 689-690). New York, NY, USA: ACM.
11. Robertson, S., & Zaragoza H. (2007). On rank-based effectiveness measures and optimization. Information Retrieval, 10(3), 321-339.
12. Sanderson, M., & Zobel J. (2005). Information retrieval system evaluation: Effort, sensitivity, and reliability. In Baeza-Yates, R. A., Ziviani, N., Marchionini, G., Moffat, A., & Tait, J. (Eds.), SIGIR (pp. 162-169). ACM.
13. Taylor, M., Guiver, J., Robertson, S., & Minka, T. (2008). Softrank: optimizing non-smooth rank metrics. In WSDM '08: Proceedings of the international conference on Web search and web data mining (pp. 77-86) New York, NY, USA: ACM.
14. Taylor, M., Zaragoza, H., Craswell, N., Robertson, S., & Burges, C. (2006). Optimisation methods for ranking functions with multiple parameters. In CIKM '06: Proceedings of the 15th ACM international conference on Information and knowledge management (pp. 585-593). New York, NY, USA: ACM.
15. Webber, W., Moffat, A., Zobel, J., & Sakai, T. (2008). Precision-at-ten considered redundant. In SIGIR (pp. 695-696). New York, NY, USA: ACM.
16. Xu, J., & Li, H. (2007). Adarank: a boosting algorithm for information retrieval. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 391-398). New York, NY, USA: ACM.
17. Yilmaz, E., & Aslam J. A. (2008). Estimating average precision when judgments are incomplete. Knowledge and Information Systems, 16(2), 173-211, August 2008.
18. Yue, Y., Finleym, T., Radlinski, F., & Joachims, T. (2007). A support vector method for optimizing average precision. In SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval (pp. 271-278). New York, NY, USA: ACM.