A new rank correlation coefficient for information retrieval

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 587-594

  Yilmaz, Emine ^a   Aslam, Javed A ^b   Robertson, Stephen ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b Northeastern University   (United States)

Author keywords

Average precision; Evaluation; Kendall's tau; Rank correlation

Indexed keywords

AVERAGE PRECISION; EVALUATION; EXPERIMENTAL DATUMS; KENDALL'S; KENDALL'S TAU; MATHEMATICAL PROPERTIES; PROBABILISTIC INTERPRETATIONS; RANK CORRELATION; RANK CORRELATION COEFFICIENTS;

ERRORS; INFORMATION RETRIEVAL; INFORMATION SERVICES; RESEARCH AND DEVELOPMENT MANAGEMENT;

CORRELATION METHODS;

EID: 57349152359     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390435     Document Type: Conference Paper

References (17)

1. J. A. Aslam, V. Pavlu, and R. Savell. A unified model for metasearch, pooling, and system evaluation. In 0. Frieder, J. Hammer, S. Quershi, and L. Seligman, editors, Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 484-491. ACM Press, November 2003.
2. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 25-32, New York, NY, USA, 2004. ACM Press.
3. B. Carterette and J. Allan. Incremental test collections. In CIKM '05: Proceedings of the 14th A CM international conference on Information and knowledge management, pages 680-687, New York, NY, USA, 2005. ACM Press.
4. T. Cover and J. Thomas. Elements of Information Theory. Wiley, 1991.
5. R. Fagin, R. Kumar, and D. Sivakumar. Comparing top k lists. In SODA '03: Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms, pages 28-36, Philadelphia, PA, USA, 2003. Society for Industrial and Applied Mathematics.
6. T. H. Haveliwala, A. Gionis, D. Klein, and P. Indyk. Evaluating strategies for similarity search on the web. In WWW '02: Proceedings of the 11th international conference on World Wide Web, pages 432-442, New York, NY, USA, 2002. ACM.
7. M. Kendall. A new measure of rank correlation. Biometrica, 30(1-2):81-89, 1938.
8. M. Melucci. On rank correlation in information retrieval evaluation. SIGIR Forum, 41(1):18-33, 2007.
9. M. Sanderson and H. Joho. Forming test collections with no system pooling. In SIGIR '04: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval, pages 33-40, New York, NY, USA, 2004. ACM.
10. M. Sanderson and I. Soboroff. Problems with kendall's tau. In SIGIR '07: Proceedings of the 30th annual international A CM SIGIR conference on Research and development in information retrieval, pages 839-840, New York, NY, USA, 2007. ACM.
11. G. S. Shieh. A weighted kendall's tau statistic. Statistics & Probability Letters, 39:17-24, 1998.
12. I. Soboroff, C. Nicholas, and P. Cahan. Ranking retrieval systems without relevance judgments. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 66-73, New Orleans, Louisiana, USA, Sept. 2001. ACM Press, New York.
13. E. M. Voorhees. Evaluation by highly relevant documents. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 74-82. ACM Press, 2001.
14. E. M. Voorhees. Overview of the TREC 2004 robust retrieval track. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004), 2004.
15. D. D. Wackerly, W. Mendenhall, and R. L. Scheaffer. Mathematical Statistics with Applications. Duxbury Advanced Series, 2002.
16. S. Wu and F. Crestani. Methods for ranking information retrieval systems without relevance judgments. In SAC '03: Proceedings of the 2003 ACM symposium on Applied computing, pages 811-816, 2003.
17. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proceedings of the Fifteenth ACM International Conference on Information and Knowledge Management. ACM Press, November 2006.