On the measurement of test collection reliability

SIGIR 2013 - Proceedings of the 36th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2013, Pages 393-402

  Urbano, Julián ^a   Marrero, Mónica ^a   Martín, Diego ^b  

^a UNIVERSIDAD CARLOS III DE MADRID   (Spain)

^b UNIVERSIDAD POLITÉCNICA DE MADRID   (Spain)

Author keywords

Evaluation; Generalizability Theory; Reliability; Test Collection; TREC

Indexed keywords

CONFIDENCE INTERVAL; EVALUATION; GENERALIZABILITY THEORIES; ORDERS OF MAGNITUDE; RELIABILITY INDICATORS; STATISTICAL THEORY; TEST COLLECTION; TREC;

INFORMATION RETRIEVAL; RELIABILITY; RESEARCH; TESTING;

RELIABILITY THEORY;

EID: 84883095495     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2484028.2484038     Document Type: Conference Paper

References (24)

1. J. Allan, J. A. Aslam, B. Carterette, V. Pavlu, and E. Kanoulas. Million Query Track 2008 Overview. In Text REtrieval Conference, 2008.
2. J. Allan, B. Carterette, J. A. Aslam, V. Pavlu, B. Dachev, and E. Kanoulas. Million Query Track 2007 Overview. In Text REtrieval Conference, 2007.
3. C. Arteaga, S. Jeyaratnam, and G. A. Franklin. Confidence Intervals for Proportions of Total Variance in the Two-Way Cross Component of Variance Model. Communications in Statistics: Theory and Methods, 11(15):1643-1658, 1982.
4. D. Banks, P. Over, and N.-F. Zhang. Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval, 1(1-2):7-34, 1999.
5. D. Bodoff. Test Theory for Evaluating Reliability of IR Test Collections. Information Processing and Management, 44(3):1117-1145, 2008.
6. D. Bodoff and P. Li. Test Theory for Assessing IR Test Collections. In ACM SIGIR, pages 367-374, 2007.
7. R. L. Brennan. Generalizability Theory. Springer, 2001.
8. C. Buckley and E. M. Voorhees. Evaluating Evaluation Measure Stability. In ACM SIGIR, pages 33-34, 2000.
9. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan. Evaluation Over Thousands of Queries. In ACM SIGIR, pages 651-658, 2008.
10. B. Carterette, V. Pavlu, E. Kanoulas, J. A. Aslam, and J. Allan. If I Had a Million Queries. In ECIR, pages 288-300, 2009.
11. L. S. Feldt. The Approximate Sampling Distribution of Kuder-Richardson Reliability Coefficient Twenty. Psychometrika, 30(3):357-370, 1965.
12. E. Kanoulas and J. A. Aslam. Empirical Justification of the Gain and Discount Function for nDCG. In ACM CIKM, pages 611-620, 2009.
13. W.-H. Lin and A. Hauptmann. Revisiting the Effect of Topic Set Size on Retrieval Error. In ACM SIGIR, pages 637-638, 2005.
14. S. Robertson and E. Kanoulas. On Per-Topic Variance in IR Evaluation. In ACM SIGIR, pages 891-900, 2012.
15. T. Sakai. On the Reliability of Information Retrieval Metrics Based on Graded Relevance. Information Processing and Management, 43(2):531-548, 2007.
16. M. Sanderson. Test Collection Based Evaluation of Information Retrieval Systems. Foundations and Trends in Information Retrieval, 4(4):247-375, 2010.
17. M. Sanderson and J. Zobel. Information Retrieval System Evaluation: Effort, Sensitivity, and Reliability. In ACM SIGIR, pages 162-169, 2005.
18. R. J. Shavelson and N. M. Webb. Generalizability Theory: A Primer. Sage Publications, 1991.
19. J. Urbano, M. Marrero, and D. Martín. A Comparison of the Optimality of Statistical Significance Tests for Information Retrieval Evaluation. In ACM SIGIR, 2013.
20. E. M. Voorhees. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. In ACM SIGIR, pages 315-323, 1998.
21. E. M. Voorhees. Topic Set Size Redux. In ACM SIGIR, pages 806-807, 2009.
22. E. M. Voorhees and C. Buckley. The Effect of Topic Set Size on Retrieval Experiment Error. In ACM SIGIR, pages 316-323, 2002.
23. E. Yilmaz, J. A. Aslam, and S. Robertson. A New Rank Correlation Coefficient for Information Retrieval. In ACM SIGIR, pages 587-594, 2008.
24. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In ACM SIGIR, pages 307-314, 1998.