Reusable test collections through experimental design

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 547-554

  Carterette, Ben ^a   Kanoulas, Evangelos ^b   Pavlu, Virgil ^c   Fang, Hui ^d  

^a University of Delaware   (United States)

^b UNIVERSITY OF SHEFFIELD   (United Kingdom)

^c Northeastern University   (United States)

^d UNIVERSITY OF DELAWARE   (United States)

Author keywords

Evaluation; Information retrieval; Reusability; Test collections

Indexed keywords

COURSE CORRECTION; EVALUATION; EXPERIMENTAL DESIGN; RELEVANCE JUDGMENT; RESEARCH AND DEVELOPMENT; SIMULATION EXPERIMENTS; STATISTICAL ANALYSIS; TEST COLLECTION;

EXPERIMENTS; INFORMATION RETRIEVAL; REUSABILITY;

TESTING;

EID: 77956034338     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835541     Document Type: Conference Paper

References (17)

1. J. Allan, J. A. Aslam, B. Carterette, V. Pavlu, and E. Kanoulas. Overview of the TREC 2008 million query track. In Proceedings of TREC, 2008.
2. J. A. Aslam and V. Pavlu. A practical sampling strategy for efficient retrieval evaluat ion, technical report.
3. S. Büttcher, C. Clarke, P. Yeung, and I. Soboroff. Reliable information retrieval evaluation with incomplete and biased judgements. In Proceedings of SIGIR, pages 63-70, 2007.
4. B. Carterette. Robust test collections for retrieval evaluation. In Proceedings of SIGIR, pages 55-62, 2007.
5. B. Carterette. On rank correlation and the distance between rankings. In Proceedings of SIGIR, 2009.
6. B. Carterette, J. Allan, and R. K. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of SIGIR, pages 268-275, 2006.
7. B. Carterette, E. Gabrilovitch, V. Josifovsky, and D. Metzler. Measuring the reusability of test collections. In Proceedings of WSDM, 2009.
8. B. Carterette, V. Pavlu, H. Fang, and E. Kanoulas. Overview of the TREC 2009 million query track. In Notebook Proceedings of TREC, 2009.
9. B. Carterette and M. D. Smucker. Hypothesis testing with incomplete relevance judgments. In Proceedings of CIKM, pages 643-652, 2007.
10. J. Cohen. Statistical Power Analysis for the Behavioral Sciences. Lawrence Erlbaum, 2nd edition, 1998.
11. D. Harman. Overview of the second text retrieval conference (trec-2). Inf. Process. Manage., 31(3):271-289, 1995.
12. J. P. A. Ionnidis. Why most published research findings are false. PLoS Med., 2(8), 2005.
13. T. Sakai. Comparing metrics across trec and ntcir: the robustness to system bias. In Proceedings of CIKM, pages 581-590, 2008.
14. E. M. Voorhees. The philosophy of information retrieval evaluation. In CLEF '01: Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, pages 355-370, London, UK, 2002. Springer-Verlag.
15. E. M. Voorhees. Overview of trec 2009. In Proceedings of TREC, 2009. Notebook draft.
16. W. Webber, A. Moffat, and J. Zobel. Statistical power in retrieval experimentation. In Proceedings of CIKM, pages 571-580, 2008.
17. J. Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of SIGIR, pages 307-314, 1998.