The effect of assessor errors on IR system evaluation

SIGIR 2010 Proceedings - 33rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2010, Pages 539-546

  Carterette, Ben ^a   Soboroff, Ian ^b  

^a University of Delaware   (United States)

^b NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

Author keywords

Assessor error; Retrieval test collections

Indexed keywords

AMAZON'S MECHANICAL TURKS; ASSESSOR ERROR; CRANFIELD; RELEVANCE JUDGMENT; SCALING-UP; SYSTEM EVALUATION; SYSTEM RANKINGS; TEST COLLECTION;

ERRORS; INFORMATION RETRIEVAL; RESEARCH; SYSTEMATIC ERRORS;

QUALITY CONTROL;

EID: 77956024152     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1835449.1835540     Document Type: Conference Paper

References (17)

1. James Allan, Javed A. Aslam, Ben Carterette, Virgil Pavlu, and Evangelos Kanoulas. Overview of the TREC 2008 million query track. In Proceedings of TREC, 2008.
2. Javed A. Aslam and Virgil Pavlu. A practical sampling strategy for efficient retrieval evaluat ion, technical report.
3. Javed A. Aslam, Virgil Pavlu, and Emine Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proceedings of SIGIR, pages 541-548, 2006.
4. Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, and Emine Yilmaz. Relevance assessment: Are judges exchangeable and does it matter? In Proceedings of SIGIR, pages 667-674, 2008.
5. Ben Carterette. Robust evaluation of information retrieval systems. In Proceedings of SIGIR, 2007.
6. Ben Carterette, James Allan, and Ramesh K. Sitaraman. Minimal test collections for retrieval evaluation. In Proceedings of SIGIR, pages 268-275, 2006.
7. Ben Carterette, Virgil Pavlu, Evangelos Kanoulas, Javed A. Aslam, and James Allan. Evaluation over thousands of queries. In Proceedings of SIGIR, pages 651-658, 2008.
8. Gordon V. Cormack, Christopher R. Palmer, and Charles L.A. Clarke. Efficient construction of large test collections. In Proceedings of SIGIR, pages 282-289, 1998.
9. Donna Harman. Overview of the fourth Text REtrieval Conference. In Proceedings of the Fourth Text REtrieval Conference (TREC-4), pages 1-24, 1995. NIST Special Publication 500-236.
10. Kenneth A. Kinney, Scott Huffman, and Juting Zhai. How evaluator domain expertise affects search result relevance judgments. In Proceedings of CIKM, pages 591-598, 2008.
11. Mark Sanderson and Justin Zobel. Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of SIGIR, pages 186-193, 2005.
12. Ian Soboroff, Charles Nicholas, and Patrick Cahan. Ranking Retrieval Systems without Relevance Judgments. In Proceedings of SIGIR, pages 66-73, 2001.
13. Ellen Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proceedings of SIGIR, pages 315-323, 1998.
14. Ellen M. Voorhees. The philosophy of information retrieval evaluation. In CLEF '01: Revised Papers from the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, pages 355-370, London, UK, 2002. Springer-Verlag.
15. Ellen M. Voorhees and Donna K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.
16. Emine Yilmaz and Javed Aslam. Estimating average precision with incomplete and imperfect relevance judgments. In Proceedings of CIKM, pages 102-111, 2006.
17. Justin Zobel. How Reliable are the Results of Large-Scale Information Retrieval Experiments? In Proceedings of SIGIR, pages 307-314, 1998.