Relevance assessment: Are judges exchangeable and does it matter?

ACM SIGIR 2008 - 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Proceedings

Volumn , Issue , 2008, Pages 667-674

  Bailey, Peter ^a   Thomas, Paul ^b   Craswell, Nick ^a   De Vries, Arjen P ^c   Soboroff, Ian ^d   Yilmaz, Emine ^e  

^a MICROSOFT   (United States)

^b CSIRO   (Australia)

^c CWI   (Netherlands)

^d NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY   (United States)

^e MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Experimentation; Measurement; Performance

Indexed keywords

BRONZE; COPPER ALLOYS; INFORMATION RETRIEVAL; INFORMATION RETRIEVAL SYSTEMS; INFORMATION SERVICES; RESEARCH AND DEVELOPMENT MANAGEMENT; SILVER;

EXPERIMENTATION; GOLD STANDARDS; INFORMATION SEEKING; IR TESTS; MEASURING SYSTEMS; PERFORMANCE; RELATIVE PERFORMANCES; RELEVANCE ASSESSMENTS; RELEVANCE JUDGEMENTS; SYSTEM RANKINGS; TEST COLLECTIONS;

STANDARDS;

EID: 57349188929     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1390334.1390447     Document Type: Conference Paper

References (25)

1. R. Artstein and M. Poesio. Inter-coder agreement for computational linguistics. Computational Linguistics, to appear.
2. J. A. Aslam, V. Pavlu, and E. Yilmaz. A statistical method for system evaluation using incomplete judgments. In Proc. SIGIR, 2006.
3. P. Bailey, N. Craswell, I. Soboroff, and A. P. de Vries. The CSIRO enterprise search test collection. SIGIR Forum, 41(2), December 2007.
4. C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proc. SIGIR, 2004.
5. R. Burgin. Variations in relevance judgments and the evaluation of retrieval performance. Information Processing & Management, 28(5):619-627, Sep-Oct 1992.
6. J. Carletta. Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics, 22(2):249-254, 1996.
7. C. W. Cleverdon. The effect of variations in relevance assessments in comparative experimental tests of index languages. Technical Report ASLIB part 2, Cranfield Institute of Technology, 1970.
8. J. Cohen. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37-46, 1960.
9. G. V. Cormack, C. R. Palmer, and C. L. A. Clarke. Efficient construction of large test collections. In Proc. SIGIR, 1998.
10. B. D. Eugenio and M. Glass. The kappa statistic: a second look. Computational Linguistics, 30(1):95-101, 2004.
11. S. P. Harter. Variations in relevance assessments and the measurement of retrieval effectiveness. JASIS, 47(1):37-49, 1996.
12. K. Järvelin and J. Kekäläinen. IR evaluation methods for retrieving highly relevant documents. In Proc. SIGIR, 2000.
13. K. S. Jones and K. van Rijsbergen. Information retrieval test collections. Journal of Documentation, 32:59-75, 1976.
14. M. E. Lesk and G. Salton. Relevance assessments and retrieval system evaluation. Information Storage and Retrieval, 4:343-359, 1969.
15. S. Mizzaro. Measuring the agreement among relevance judges. In Proc. MIRA 99: Evaluating Interactive Information Retrieval, April 1999.
16. R. Rietveld and R. van Hout. Statistical Techniques for the Study of Language and Language Behaviour. Mouton de Gray ter, 1993.
17. S. Sigel and N. J. Castellan. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, 1988.
18. I. Soboroff. A comparison of pooled and sampled relevance judgments. In Proc. SIGIR, 2007.
19. E. Sormunen. Liberal relevance criteria of TREC: counting on negligible documents? In Proc. SIGIR, 2002.
20. A. Trotman and D. Jenkinson. IR Evaluation Using Multiple Assessors per Topic. In Proc. ADCS, 2007.
21. A. Trotman, N. Pharo, and D. Jenkinson. Can we at least agree on something? In Proc. SIGIR Workshop on Focused Retrieval, 2007.
22. E. M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. In Proc. SIGIR, 1998.
23. E. M. Voorhees and D. Harman. Overview of the Fifth Text REtrieval Conference (TREC-5). NIST, 1996.
24. E. Yilmaz and J. A. Aslam. Estimating average precision with incomplete and imperfect judgments. In Proc. CIKM, 2006.
25. E. Yilmaz, E. Kanoulas, and J. Aslam.. A simple and efficient sampling method for estimating AP and NDCG. In Proc. SIGIR, 2008.