Implications of inter-rater agreement on a student information retrieval evaluation

LWA 2010 - Lernen, Wissen und Adaptivitat - Learning, Knowledge, and Adaptivity, Workshop Proceedings

Volumn , Issue , 2010, Pages 213-219

  Schaer, Philipp ^a   Mayr, Philipp ^a   Mutschke, Peter ^a  

^a GESIS LEIBNIZ INSTITUTE FOR THE SOCIAL SCIENCES   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

CO-WORD ANALYSIS; DOMAIN EXPERTS; INFORMATION PROFESSIONALS; INTER-RATER AGREEMENTS; QUERY EXPANSION MECHANISM; RE-RANKING; RELEVANCE ASSESSMENTS;

DIGITAL LIBRARIES; DIGITAL STORAGE; INFORMATION RETRIEVAL; INFORMATION SERVICES; METADATA;

STUDENTS;

EID: 84874002324     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (15)

1. Omar Alonso and Stefano Mizzaro. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. Proceedings of the SIGIR 2009 Workshop on The Future of IR Evaluation : 15-16, 2009
2. Azzah Al-Maskari, Mark Sanderson, and Paul Clough. Relevance judgments between TREC and Non-TREC assessors. Proceedings of the 31st annual international ACM SIGIR, 2008
3. Marcia J. Bates. Where Should the Person Stop and the Information Search Interface Start? Information Processing & Management, 26 575-591, 1990
4. Efthimis N. Efthimiadis. Query expansion. Annual Review of Information Systems and Technology (ARIST), Vol. 31, 121-187, 1996
5. Nicola Ferro and Carol Peters. CLEF 2009 Ad Hoc Track Overview: TEL & Persian Tasks. Working Notes for the CLEF 2009 Workshop, Corfu, Greece, 2009
6. Joseph L. Fleiss. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378-382, 1971
7. Werner Greve and Dirk Wentura. Wissenschaftliche Beobachtung: Eine Einführung. Weinheim, PVU/Beltz, 1997
8. J. Richard Landis and Gary G. Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 33(1), 159-174, 1977
9. Philipp Mayr, Peter Mutschke, and Vivien Petras. Reducing semantic complexity in distributed digital libraries: Treatment of term vagueness and document re-ranking. Library Review, 57(3), 213-224, 2008
10. Philipp Mayr. Re-Ranking auf Basis von Bradfordizing für die verteilte Suche in Digitalen Bibliotheken. Humboldt-Universität zu Berlin, Berlin, 2009
11. Peter Mutschke. Autorennetzwerke: Netzwerkanalyse als Mehrwertdienst für Informationssysteme. In: Information zwischen Kultur und Marktwirtschaft (ISI 2004), 141-162, 2004
12. Henning Müller, Jayashree Kalpathy-Cramer, Ivan Eggel, Steven Bedrick, Saïd Radhouani, Brian Bakke Jr., Charles E. Kahn, and William Hersh. Overview of the CLEF 2009 Medical Image Retrieval Track. Working Notes for the CLEF 2009 Workshop, Corfu, Greece, 2009
13. Vivien Petras. Translating Dialects in Search: Mapping between Specialized Languages of Discourse and Documentary Languages. University of California, Berkley, 2006
14. Ellen M. Voorhees. Variations in relevance judgments and the measurement of retrieval effectiveness. Information Processing & Management. 36(5): 697-716, 2000
15. Ellen M. Voorhees and Donna K. Harman (Eds.). TREC: Experiment and Evaluation in Information Retrieval: The MIT Press, 2005