Measuring assessor accuracy: A comparison of NIST assessors and user study participants

SIGIR'11 - Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval

Volumn , Issue , 2011, Pages 1231-1232

  Smucker, Mark D ^a   Jethani, Chandra Prakash ^b  

^a UNIVERSITY OF WATERLOO   (Canada)

^b UNIVERSITY OF WATERLOO   (Canada)

Author keywords

Assessors; Relevance; Signal detection theory

Indexed keywords

INFORMATION RETRIEVAL;

ASSESSORS; INTERACTIVE RETRIEVAL SYSTEMS; NON-RELEVANT DOCUMENTS; RELEVANCE; SIGNAL DETECTION THEORY; TRADE OFF; TRUE POSITIVE RATES; USER STUDY;

ECONOMIC AND SOCIAL EFFECTS;

EID: 80052119348     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2009916.2010134     Document Type: Conference Paper

References (3)

1. O. Alonso and S. Mizzaro. Can we get rid of TREC assessors? Using Mechanical Turk for relevance assessment. In Proc. SIGIR'09 FIRE Workshop, pages 15-16, July 2009.
2. N. Macmillan and C. Creelman. Detection theory: a user's guide. Lawrence Erlbaum Associates, 2005.
3. M. D. Smucker and C. Jethani. Human performance and retrieval precision revisited. In SIGIR'10. ACM, 2010.