An analysis of crowd workers mistakes for specific and complex relevance assessment task

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2013, Pages 1873-1876

  Anderton, Jesse ^a   Bashir, Maryam ^a   Pavlu, Virgil ^a   Aslam, Javed A ^a  

^a Northeastern University   (United States)

Author keywords

Crowdsourcing; Preference judgment

Indexed keywords

AD HOC TRACKS; AMAZON'S MECHANICAL TURKS; CROWDSOURCING; EXPERT JUDGEMENT; EXPERT JUDGMENT; PREFERENCE JUDGMENT; RELEVANCE ASSESSMENTS; WEB SEARCHES;

KNOWLEDGE MANAGEMENT;

EID: 84889603841     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2505515.2507884     Document Type: Conference Paper

References (5)

1. C. Buckley, D. Dimmick, I. Soboroff, and E. M. Voorhees. Bias and the limits of pooling. In SIGIR, pages 619-620, 2006.
2. B. Carterette, P. N. Bennett, D. M. Chickering, and S. T. Dumais. Here or there. In ECIR, 2008.
3. M. Hosseini, I. J. Cox, N. Milić-Frayling, G. Kazai, and V. Vinay. On aggregating labels from multiple crowd workers to infer relevance of documents. In ECIR. Springer-Verlag, 2012.
4. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In SIGKDD Workshop on Human Computation. ACM, 2010.
5. M. D. Smucker, G. Kazai, and M. Lease. Overview of the trec 2012 crowdsourcing track. In TREC 2012.