Using crowdsourcing for TREC relevance assessment

Information Processing and Management

Volumn 48, Issue 6, 2012, Pages 1053-1066

  Alonso, Omar ^a   Mizzaro, Stefano ^b  

^a MICROSOFT   (United States)

^b UNIVERSITY OF UDINE   (Italy)

Author keywords

Amazon Mechanical Turk; Crowdsourcing; Experimental design; IR evaluation; Relevance assessment; Test collections; TREC

Indexed keywords

CROWDSOURCING; MECHANICAL TURKS; RELEVANCE ASSESSMENTS; TEST COLLECTION; TREC;

DESIGN OF EXPERIMENTS;

EXPERIMENTS;

EID: 84865695467     PISSN: 03064573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.ipm.2012.01.004     Document Type: Article

References (40)

1. Alonso, O.; & Baeza-Yates, R. (2011). Design and implementation of relevance assessments using crowdsourcing. In Proceedings of the European conference on Information Retrieval (ECIR) (pp. 153-164).
2. Alonso, O.; & Lease, M. (2011). Crowdsourcing for information retrieval: Principles, methods and applications, SIGIR tutorial. In: Proceedings of the 34th ACM SIGIR conference (pp. 1299-1300).
3. Alonso, O.; & Mizzaro, S. (2009a). Relevance criteria for E-commerce. A crowdsourcing-based experimental analysis. In Proceedings of the 32nd ACM SIGIR conference (pp. 760-761).
4. Alonso, O.; & Mizzaro, S. (2009b). Can we get rid of TREC assessors? Using mechanical Turk for relevance assessment. In Proceedings of the 32nd ACM SIGIR workshop on the future of IR, evaluation (pp. 15-16).
5. O. Alonso, D. Rose, and B. Stewart Crowdsourcing for relevance evaluation SIGIR Forum 42 2 2008 9 15
6. O. Alonso, R. Schenkel, and M. Theobald Crowdsourcing assessments for XML ranked retrieval Proceedings of the European Conference on Information Retrieval (ECIR) 2010 2010 623 626
7. Aslam, J.A.; Pavlu, V.; & Yilmaz, E. (2006). A statistical method for system evaluation using incomplete judgments. In Proceedings of the 29th ACM SIGIR conference (pp. 541-548).
8. Bailey, P.; Craswell, N.; Soboroff, I.; Thomas, P.; de Vries, A.P.; & Yilmaz, E. (2008). Relevance assessment: Are judges exchangeable and does it matter. In Proceedings of the 31st ACM SIGIR conference (pp. 667-674).
9. N. Bradburn, S. Sudman, and B. Wansink Asking questions: The definitive guide to questionnaire design 2004 Jossey-Bass
10. J. Callan, J. Allan, C.L.A. Clarke, S. Dumais, D.A. Evans, and M. Sanderson Meeting of the MINDS: An information retrieval research agenda SIGIR Forum 41 2 2007 25 34
11. Callison-Burch, C. (2009). Fast, cheap, and creative: Evaluating translation quality using Amazon's mechanical turk. In Proceedings of the 2009 conference on empirical methods in natural language processing (pp. 286-295).
12. Carterette, B.; & Soboroff, I. (2010). The effect of assessor error on IR system evaluation. In Proceedings of the 33rd ACM SIGIR conference (pp. 539-546).
13. B. Carterette, P. Bennet, D.M. Chickering, and S. Dumais Here or there: Preference judgments for relevance Proceedings of the European Conference on Information Retrieval (ECIR) 2008 2008 16 27
14. Carterette, B.; Pavlu, V.; Kanoulas, E.; Aslam, J.A.; & Allan, J. (2008). Evaluation over thousands of queries. In Proceedings of the 31st ACM SIGIR conference (pp. 651-658).
15. Carvalho, V.; Lease, M.; & Yilmaz, E. (Eds.) (2010). Proceedings of the 32nd ACM SIGIR workshop on crowdsourcing for relevance, evaluation, 2010.
16. J. Cohen A coefficient for agreement for nominal scales Education and Psychological Measurement 20 1960 37 46
17. Cormack, G.V.; Palmer, C.R.; & Clarke, C.L.A. (1998). Efficient construction of large test collections. In Proceedings of the 21st ACM SIGIR conference (pp. 282-289).
18. J.L. Fleiss Measuring nominal scale agreement among many raters Psychological Bulletin 76 5 1971 378 382
19. Grady, C.; & Lease, M. (2010). Crowdsourcing document relevance assessment with mechanical turk. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's mechanical turk (pp. 172-179).
20. J. Guiver, S. Mizzaro, and S. Robertson A few good topics: Experiments in topic set reduction for retrieval evaluation ACM Transactions on Information Systems 27 4 2009 1 26
21. J. Howe Crowdsourcing: Why the power of the crowd is driving the future of business 2008 Crown Business New York
22. Kazai, G.; Milic-Frayling, N.; & Costello, J. (2009). Towards methods for the collective gathering and quality control of relevance assessments. In Proceedings of the 32nd ACM SIGIR conference (pp: 452-459).
23. Kazai, G.; Kamps, J.; Koolen, M.; & Milic-Frayling, N. (2011). Crowdsourcing for book search evaluation: Impact of HIT design on comparative system ranking. In Proceedings of the 34th ACM SIGIR conference (pp. 205-214). Beijing, China, ACM.
24. Kittur, A.; Chi, E.H.; & Suh, B. (2008). Crowdsourcing user studies with mechanical turk. In CHI '08: Proceeding of the 26th ACM SIGCHI conference (pp. 453-456).
25. K. Krippendorff Estimating the reliability, systematic error, and random error of interval data Educational and Psychological Measurement 30 1 1970 61 70
26. Lease, M.; Sorokin, A.; & Yilmaz, E. (Eds.) (2011). Proceedings of the 33rd ACM SIGIR workshop on crowdsourcing for information retrieval.
27. M. Lease, and G. Kazai Overview of the TREC 2011 crowdsourcing track 2011 TREC
28. McCreadie, R.; Macdonald, C.; & Ounis, I. (2010). Crowdsourcing a News query classification dataset. In Proceedings of CSE 2010 workshop at SIGIR.
29. McCreadie, R.; Macdonald, C.; & Ounis, I. (2011). Crowdsourcing blog track top news judgments at TREC. In Proceedings of CSDM workshop at WSDM 2011.
30. J. Nielsen Usability engineering 1993 Morgan-Kaufmann
31. Nowak, S.; Rüger, S. (2010). How reliable are annotations via crowdsourcing? A study about inter-annotator agreement for multi-label image annotation. In Proceedings of the international ACM conference on multimedia, information retrieval (pp. 557-566).
32. Sanderson, M. & Zobel, J. (2005). Information retrieval system evaluation: Effort, sensitivity, and reliability. In Proceedings of the 28th ACM SIGIR conference (pp. 162-169).
33. M. Sanderson Test collection based evaluation of information retrieval systems Foundations and Trends in Information Retrieval 4 4 2010 247 375
34. Smucker, M.; & Prakash Jethani, C. (2011). Measuring assessor accuracy: A comparison of NIST assessors and user study participants. In: Proceedings of the 34th ACM SIGIR conference (pp. 1231-1232).
35. Snow, R.; O'Connor, B.; Jurafsky, D.; & Ng, A.Y. (2008). Cheap and fast but is it good? Evaluating non-expert annotations for natural language tasks. In Conference on empirical methods on natural language processing (pp. 254-263).
36. Soboroff, I.; Nicholas, C.; & Cahan, P. (2001). Ranking retrieval systems without relevance judgments. In Proceedings of the 24th ACM SIGIR conference (pp. 66-73).
37. Sormunen, E. (2002). Liberal relevance criteria of TREC: Counting on negligible documents? In Proceedings of the 25th ACM SIGIR conference (pp. 324-330).
38. Stemler, S.E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). < http://PAREonline.net/getvn. asp?v=9&n=4 > Retrieved 01.10.10.
39. E. Voorhees Variations in relevance judgments and the measurement of retrieval effectiveness Information Processing and Management 36 5 2000 697 716
40. Voorhees, E. (2001). The philosophy of information retrieval evaluation. In CLEF '01 proceedings (pp. 355-370).