Obtaining high-quality relevance judgments using crowdsourcing

IEEE Internet Computing

Volumn 16, Issue 5, 2012, Pages 20-27

  Vuurens, Jeroen B P ^a   De Vries, Arjen P ^b  

^a THE HAGUE UNIVERSITY OF APPLIED SCIENCES   (Netherlands)

^b CWI   (Netherlands)

Author keywords

crowdsourcing; judgment; quality; relevance; spam

Indexed keywords

CROWDSOURCING; GROUND TRUTH; HIGH QUALITY; JUDGMENT; RELEVANCE; RELEVANCE JUDGMENT; RELEVANT DOCUMENTS; SPAM; SPAMMERS; TEST SETS;

IMAGE QUALITY; INTERNET;

QUALITY CONTROL;

EID: 84867482244     PISSN: 10897801     EISSN: None     Source Type: Journal    
DOI: 10.1109/MIC.2012.71     Document Type: Article

References (18)

1. E.M. Voorhees, "Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness," Information Processing and Management, vol. 36, Elsevier, 2000, pp. 697-716.
2. O. Alonso, D.E. Rose, and B. Stewart, "Crowdsourcing for Relevance Evaluation," SIGIR Forum, vol. 42, no. 2, 2008, pp. 9-15.
3. P.G. Ipeirotis, F. Provost, and J. Wang, "Quality Management on Amazon Mechanical Turk," Proc. ACM SIGKDD Workshop Human Computation, ACM, 2010, pp. 64-67.
4. R. Blanco et al., "Repeatable and Reliable Search System Evaluation Using Crowdsourcing," Proc. 34th Ann. Int'l ACM SIGIR, Conf. Research and Development in Information Retrieval, ACM, 2011, pp. 923-932.
5. A. Kittur, E.H. Chi, and B. Suh, "Crowdsourcing User Studies with Mechanical Turk," Proc. 26th Ann. SIGCHI Conf. Human Factors in Computing Systems (CHI 08), ACM, 2008, pp. 453-456.
6. J.B.P. Vuurens, A.P. de Vries, and C. Eickhoff, "How Much Spam Can You Take? An Analysis of Crowdsourcing Results to Increase Accuracy," Proc. SIGIR Workshop Crowdsourcing for Information Retrieval (CIR 11), ACM, 2011, pp. 21-26
7. J. Le et al., "Ensuring Quality in Crowdsourced Search Relevance Evaluation," Proc. SIGIR Workshop Crowdsourcing for Search Evaluation (CSE 10), ACM, 2010, pp. 17-20.
8. G. Kazai, J. Kamps, and N. Milic-Frayling, "Worker Types and Personality Traits in Crowdsourcing Relevance Labels," Proc. 20th ACM Conf. Information and Knowledge Management (CIKM 11), ACM, 2011, pp. 1941-1944.
9. D. Zhu and B. Carterette, "An Analysis of Assessor Behavior in Crowdsourced Preference Judgments," Proc. 33rd ACM SIGIR Workshop Crowdsourcing for Search Evaluat ion (CSE 10), ACM, 2010, pp. 21-26.
10. C. Eickhoff and A.P. de Vr ies, "Increasing Cheat Robustness of Crowdsourcing Tasks," Advances in Information Retrieval, Spr inger, to appear, 2012.
11. G. Kazai, "In Search of Quality in Crowdsourcing for Search Engine Evaluation," Advances in Information Retrieval, Springer, 2011, pp. 165-176.
12. O. Dekel and O. Shamir, "Vox Populi: Collecting High-Quality Labels from a Crowd," Proc. 22nd Ann. Conf. Learning Theory (COLT 09), 2009.
13. J. Rzeszotarski and A. Kittur, "Instrumenting the Crowd: Using Implicit Behavioral Measures to Predict Task Performance," ACM Symp. User Interface Software and Technology, ACM, 2011, pp. 13-22.
14. J. Downs et al., "Are Your Participants Gaming the System? Screening Mechanical Turk Workers," Proc. 28th Int'l Conf. Human Factors in Computing Systems (CHI 10), ACM, 2010, pp. 2399-2402.
15. K. Krippendorff and M.A. Bock, The Content Analysis Reader, SAGE Publications, 2009.
16. M.A. Kouritzin et al., "On Detecting Fake Coin Flip Sequences," IMS Collections - Markov Processes and Related Topics: A Festschrift for Thomas G. Kurtz, vol. 4, IMS Collections, 2008, pp. 107-122.
17. F. Scholer, A. Turpin, and M. Sanderson, "Quantifying Test Collection Quality Based on the Consistency of Relevance Judgments," Proc. 34th Ann. ACM Special Interest Group on Information Retrieval, ACM, 2011, pp. 1063-1072.
18. Sheng V.S., Provost F., Ipeirotis P.G., Get another label? Improving data quality and data mining using multiple, noisy labelers. Proc. 14th ACM SIGKDD Int'l Conf. Knowledge Discovery and Data Mining (KDD 08), ACM 2008, 614-622.