The face of quality in crowdsourcing relevance labels: Demographics, personality and labeling accuracy

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 2583-2586

  Kazai, Gabriella ^a   Kamps, Jaap ^b   Milic Frayling, Natasa ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

crowdsourcing; demographics; personality traits; worker accuracy

Indexed keywords

CONTROL TECHNIQUES; CROWDSOURCING; DEMOGRAPHICS; OUTPUT QUALITY; PERSONALITY TRAITS; SPECIFIC TASKS; WORKER ACCURACY;

INFORMATION RETRIEVAL SYSTEMS; KNOWLEDGE MANAGEMENT; POPULATION STATISTICS; QUALITY ASSURANCE;

QUALITY CONTROL;

EID: 84871089459     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2396761.2398697     Document Type: Conference Paper

References (21)

1. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42:9-15, November 2008.
2. P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz. Relevance assessment: are judges exchangeable and does it matter. In Proc. of SIGIR 2008, pages 667-674, 2008.
3. D. Chandler and A. Kapelner. Breaking monotony with meaning: Motivation in crowdsourcing markets. Technical report, Working paper, University of Chicago, 2010.
4. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening mechanical turk workers. In Proc. of CHI 2010, pages 2399-2402. ACM, 2010.
5. C. Grady and M. Lease. Crowdsourcing document relevance assessment with mechanical turk. In Proc. of CSLDAMT'10, pages 172-179, 2010.
6. P. G. Ipeirotis. Analyzing the amazon mechanical turk marketplace. XRDS, 17:16-21, 2010.
7. P. G. Ipeirotis, F. Provost, and J. Wang. Quality management on amazon mechanical turk. In Proc. of HCOMP'10, pages 64-67, 2010. ACM.
8. O. P. John, L. P. Naumann, and C. J. Soto. Paradigm shift to the integrative big-five trait taxonomy: History, measurement, and conceptual issues. In Handbook of personality: Theory and research, chapter 4, pages 114-212. Guilford Press, New York NY, 2008.
9. G. Kazai, M. Koolen, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: At the mercy of crowdsourcing. In INEX 2010 Workshop Pre-proceedings, pages 89-99, 2010.
10. G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: Impact of quality on comparative system ranking. In Proc. of SIGIR 2011. ACM, 2011.
11. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with mechanical turk. In Proc. of CHI 2008, pages 453-456, 2008. ACM.
12. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 21-26, 2010.
13. W. Mason and D. J. Watts. Financial incentives and the "performance of crowds". In Proc. of HCOMP'09, pages 77-85, 2009. ACM.
14. B. Rammstedt and O. P. John. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41:203-212, 2007.
15. J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in mechanical turk. In Proc. of CHI 2010, Extended Abstracts Volume, pages 2863-2872. ACM, 2010.
16. A. Shaw, J. Horton, and D. Chen. Designing incentives for inexpert human raters. In Proc. of CSCW'11, 2011.
17. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proc. of KDD'08, pages 614-622, 2008.
18. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. of the EMNLP'08, pages 254-263, 2008.
19. E. M. Voorhees and D. K. Harman, editors. TREC: Experimentation and Evaluation in Information Retrieval. MIT Press, 2005.
20. J. Vuurens, A. de Vries, and C. Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. of the ACM SIGIR Workshop on Crowdsourcing for Information Retrieval, 2011. ACM.
21. D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In SIGIR 2010 Workshop on Crowdsourcing for Search Evaluation, pages 17-20, 2010.