Worker types and personality traits in crowdsourcing relevance labels

International Conference on Information and Knowledge Management, Proceedings

Volumn , Issue , 2011, Pages 1941-1944

  Kazai, Gabriella ^a   Kamps, Jaap ^b   Milic Frayling, Natasa ^a  

^a MICROSOFT RESEARCH   (United Kingdom)

^b UNIVERSITY OF AMSTERDAM   (Netherlands)

Author keywords

bfi test; crowdsourcing relevance labels; worker typology

Indexed keywords

BEHAVIORAL OBSERVATION; BFI TEST; BIG FIVE; COMPLETION TIME; CROWDSOURCING; IN-DEPTH UNDERSTANDING; OUTPUT QUALITY; PERSONALITY DIMENSIONS; PERSONALITY TRAITS; WORKER TYPOLOGY;

BENCHMARKING; KNOWLEDGE MANAGEMENT; QUALITY CONTROL;

HUMAN RESOURCE MANAGEMENT;

EID: 83055165986     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2063576.2063860     Document Type: Conference Paper

References (21)

1. O. Alonso and R. A. Baeza-Yates. Design and implementation of relevance assessments using crowdsourcing. In Proc. ECIR'11, pages 153-164, 2011.
2. O. Alonso, D. E. Rose, and B. Stewart. Crowdsourcing for relevance evaluation. SIGIR Forum, 42:9-15, November 2008.
3. B. Carterette and I. Soboroff. The effect of assessor error on ir system evaluation. In Proc. SIGIR'10, pages 539-546. ACM, 2010.
4. J. S. Downs, M. B. Holbrook, S. Sheng, and L. F. Cranor. Are your participants gaming the system?: screening Mechanical Turk workers. In Proc. CHI'10, pages 2399-2402, 2010.
5. S. D. Gosling, S. Gaddis, and S. Vazire. Personality impressions based on Facebook profiles. Psychology, pages 1-4, 2007.
6. C. Grady and M. Lease. Crowdsourcing document relevance assessment with Mechanical Turk. In Proc. CSLDAMT'10, pages 172-179, 2010.
7. J. Howe. Crowdsourcing: Why the Power of the Crowd Is Driving the Future of Business. Crown Publishing Group, 2008.
8. J.-H. Huang and Y.-C. Yang. The relationship between personality traits and online shopping motivations. Social Behavior and Personality, 38:673-680, 2010.
9. P. G. Ipeirotis. Analyzing the Amazon Mechanical Turk marketplace. XRDS, 17:16-21, 2010.
10. O. P. John, L. P. Naumann, and C. J. Soto. Paradigm shift to the integrative big-five trait taxonomy. In Handbook of personality, chapter 4, pages 114-212. Guilford Press, New York NY, 2008.
11. G. Kazai. In search of quality in crowdsourcing for search engine evaluation. In Proc. ECIR'11, pages 165-176, 2011.
12. G. Kazai, J. Kamps, M. Koolen, and N. Milic-Frayling. Crowdsourcing for book search evaluation: impact of HIT design on comparative system ranking. In Proc. SIGIR'11, pages 205-214, 2011.
13. G. Kazai, M. Koolen, J. Kamps, A. Doucet, and M. Landoni. Overview of the INEX 2010 book track: Scaling up the evaluation using crowdsourcing. In Proc. INEX'10, pages 101-120, 2011.
14. A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user studies with Mechanical Turk. In Proc. CHI'08, CHI '08, pages 453-456, 2008.
15. M. Kosinski, F. Radlinski, and P. Kohli. Personality and online behavior. In Proc. CIKM'11, 2011. ACM.
16. J. Le, A. Edmonds, V. Hester, and L. Biewald. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. CSE'10, pages 21-26, 2010.
17. B. Rammstedt and O. P. John. Measuring personality in one minute or less: A 10-item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41:203-212, 2007.
18. J. Ross, L. Irani, M. S. Silberman, A. Zaldivar, and B. Tomlinson. Who are the crowdworkers?: shifting demographics in Mechanical Turk. In Proc. CHI 2010, pages 2863-2872. ACM, 2010.
19. R. Snow, B. O'Connor, D. Jurafsky, and A. Y. Ng. Cheap and fast - but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. EMNLP'08, pages 254-263, 2008.
20. J. Vuurens, A. P. de Vries, and C. Eickhoff. How much spam can you take? an analysis of crowdsourcing results to increase accuracy. In Proc. ACM SIGIR Workshop on Crowdsourcing for Information Retrieval (CIR'11), pages 21-26, 2011. ACM.
21. D. Zhu and B. Carterette. An analysis of assessor behavior in crowdsourced preference judgments. In Proc. CSE'10, pages 17-20, 2010.