The relationship between motivation, monetary compensation, and data quality among US- and India-based workers on Mechanical Turk

Behavior Research Methods

Volumn 47, Issue 2, 2015, Pages 519-528

  Litman, Leib ^a   Robinson, Jonathan ^a   Rosenzweig, Cheskie ^a  

^a TOURO COLLEGE   (United States)

Author keywords

Amazon Mechanical Turk; Data quality; Research methods

Indexed keywords

ADULT; CROWDSOURCING; ECONOMICS; FEMALE; HUMAN; INDIA; INTERNET; MALE; MARKETING; METHODOLOGY; MOTIVATION; PROCEDURES; SALARY AND FRINGE BENEFIT; SOCIOLOGY; STANDARDS; TASK PERFORMANCE; UNITED STATES;

ADULT; CROWDSOURCING; FEMALE; HUMANS; INDIA; INTERNET; MALE; MARKETING; MOTIVATION; RESEARCH DESIGN; SALARIES AND FRINGE BENEFITS; SOCIAL SCIENCES; TASK PERFORMANCE AND ANALYSIS; UNITED STATES;

EID: 84939895552     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-014-0483-x     Document Type: Article

References (33)

1. Antin, J., & Shaw, A. (2012). Social desirability bias and self-reports of motivation: A study of Amazon Mechanical Turk in the US and India. In CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2925–2934). New York, NY: ACM.
2. Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. com's Mechanical Turk. Political Analysis, 20, 351–368.
3. Bohannon, J. (2011). Human subject research: Social science for pennies. Science, 334, 307. doi:10.1126/science.334.6054.307
4. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s Mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6, 3–5. doi:10.1177/1745691610393980
5. Bureau of Democracy, Human Rights and Labor. (2012). Country Reports on Human Rights Practices, India. Retrieved December 22, 2013, from U.S. Department of State website, www.state.gov/j/drl/rls/hrrpt/2012humanrightsreport/index
6. Chandler, J., Mueller, P., & Paolacci, G. (2013). Methodological concerns and advanced uses of crowdsourcing in psychological research. Manuscript submitted for publication. http://www.jessechandler.com/uploads/2/8/0/5/2805897/mturk_adv_methods.pdf
7. Chouliarakis, G., & Correa-López, M. (2014). A fair wage model of unemployment with inertia in fairness perceptions. Oxford Economic Papers, 66(1), 88–114.
8. DeScioli, P., Christner, J., & Kurzban, R. (2011). The omission strategy. Psychological Science, 22, 442–446.
9. Downs, J. S., Holbrook, M. B., Sheng, S., & Cranor, L. F. (2010). Are your participants gaming the system? Screening Mechanical Turk Workers. In CHI ’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 2399–2402). New York, NY: ACM.
10. Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5, 159–163.
11. Falk, A., Fehr, E., & Zehnder, C. (2006). Fairness perceptions and reservation wages—The behavioral effects of minimum wage laws. The Quarterly Journal of Economics, 121, 1347–1381.
12. Faridani, S., Hartmann, B., & Ipeirotis, P. G. (2011). What’s the right price? Pricing tasks for finishing on time. In Workshops at the Twenty-Fifth AAAI Conference on Artificial Intelligence (pp. 26–31). Menlo Park, CA: AAAI.
13. Gardner, R. M., Brown, D. L., & Boice, R. (2012). Using Amazon’s Mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9, 532–534.
14. Government of India, Ministry of Labor and Employment. (2011).Second annual report to the people on employment. Retrieved December 22, 2013, from the Government of India website, dget.gov.in/publications/annualreportemployment2011.pdf.
15. Horton, J. J., & Chilton, L. B. (2010). The labor economics of paid crowdsourcing. In EC ’10: Proceedings of the 11th ACM Conference on Electronic Commerce (pp. 209–218). New York, NY: ACM. doi:10.1145/1807342.1807376
16. Ipeirotis, P. (2010). Demographics of Mechanical Turk [Working Article]. New York University. http://hdl.handle.net/2451/29585, http://scholar.google.com/scholar?cluster=8873729056826117132&hl=en&as_sdt=0,33
17. Jenkins, G. D. J., Mitra, A., Gupta, N., & Shaw, J. D. (1998). Are financial incentives related to performance? A meta-analytic review of empirical research. Journal of Applied Psychology, 83, 777–787.
18. John, O. P., Robins, R. W., & Pervin, L. A. (2008). Handbook of personality, third edition: Theory and research. New York, NY: Guilford Press.
19. Johnson, D. R., & Borden, L. A. (2012). Participants at your fingertips: Using Amazon’s Mechanical Turk to increase student-faculty collaborative research. Teaching of Psychology, 39, 245–251.
20. Jones, D. N., & Paulhus, D. L. (2011). The role of impulsivity in the Dark Triad of personality. Personality and Individual Differences, 51, 679–682.
21. Kazai, G., Kamps, J., & Milic-Frayling, N. (2013). An analysis of human factors and label accuracy in crowdsourcing relevance judgements. Information Retrieval Journal, 16, 138–178. doi:10.1007/s10791-012-9205-0
22. Marge, M., Banerjee, S., & Rudnicky, A. I. (2010). Using the Amazon Mechanical Turk for transcription of spoken language. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP) (pp. 5270–5273). Los Alamitos, CA: IEEE Press. doi:10.1109/ICASSP.2010.5494979
23. Mason, W., & Watts, D. J. (2010). Financial incentives and the performance of crowds. ACM SigKDD Explorations Newsletter, 11(2), 100–108.
24. Nyhan, B., & Reifler, J. (2011). Opening the political mind? The effects of self-affirmation and graphical information on factual misperceptions. Unpublished manuscript. Hanover, NH: Dartmouth College.
25. Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.
26. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411–419.
27. Paulhus, D. L., & Carey, J. M. (2011). The FAD-Plus: measuring lay beliefs regarding free will and related constructs. Journal of Personality Assessment, 93, 96–104.
28. Rani, U., & Belser, P. (2012). The effectiveness of minimum wages in developing countries: The case of India. International Journal of Labour Research, 4, 45–66.
29. Ross, J., Irani, L., Silberman, M. S., Zaldivar, A., & Tomlinson, B. (2010). Who are the crowdworkers? Shifting demographics in Mechanical Turk. In CHI ’10 Extended Abstracts on Human Factors in Computing Systems (pp. 2863–2872). New York, NY: ACM.
30. Schmitt, D. P., Allik, J., McCrae, R. R., & Benet-Martinez, V. (2007). The geographic distribution of Big Five Personality traits: Patterns and profiles of human self-description across 56 nations. Journal of Cross-Cultural Psychology, 38, 173–212.
31. Shapiro, D. N., Chandler, J., & Mueller, P. A. (2013). Using Mechanical Turk to study clinical populations. Clinical Psychological Science, 1, 213–220.
32. Shenhav, A., Rand, D. G., & Greene, J. D. (2012). Divine intuition: Cognitive style influences belief in God. Journal of Experimental Psychology: General, 141, 423–428. doi:10.1037/a0025391
33. Silberman, M. S., Ross, J., Irani, L., & Tomlinson, B. (2010). Sellers’ problems in human computation markets. In Proceedings of the ACM SIGKDD Workshop on Human Computation (pp. 18–21). New York, NY: ACM.