MTurk character misrepresentation: Assessment and solutions

Journal of Consumer Research

Volumn 44, Issue 1, 2017, Pages 211-230

  Wessling, Kathryn Sharpe ^a   Huber, Joel ^b   Netzer, Oded ^c  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

^b DUKE UNIVERSITY   (United States)

^c Och Spine at New York Presbyterian Hospitals   (United States)

Author keywords

Amazon Mechanical Turk; Deception; Panel; Screener questions; Theory driven sample

Indexed keywords

EID: 85020066608     PISSN: 00935301     EISSN: None     Source Type: Journal    
DOI: 10.1093/jcr/ucx053     Document Type: Article

References (34)

1. Barriga, Alvaro Q., Elizabeth M. Morrison, Albert K. Liau, and John C. Gibbs (2001), "Moral Cognition: Explaining the Gender Difference in Antisocial Behavior," Merrill-Palmer Quarterly, 47 (4), 532-62.
2. Bederson, Benjamin B. and Alexander J. Quinn (2011), "Web Workers Unite! Addressing Challenges of Online Laborers," in CHI'11 Extended Abstracts on Human Factors in Computing Systems, New York: ACM, 97-106.
3. Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz (2012), "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk," Political Analysis, 20 (3), 351-68.
4. Brawley, Alice M. and Cynthia L. S. Pury (2016), "Work Experiences on MTurk: Job Satisfaction, Turnover, and Information Sharing," Computers in Human Behavior, 54, 531-46.
5. Burke, Daniel (2016), "Pope Suggests Trump 'Is Not Christian,'" CNN Politics, February 18, 2016, http://www.cnn.com/2016/02/18/politics/pope-francis-trump-christian-wall/.
6. Chandler, Jesse, Pam Mueller, and Gabriele Paolacci (2014), "Nonnaïveté among Amazon Mechanical Turk Workers: Consequences and Solutions for Behavioral Researchers," Behavior Research Methods, 46 (1), 112-30.
7. Chandler, Jesse and Gabriele Paolacci (2017), "Lie for a Dime: When Most Prescreening Responses Are Honest but Most Study Participants Are Imposters," Society Psychological and Personality Science, forthcoming.
8. Chandler, Jesse, Gabriele Paolacci, Pam Mueller, Eyal Peer, and Kate A. Ratliff(2015), "Using Nonnaïve Participants Can Reduce Effect Sizes," Psychological Science, 26 (7), 1131-9.
9. Cheung, Janelle H., Deanna K. Burns, Robert R. Sinclair, and Michael Sliter (2016), "Amazon Mechanical Turk in Organizational Psychology: An Evaluation and Practical Recommendations," Journal of Business and Psychology, 1-15.
10. Crowne, Douglas P. and David Marlowe (1960), "A New Scale of Social Desirability Independent of Psychopathology," Journal of Consulting Psychology, 24 (4), 349-54.
11. Dholakia, Utpal (2015), "My Experience as an Amazon Mechanical Turk (MTurk) Worker," July 20, https://www.linkedin.com/pulse/my-experience-amazon-mechanical-turkmturk-worker-utpal-dholakia.
12. Farrell, Anne M., Jonathan H. Grenier, and Justin Leiby (2017), "Scoundrels or Stars? Theory and Evidence on the Quality of Workers in Online Labor Markets," Accounting Review, 92 (1), 92-114.
13. Gleibs, Ilka H. (2016), "Are All 'Research Fields' Equal? Rethinking Practice for the Use of Data from Crowdsourcing Market Places," Behavior Research Methods, 1-10.
14. Goodman, Joseph K., Cynthia E. Cryder, and Amar Cheema (2013), "Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples," Journal of Behavioral Decision Making, 26 (3), 213-24.
15. Goodman, Joseph K. and Gabriele Paolacci (2017), "Crowdsumers Take Over: Towards Valid Crowdsourcing of Consumer Research," Journal of Consumer Research, forthcoming.
16. Gosling, Samuel D., Peter J. Rentfrow, and William B. Swann (2003), "A Very Brief Measure of the Big-Five Personality Domains," Journal of Research in Personality, 37 (6), 504-28.
17. Gould, Stephen J. (1988), "Consumer Attitudes Toward Health and Health Care: A Differential Perspective," Journal of Consumer Affairs, 22 (1), 96-118.
18. Graham, Jesse, Brian A. Nosek, Jonathan Haidt, Ravi Iyer, Spassena Koleva, and Peter H. Ditto (2011), "Mapping the Moral Domain," Journal of Personality and Social Psychology, 101 (2), 366-85.
19. Haidt, Jonathan, Jesse Graham, and Craig Joseph (2009), "Above and Below Left-Right: Ideological Narratives and Moral Foundations," Psychological Inquiry, 20 (2-3), 110-9.
20. Hauser, David J. and Norbert Schwarz (2016), "Attentive Turkers: MTurk Participants Perform Better on Online Attention Checks Than Do Subject Pool Participants," Behavior Research Methods 48 (1), 400-7.
21. John, Oliver P. and Sanjay Srivastava (1999), "The Big Five Trait Taxonomy: History, Measurement, and Theoretical Perspectives," Handbook of Personality: Theory and Research, 2, 102-38.
22. Leeper, Thomas J. (2017), "MTurkR: R Client for the MTurk Requester API, 2017," R Client for MTurk Requester API, https://cran.r-project.org/web/packages/MTurkR/MTurkR.pdf. R package version 0.8.0.
23. Leitch, Will (2004), "Group Thinker," New York Magazine, June 21, http://nymag.com/nymetro/shopping/features/9299/#comments.
24. Litman, Leib, Johnathan Robinson, and Tzvi Abberbock (2016), "TurkPrime.com: A Versatile Crowdsourcing Data Acquisition Platform for the Behavioral Sciences," Behavior Research Methods, forthcoming.
25. Martin, David, Benjamin V. Hanrahan, Jacki O'Neill, and Neha Gupta (2014), "Being a Turker," in Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work and Social Computing, New York: ACM, 224-35.
26. Paolacci, Gabriele, Jesse Chandler, and Panagiotis G. Ipeirotis (2010), "Running Experiments on Amazon Mechanical Turk," Judgment and Decision Making, 5 (5), 411-9.
27. Peer, Eyal, Joachim Vosgerau, and Alessandro Acquisti (2014), "Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk," Behavior Research Methods, 46, 1023-31.
28. Richins, Marsha L (2004), "The Material Values Scale: Measurement Properties and Development of a Short Form," Journal of Consumer Research, 31 (1), 209-19.
29. Ross, Joel, Lilly Irani, M. Silberman, Andrew Zaldivar, and Bill Tomlinson (2010), "Who Are the Crowdworkers?: Shifting Demographics in Mechanical Turk," in CHI'10 Extended Abstracts on Human Factors in Computing Systems, New York: ACM, 2863-72.
30. Sawtooth Software (2013), "The CBC System for Choice-Based Conjoint Analysis," Technical Paper Series, https://www.sawtoothsoftware.com/download/techpap/cbctech.pdf.
31. Sharpe, Kathryn, Richard Staelin, and Joel Huber (2008), "Using Extremeness Aversion to Fight Obesity: Policy Implications of Context Dependent Demand," Journal of Consumer Research, 35 (3), 406-22.
32. Tong, Betty C., Joel Huber, Deborah D. Ascheim, John Puskas, T. Bruce Ferguson Jr., Eugene Blackstone, and Peter K. Smith (2012) "Weighting Composite Endpoints in Clinical Trials: Essential Evidence from the Heart Team," Annals of Thoracic Surgery, 94 (6), 1908-13.
33. Wessling, Kathryn Sharpe, Oded Netzer, and Joel Huber (2016), "Customer Response to Within-Chain Price Hikes," working paper.
34. Zhou, Haotian and Ayelet Fishbach (2016), "The Pitfall of Experimenting on the Web: How Unattended Selective Attrition Leads to Surprising (Yet False) Research Conclusions," Journal of Personality and Social Psychology, 111 (4), 493-504.