Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples

Journal of Behavioral Decision Making

Volumn 26, Issue 3, 2013, Pages 213-224

  Goodman, Joseph K ^a   Cryder, Cynthia E ^a   Cheema, Amar ^b  

^a WASHINGTON UNIVERSITY IN ST LOUIS   (United States)

^b UNIVERSITY OF VIRGINIA   (United States)

Author keywords

External validity; Online research; Research methods; Sampling; Surveys

Indexed keywords

EID: 84879603028     PISSN: 08943257     EISSN: 10990771     Source Type: Journal    
DOI: 10.1002/bdm.1753     Document Type: Article

References (38)

1. Barron, G., & Erev, I. (2003). Small feedback-based decisions and their limited correspondence to description-based decisions. Journal of Behavioral Decision Making, 16, 215-233.
2. Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon's Mechanical Turk: A new source of cheap, yet high-quality, data? Perspectives on Psychological Science, 6, 3-5.
3. Chatterjee, S., Heath, T. B., Milberg, S. J., & France, K. R. (2000). The differential processing of price in gains and losses: The effects of frame and need for cognition. Journal of Behavioral Decision Making, 13, 61-75.
4. Cryder, C., & Loewenstein, G. (2010). The time versus money scale. Unpublished data, Olin Business School, Washington University in St. Louis.
5. Eriksson, K., & Simpson, B. (2010). Emotional reactions to losing explain gender differences in entering a risky lottery. Judgment and Decision Making, 5, 159-163.
6. Frederick, S. (2005). Cognitive reflection and decision making. Journal of Economic Perspectives, 19, 25-42.
7. Friedman, T. L. (2005). The world is flat: A brief history of the twenty-first century. New York, NY: Farrar, Straus, and Giroux.
8. Gosling, S. D., Rentfrow, P. J., & Swann Jr., W. B. (2003). A very brief measure of the big-five personality domains. Journal of Research in Personality, 37, 504-528.
9. Hertwig, R., Barron G., Weber, E. U., & Erev, I. (2004). Decisions from experience and the effect of rare events in risky choice. Psychological Science, 8, 534-539.
10. Horton J. J., Rand D. G., & Zeckhauser R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14, 399-425.
11. Ipeirotis, P. (2010). Demographics of Mechanical Turk. (CeDER Working Paper-10-01). New York University. Retrieved from http://hdl.handle.net/2451/29585.
12. John, O. P., & Srivistava, S. (1999). The big five trait taxonomy: History, measurement, and theoretical perspectives. In L. A. Pervin & O. P. John (Eds.), Handbook of personality: Theory and research (pp. 102-138). New York/London: Guilford Press.
13. Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47, 263-292.
14. LeBoeuf, R. A., & Shafir, E. (2003). Deep thoughts and shallow frames: On the susceptibility to framing effects. Journal of Behavioral Decision Making, 16, 77-92.
15. Loewenstein, G. (1988). Frames of mind in intertemporal choice. Management Science, 34, 200-214.
16. Lynch Jr., J. G. (1982). On the external validity of experiments in consumer research. Journal of Consumer Research, 9, 225-239.
17. Malkoc, S. A., & Zauberman G. (2006). Deferring versus expediting consumption: The effect of outcome concreteness on sensitivity to time horizon. Journal of Marketing Research, 43, 618-627.
18. Mason, W. & Suri, S. (2012). Conducting behavioral research on Amazon's Mechanical Turk. Behavioral Research Methods, 44, 1-23.
19. Mazar, N., Amir, O., Ariely, D. (2008). The dishonesty of honest people: A theory of self-concept maintenance. Journal of Marketing Research, 45, 633-644.
20. Nenkov, G., Morrin, M., Ward, A., Schwartz, B., & Hulland J. (2008). A short form of the maximization scale: Factor structure, reliability, and validity studies. Judgment and Decision Making, 3, 371-388.
21. Oppenheimer, D., Meyvis T., & Davidenko N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867-872.
22. Oppenheimer, D. M., LeBoeuf, R. A., & Brewer, N. T. (2008). Anchors aweigh: A demonstration of cross-modality anchoring and magnitude priming. Cognition, 106, 13-26.
23. Paolacci, G., Chandler, J., & Ipeirotis, P. G. (2010). Running experiments on Amazon Mechanical Turk. Judgment and Decision Making, 5, 411-419.
24. Parker, A., & Fischoff, B. (2005). Decision-making competence: External validation through an individual-differences approach. Journal of Behavioral Decision Making, 18, 1-27.
25. Peterson, R. A. (2001). On the use of college students in social science research: Insights from a second-order meta-analysis. Journal of Consumer Research, 28, 450-461.
26. Rand, D. G. (2011). The promise of Mechanical Turk: How online labor markets can help theorists run behavioral experiments. Journal of Theoretical Biology, DOI: 10.1016/j.jtbi.2011.03.004.
27. Richins, M. L. (2004). The material values scale: Measurement properties and development of a short form. Journal of Consumer Research, 31, 209-219.
28. Rick, S. I., Cryder, C. E., & Loewenstein, G. (2008). Tightwads and spendthrifts. Journal of Consumer Research, 34, 767-782.
29. Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001). Measuring global self-esteem: Construct validation of a single-item measure and the Rosenberg Self-Esteem Scale. Personality and Social Psychology Bulletin, 27, 151-161.
30. Rosenthal, R., Rosnow, R. L., & Rubin, D. B. (2000). Contrasts and effect sizes in behavioral research: A correlational approach. Cambridge: Cambridge University Press.
31. Shu, L. L., Gino, F., & Bazerman, M. H. (2011). Dishonest deed, clear conscience: When cheating leads to moral disengagement and motivated forgetting. Personality and Social Psychology Bulletin, 37, 330-349.
32. Suri, S., & Watts, D. J. (2011). Cooperation and contagion in Web-based, networked public goods experiments. PLoS One, 6(3), 1-8.
33. Thaler, R. H. (1981). Some empirical evidence on dynamic inconsistency. Economic Letters, 8, 201-207.
34. Trzesniewski, K. H., Donnellan, M. B., & Robins, R. W. (2003). Stability of self-esteem across the life-span. Journal of Personality and Social Psychology, 84, 205-220.
35. Tversky, A., & Kahneman D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 211, 453-458.
36. Tversky, A., & Kahneman D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458.
37. Tversky, A., & Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of Uncertainty. Journal of Risk and Uncertainty, 5, 297-323.
38. Winer, R. S. (1999). Experimentation in the 21st century: The importance of external validity. Journal of the Academy of Marketing Science, 27, 349-358.