An Analysis of Data Quality: Professional Panels, Student Subject Pools, and Amazon’s Mechanical Turk

Journal of Advertising

Volumn 46, Issue 1, 2017, Pages 141-155

  Kees, Jeremy ^a   Berry, Christopher ^b   Burton, Scot ^b   Sheehan, Kim ^c  

^a VILLANOVA UNIVERSITY   (United States)

^b UNIVERSITY OF ARKANSAS   (United States)

^c UNIVERSITY OF OREGON   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 85010705185     PISSN: 00913367     EISSN: None     Source Type: Journal    
DOI: 10.1080/00913367.2016.1269304     Document Type: Article

References (48)

1. Amir, Ofra, David G., Rand, and Ya'akov, Kobi Gal (2012), “Economic Games on the Internet:The Effect of $1 Stakes,” PLOS ONE, 7 (2), e31461.
2. Bartneck, Christoph, Andreas, Duenser, Elena, Moltchanova, and Karolina, Zawieska (2015), “Comparing the Similarity of Responses Received from Studies in Amazon's Mechanical Turk to Studies Conducted Online and with Direct Recruitment,” PLOS ONE, 10 (4), e0121595.
3. Berinsky, Adam J., Gregory A., Huber, and Gabriel, Lenz (2012), “Evaluating Online Labor Markets for Experimental Research:Amazon.com's Mechanical Turk,” Political Analysis, 20 (3), 351–68.
4. Buhrmester, Michael, Tracy, Kwang, and Samuel D., Gosling (2011), “Amazon's Mechanical Turk:A New Source of Inexpensive, Yet High-Quality, Data?,” PLOS ONE, 6 (1), 3–5.
5. Casler, Krista, Lydia, Bickel, and Elizabeth, Hackett (2013), “Separate but Equal? A Comparison of Participants and Data Gathered via Amazon's MTurk, Social Media, and Face-to-Face Behavioral Testing,” Computers in Human Behavior, 29 (6), 2156–60.
6. Chandran, Sucharita, and Geeta, Menon (2004), “When a Day Means More than a Year:Effects of Temporal Framing on Judgments of Health Risk,” Journal of Consumer Research, 31 (2), 375–89.
7. Clifford, Scott, Ryan M., Jewell, and Philip D., Waggoner (2015), “Are Samples Drawn from Mechanical Turk Valid for Research on Political Ideology?,” Research and Politics, 2 (4), 1–9.
8. Crump, Matthew J.C., John V., McDonnell, and Todd M., Gureckis (2013), “Evaluating Amazon's Mechanical Turk as a Tool for Experimental Behavioral Research,” PLOS ONE, 8 (3), e57410.
9. Feldt, Leonard S., (1969), “A Test of the Hypothesis That Cronbach's Alpha or Kuder-Richardson Coefficient Twenty is the Same for Two Tests,” Psychometrika, 34 (3), 363–73.
10. Ferguson, Nakeisha S., and Janee N., Burkhalter (2015), “Yo, DJ, That's My Brand:An Examination of Consumer Response to Brand Placements in Hip-Hop Music,” Journal of Advertising, 44 (1), 47–57.
11. Goodman, Joseph K., Cynthia E., Cryder, and Amar, Cheema (2013), “Data Collection in a Flat World:The Strengths and Weaknesses of Mechanical Turk Samples,” Journal of Behavioral Decision Making, 26 (3), 213–24.
12. Hamby, Tyler, and Wyn, Taylor (2016), “Survey Satisficing Inflates Reliability and Validity Measures:An Experimental Comparison of College and Amazon Mechanical Turk Samples,” Educational and Psychological Measurement, 76 (6), 912–32.
13. Hauser, David J., and Norbert, Schwarz (2016), “Attentive Turkers:MTurk Participants Perform Better on Online Attention Checks Than Subject Pool Participants,” Behavior Research Methods, 48 (1), 400–407.
14. Heen, M.S.J., Joel D., Lieberman, and Terence D., Miethe (2014), A Comparison of Different Online Sampling Approaches for Generating National Samples (Report CCJP 2014–01). Las Vegas:University of Nevada, Las Vegas, Center for Crime and Justice Policy, http://www.unlv.edu/sites/default/files/page_files/27/ComparisonDifferentOnlineSampling.pdf
15. Horton, John J., David G., Rand, and Richard J., Zeckhauser (2011), “The Online Laboratory:Conducting Experiments in a Real Labor Market,” Experimental Economics, 14, 399–425.
16. Huff, Connor, and Dustin, Tingley (2015), “‘Who Are These People?’ Evaluating the Demographic Characteristics and Political Preferences of MTurk Survey Respondents,” Research and Politics, 2 (3), 1–12.
17. Iacobucci, Dawn, Steven S., Posavac, Frank R., Kardes, Matthew J., Schneider, and Deidre L., Popovich (2015), “Toward a More Nuanced Understanding of the Statistical Properties of a Median Split,” Journal of Consumer Psychology, 25 (4), 652–65.
18. Joireman, Jeff A., Alan, Strathman, and Daniel, Balliet (2006), “Considering Future Consequences:An Integrative Model,” in Judgments Over Time:The Interplay of Thoughts, Feelings, and Behaviors, Lawrence J., Sanna and Edward, Chin-Ho Chang, eds., Oxford:Oxford University Press, 82–99.
19. Kareklas, Ioannis, Darrel D., Muehling, and T.J., Weber (2015), “Reexamining Health Messages in the Digital Age:A Fresh Look at Source Credibility Effects,” Journal of Advertising, 44 (2), 88–104.
20. Kees, Jeremy, Scot, Burton, and Andrea, Heintz Tangari (2010), “The Impact of Regulatory Focus, Temporal Orientation, and Fit on Consumer Responses to Health-Related Advertising,” Journal of Advertising, 39 (1), 19–34.
21. Kulkarni, Atul A., and Hong, Yuan (2015), “Effect of Ad-Irrelevant Distance Cues on Persuasiveness of Message Framing,” Journal of Advertising, 44 (3), 254–63.
22. Kung, Franki Y.H., Young-Hoon, Kim, Daniel Y.-J., Yang, and Shirley Y.Y., Cheng (2016), “The Role of Regulatory Fit in Framing Effective Negative Feedback across Cultures,” Journal of Cross-Cultural Psychology, 47 (5), 696–712.
23. Landers, Richard N., and Tara S., Behrend (2015), “An Inconvenient Truth:Arbitrary Distinctions between Organizational, Mechanical Turk, and Other Convenience Samples,” Industrial and Organizational Psychology, 8 (2), 142–64.
24. Lee, Angela Y., and Jennifer L., Aaker (2004), “Bringing the Frame into Focus:The Influence of Regulatory Fit on Processing Fluency and Persuasion,” Journal of Personality and Social Psychology, 86 (2), 205–218.
25. Leonhardt, James M., Jesse R., Catlin, and Dante M., Pirouz (2015), “Is Your Product Facing the Ad's Center? Facing Direction Affects Processing Fluency and Ad Evaluation,” Journal of Advertising, 44 (4), 315–25.
26. Marvit, Moshe Z., (2014), “How Crowdworkers Become Ghosts in the Digital Machine,” The Nation, February 5, http://www.thenation.com/article/178241/how-crowdworkers-became-ghosts-digital-machine
27. Maynard, Andrew, (2014), “Has Anyone Heard of Bisphenol A,” 2020 Science, December 1, http://2020science.org/2014/12/01/anyone-heard-bpa/
28. Minton, Elizabeth, Christopher, Lee, Ulrich, Orth, Chung-Hyun, Kim, and Lynn, Kahle (2012), “Sustainable Marketing and Social Media:A Cross-Country Analysis of Motives for Sustainable Behaviors,” Journal of Advertising, 41 (4), 69–84.
29. Mohanty, Praggyan, and S., Ratneshwar (2015), “Did You Get It? Factors Influencing Subjective Comprehension of Visual Metaphors in Advertising,” Journal of Advertising, 44 (3), 232–42.
30. Muehling, Darrel D., David E., Sprott, and Abdullah J., Sultan (2014), “Exploring the Boundaries of Nostalgic Advertising Effects:A Consideration of Childhood Brand Exposure and Attachment on Consumers' Responses to Nostalgia-Themed Advertisements,” Journal of Advertising, 43 (1), 73–84.
31. Newton, Joshua D., Jimmy, Wong, and Fiona J., Newton (2015), “The Social Status of Health Message Endorsers Influences the Health Intentions of the Powerless,” Journal of Advertising, 44 (2), 151–60.
32. Oppenheimer, Daniel M., Tom, Meyvis, and Nicolas, Davidenko (2009), “Instructional Manipulation Checks:Detecting Satisficing to Increase Statistical Power,” Journal of Experimental Social Psychology, 45 (4), 867–72.
33. Paolacci, Gabriele, Jesse, Chandler, and Panagiotis G., Ipeirotis (2010), “Running Experiments on Amazon Mechanical Turk,” Judgment and Decision Making, 5 (5), 411–19.
34. Peer, Eyal, Sonam, Samat, Laura, Brandimarte, and Alessandro, Acquisti (2015), “Beyond the Turk:An Empirical Comparison of Alternative Platforms for Crowdsourcing Online Behavioral Research,” SSRN, April 14, http://ssrn.com/abstract=2594183
35. ———, Joachim Vosgerau, and Alessandro, Acquisti (2014), “Reputation as a Sufficient Condition for Data Quality on Amazon Mechanical Turk,” Behavior Research Methods, 46 (4), 1023–31.
36. Peterson, Robert A., and Dwight R., Merunka (2014), “Convenience Samples of College Students and Research Reproducibility,” Journal of Business Research, 67 (5), 1035–41.
37. Pounders, Kathrynn R., Seungae, Lee, and Mike, Mackert (2015), “Matching Temporal Frame, Self-View, and Message Frame Valence:Improving Persuasiveness in Health Communications,” Journal of Advertising, 44 (4), 388–402.
38. Qualtrics. (2016). Q Support:Survey Protection. www.qualtrics.com/support/survey-platform/survey-module/survey-options/survey-protection/
39. Roulin, Nicolas, (2015), “Don't Throw the Baby Out with the Bathwater:Comparing Data Quality of Crowdsourcing, Online Panels, and Student Samples,” Industrial and Organizational Psychology, 8 (2), 190–96.
40. Sackett, Paul R., and James R., Larson (1990), “Research Strategies and Tactics in Industrial and Organizational Psychology,” in Handbook of Industrial and Organizational Psychology, Marvin D., Dunnette and Leanetta M., Hough, eds., Palo Alto, CA:Consulting Psychologists Press, 419–89.
41. Schuman, Howard, and Graham, Kalton (1985), “Survey Methods,” in Handbook of Social Psychology, Vol. 1, Gardner, Lindzey and Elliot, Aronson, eds., New York:Random House, 635–97.
42. Sheehan, Kim, and Matthew, Pittman (2016), The Academic Researcher's Guide to Mechanical Turk, Irvine, CA:Melvin and Leigh.
43. Simons, Daniel J., and Christopher F., Chabris (2012), “Common (Mis) Beliefs about Memory:A Replication and Comparison of Telephone and Mechanical Turk Survey Methods,” PLOS ONE, 7 (12), e51876.
44. Smith, Scott M., Catherine A., Roster, Linda L., Golden, and Gerald S., Albaum (2016), “A Multi-Group Analysis of Online Survey Respondent Data Quality:Comparing a Regular USA Consumer Panel to MTurk Samples,” Journal of Business Research, 69 (8), 3139–48.
45. Sprouse, Jon, (2011), “A Validation of Amazon Mechanical Turk for the Collection of Acceptability Judgments in Linguistic Theory,” Behavior Research Methods, 43 (1), 155–67.
46. Strathman, Alan, Faith, Gleicher, David S., Boninger, and Scott C., Edwards (1994), “The Consideration of Future Consequences:Weighing Immediate and Distant Outcomes of Behavior,” Journal of Personality and Social Psychology, 66 (4), 742–52.
47. Suri, Siddharth, and Duncan J., Watts (2011), “Cooperation and Contagion in Web-Based, Networked Public Goods Experiments,” PLOS ONE, 6 (3), e16836.
48. TurkPrime. (2016), “Run Large Surveys as Multiple Time-Released Smaller HITs,” http://www.turkprime.com/Home/MicroBatch