Are samples drawn from mechanical turk valid for research on political ideology?

Research and Politics

Volumn 2, Issue 4, 2015, Pages

  Clifford, Scott ^a   Jewell, Ryan M ^a   Waggoner, Philip D ^a  

^a UNIVERSITY OF HOUSTON   (United States)

Author keywords

External validity; Mechanical turk; Sample comparison

Indexed keywords

EID: 85043532872     PISSN: None     EISSN: 20531680     Source Type: Journal    
DOI: 10.1177/2053168015622072     Document Type: Article

References (46)

1. Ahler DJ (2014) Self-fulfilling misperceptions of public polarization. The Journal of Politics 76(03): 607–620.
2. Arceneaux K (2012) Cognitive biases and the strength of political arguments. American Journal of Political Science 56(2): 271–285.
3. Battaglini M, Morton RB and Palfrey T (2007) Efficiency, equity, and timing of voting mechanisms. American Political Science Review 101(03): 409–424.
4. Benjamini Y and Hochberg Y (1995) Controlling the False discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society 57(1): 289–300.
5. Benjamini Y and Yekutieli D (2001) The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics 29(4): 1165–1188.
6. Berinsky AJ, Huber GA and Lenz GS (2012) Evaluating online labor markets for experimental research: Amazon.com’s Mechanical Turk. Political Analysis 20(3): 351–368.
7. Brüggen E and Dholakia UM (2010) Determinants of Participation and response effort in web panel surveys. Journal of Interactive Marketing 24(3): 239–250.
8. Buhrmester M, Kwang T and Gosling SD (2011) Amazon’s Mechanical Turk: A new source of inexpensive, yet high-qual-ity, data? Perspectives on Psychological Science 6(1): 3–5.
9. Bullock JG (2011) Elite influence on public opinion in an informed electorate. American Political Science Review 105(03): 496–515.
10. Chandler J, Mueller P and Paolacci G (2014) Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods 46(1): 112–130.
11. Chong D and Druckman JN (2012) Counterframing effects. The Journal of Politics 75(1): 1–16.
12. Clifford S (2014) Linking Issue stances and trait inferences: A theory of moral exemplification. The Journal of Politics 76(3): 698–710.
13. Clifford S and Jerit J (2014) Is There a cost to convenience? An experimental comparison of data quality in laboratory and online studies. Journal of Experimental Political Science 1(2): 1–12.
14. Cohen J (1988) Statistical Power Analysis for the Behavioral Sciences. New Jersey: Lawrence Erlbaum Associates.
15. Costa PT and McCrae RR (1992) Four ways five factors are basic. Personality and Individual Differences 13(6): 653–665.
16. Crump MJC, McDonnell JV and Gureckis TM (2013) Evaluating Amazon’s Mechanical Turk as a tool for experimental behavioral research. PloS one 8(3): e57410.
17. Dollinger SJ and Leong FTL (1993) Volunteer bias and the five-factor model. The Journal of Psychology 127(1): 29–36.
18. Druckman JN, Peterson E and Slothuus R (2013) How elite partisan polarization affects public opinion formation. American Political Science Review 107(1): 57–79.
19. Feldman S and Johnston C (2014) Understanding the determinants of political ideology: Implications of structural complexity. Political Psychology 35(3): 337–358.
20. Feldman S and Stenner K (1997) Perceived Threat and authoritarianism. Political Psychology 18(4): 741–770.
21. Gerber AS, et al. (2013) Big Five personality traits and responses to persuasive appeals: Results from voter turnout experiments. Political Behavior 35(4): 687–728.
22. Gerber AS, et al. (2010) Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review 104(1): 111–133.
23. Goodman JK, Cryder CE and Cheema A (2013) Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making 26(3): 213–224.
24. Gosling SD, Rentfrow PJ and Swann WB (2003) A very brief measure of the Big-Five personality domains. Journal of Research in Personality 37(6): 504–528.
25. Grimmer J, Messing S and Westwood SJ (2012) How words and money cultivate a personal vote: the effect of legislator credit claiming on constituent credit allocation. American Political Science Review 106(4): 703–719.
26. Hainmueller J, Hangartner D and Yamamoto T (2015) Validating vignette and conjoint survey experiments against real-world behavior. Proceedings of the National Academy of Sciences of the United States of America 112(8): 2395–2400.
27. Hauser DJ and Schwarz N (forthcoming) Attentive Turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior research methods.
28. Hauser DJ and Schwarz N (2015) It’s a trap! Instructional Manipulation checks prompt systematic thinking on “tricky” tasks. SAGE Open 5(2): 1–6.
29. Henry PJ and Sears DO (2002) The Symbolic Racism 2000 Scale. Political Psychology 23(2): 253–283.
30. Horton JJ, Rand DG and Zeckhauser RJ (2011) The online laboratory: Conducting experiments in a real labor market. Experimental Economics 14(3): 399–425.
31. Huber GA and Paris C (2013) Assessing the Programmatic Equivalence Assumption in question wording experiments: Understanding why Americans like assistance to the poor more than welfare. Public Opinion Quarterly 77(1): 385–397.
32. Huff C and Tingley D (2015) “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics 2(3): 1–12.
33. Jerit J, Barabas J and Clifford S (2013) Comparing contemporaneous laboratory and field experiments on media effects. Public Opinion Quarterly 77(1): 256–282.
34. Johnston CD, Newman BJ and Velez Y (2015) Ethnic change, personality, and polarization over immigration in the American public. Public Opinion Quarterly 79(3): 662–686.
35. Kahan DM (2013) What’s a “valid” sample? Problems with Mechanical Turk study samples, part 1. Cultural Cognition Project Blog. Available at: http://www.culturalcognition. net/blog/2013/7/8/whats-a-valid-sample-problems-with-mechanical-turk-study-sam.html (accessed 3 May 2015).
36. Krupnikov Y and Levine AS (2014) Cross-sample comparisons and external validity. Journal of Experimental Political Science 1(1): 59–80.
37. McDermott R (2002) Experimental methodology in political science. Political Analysis 10(4): 325–342.
38. Mondak JJ (2010) Personality and the Foundations of Political Behavior. New York: Cambridge University Press.
39. Mondak JJ and Halperin KD (2008) A framework for the study of personality and political behaviour. British Journal of Political Science 38(2): 335–362.
40. Nyhan B and Reifler J (2010) When corrections fail: The persistence of political misperceptions. Political Behavior 32(2): 303–330.
41. Paolacci G, Chandler J and Ipeirotis PG (2010) Running experiments on Amazon Mechanical Turk. Judgment and Decision Making 5(5): 411–419.
42. Rogelberg SG, et al. (2003) Profiling active and passive nonre-spondents to an organizational survey. Journal of Applied Psychology 88(6): 1104–1114.
43. Scherer AM, Windschitl PD and Graham J (2014). An ideological house of mirrors political stereotypes as exaggerations of motivated social cognition differences. Social Psychological and Personality Science 6(2): 201–209.
44. Searles K and Ryan JB (2015) Researchers are rushing to Amazon’s Mechanical Turk. Should they? Washington Post. Available at: http://www.washingtonpost.com/blogs/monkey-cage/wp/2015/05/04/researchers-are-rushing-to-amazons-mechanical-turk-should-they/(accessed 4 May 2015).
45. Shapiro DN, Chandler J and Mueller PA (2013) Using Mechanical Turk to study clinical populations. Clinical Psychological Science 1(2): 213–220.
46. Weinberg JD, Freese J and McElhattan D (2014) Comparing data characteristics and results of an online factorial survey between a population-based and Crowdsource-recruited sample. Sociological Science 1: 292–310.