The Generalizability of Survey Experiments

Journal of Experimental Political Science

Volumn 2, Issue 2, 2015, Pages 109-138

  Mullinix, Kevin J ^a   Leeper, Thomas J ^b   Druckman, James N ^c   Freese, Jeremy ^c  

^a APPALACHIAN STATE UNIVERSITY   (United States)

^b LONDON SCHOOL OF ECONOMICS   (United Kingdom)

^c NORTHWESTERN UNIVERSITY   (United States)

Author keywords

causal inference; sampling; Survey experiments

Indexed keywords

EID: 85012122035     PISSN: 20522630     EISSN: 20522649     Source Type: Journal    
DOI: 10.1017/XPS.2015.19     Document Type: Article

References (78)

1. Ahler, Douglas J. 2014. "Self-Fulfilling Misperceptions of Public Polarization." The Journal of Politics 76(3): 607-20.
2. Baker, Reg. et al. 2010. "Research Synthesis: AAPOR Report on Online Panels." Public Opinion Quarterly 74(4): 1-71.
3. Barabas, Jason and Jennifer Jerit. 2010. "Are Survey Experiments Externally Valid?" American Political Science Review 104 (May): 226-242.
4. Benoit, Kenneth, Drew Conway, Benjamin E. Lauderdale, Michael Laver, and Slava Mikhaylov. 2015. "Crowd-Sourced Text Analysis: Reproducible and Agile Production of Political Data." American Political Science Review: Forthcoming.
5. Berger, Arthur Asa. 2014. Media and Communications Research Methods: An Introduction to Qualitative and Quantitative Approaches. Los Angeles: Sage Publication, Inc.
6. Berinsky, Adam J., Gregory A. Huber, and Gabriel S. Lenz. 2012. "Evaluating Online Labor Markets for Experimental Research: Amazon.com's Mechanical Turk." Political Analysis 20 (Summer): 351-68.
7. Berinsky, Adam J., Michele F. Margolis, and Michael W. Sances. 2014. "Separating the Shirkers from the Workers? Making Sure Respondents Pay Attention on Self-Administered Surveys." American Journal of Political Science 58(3): 739-53.
8. Bloom, Howard S. 2005. Learning More from Social Experiments. New York: Russell Sage Foundation.
9. Bohannon, John. 2011. "Social Science for Pennies." Science 334 (October): 307.
10. Brady, Henry E. 2000. "Contributions of Survey Research to Political Science." PS: Political Science & Politics 33(1): 47-57.
11. Broockman, David E. and Donald P. Green. 2013. "Do Online Advertisements Increase Political Candidates' Name Recognition or Favorability? Evidence from Randomized Field Experiments." Political Behavior 36(2): 263-89.
12. Callegaro, Mario, Reg Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, and Paul J. Lavrakas. 2014. "Online Panel Research: History, Concepts, Applications, and a Look at the Future." In Online Panel Research: A Data Quality Perspective, eds. Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, and Paul J. Lavrakas. West Sussex, United Kingdom: John Wiley & Sons Ltd.
13. Campbell, Donald T. 1969. "Prospective: Artifact and Control." In Artifact in Behavioral Research, eds. Robert Rosenthal and Robert Rosnow. New York: Academic Press.
14. Cassese, Erin C., Leonie Huddy, Todd K. Hartman, Liliana Mason, and Christopher R. Weber. 2013. "Socially Mediated Internet Surveys: Recruiting Participants for Online Experiments." PS: Political Science and Politics 46(4): 1-10.
15. Chandler, Jesse, Pam Mueller, and Gabriele Paolacci. 2014. "Nonnaiveté Among Amazon Mechanical Turk Workers: Consequences and Solution for Behavioral Researchers." Behavior Research Methods 46(1): 112-30.
16. Chong, Dennis and James N. Druckman. 2007a. "Framing Public Opinion in Competitive Democracies." American Political Science Review 101(4): 637-55.
17. Chong, Dennis and James N. Druckman. 2007b. "Framing Theory." Annual Review of Political Science 10(1): 103-26.
18. Clifford, Scott and Jennifer Jerit. 2015. "Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies." Journal of Experimental Political Science 1(2): 120-31.
19. Coppock, Alexander and Donald P. Green. 2015. "Assessing the Correspondence Between Experimental Results Obtained in the Lab and Field: A Review of Recent Social Science Research." Political Science Research and Methods 3(1): 113-31.
20. Druckman, James N. 2001. "The Implications of Framing Effects for Citizen Competence." Political Behavior 23(3): 225-56.
21. Druckman, James N. 2004. "Priming the Vote: Campaign Effects in a US Senate Election." Political Psychology 25: 577-94.
22. Druckman, James N. and Arthur Lupia. 2012. "Experimenting with Politics." Science 335 (March): 1177-79.
23. Druckman, James N. and Cindy D. Kam. 2011. "Students as Experimental Participants: A Defense of the 'Narrow Data Base'." In Cambridge Handbook of Experimental Political Science, eds. J. N. Druckman, D. P. Green, J. H. Kuklinski, and A. Lupia. New York: Cambridge University Press, 41-57.
24. Druckman, James N., Donald P. Green, James H. Kuklinski, and Arthur Lupia. 2006. "The Growth and Development of Experimental Research in Political Science." American Political Science Review 100 (November): 627-35.
25. Druckman, James N., Erik Peterson, and Rune Slothuus. 2013. "How Elite Partisan Polarization Affects Public Opinion Formation." American Political Science Review 107(1): 57-79.
26. Dynamo. 2014. "Guidelines for Academic Requesters." Retrieved 6 October 2015 from (http://wiki.wearedynamo.org/index.php/Guidelines-for-Academic-Requesters), Accessed October 6, 2015.
27. Egami, Naoki and Kosuke Imai. 2015. "Causal Interaction in High-Dimension." Working paper.
28. Entman, Robert M. 1993. "Framing: Toward Clarification of a Fractured Paradigm." Journal of Communication 43(4): 51-58.
29. Fowler, Anthony and Michele Margolis. 2014. "The Political Consequences of Uninformed Voters." Electoral Studies 34: 100-10.
30. Franco, Annie, Neil Malhotra, and Gabor Simonovits. 2014. "Publication Bias in the Social Sciences: Unlocking the File Drawer." Science 345 (August): 1502-5.
31. Freese, Jeremy, Adam Howat, Kevin J. Mullinix, and James N. Druckman. 2015. "Limitations of Screening Methods to Obtain Representative Samples Using Online Labor Markets." Working Paper, Northwestern University.
32. Gamson, William A. and Andre Modigiliani. 1989. "Media Discourse and Public Opinion on Nuclear Power: A Constructionist Approach." American Journal of Sociology 95(1): 1-37.
33. Gelman, Andrew and Hal Stern. 2006. "The Difference Between 'Significant' and 'Not Significant' is not Itself Statistically Significant." The American Statistician 60(4): 328-31.
34. Gerber, Alan S. and Donald P. Green. 2008. "Field Experiments and Natural Experiments." In Oxford Handbook of Political Methodology, eds. J. M. Box-Steffensmeier, H. E. Brady, and D. Collier. New York: Oxford University Press, 357-81.
35. Gerber, Alan S. and Donald P. Green. 2011. Field Experiments: Design, Analysis, and Interpretation. New York: W.W. Norton & Company.
36. Gerring, John. 2012. Social Science Methodology: A Unified Framework. New York: Cambridge University Press.
37. GfK. 2013. "Knowledge Panel Design Summary." Available at: https://www.gfk.com/Documents/GfK-KnowledgePanel.pdf. Last accessed 20 November 2015.
38. Goodman, Joseph K., Cynthia E. Cryder, and Amar Cheema. 2012. "Data Collection in a Flat World: The Strengths and Weaknesses of Mechanical Turk Samples." Journal of Behavioral Decision Making 26: 213-24.
39. Green, Donald P. and Holger L. Kern. 2012. "Modeling Heterogeneous Treatment Effects in Survey Experiments with Bayesian Additive Regression Trees." Public Opinion Quarterly 76(3): 491-511.
40. Henrich, Joseph, Steven J. Heine, and Ara Norenzayan. 2010. "The Weirdest People in the World?" Behavioral and Brain Sciences 33 (April): 61-83.
41. Hillygus, D. Sunshine, Natalie Jackson, and McKenzie Young. 2014. "Professional Respondents in Nonprobability Online Panels." In Online Panel Research: A Data Quality Perspetive, eds. Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, and Paul J. Lavrakas. West Sussex, United Kingdom: John Wiley & Sons Ltd.
42. Holt, Charles A. 2006. Markets, Games, and Strategic Behavior: Recipes for Interactive Learning. Reading, MA: Addison-Wesley.
43. Horton, John J., David G. Rand, and Richard J. Zeckhauser. 2011. "The Online Laboratory: Conducting Experiments in a Real Labor Market." Experimental Economics 14(3): 399-425.
44. Hovland, Carl I. 1959. "Reconciling Conflicting Results Derived from Experimental and Survey Studies of Attitude Change." The American Psychologist 14: 8-17.
45. Huber, Gregory A., Seth J. Hill, and Gabriel S. Lenz. 2012. "Sources of Bias in Retrospective Decision-Making: Experimental Evidence of Voters' Limitations in Controlling Incumbents." American Political Science Review 106(4): 720-41.
46. Huff, Connor and Dustin Tingley. 2015. "'Who are these people?' Evaluating the demographic characteristics and political preferences of MTurk survey respondents." Research & Politics 2(3): 1-11. DOI: 10.1177/2053168015604648.
47. Iyengar, Shanto. 1991. Is Anyone Responsible? How Television Frames Political Issues. Chicago, IL: The University of Chicago Press.
48. Jerit, Jennifer, Jason Barabas, and Scott Clifford. 2013. "Comparing Contemporaneous Laboratory and Field Experiments on Media Effects." Public Opinion Quarterly 77(1): 256-82.
49. Kam, Cindy D., Jennifer R. Wilking, and Elizabeth J. Zechmeister. 2007. "Beyond the 'Narrow Data Base': Another Convenience Sample for Experimental Research." Political Behavior 29(4): 415-40.
50. Keeter, Scott, Courtney Kennedy, Michael Dimock, Jonathan Best and Peyton Craighill. 2006. "Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey." Public Opinion Quarterly 70(5): 759-79.
51. Klar, Samara. 2013. "The Influence of Competing Identity Primes on Political Preferences." Journal of Politics 75(4): 1108-24.
52. Klar, Samara, Joshua Robison, and James N. Druckman. 2013. "Political Dynamics of Framing." In New Directions in Media and Politics, ed. Travis N. Ridout. New York: Routledge, 173-192.
53. Klein, Richard A. et al. 2014. "Investigating Variation in Replicability: A 'Many Labs' Replication Project." Social Psychology 45: 142-52.
54. Kraft, Peter. 2008. "Curses-Winner's and Otherwise-in Genetic Epidemiology." Epidemiology 19 (September): 649-51.
55. Kriss, Peter H. and Roberto Weber. 2013. "Organizational Formation and Change: Lessons from Economic Laboratory Experiments." In Handbook of Economic Organization: Integrating Economic and Organizational Theory, ed. A. Grandori. Northampton: Edward Elgar Publishing Limited, 245-72.
56. Krupnikov, Yanna and Adam Seth Levine. 2014. "Cross-Sample Comparisons and External Validity." Journal of Experimental Political Science 1 (Spring): 59-80.
57. Lupia, Arthur. 2014. "The 2013 Ithiel de Sola Pool Lecture: What is the Value of Social Science? Challengers for Researchers and Government Funders." PS: Political Science & Politics 47 (January): 1-7.
58. Malhotra, Neil and Alexander G. Kuo. 2008. "Attributing Blame: The Public's Response to Hurricane Katrina." The Journal of Politics 70(1): 120-35.
59. McDermott, Rose. 2002. "Experimental Methodology in Political Science." Political Analysis, 10: 325-42.
60. Morawski, Jill G. 1988. The Rise of Experimentation in American Psychology. New Haven: Yale University Press.
61. Mutz, Diana C. 2011. Population-Based Survey Experiments. Princeton: Princeton University Press.
62. Nelson, Thomas E., Rosalee A. Clawson, and Zoe M. Oxley. 1997. "Media Framing of a Civil Liberties Conflict and Its Effect on Tolerance." American Political Science Review 91(3): 567-83.
63. Nock, Steven L. and Thomas M. Guterbock. 2010. "Survey experiments." In Handbook of Survey Research, eds. P. V. Marsden, and J. D. Wright. Emerald, UK, 837-64.
64. Open Science Collaboration. 2015. "Estimating the Reproducibility of Psychological Science." Science 349: 943.
65. Paolacci, Gabriele, Jesse Chandler, and Panagiotis G. Ipeirotis. 2010. "Running Experiments on Amazon Mechanical Turk." Judgment and Decision Making, 5(August): 411-9.
66. Pew. 2012. Assessing the Representativeness of Public Opinion Surveys. Available at: http://www.people-press.org/2012/05/15/assessing-the-representativeness-of-publicopinion-surveys/. Last accessed 20 November 2015.
67. Rand, David G., Alexander Peysakhovich, Gordon T. Kraft-Todd, George E. Newman, Owen Wurzbacher, Martin A. Nowak, and Joshua D. Greene. 2014. "Social Heuristics Shape Intuitive Cooperation." Nature Communications 5: 1-12.
68. Redlawsk, David P., Andrew J. Civettini, and Karen M. Emmerson. 2010. "The Affective Tipping Point: Do Motivated Reasoners Ever 'Get It'?" Political Psychology 31: 563-593.
69. Riker, William H. 1996. The Strategy of Rhetoric: Campaigning for the American Constitution. New Haven, CT: Yale University Press.
70. Ross, Joel, Lily Irani, M. Six Silberman, Andrew Zaldivar, and Bill Tomlinson. 2010. "Who are the Crowdworkers? Shifting Demographics in Amazon Mechanical Turk. In CHI EA 2010, New York: ACM Press, 2863-72.
71. Sears, David O. 1986. "College Sophomores in the Laboratory: Influences of a Narrow Data Base on Social Psychology's View of Human Nature." Journal of Personality and Social Psychology 51: 515-530.
72. Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2001. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton-Mifflin.
73. Sniderman, Paul. 2011. The Logic and Design of the Survey Experiment: An Autobiography of a Methodological Innovation." In Cambridge Handbook of Experimental Political Science, eds. J. N. Druckman, D. P. Green, J. H. Kuklinski, and A. Lupia. New York: Cambridge University Press, 102-14.
74. Steinmetz, Stephanie, Annamaria Bianchi, Kea Tijdens, and Silvia Biffignandi. 2014. "Improving Web Surveys Quality: Potentials and Constraints of Propensity Score Adjustments." In Online Panel Research: A Data Quality Perspetive, eds. Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, and Paul J. Lavrakas. West Sussex, United Kingdom: John Wiley & Sons Ltd.
75. Valentino, Nicholas A., Michael W. Traugott, and Vincent L. Hutchings. 2002. "Group Cues and Ideological Constraint: A Replication of Political Advertising Effects Studies in the Lab and in the Field." Political Communicaton 19(1): 29-48.
76. Wang, Wei, David Rothschild, Sharad Goel, and Andrew Gelman. 2015. "Forecasting Elections with Non-representative Polls." International Journal of Forecasting 31(3): 980-991.
77. Weinberg, Jill D., Jeremy Freese, and David McElhattan. 2014. "Comparing Demographics, Data Quality, and Results of an Online Factorial Survey Between a Population-Based and a Crowdsource-Recruited Sample." Sociological Science 1: 292-310.
78. Wright, James D., and Peter V Marsden. 2010. "Survey Research and Social Science: History, Current Practice, and Future Prospects." In Handbook of Survey Research, eds. P. V. Marsden, and J. D. Wright. Emerald, UK, 3-26.