Standards for Experimental Research: Encouraging a Better Understanding of Experimental Methods

Journal of Experimental Political Science

Volumn 2, Issue 2, 2015, Pages 192-215

  Mutz, Diana C ^a   Pemantle, Robin ^a  

^a UNIVERSITY OF PENNSYLVANIA   (United States)

Author keywords

manipulation check; Randomization check; response rates; standards

Indexed keywords

EID: 84982835999     PISSN: 20522630     EISSN: 20522649     Source Type: Journal    
DOI: 10.1017/XPS.2015.4     Document Type: Article

References (44)

1. Abelson, R. 1995. Statistics as Principled Argument. Hillsdale, NJ: L. Erlbaum Associates.
2. Arceneaux, K. 2012. "Cognitive Biases and the Strength of Political Arguments." American Journal of Political Science 56(2): 271-85.
3. Arceneaux, K. and R. Kolodny. 2009. "Educating the Least Informed: Group Endorsements in a Grassroots Campaign." American Journal of Political Science 53(4): 755-70.
4. Barabas, J., W. Pollock and J. Wachtel. 2011. "Informed Consent: Roll-Call Knowledge, the Mass Media, and Political Representation." Paper Presented at the Annual Meeting of the American Political Science Association, Seattle, WA, Sept. 1-4. (http://www.jasonbarabas.com/images/BarabasPollockWachtel-RewardingRepresentation.pdf), accessed September 1, 2014.
5. Berinsky, A. J., M. F. Margolis and M. W. Sances. 2014. "Separating the Shirkers from the Wokers? Making Sure Respondents Pay Attention on Self-Administered Surveys." American Journal of Political Science 58(3): 739-53.
6. Berinsky, A. J. and T. Mendelberg. 2005. "The Indirect Effects of Discredited Stereotypes in Judgments of Jewish Leaders." American Journal of Political Science 49(4): 845-64.
7. Boers, M. 2011. "In randomized Trials, Statistical Tests are not Helpful to Study Prognostic (im)balance at Baseline." Lett Ed Rheumatol 1(1): e110002. doi:10.2399/ler.11.0002.
8. Butler, D. M. and D. E. Broockman. 2011. "Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators." American Journal of Political Science 55(3): 463-77.
9. Cover, A. D. and B. S. Brumberg. 1982. "Baby Books and Ballots: The Impact of Congressional Mail on Constituent Opinion." The American Political Science Review 76(2): 347-59.
10. Cozby, P. C. 2009. Methods of Behavioral Research. (10th ed.). New York, NY:McGraw-Hill.
11. Fisher, R. A. 1935. The Design of Experiments. London: Oliver and Boyd.
12. Foschi, M. 2007. "Hypotheses, Operationalizations, and Manipulation Checks." Chapter 5, In Laboratory Experiment in the Social Sciences, eds. M. Webster and J. Sell, (pp.113-140). New York: Elsevier.
13. Franco, A., N. Malhotra and G. Simonovits. 2014. "Publication Bias in the Social Sciences: Unlocking the File Drawer." Science 345(6203): 1502-5.
14. Franklin, C. 1991. "Efficient Estimation in Experiments." Political Methodologist 4(1): 13-15.
15. Gerber, A., K. Arceneaux, C. Boudreau, C. Dowling, S. Hillygus, T. Palfrey, D. R. Biggers and D. J. Hendry. 2014. "Reporting Guidelines for Experimental Research: A Report from the Experimental Research Section Standards Committee." Journal of Experimental Political Science 1(1): 81-98.
16. Gosnell, H. F. 1942. Grass Roots Politics. Washington, DC: American Council on Public Affairs.
17. Green, D. P. and A. S. Gerber. 2002. "Reclaiming the Experimental Tradition in Political Science." In Political Science: The State of the Discipline, 3rd Edition. eds. H. V. Milner and I. Katznelson, (pp.805-32). New York: W.W. Norton & Co.
18. Harbridge, L., N. Malhotra and B. F. Harrison. 2014. "Public Preferences for Bipartisanship in the Policymaking Process." Legislative Studies Quarterly 39(3): 327-55.
19. Hiscox, M. J. 2006. "Through a Glass and Darkly: Attitudes Toward International Trade and the Curious Effects of Issue Framing." International Organization 60(3): 755-80.
20. Hutchings, V. L., N. A. Valentino, T. S. Philpot and I. K. White. 2004. "The Compassion Strategy: Race and the Gender Gap in Campaign 2000." Public Opinion Quarterly 68(4): 512-41.
21. Hyde, S. D. 2010. "Experimenting in Democracy Promotion: International Observers and the 2004 Presidential Elections in Indonesia." Perspectives on Politics 8(2): 511-27.
22. Imai, K., G. King and E. Stuart. 2008. "Misunderstandings between Experimentalists and Observationalists about Causal Inference." Journal of the Royal Statistical Society, Series A, 171(2): 481-502.
23. Ioannidis, J. P. A. 2005. "Why Most Published Research Findings Are False." PLoS Medicine 2(8): e124. doi:10.1371/journal.pmed.0020124.
24. Kessler, J. B. and S. Meier. 2014. "Learning from (Failed) Replications: Cognitive Load Manipulations and Charitable Giving." Journal of Economic Behavior and Organization 102(June): 10-13.
25. Ladd, J. M. 2010. "The Neglected Power of Elite Opinion Leadership to Produce Antipathy Toward the News Media: Evidence from a Survey Experiment." Political Behavior 32(1): 29-50.
26. Malhotra, N. and E. Popp. 2012. "Bridging Partisan Divisions over Antiterrorism Policies: The Role of Threat Perceptions." Political Research Quarterly 65(1): 34-47.
27. Michelbach, P. A., J. T. Scott, R. E. Matland and B. H. Bornstein. 2003. "Doing Rawls Justice: An Experimental Study of Income Distribution Norms." American Journal of Political Science 47(3): 523-39.
28. Moher, D., S. Hopewell, K. F. Schulz, V. Montori, P. C. Gøtzsche, P. J. Devereaux, D. Elbourne, M. Egger and D. G. Altman. CONSORT. 2010. "Explanation and Elaboration: Updated Guidelines for Reporting Parallel Group Randomised Trials." Journal of Clinical Epidemiology 63(8): e1-e37.
29. Morgan, K. and D. Rubin. 2012. "Rerandomization to improve covariate balance in experiments." Annals of Statistics 40(2): 1263-82.
30. Mutz, D. C. and R. Pemantle. 2011. "The Perils of Randomization Checks in the Analysis of Experiments." Paper presented at the Annual Meetings of the Society for Political Methodology, (July 28-30). (http://www.math.upenn.edu/-pemantle/papers/Preprints/perils.pdf), accessed September 1, 2014.
31. Newsted, P. R., P. Todd and R. W. Zmud. 1997. "Measurement Issues in the Study of Human Factors in Management Information Systems." Chapter 16, In Human Factors in Management Information System, ed. J. Carey, (pp.211-242). New York, USA: Ablex.
32. Perdue, B. C. and J. O. Summers. 1986. "Checking the Success of Manipulations in Marketing Experiments." Journal of Marketing Research 23(4): 317-26.
33. Permutt, T. 1990. "Testing for Imbalance of Covariates in Controlled Experiments." Statistics in Medicine 9(12): 1455-62.
34. Prior, M. 2009. "Improving Media Effects Research through Better Measurement of News Exposure." Journal of Politics 71(3): 893-908.
35. Sances, M. W. 2012. "Is Money in Politics Harming Trust in Government? Evidence from Two Survey Experiments." (http://www.tessexperiments.org/data/SancesSSRN.pdf), accessed January 20, 2015.
36. Sansone, C., C. C. Morf and A. T. Panter. 2008. The Sage Handbook of Methods in Social Psychology. Thousand Oaks, CA: Sage Publications.
37. Saperstein, A. and A. M. Penner. 2012. "Racial Fluidity and Inequality in the United States." American Journal of Sociology 118(3): 676-727.
38. Scherer, N. and B. Curry. 2010. "Does Descriptive Race Representation Enhance Institutional legitimacy? The Case of the U.S. Courts." Journal of Politics 72(1): 90-104.
39. Schulz, K. F., D. G. Altman, D. Moher, for the CONSORT Group. CONSORT 2010. "Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials." British Medical Journal 340: c332.
40. Senn, S. 1994. "Testing for Baseline Balance in Clinical Trials." Statistics in Medicine 13: 1715-26.
41. Senn, S. 2013. "Seven Myths of Randomisation in Clinical Trials." Statistics in Medicine 32(9): 1439-50. doi: 10.1002/sim.5713. Epub 2012 Dec 17.
42. Statisticalmisses.nl, 2013. (http://www.statisticalmisses.nl/index.php/frequently-askedquestions/84-why-are-significance-tests-of-baseline-differences-a-very-bad-idea), accessed January 21, 2015. As attributed to Senn (2013).
43. Thye, S. 2007. "Logic and Philosophical Foundations of Experimental Research in the Social Sciences." Chapter 3, In Laboratory Experiments in the Social Sciences, (pp.57-86). Burlington, MA: Academic Press.
44. Valentino, N. A., V. L. Hutchings and I. K. White. 2002. "Cues That Matter: How Political Ads Prime Racial Attitudes during Campaigns." The American Political Science Review 96(1): 75-90.