Testing What Matters (If You Must Test at All): A Context-Driven Approach to Substantive and Statistical Significance

American Journal of Political Science

Volumn 59, Issue 3, 2015, Pages 775-788

  Gross, Justin H ^a  

^a UNIVERSITY OF NORTH CAROLINA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84933566984     PISSN: 00925853     EISSN: 15405907     Source Type: Journal    
DOI: 10.1111/ajps.12149     Document Type: Article

References (45)

1. Altman, Morris, ed. 2004. "Statistical Significance" Special issue. Journal of Socio-Economics 33(5): 523-675.
2. Bakan, David. 1966. "The Test of Significance in Psychological Research." Psychological Bulletin 66(6): 423-37.
3. Berkson, Joseph. 1942. "Tests of Significance Considered as Evidence." Journal of the American Statistical Association 37(219): 325-35.
4. Carver, Ronald P. 1978. "The Case Against Statistical Significance Testing." Harvard Educational Review 48(3): 378-99.
5. Chow, Siu L. 1998. "Precis of Statistical Significance: Rationale, Validity, and Utility." Behavioral and Brain Sciences 21(2): 169-94.
6. Cohen, J. 1994. "The Earth Is Round (p < .05)." American Psychologist 49(12): 997-1003.
7. Copay, Anne G., Brian R. Subach, Steven D. Glassman, David W. Polly Jr., and Thomas C. Schuler . 2007. "Understanding the Minimum Clinically Important Difference: A Review of Concepts and Methods." The Spine Journal 7(5): 541-46.
8. DeGroot, Morris H., and Mark J. Schervish . 2002. Probability and Statistics. Boston, MA: Addison-Wesley.
9. Esarey, Justin, and Nathan Danneman . Forthcoming. "A Quantitative Method for Substantive Robustness Assessment." Political Science Research and Methods. http://jee3.web.rice.edu/riskstats.pdf.
10. Eysenck, Hans J. 1960. "The Concept of Statistical Significance and the Controversy about One-Tailed Tests." Psychological Review 67(4): 269-71.
11. Falk, Ruma, and Charles W. Greenbaum . 1995. "Significance Tests Die Hard: The Amazing Persistence of a Probabilistic Misconception." Theory & Psychology 5(1): 75-98.
12. Frick, Robert W. 1996. "The Appropriate Use of Null Hypothesis Testing." Psychological Methods 1(4): 379-90.
13. Gerber, Alan S., Donald P. Green, and David Nickerson . 2001. "Testing for Publication Bias in Political Science." Political Analysis 9(4): 385-92.
14. Gigerenzer, Gerd. 1993. "The Superego, the Ego, and the Id in Statistical Reasoning." In A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues, ed. Gideon Keren and Charles Lewis. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc., pp. 311-39.
15. Gigerenzer, Gerd. 1998. "We Need Statistical Thinking, Not Statistical Rituals." Behavioral and Brain Sciences 21(2): 199-200.
16. Gigerenzer, Gerd, and Zeno Swijtink . 1990. The Empire of Chance: How Probability Changed Science and Everyday Life. New York: Cambridge University Press.
17. Gill, Jeff. 1999. "The Insignificance of Null Hypothesis Significance Testing." Political Research Quarterly 52(3): 647-74.
18. Harlow, Lisa L., Stanley A. Mulaik, and James H. Steiger . 1997. What If There Were No Significance Tests? Mahwah, NJ: Erlbaum.
19. Hoijtink, Herbert. 2011. Informative Hypotheses: Theory and Practice for Behavioral and Social Scientists. Boca Raton, FL: CRC Press.
20. Hoijtink, Herbert, Irene Klugkist, and Paul A. Boelen . 2008. Bayesian Evaluation of Informative Hypotheses. New York: Springer.
21. Ioannidis, John P. A. 2005. "Why Most Published Research Findings Are False." PLoS Medicine 2(8): e124.
22. Iyengar, Shanto, and Adam Simon . 1993. "News Coverage of the Gulf Crisis and Public Opinion." Communication Research 20(3): 365-83.
23. Jaeschke, Roman, Joel Singer, and Gordon H. Guyatt . 1989. "Measurement of Health Status: Ascertaining the Minimal Clinically Important Difference." Controlled Clinical Trials 10(4): 407-15.
24. Jeffreys, Sir Harold. 1961. Theory of Probability. Oxford: Oxford University Press.
25. Kish, Leslie. 1959. "Some Statistical Problems in Research Design." American Sociological Review 24(3):328-38.
26. Levin, Joel R. 1998. "What If There Were No More Bickering about Statistical Significance Tests?" Research in the Schools 5(2): 43-53.
27. Mayo, Deborah G., and Aris Spanos . 2006. "Severe Testing as a Basic Concept in a Neyman-Pearson Philosophy of Induction." British Journal for the Philosophy of Science 57(2): 323-57.
28. Meehl, Paul E. 1978. "Theoretical Risks and Tabular Asterisks: Sir Karl, Sir Ronald, and the Slow Progress of Soft Psychology." Journal of Consulting and Clinical Psychology 46(4): 806-34.
29. Miller, Warren E., Donald R. Kinder, Steven J. Rosenstone, and the National Election Studies. 1999. American National Election Studies, 1991 Pilot Election Study [Data set]. Ann Arbor, MI: University of Michigan, Center for Political Studies.
30. Mogie, Michael. 2004. "In Support of Null Hypothesis Significance Testing." Proceedings of the Royal Society of London: Series B, Biological Sciences (Suppl. 3): S82-84.
31. Morrison, Denton E., and Ramon E. Henkel, eds. 1970. The Significance Test Controversy: A Reader. New Brunswick, NJ: Transaction.
32. Prior, Markus. 2009. "The Immensely Inflated News Audience: Assessing Bias in Self-Reported News Exposure." Public Opinion Quarterly 73(1): 130-43.
33. Rainey, Carlisle. Forthcoming. "Arguing for a Negligible Effect." American Journal of Political Science. http://onlinelibrary.wiley.com.libproxy.lib.unc.edu/doi/10.1111/ajps.12102/full.
34. Revicki, Dennis, Ron D . Hays, David Cella, and Jeff Sloan. 2008. "Recommended Methods for Determining Responsiveness and Minimally Important Differences for Patient-Reported Outcomes." Journal of Clinical Epidemiology 61(2): 102-09.
35. Rosenthal, Robert. 1979. "The 'File Drawer Problem' and Tolerance for Null Results." Psychological Bulletin 86(3): 638-41.
36. Rozeboom, William W. 1960. "The Fallacy of the Null-Hypothesis Significance Test." Psychological Bulletin 57(5): 416-28.
37. Schrodt, Philip A. 2006. "Beyond the Linear Frequentist Orthodoxy." Political Analysis 14(3): 335-39.
38. Serlin, Ronald C., and Daniel K. Lapsley . 1985. "Rationality in Psychological Research: The Good-Enough Principle." American Psychologist 40(1): 73-83.
39. Steiger, James H., and Rachel T. Fouladi . 1997. "Noncentrality Interval Estimation and the Evaluation of Statistical Models." In What If There Were No Significance Tests? ed. Lisa L. Harlow, Stanley A. Mulaik, and James H. Steiger . Mahwah, NJ: Erlbaum, pp. 221-57.
40. Thompson, Bruce. 2004. "The 'Significance' Crisis in Psychology and Education." Journal of Socio-Economics 33(5): 607-13.
41. Ward, Michael D., Brian D. Greenhill, and Kristin M. Bakke . 2010. "The Perils of Policy by p-value: Predicting Civil Conflicts." Journal of Peace Research 47(4): 363-75.
42. Wells, George, Dorcas Beaton, Beverly Shea, Maarten Boers, Lee Simon, Vibeke Strand, Peter Brooks, and Peter Tugwell . 2001. "Minimal Clinically Important Differences: Review of Methods." Journal of Rheumatology 28(2):406-12.
43. Wilkinson, Leland, and APA Board of Scientific Affairs Task Force on Statistical Inference. 1999. "Statistical Methods in Psychology Journals: Guidelines and Explanations." American Psychologist 54(8): 594-604.
44. Yates, Frank. 1951. "The Influence of Statistical Methods for Research Workers on the Development of the Science of Statistics." Journal of the American Statistical Association 46(253): 19-34.
45. Ziliak, Stephen T., and Deirdre N. McCloskey . 2008. The Cult of Statistical Significance: How the Standard Error Costs Us Jobs, Justice, and Lives. Ann Arbor, MI: University of Michigan Press.