Null hypothesis significance testing and Type I error: The domain problem

New Ideas in Psychology

Volumn 45, Issue , 2017, Pages 19-27

  Trafimow, David ^a   Earp, Brian D ^b  

^a NEW MEXICO STATE UNIVERSITY   (United States)

^b YALE UNIVERSITY   (United States)

Author keywords

Across what; Null hypothesis significance testing procedure; Statistical control; Type I error; Type II error

Indexed keywords

EID: 85012299288     PISSN: 0732118X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.newideapsych.2017.01.002     Document Type: Article

References (75)

1. Abelson, R.P., A retrospective on the significance test ban of 1999 (If there were no significance tests, they would be invented). Harlow, L., Mulaik, S.A., Steiger, J.H., (eds.) What if there were no significance tests?, 1997, Erlbaum, Mahwah, NJ, 117–141.
2. Anderson, Christopher J., Bahník, Štěpán, Barnett-Cowan, Michael, Bosco, Frank A., Chandler, Jesse, Chartier, Christopher R., et al. Response to Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 2016, 10.1126/science.aad9163.
3. Asimov, I., The universe: From flat earth to quasar. 1966, Avon Books, New York.
4. Bakan, D., The test of significance in psychological research. Psychological Bulletin 66 (1966), 423–437.
5. Bakker, M., van Dijk, A., Wicherts, J.M., The rules of the game called psychological science. Perspectives on Psychological Science 7:6 (2012), 543–554.
6. Button, K.S., Ioannidis, J.P., Mokrysz, C., Nosek, B.A., Flint, J., Robinson, E.S., et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14:5 (2013), 365–376.
7. Carver, R.P., The case against statistical significance testing. Harvard Educational Review 48 (1978), 378–399.
8. Carver, R.P., The case against statistical significance testing, revisited. The Journal of Experimental Education 61 (1993), 287–292.
9. Chow, S.L., Précis of statistical significance: Rationale, validity, and utility. Behavioral and Brain Sciences 21 (1998), 169–239.
10. Cohen, J., The earth is round (p <.05). American Psychologist 49 (1994), 997–1003.
11. Cumming, G., Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. 2012, Routledge, New York.
12. Earp, B.D., Psychology is not in crisis? Depends on what you mean by “crisis”. Huffington Post, 2015, September 2 Available at: http://www.huffingtonpost.com/brian-earp/psychology-is-not-in-crisis_b_8077522.html.
13. Earp, B.D., What did the OSC replication initiative reveal about the crisis in psychology? An open review of the draft paper entitled “Replication initiatives will not salvage the trustworthiness of psychology” by James C. Coyne. BMC Psychology 4:28 (2016), 1–19.
14. Earp, B.D., Everett, J.A.C., How to fix psychology's replication crisis. The Chronicle of Higher Education, 2015, October 26 Available at: http://chronicle.com/article/How-to-Fix-Psychology-s/233857.
15. Earp, B.D., Everett, J.A.C., Madva, E.N., Hamlin, J.K., Out, damned spot: Can the “Macbeth Effect” be replicated?. Basic and Applied Social Psychology 36:1 (2014), 91–98.
16. Earp, B.D., Trafimow, D., Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology 6 (2015), 1–11.
17. Edwards, W., Lindman, H., Savage, L.J., Bayesian statistical inference for psychological research. Psychological Review 70 (1963), 193–242.
18. Einstein, A., Relativity: The special and the general theory (Robert W. Lawson, Trans.). 1961, Crown Publishers, New York.
19. Everett, J.A.C., Earp, B.D., A tragedy of the (academic) commons: Interpreting the replication crisis in psychology as a social dilemma for early-career researchers. Frontiers in Psychology 6:1152 (2015), 1–4.
20. Feyerabend, P., Against method. 3rd ed., 1993, Verso, London.
21. Fidler, F., Should psychology abandon p values and teach CIs instead? Evidence-based reforms in statistics education. 2006 Available at: http://iase-web.org/documents/papers/icots7/5E4_FIDL.pdf.
22. Fidler, F., Loftus, G.R., Why figures with error bars should replace p values: Some conceptual arguments and empirical demonstrations. Zeitschrift für Psychologie/Journal of Psychology 217:1 (2009), 27–37, 10.1027/0044-3409.217.1.27.
23. Fisher, R.A., Statistical methods for research workers. 1925, Oliver and Boyd, Edinburgh, Scotland.
24. Fisher, R.A., Inverse probability. Proceedings of the Cambridge Philosophical Society 26 (1930), 528–538.
25. Fisher, R.A., The fiducial argument in statistical inference. Annals of Eugenics 6 (1935), 391–398.
26. Fisher, R.A., The design of experiments. 6th ed., 1951, Hafner Press, New York.
27. Fisher, R.A., Statistical methods and scientific inference. 3rd ed., 1973, Hafner Press, New York.
28. Gilbert, D.T., King, G., Pettigrew, S., Wilsond, T.D., Comment on “Estimating the reproducibility of psychological science”. Science, 351(6277), 2016, 1037, 10.1126/science.aad7243.
29. Good, P., Introduction to statistics through resampling methods and R/S-PLUS. 2005, Wiley.
30. Greenwald, A.G., Consequences of prejudice against the null hypothesis. Psychological Bulletin 82:1 (1975), 1–20.
31. Grice, J.W., Observation oriented modeling: Analysis of cause in the behavioral sciences. 2011, Academic Press, USA.
32. Hagen, R.L., In praise of the null hypothesis statistical test. American Psychologist 52 (1997), 15–24.
33. Hamilton, W.C., Statistics in physical science: Estimation, hypothesis testing, and least squares. 1964, The Ronald Press Company, New York.
34. Hays, W.L., Statistics. 5th ed., 1994, Harcourt Brace College, Fort Worth, Texas.
35. Howson, C., Urbach, P., Scientific reasoning: The Bayesian approach. 1989, Open Court, La Salle, IL.
36. Howson, C., Urbach, P., Probability, uncertainty and the practice of statistics. Wright, G., Ayton, P., (eds.) Subjective probability, 1994, Wiley, Chichester, England, 39–51.
37. John, L.K., Loewenstein, G., Prelec, D., Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23:5 (2012), 524–532.
38. Kass, R.E., Raftery, A.E., Bayes factors. Journal of the American Statistical Association 90 (1995), 773–795.
39. Krueger, J., Null hypothesis significance testing: On the survival of a flawed method. American Psychologist 56 (2001), 16–26.
40. Kruschke, J.K., Doing bayesian data analysis: A tutorial with R and BUGS. 2011, Academic Press, USA.
41. Lachman, R., Lachman, J.L., Butterfield, E.C., Cognitive psychology and information processing: An introduction. 1979, Lawrence Erlbaum Associates, Publishers, Hillsdale, N. J.
42. Lakatos, I., The methodology of scientific research programmes. 1978, Cambridge University Press, Cambridge, UK.
43. Mayo, D., Error and the growth of experimental knowledge. 1996, The University of Chicago Press, Chicago.
44. Meehl, P.E., Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science 34 (1967), 103–115.
45. Meehl, P.E., Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 46 (1978), 806–834.
46. Meehl, P.E., Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant using it. Psychological Inquiry 1 (1990), 108–141.
47. Meehl, P.E., The problem is epistemology, not statistics: Replace significance tests by confidence intervals and quantify accuracy of risky numerical predictions. 1997 In L.
48. Mulaik, S.A., Raju, N.S., Harshman, R.A., There is a time and place for significance testing. Harlow, L., Mulaik, S.A., Steiger, J.H., (eds.) What if there were no significance tests?, 1997, Erlbaum, Mahwah, NJ, 65–115.
49. Neyman, J., Pearson, E.S., On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika 20A (1928), 175–240.
50. Neyman, J., Pearson, E.S., The testing of statistical hypotheses in relation to probabilities a priori. Proceedings of the Cambridge Philosophical Society 29 (1933), 492–510.
51. Nickerson, R.S., Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods 5 (2000), 241–301.
52. Open Science Collaboration, Estimating the reproducibility of psychological science. Science, 349(6251), 2015 aac4716.
53. Palmer, K.M., Psychology is in crisis over whether it's in crisis. Wired, 2016, March 3 Available at: http://www.wired.com/2016/03/psychology-crisis-whether-crisis/.
54. Pashler, H., Wagenmakers, E.J., Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence?. Perspectives on Psychological Science 7:6 (2012), 528–530.
55. Popper, K., Logik der Forschung [The logic of scientific discovery]. 1959, Basic Books, New York (Original work published 1934).
56. Preston, J., Feyerabend: Philosophy, science and society. 1997, Blackwell Publishing, Incorporated. Publishers, Malden, M. A.
57. Rosenthal, R., Rosnow, R.L., Essentials of behavioral research: Methods and data analysis. 2nd ed., 1991, McGraw-Hill, USA.
58. Rozeboom, W.W., The fallacy of the null-hypothesis significance test. Psychological Bulletin 57 (1960), 416–428.
59. Rozeboom, W.W., Good science is abductive, not hypothetico-deductive. Harlow, L., Mulaik, S.A., Steiger, J.H., (eds.) What if there were no significance tests?, 1997, Erlbaum, Mahwah, NJ, 335–391.
60. Schmidt, F.L., Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods 1 (1996), 115–129.
61. Schmidt, F.L., Hunter, J.E., Eight objections to the discontinuation of significance testing in the analysis of research data. Harlow, L., Mulaik, S.A., Steiger, J.H., (eds.) What if there were no significance tests?, 1997, Erlbaum, Mahwah, NJ, 37–64.
62. Seidenfeld, T., R. A. Fisher's fiducial Argument and Bayes' theorem. Statistical Science 7:3 (1992), 358–368 http://links.jstor.org/sici?sici=0883-4237%28199208%297%3A3%3C358%3ARAFFAA%3E2.0.CO%3B2-G.
63. Simmons, J.P., Nelson, L.D., Simonsohn, U., False-positive Psychology: Undisclosed flexibility in data collection and Analysis Allows presenting anything as significant. Psychological Science 22:11 (2011), 1359–1366.
64. Spellman, B.A., A short (personal) future history of revolution 2.0. Perspectives on Psychological Science 10:6 (2015), 886–899.
65. Trafimow, D., Hypothesis testing and theory evaluation at the boundaries: Surprising insights from Bayes's theorem. Psychological Review 110 (2003), 526–535.
66. Trafimow, D., Using epistemic ratios to evaluate Hypotheses: An imprecision penalty for imprecise hypotheses. Genetic, Social, and General Psychology Monographs 132 (2006), 431–462.
67. Trafimow, D., Earp, B.D., Badly specified theories are not responsible for the replication crisis in social psychology. Theory & Psychology 26:4 (2016), 540–548.
68. Trafimow, D., Marks, M., Editorial. Basic and Applied Social Psychology 37:1 (2015), 1–2.
69. Trafimow, D., Marks, M., Editorial. Basic and Applied Social Psychology 38:1 (2016), 1–2.
70. Trafimow, D., Using the coefficient of confidence to make the philosophical switch from a posteriori to a priori inferential statistics. Educational and Psychological Measurement, 2016 in press.
71. Valentine, J.C., Aloe, A.M., Lau, T.S., Life after NHST: How to describe your data without “p-ing” everywhere. Basic and Applied Social Psychology 37:5 (2015), 260–273 http://dx.doi.org/10.1080/01973533.2015.1060240.
72. Wasserstein, R.L., Lazar, N.A., The ASA's statement on p-values: Context, process, and purpose. The American Statistician, 2016, 10.1080/00031305.2016.1154108.
73. Wolfson, R., Simply Einstein: Relativity demystified. 2003, W.W. Norton, New York.
74. Woodside, A., The good practices manifesto: Overcoming bad practices pervasive in current research in business. Journal of Business Research 69:2 (2016), 365–381.
75. Yong, E., Replication studies: Bad copy. Nature 485:7398 (2012), 298–300 http://www.nature.com/news/replication-studies-bad-copy-1.10634.