The Harm Done to Reproducibility by the Culture of Null Hypothesis Significance Testing

American Journal of Epidemiology

Volumn 186, Issue 6, 2017, Pages 627-635

  Lash, Timothy L ^a,b  

^a EMORY UNIVERSITY   (United States)

^b AARHUS UNIVERSITY HOSPITAL   (Denmark)

Author keywords

data interpretation; epidemiologic methods; epidemiologic research design; null hypothesis; reproducibility of results; significance testing; statistical; statistical significance; statistics

Indexed keywords

DATA ASSIMILATION; EPIDEMIOLOGY; HYPOTHESIS TESTING; LITERATURE REVIEW; PROJECT ASSESSMENT; RESEARCH METHOD; STAKEHOLDER; STATISTICAL ANALYSIS;

ARTICLE; DATA ANALYSIS; DATA INTERPRETATION; DATA PROCESSING; ERROR; FALSE NEGATIVE RESULT; INNOVATIVE RESEARCH; MEDICAL RESEARCH; MUNDANE RESEARCH; NULL HYPOTHESIS; PROBABILITY; QUANTITATIVE ANALYSIS; REPRODUCIBILITY; SENSITIVITY AND SPECIFICITY; STATISTICAL ANALYSIS; STATISTICAL INFERENCE; STATISTICAL PARAMETERS; STATISTICAL SIGNIFICANCE; STUDY DESIGN; SYSTEMATIC ERROR; TYPE I ERROR; TYPE II ERROR; FALSE POSITIVE RESULT; HUMAN; METHODOLOGY; PREDICTIVE VALUE; RESEARCH; STANDARDS;

DATA INTERPRETATION, STATISTICAL; FALSE POSITIVE REACTIONS; HUMANS; PREDICTIVE VALUE OF TESTS; PROBABILITY; REPRODUCIBILITY OF RESULTS; RESEARCH; RESEARCH DESIGN;

EID: 85029691663     PISSN: 00029262     EISSN: 14766256     Source Type: Journal    
DOI: 10.1093/aje/kwx261     Document Type: Article

References (69)

1. Ioannidis JP. How to make more published research true. PLoS Med. 2014;11(10): e1001747.
2. Unreliable research: trouble at the lab. Economist. October 18, 2013.
3. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014;505(7485): 612-613.
4. Nosek BA, Alter G, Banks GC, et al. Promoting an open research culture. Science. 2015;348(6242): 1422-1425.
5. Journals unite for reproducibility. Nature. 2014;515(7525): 7.
6. Office of Extramural Research, National Institutes of Health. Rigor and reproducibility. 2016. http: //grants. nih. gov/ reproducibility/index. htm#guidance. Updated August 18, 2016. Accessed July 6, 2016.
7. Lash TL. Declining the Transparency and Openness Promotion Guidelines. Epidemiology. 2015;26(6): 779-780.
8. Stedman RC, Beckley TM. If we knew what it was we were doing, it would not be called research, would it Soc Nat Resour. 2007;20(10): 939-943.
9. Ten Hagen KG. Novel or reproducible: that is the question. Glycobiology. 2016;26(5): 429.
10. Goodman SN. A comment on replication, p-values and evidence. Stat Med. 1992;11(7): 875-879.
11. Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci USA. 2013;110(48): 19313-19317.
12. Rothman KJ, Greenland S, Lash TL. Precision and statistics in epidemiologic studies. In: Rothman KJ, Greenland S, Lash TL, eds. Modern Epidemiology. 3rd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2008: 148-167.
13. Wasserstein RL, Lazar NA. The ASA's statement on p-values: context, process, and purpose [editorial]. Am Stat. 2016;70(2): 129-133.
14. Greenland S, Senn SJ, Rothman KJ, et al. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016;31(4): 337-350.
15. Rothman KJ. A show of confidence. N Eng J Med. 1978; 299(24): 1362-1363.
16. Savitz DA, Tolo KA, Poole C. Statistical significance testing in the American Journal of Epidemiology, 1970-1990. Am J Epidemiol. 1994;139(10): 1047-1052.
17. Poole C. Low P-values or narrow confidence intervals: which are more durable Epidemiology. 2001;12(3): 291-294.
18. O'Boyle EH, Banks GC, Gonzalez-Mulé E. The chrysalis effect: how ugly initial results metamorphosize into beautiful articles. J Manag. 2017;43(2): 376-399.
19. Pocock SJ, Hughes MD. Practical problems in interim analyses, with particular regard to estimation. Control Clin Trials. 1989;10(4 suppl): 209S-221S.
20. Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010; 303(12): 1180-1187.
21. Open Science Collaboration. Estimating the reproducibility of psychological science. Science. 2015;349(6251): aac4716.
22. Sterling TD. Publication decisions and their possible effects on inferences drawn from tests of significance-or vice versa. J Am Stat Assoc. 1959;54(285): 30-34.
23. Begg CB. A measure to aid in the interpretation of published clinical trials. Stat Med. 1985;4(1): 1-9.
24. Rothman KJ. Significance questing. Ann Intern Med. 1986; 105(3): 445-447.
25. Lash TL, Fox MP, MacLehose RF, et al. Good practices for quantitative bias analysis. Int J Epidemiol. 2014;43(6): 1969-1985.
26. Lash TL, Fox MP, Fink AK. Applying Quantitative Bias Analysis to Epidemiologic Data. New York, NY: Springer Publishing Company; 2009.
27. Greenland S, Lash TL. Bias analysis. In: RothmanKJ, Greenland S, Lash TL, eds. Modern Epidemiology. 3rd ed. Philadelphia, PA: LippincottWilliams&Wilkins; 2008: 345-380.
28. Henrion M, Fischhoff B. Assessing uncertainty in physical constants. Am J Phys. 1986;54(9): 791-798.
29. Lash TL. Heuristic thinking and inference from observational epidemiology. Epidemiology. 2007;18(1): 67-72.
30. Birge RT. The general physical constants: as of August 1941 with details on the velocity of light only. Rep Prog Phys. 1941; 8: 90-134.
31. HenrionM, Fischoff B. Assessing uncertainty in physical constants. In: Gilovich T, Griffin D, Kahneman D, eds. Heuristics and Biases: The Psychology of Intuitive Judgment. New York, NY: Cambridge University Press; 2002: 666-677.
32. Cornfield J, Haenszel W, Hammond EC, et al. Smoking and lung cancer: recent evidence and a discussion of some questions. J Natl Cancer Inst. 1959;22(1): 173-203.
33. Bross I. Misclassification in 2×2 tables. Biometrics. 1954; 10(4): 478-486.
34. Bross ID. Spurious effects from an extraneous variable. J Chronic Dis. 1966;19(6): 637-647.
35. Lyles RH, Lin J. Sensitivity analysis for misclassification in logistic regression via likelihood methods and predictive value weighting. Stat Med. 2010;29(22): 2297-2309.
36. Gustafson P. Measurement Error andMisclassification in Statistics and Epidemiology. Boca Raton, FL: CRC Press; 2003.
37. Greenland S. Interval estimation by simulation as an alternative to and extension of confidence intervals. Int J Epidemiol. 2004; 33(6): 1389-1397.
38. Fox MP, Lash TL, Greenland S. A method to automate probabilistic sensitivity analyses of misclassified binary variables. Int J Epidemiol. 2005;34(6): 1370-1376.
39. Sturmer T, Schneeweiss S, Avorn J, et al. Adjusting effect estimates for unmeasured confounding with validation data using propensity score calibration. Am J Epidemiol. 2005; 162(3): 279-289.
40. Spiegelman D, Carroll RJ, Kipnis V Efficient regression calibration for logistic regression in main study/internal validation study designs with an imperfect reference instrument. Stat Med. 2001;20(1): 139-160.
41. MacLehose RF, Gustafson P. Is probabilistic bias analysis approximately Bayesian Epidemiology. 2012;23(1): 151-158.
42. McCandless LC, Gustafson P, Levy A. Bayesian sensitivity analysis for unmeasured confounding in observational studies. Stat Med. 2007;26(11): 2331-2347.
43. FoxMP, Lash TL. On the need for quantitative bias analysis in the peer-review process. Am J Epidemiol. 2017;185(10): 865-868.
44. Burstyn I, Yang Y, Schnatter AR. Effects of non-differential exposure misclassification on false conclusions in hypothesisgenerating studies. Int J Environ Res Public Health. 2014; 11(10): 10951-10966.
45. Greenland S. Bayesian perspectives for epidemiological research: I. Foundations and basic methods. Int J Epidemiol. 2006;35(3): 765-775.
46. MacLehose RF, Hamra GB. Applications of Bayesian methods to epidemiologic research. Curr Epidemiol Rep. 2014;1(3): 103-109.
47. Early Breast Cancer Trialists' Collaborative Group (EBCTCG), Peto R, Davies C, et al. Comparisons between different polychemotherapy regimens for early breast cancer: meta-analyses of long-term outcome among 100, 000 women in 123 randomised trials. Lancet. 2012;379(9814): 432-444.
48. Lash TL, Lien EA, Sørensen HT, et al. Genotype-guided tamoxifen therapy: time to pause for reflection Lancet Oncol. 2009;10(8): 825-833.
49. Myrand SP, Sekiguchi K, Man MZ, et al. Pharmacokinetics/ genotype associations for major cytochrome P450 enzymes in native and first-and third-generation Japanese populations: comparison with Korean, Chinese, and Caucasian populations. Clin Pharmacol Ther. 2008;84(3): 347-361.
50. Goetz MP, Knox SK, Suman VJ, et al. The impact of cytochrome P450 2D6 metabolism in women receiving adjuvant tamoxifen. Breast Cancer Res Treat. 2007;101(1): 113-121.
51. Ahern TP, Hertz DL, Damkier P, et al. Cytochrome P-450 2D6 (CYP2D6) genotype and breast cancer recurrence in tamoxifen-treated patients: evaluating the importance of loss of heterozygosity. Am J Epidemiol. 2017;185(2): 75-85.
52. Gelman A. The connection between varying treatment effects and the crisis of unreplicable research: a Bayesian perspective. J Manag. 2015;41(2): 632-643.
53. Kenall A, Edmunds S, Goodman L, et al. Better reporting for better research: a checklist for reproducibility. Gigascience. 2015;4: 32.
54. Schmidt CW. Research wranglers: initiatives to improve reproducibility of study findings. Environ Health Perspect. 2014;122(7): A188-A191.
55. George BJ, Sobus JR, Phelps LP, et al. Raising the bar for reproducible science at the US Environmental Protection Agency Office of Research and Development. Toxicol Sci. 2015;145(1): 16-22.
56. Motulsky HJ. Common misconceptions about data analysis and statistics. Pharmacol Res Perspect. 2015;3(1): e00093.
57. Rothman KJ. Disengaging from statistical significance. Eur J Epidemiol. 2016;31(5): 443-444.
58. McShane BB, Gal D. Blinding us to the obvious The effect of statistical training on the evaluation of evidence. Manage Sci. 2016;62(6): 1707-1718.
59. Cumming G. The new statistics: why and how. Psychol Sci. 2014;25(1): 7-29.
60. Lang JM, Rothman KJ, Cann CI. That confounded P-value. Epidemiology. 1998;9(1): 7-8.
61. Stang A, Deckert M, Poole C, et al. Statistical inference in abstracts of major medical and epidemiology journals 1975-2014: a systematic review. Eur J Epidemiol. 2017;32(1): 21-29.
62. Trafimow D. Editorial. Basic Appl Soc Psych. 2014;36(1): 1-2.
63. Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol. 2010;25(4): 225-230.
64. Siegfried T. P value ban: small step for a journal, giant leap for science. ScienceNews. March 17, 2015.
65. Stack Exchange, Inc. References containing arguments against null hypothesis significance testing 2016. http: //stats. stackexchange. com/questions/10510/references-containingarguments-against-null-hypothesis-significance-testing. Accessed December 21, 2016.
66. Weinberg CR. It's time to rehabilitate the P-value. Epidemiology. 2001;12(3): 288-290.
67. Glisson C. In defense of statistical tests of significance. Soc Serv Rev. 1985;59(3): 377-386.
68. Murtaugh PA. In defense of P values. Ecology. 2014;95(3): 611-617.
69. Hoover KD, Siegler MV. Sound and fury: McCloskey and significance testing in economics. J Econ Methodol. 2008; 15(1): 1-37.