Manipulating the alpha level cannot cure significance testing

Frontiers in Psychology

Volumn 9, Issue MAY, 2018, Pages

  Trafimow, David ^a   Amrhein, Valentin ^b,c   Areshenkoff, Corson N ^d   Barrera Causil, Carlos J ^e   Beh, Eric J ^f   Bilgiç, Yusuf K ^g   Bono, Roser ^h   Bradley, Michael T ⁱ   Briggs, William M ^j   Cepeda Freyre, Héctor A ^k   Chaigneau, Sergio E ^l   Ciocca, Daniel R ^m   Correa, Juan C ⁿ   Cousineau, Denis ^o   de Boer, Michiel R ^p   Dhar, Subhra S ^q   Dolgov, Igor ^a   Gómez Benito, Juana ^h   Grendar, Marian ^r,s   Grice, James W ^t   Guerrero Gimenez, Martin E ^m   Gutiérrez, Andrés ^u   Huedo Medina, Tania B ^v   Jaffe, Klaus ^w   Janyan, Armina ^x,y   Karimnezhad, Ali ^o   Korner Nievergelt, Fränzi ^c,z   Kosugi, Koji ^aa   Lachmair, Martin ^ab   Ledesma, Rubén D ^ac,ad   Limongi, Roberto ^ae,af   Liuzza, Marco T ^ag   Lombardo, Rosaria ^ah   Marks, Michael J ^a   Meinlschmidt, Gunther ^ai,aj,ak   Nalborczyk, Ladislas ^al,am   Nguyen, Hung T ^a   Ospina, Raydonal ^an   Perezgonzalez, Jose D ^ao   Pfister, Roland ^ap   Rahona, Juan J ^ab   Rodríguez Medina, David A ^aq   Romão, Xavier ^ar   Ruiz Fernández, Susana ^ab,as,at   Suarez, Isabel ^au   Tegethoff, Marion ^ak   Tejo, Mauricio ^av   van de Schoot, Rens ^aw,ax   Vankov, Ivan I ^x   Velasco Forero, Santiago ^ay   Wang, Tonghui ^a   Yamada, Yuki ^az   Zoppino, Felipe C M ^m   Marmolejo Ramos, Fernando ^ba   more..

^a NEW MEXICO STATE UNIVERSITY   (United States)

^b UNIVERSITY OF BASEL   (Switzerland)

^c SWISS ORNITHOLOGICAL INSTITUTE   (Switzerland)

^d QUEEN S UNIVERSITY   (Canada)

^e Metropolitan Institute of Technology   (Colombia)

^f UNIVERSITY OF NEWCASTLE   (Australia)

^g STATE UNIVERSITY OF NEW YORK   (United States)

^h UNIVERSITY OF BARCELONA   (Spain)

ⁱ UNIVERSITY OF NEW BRUNSWICK   (Canada)

^j Independent Researcher   (United States)

^k BENEMÉRITA UNIVERSIDAD AUTÓNOMA DE PUEBLA   (Mexico)

^l UNIVERSIDAD ADOLFO IBÁÑEZ   (Chile)

^m INSTITUTO ARGENTINO DE NIVOLOGÍA   (Argentina)

ⁿ UNIVERSIDAD NACIONAL DE COLOMBIA   (Colombia)

^o UNIVERSITY OF OTTAWA   (Canada)

^p UNIVERSITY OF AMSTERDAM   (Netherlands)

^q INDIAN INSTITUTE OF TECHNOLOGY KANPUR   (India)

^r University Hospital in Martin   (Slovakia)

^s INSTITUTE OF MEASUREMENT SCIENCE   (Slovakia)

^t OKLAHOMA STATE UNIVERSITY   (United States)

^u UNIVERSIDAD SANTO TOMÁS   (Colombia)

^v UNIVERSITY OF CONNECTICUT   (United States)

^w UNIVERSIDAD SIMÓN BOLÍVAR   (Venezuela)

^x NEW BULGARIAN UNIVERSITY   (Bulgaria)

^y TOMSK STATE UNIVERSITY   (Russian Federation)

^z Oikostat GmbH   (Switzerland)

^aa SENSHU UNIVERSITY   (Japan)

^ab KNOWLEDGE MEDIA RESEARCH CENTER   (Germany)

^ac Facultad de Ciencias Naturales e Instituto M Lillo   (Argentina)

^ad UNIVERSIDAD NACIONAL DE MAR DEL PLATA   (Argentina)

^ae PONTIFICIA UNIVERSIDAD CATÓLICA DE VALPARAÍSO   (Chile)

^af UNIVERSIDAD TECNOLÓGICA DE CHILE INACAP   (Chile)

^ag UNIVERSITY MAGNA GRAECIA OF CATANZARO   (Italy)

^ah SECOND UNIVERSITY OF NAPLES   (Italy)

^ai UNIVERSITY HOSPITAL BASEL   (Switzerland)

^aj INTERNATIONAL PSYCHOANALYTIC UNIVERSITY   (Germany)

^ak UNIVERSITY OF BASEL   (Switzerland)

^al UNIVERSITÉ PIERRE MENDÈS FRANCE   (France)

^am GHENT UNIVERSITY   (Belgium)

^an FEDERAL UNIVERSITY OF PERNAMBUCO   (Brazil)

^ao MASSEY UNIVERSITY   (New Zealand)

^ap UNIVERSITY OF WÜRZBURG   (Germany)

^aq UNIVERSIDAD NACIONAL AUTÓNOMA DE MÉXICO   (Mexico)

^ar UNIVERSITY OF PORTO   (Portugal)

^as FOM UNIVERSITY OF APPLIED SCIENCES   (Germany)

^at UNIVERSITY OF TÜBINGEN   (Germany)

^au UNIVERSIDAD DEL NORTE   (Colombia)

^av UNIVERSIDAD DE PLAYA ANCHA   (Chile)

^aw UTRECHT UNIVERSITY   (Netherlands)

^ax NORTH WEST UNIVERSITY   (South Africa)

^ay PSL RESEARCH UNIVERSITY   (France)

^az KYUSHU UNIVERSITY   (Japan)

^ba UNIVERSITY OF ADELAIDE   (Australia)

more..

Author keywords

Decision making; Null hypothesis testing; P value; Significance testing; Statistical significance

Indexed keywords

EID: 85047014418     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2018.00699     Document Type: Article

References (65)

1. Amrhein, V., and Greenland, S. (2018). Remove, rather than redefine, statistical significance. Nat. Hum. Behav. 2:4. doi: 10.1038/s41562-017-0224-0
2. Amrhein, V., Korner-Nievergelt, F., and Roth, T. (2017). The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research. PeerJ. 5:e3544. doi: 10.7717/peerj.3544
3. Amrhein, V., Trafimow, D., and Greenland, S. (2018). Abandon statistical inference. PeerJ Preprints 6:e26857v1. doi: 10.7287/peerj.preprints.26857v1
4. Balluerka, N., Gómez, J., and Hidalgo, D. (2005). The controversy over null hypothesis significance testing revisited. Methodology 1, 55-77. doi: 10.1027/1614-1881.1.2.55
5. Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., et al. (2018). Redefine statistical significance. Nat. Hum. Behav. 2, 6-10. doi: 10.1038/s41562-017-0189-z
6. Berk, R. A., and Freedman, D. A. (2003). "Statistical assumptions as empirical commitments," in Law, Punishment, and Social Control: Essays in Honor of Sheldon Messinger, 2nd Edn., eds T. G. Blomberg and S. Cohen (New York, NY: Aldine de Gruyter), 235-254
7. Bhardwaj, S. S., Camacho, F., Derrow, A., Fleischer, A. B., and Feldman, S. R. (2004). Statistical significance and clinical relevance. Arch. Dermatol. 140, 1520-1523. doi: 10.1001/archderm.140.12.1520
8. Bradley, M. T., and Brand, A. (2016). Significance testing needs a taxonomy: or how the Fisher, Neyman-Pearson controversy resulted in the inferential tail wagging the measurement dog. Psychol. Rep. 119, 487-504. doi: 10.1177/0033294116662659
9. Briggs, W. M. (2016). Uncertainty: The Soul of Modeling, Probability and Statistics. New York, NY: Springer
10. Buhl-Mortensen, L. (1996). Type-II statistical errors in environmental science and the precautionary principle. Mar. Pollut. Bull. 32, 528-531. doi: 10.1016/0025-326X(96)00036-7
11. Button, K. S., Ioannidis, J. P., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14:365376. doi: 10.1038/nrn3475
12. Cohen, J. (1994). The earth is round (p < 0.05). Am. Psychol. 49, 997-1003
13. Cumming, G. (2012). Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis. New York, NY: Routledge
14. Erceg-Hurn, D. M., Wilcox, R. R., and Keselman, H. J. (2013). "Robust statistical estimation," in The Oxford Handbook of Quantitative Methods, Vol. 1, ed T. Little (New York, NY: Oxford University Press), 388-406
15. Ferrill, M. J., Brown, D. A., and Kyle, J. A. (2010). Clinical versus statistical significance: interpreting P values and confidence intervals related to measures of association to decision making. J. Pharm. Pract. 23, 344-351. doi: 10.1177/0897190009358774
16. Fethney, J. (2010). Statistical and clinical significance, and how to use confidence intervals to help interpret both. Austr. Crit. Care 23, 93-97. doi: 10.1016/j.aucc.2010.03.001
17. Field, A. P., and Wilcox, R. R. (2017). Robust statistical methods: a primer for clinical psychology and experimental psychopathology researchers. Behav. Res. Ther. 98, 19-38. doi: 10.1016/j.brat.2017.05.013
18. Fisher, R. A. (1925). Statistical Methods for Research Workers, 1st Edn. Edinburgh: Oliver and Boyd
19. Fisher, R. A. (1937). The Design of Experiments, 2nd Edn. Edinburgh: Oliver and Boyd
20. Fisher, R. A. (1973). Statistical Methods and Scientific Inference, 3rd Edn. London: Macmillan
21. Gigerenzer, G., and Marewski, J. N. (2015). Surrogate science: the idol of a universal method for scientific inference. J. Manage. 41, 421-440. doi: 10.1177/0149206314547522
22. Goodman, S. N. (1992). A comment on replication, p-values and evidence. Stat. Med. 11, 875-879. doi: 10.1002/sim.4780110705
23. Greenland, S. (2017). The need for cognitive science in methodology. Am. J. Epidemiol. 186, 639-645. doi: 10.1093/aje/kwx259
24. Greenland, S. (2018). The unconditional information in P-values, and its refutational interpretation via S-values. Retrieved from: https://tinyurl.com/greenland2018
25. Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., et al. (2016). Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur. J. Epidemiol. 31, 337-350. doi: 10.1007/s10654-016-0149-3
26. Grice, J. W. (2017). Comment on Locascio's results blind manuscript evaluation proposal. Basic Appl. Soc. Psych. 39, 254-255. doi: 10.1080/01973533.2017.1352505
27. Halsey, L. G., Curran-Everett, D., Vowler, S. L., and Drummond, G. B. (2015). The fickle P value generates irreproducible results. Nat. Methods 12, 179-185. doi: 10.1038/nmeth.3288
28. Huber, P. J. (1972). Robust statistics: a review. Ann. Math. Stat. 43, 1041-1067. doi: 10.1214/aoms/1177692459
29. Hyman, M. (2017). Can "results blind manuscript evaluation" assuage "publication bias"? Basic Appl. Soc. Psych. 39, 247-251. doi: 10.1080/01973533.2017.1350581
30. Kline, R. (2017). Comment on Locascio, results blind science publishing. Basic Appl. Soc. Psychol. 39, 256-257. doi: 10.1080/01973533.2017.1355308
31. Konijn, E. A., van de Schoot, R., Winter, S. D., and Ferguson, C. J. (2015). Possible solution to publication bias through Bayesian statistics, including proper null hypothesis testing. Commun. Methods Meas. 9, 280-302. doi: 10.1080/19312458.2015.1096332
32. Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., et al. (2018). Justify your alpha. Nat. Hum. Behav. 2, 168-171. doi: 10.1038/s41562-018-0311-x
33. Lemons, J., and Victor, R. (2008). "Uncertainty in river restoration," in River Restoration: Managing the Uncertainty in Restoring Physical Habitat, eds S. Darby and D. Sear (Chichester: John Wiley and Sons), 3-13
34. Lemons, J., Shrader-Frechette, K., and Cranor, C. (1997). The precautionary principle: scientific uncertainty and type I and type II errors. Found. Sci. 2, 207-236. doi: 10.1023/A:1009611419680
35. Lieberman, M. D., and Cunningham, W. A. (2009). Type I and Type II error concerns in fMRI research: re-balancing the scale. Soc. Cogn. Affect. Neurosci. 4, 423-428. doi: 10.1093/scan/nsp052
36. Locascio, J. (2017a). Results blind science publishing. Basic Appl. Soc. Psychol. 39, 239-246. doi: 10.1080/01973533.2017.1336093
37. Locascio, J. (2017b). Rejoinder to responses to "results-blind publishing." Basic Appl. Soc. Psychol. 39, 258-261. doi: 10.1080/01973533.2017.1356305
38. Lykken, D. T. (1968). Statistical significance in psychological research. Psychol. Bull. 70, 151-159. doi: 10.1037/h0026141
39. Marks, M. J. (2017). Commentary on Locascio 2017. Basic Appl. Soc. Psych. 39, 252-253. doi: 10.1080/01973533.2017.1350580
40. Mayo, D. (1996). Error and the Growth of Experimental Knowledge. Chicago, IL: The University of Chicago Press
41. McShane, B. B., Gal, D., Gelman, A., Robert, C., and Tackett, J. L. (2018). Abandon statistical significance. arXiv:1709.07588v2
42. Meehl, P. E. (1967). Theory-testing in psychology and physics: a methodological paradox. Philos. Sci. 34, 103-115. doi: 10.1086/288135
43. Melton, A. (1962). Editorial. J. Exp. Psychol. 64, 553-557. doi: 10.1037/h0045549
44. Michelson, A. A., and Morley, E. W. (1887). On the relative motion of earth and luminiferous ether. Am. J. Sci. 34, 233-245. doi: 10.2475/ajs.s3-34.203.333
45. Miller, J., and Ulrich, R. (2016). Optimizing research payoff. Perspect. Psychol. Sci. 11, 664-691. doi: 10.1177/1745691616649170
46. Mudge, J. F., Baker, L. F., Edge, C. B., and Houlahan, J. E. (2012). Setting an optimal a that minimizes errors in null hypothesis significance tests. PLoS ONE 7:e32734. doi: 10.1371/journal.pone.0032734
47. Mullinix, K. J., Leeper, T. J., Druckman, J. N., and Freese, J. (2015). The generalizability of survey experiments. J. Exp. Political Sci. 2, 109-138. doi: 10.1017/XPS.2015.19
48. Myhr, A. I. (2010). A precautionary approach to genetically modified organisms: challenges and implications for policy and science. J. Agricult. Environ. Ethics 23, 501-525. doi: 10.1007/s10806-010-9234-x
49. Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. Ser. A 231, 289-337. doi: 10.1098/rsta.1933.0009
50. Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631. doi: 10.1177/1745691612459058
51. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349:aac4716. doi: 10.1126/science.aac4716
52. Page, P. (2014). Beyond statistical significance: Clinical interpretation of rehabilitation research literature. Int. J. Sports Phys. Ther. 9:72
53. Portnoy, S., and He, X. (2000). A robust journey in the new millennium. J. Am. Stat. Assoc. 95, 1331-1335. doi: 10.1080/01621459.2000.10474342
54. Rice, S., and Trafimow, D. (2010). How many people have to die for a type II error? Theor. Issues Ergon. Sci. 11, 387-401. doi: 10.1080/14639220902853096
55. Rousseeuw, P. J. (1991). Tutorial to robust statistics. J. Chemom. 5, 1-20. doi: 10.1002/cem.1180050103
56. Sawilowsky, S. (2003). Deconstructing arguments from the case against hypothesis testing. J. Modern Appl. Stat. Methods 2, 467-474. doi: 10.22237/jmasm/1067645940
57. Senn, S. (2002). A comment on replication, p-values and evidence. Stat. Med. 21, 2437-2444. doi: 10.1002/sim.1072
58. Smaldino, P. E., and McElreath, R. (2016). The natural selection of bad science. R. Soc. Open Sci. 3:160384. doi: 10.1098/rsos.160384
59. Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: three suggested reforms. Educ. Res. 25, 26-30. doi: 10.2307/1176337
60. Trafimow, D. (2017). Using the coefficient of confidence to make the philosophical switch from a posteriori to a priori inferential statistics. Educ. Psychol. Meas. 77, 831-854. doi: 10.1177/0013164416667977
61. Trafimow, D., and Earp, B. D. (2017). Null hypothesis significance testing and the use of P values to control the Type I error rate: the domain problem. New Ideas Psychol. 45, 19-27. doi: 10.1016/j.newideapsych.2017.01.002
62. Trafimow, D., and MacDonald, J. A. (2017). Performing inferential statistics prior to data collection. Educ. Psychol. Meas. 77, 204-219. doi: 10.1177/0013164416659745
63. Tukey, J. W. (1979). "Robust techniques for the user," in Robustness in Statistics, eds R. L. Launer and G. N. Wilkinson (New York, NY: Academic Press), 103-106
64. Vankov, I., Bowers, J., and Munafò, M. R. (2014). On the persistence of low power in psychological science. Q. J. Exp. Psychol. 67, 1037-1040. doi: 10.1080/17470218.2014.885986
65. Zwaan, R. A., Pecher, D., Paolacci, G., Bouwmeester, S., Verkoeijen, P., Dijkstra, K., et al. (2017). Participant Nonnaiveté and the reproducibility of cognitive psychology. Psychon. Bull. Rev. doi: 10.3758/s13423-017-1348-y