A tutorial on hunting statistical significance by chasing N

Frontiers in Psychology

Volumn 7, Issue SEP, 2016, Pages

  Szucs, Denes ^a  

^a UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

Bias and data dredging; False positive error; N hacking; Null hypothesis significance testing (NHST); P hacking; Replication crisis; Type I error

Indexed keywords

EID: 84992694576     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2016.01444     Document Type: Article

References (50)

1. Anscombe, F. J. (1963). Sequential medical trials. J. Am. Stat. Assoc. 58, 365-383. doi: 10.1080/01621459.1963.10500851
2. Bakan, D. (1966). The test of significance in psychological research. Psychol. Bull. 66, 423-437. doi: 10.1037/h0020412
3. Bakker, M., Van Dijk, A., and Wicherts, J. M. (2012). The rules of the game called psychological science. Perspect. Psychol. Sci. 7, 543-554. doi: 10.1177/1745691612459060
4. Bakker, M., and Wicherts, J. M. (2014). Outlier removal, sum scores, and the inflation of Type I error rate in independent samples t tests: the power of alternatives and recommendations. Psychol. Methods 19, 409-427. doi: 10.1037/met0000014
5. Barnett, V., and Lewis, T. (1994). Outliers in Statistical Data. Chichester: Wiley.
6. Benjamini, Y. (2010). Simultaneous and selective inference: current successes and future challenges. Biom. J. 52, 708-721. doi: 10.1002/bimj.200900299
7. Benjamini, Y., and Hochberg, Y. (1995). Controllingth ef alsed iscoveryra te:a practicalan d powerfulap proacht o multiplete sting. R. Statist. Soc. B 57, 89-300.
8. Benjamini, Y., and Yekutieli, D. (2001). The control of false discovery rate in multiple testing under dependency. Ann. Stat. 29, 1165-1188. doi: 10.1186/1471-2105-9-114
9. Bennett, C. M., Wolford, G. L., and Miller, M. B. (2009). The principled control of false positives in neuroimaging. Soc. Cogn. Affect. Neurosci. 4, 417-442. doi: 10.1093/scan/nsp053
10. Bruns, S., and Ioannidis, J. P. (2016). p-Curve and p-Hacking in observational research. PLoS ONE 11:e0149144. doi: 10.1371/journal.pone.0149144
11. Button, K. S., Ioannidis, J., Mokrysz, C., and Nosek, B. A. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365-376. doi: 10.1038/nrn3475
12. Curran-Everett, D. (2000). Multiple comparisons: philosophies and illustrations. Am. J. Physiol. Regul. Integr. Comp. Physiol. 279, R1-R8.
13. Demets, D. L., and Gordon-Lan, K. K. (1994). Interim analysis: the alpha spending function approach. Stat. Med. 13, 1341-1352. doi: 10.1002/sim.4780131308
14. Fanelli, D. (2010). 'Positive' results increase down the hierarchy of sciences. PLoS ONE. 5:e10068. doi: 10.1371/journal.pone.0010068
15. Fanelli, D. (2012). Negative results are disappearing form most disciplines and countries. Scientometrics 90, 891-904. doi: 10.1007/s11192-011-0494-7
16. Francis, G. (2013). Replication, statistical consistency and publication bias. J. Math. Psychol. 57, 153-169. doi: 10.1016/j.jmp.2013.02.003
17. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., and Rubin, D. (2014). Bayesian Data Analysis. Boca Raton, FL: CRC Press.
18. Gelman, A., Hill, J., and Yajima, M. (2012). Why we (usually) do not have to worry about multiple comparisons. J. Res. Educ. Eff. 5, 189-211. doi: 10.1080/19345747.2011.618213
19. Gleason, A. M., and Hughes Hallett, D. (1992). The Calculus Consortium Based at Harvard University. A Newsletter of the Calculus Consortium Based at Harvard University. Cambridge, MA: Harvard University.
20. Goeman, J. J., and Solari, A. (2014). Multiple hypothesis testing in genomics. Stat. Med. 20, 1946-1978. doi: 10.1002/sim.6082
21. Goodman, S. N. (1999). Toward evidence-based medical statistics 1: the p value fallacy. Ann. Intern. Med. 130, 995-1004. doi: 10.7326/0003-4819-130-12-199906150-00008
22. Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Med. 2:e124. doi: 10.1371/journal.pmed.0020124
23. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology 19, 640-648. doi: 10.1097/EDE.0b013e31818131e7
24. Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., et al. (2014). Increasing value and reducing waste and research design, conduct and analysis. Lancet 383, 166-175. doi: 10.1016/S0140-6736(13)62227-8
25. Ioannidis, J. P. A., and Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clin. Trials 4, 245-253. doi: 10.1177/1740774507079441
26. Khoury, M. J., and Ioannidis, J. P. A. (2014). Big data meets public health. Science 346, 1054-1055. doi: 10.1126/science.aaa2709
27. Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., and Baker, C. I. (2009). Circular analysis in systems neuroscience-the dangers of double dipping. Nat. Neurosci. 12, 535-540. doi: 10.1038/nn.2303
28. Lykken, D. T. (1968). Statistical significance in psychological research. Psychol. Bull. 70, 151-159. doi: 10.1037/h0026141
29. MacKay, D. J. C. (2003). Information Theory, Inference and Learning Algorithms. Cambridge: Cambridge University Press.
30. Meehl, P. E. (1967). Theory testing in psychology and physics: a methodological paradox. Philos. Sci. 34, 103-115. doi: 10.1086/288135
31. Neyman, J., and Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philos. Trans. R. Soc. Lond. Ser. A 231, 289-337. doi: 10.1098/rsta.1933.0009
32. Nichols, T., and Hayasaka, S. (2003). Controlling the familywise error rate in neuroimaging: a comparative review. Stat. Methods Med. Res. 12, 419-446. doi: 10.1191/0962280203sm341ra
33. Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349:943. doi: 10.1126/science.aac4716
34. Pollard, P., and Richardson, J. T. E. (1987). On the probability of making Type-I errors. Psychol. Bull. 102, 159-163.
35. Rossi, J. S. (1990). Statistical power of psychological research: what have we gained in 20 years? J. Consult. Clin. Psychol. 58, 646-656. doi: 10.1037/0022-006X.58.5.646
36. Rozeboom, W. W. (1960). The fallacy of the null hypothesis significance test. Psychol. Bull. 57, 416-428. doi: 10.1037/h0042040
37. Sedlmeier, P., and Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of the studies? Psychol. Bull. 105, 309-316. doi: 10.1037/0033-2909.105.2.309
38. Sellke, T., Bayarri, M. J., and Berger, J. O. (2001). Calibration of p values for testing precise null hypotheses. Am. Stat. 55, 62-71. doi: 10.1198/000313001300339950
39. Shaffer, J. P. (1995). Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561-584. doi: 10.1146/annurev.ps.46.020195.003021
40. Shih, W. J., and Aisner, J. (2016). Statistical Design and Analysis of Clinical Trials. Boca Raton, FL: Chapman and Hall/CRC.
41. Silberzahn, R., and Uhlmann, E. L. (2015). Crowdsourced research: many hands make tight work. Nature 526, 189-191. doi: 10.1038/526189a
42. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632
43. Simonsohn, U., Nelson, L. D., and Simmons, J. P. (2014). P-curve: a key to the file drawer. J. Exp. Psychol. Gen. 143, 534-547. doi: 10.1037/a0033242
44. Sivia, D. S., and Skilling, J. (2006). Data Analysis: A Bayesian tutorial. Oxford: Oxford University Press.
45. Vul, E., Harris, C., Winkielman, P., and Pashler, H. (2009). Puzzlingly high correlations in fMRI studies of emotion, personality and social cognition. Perspect. Psychol. Sci. 4, 274-324. doi: 10.1111/j.1745-6924.2009.01132.x
46. Wagenmakers, E. J. (2007). A practical solution to the pervasive problem of p values. Psychon. Bull. Rev. 14, 779-804. doi: 10.3758/BF03194105
47. Waller, N. G. (2004). The fallacy of the null hypothesis in soft psychology. Appl. Prev. Psychol. 11, 83-86. doi: 10.1016/j.appsy.2004.02.015
48. Whitehead, J. (1997). The Design and Analysis of Sequential Clinical Trials. Hoboken, NJ: Wiley.
49. Whitehead, J. (1999). A unified theory for sequential clinical trials. Stat. Med. 18, 2271-2286. doi: 10.1002/(SICI)1097-0258(19990915/30)18:17/18<2271::AID-SIM254>3.0.CO;2-Z
50. Wilcox, R. R. (1998). How many discoveries have been lost by ignoring modern statistical methods? Am. Psychol. 53, 300-314. doi: 10.1037/0003-066X.53.3.300