Is the call to abandon p-values the red herring of the replicability crisis?

Frontiers in Psychology

Volumn 6, Issue MAR, 2015, Pages

  Savalei, Victoria ^a   Dunn, Elizabeth ^a  

^a UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Bayes factors; Confidence intervals (CIs); Crisis of replicability; Null hypothesis significance testing (NHST); P values

Indexed keywords

EID: 84926431253     PISSN: None     EISSN: 16641078     Source Type: Journal    
DOI: 10.3389/fpsyg.2015.00245     Document Type: Note

References (46)

1. Abelson, R. P. (1997). "A retrospective on the significance test ban of 1999 (if there were no significance tests, they would be invented)," in What If There Were No Significance Tests?, eds L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Mahwah, NJ: Lawrence Erlbaum), 117-144.
2. Baril, G. L., and Cannon, J. T. (1995). What is the probability that null hypothesis testing is meaningless? Am. Psychol. 50, 1098-1099. doi: 10.1037/0003-066X.50.12.1098.b.
3. Belia, S., Fidler, F., Williams, J., and Cumming, G. (2005). Researchers misunderstand confidence intervals and standard error bars. Psychol. Methods 10, 389-396. doi: 10.1037/1082-989X.10.4.389.
4. Bem, D. J. (2011). Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J. Pers. Soc. Psychol. 100, 407-425. doi: 10.1037/a0021524.
5. Bem, D. J., Utts, J., and Johnson, W. O. (2011). Must psychologists change the way they analyze their data? J. Pers. Soc. Psychol. 101, 716-719. doi: 10.1037/a0024777.
6. Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., et al. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365-376. doi: 10.1038/nrn3475.
7. Cohen, J. (1994). The earth is round (p<.05). Am. Psychol. 49, 997-1003. doi: 10.1037/0003-066X.49.12.997.
8. Cortina, J. M., and Dunlap, W. P. (1997). On the logic and purpose of significance testing. Psychol. Methods 2, 161-172. doi: 10.1037/1082-989X.2.2.161.
9. Coulson, M., Healey, M., Fidler, F., and Cumming, G. (2010). Confidence intervals permit, but do not guarantee, better inference than statistical significance testing. Front. Psychol. 1:26. doi: 10.3389/fpsyg.2010.00026.
10. Cumming, G. (2014). The new statistics: why and how. Psychol. Sci. 25, 7-29. doi: 10.1177/0956797613504966.
11. Dienes, Z. (2011). Bayesian versus orthodox statistics: which side are you on? Perspect. Psychol. Sci. 6, 274-290. doi: 10.1177/1745691611406920.
12. Dienes, Z. (2014). How Bayes Factors Change Scientific Practice. Available online at: http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/publications.html.
13. Fidler, F., and Loftus, G. R. (2009). Why figures with error bars should replace p values: some conceptual arguments and empirical demonstrations. J. Psychol. 217, 27-37. doi: 10.1027/0044-3409.217.1.27.
14. Frick, R. W. (1995). A problem with confidence intervals. Am. Psychol. 50, 1102-1103. doi: 10.1037/0003-066X.50.12.1102.
15. Frick, R. W. (1996). The appropriate use of null hypothesis testing. Psychol. Methods 1, 379-390. doi: 10.1037/1082-989X.1.4.379.
16. Hagen, R. L. (1997). In praise of the null hypothesis statistical test. Am. Psychol. 52, 15-24. doi: 10.1037/0003-066X.52.1.15.
17. Harlow, L. L., Mulaik, S. A., and Steiger, J. H. (eds.). (1997). What If There Were No Significance Tests? Mahwah, NJ: Lawrence Erlbaum.
18. Harris, R. J. (1997). "Reforming significance testing via three-valued logic," in What If There Were No Significance Tests?, eds L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Mahwah, NJ: Lawrence Erlbaum), 145-174.
19. Hoekstra, R., Johnson, A., and Kiers, H. A. L. (2012). Confidence intervals make a difference: effects of showing confidence intervals on inferential reasoning. Educ. Psychol. Meas. 72, 1039-1052. doi: 10.1177/0013164412450297.
20. Hoekstra, R., Morey, R. D., Rouder, J. N., and Wagenmakers, E.-J. (2014). Robust misinterpretation of confidence intervals. Psychon. Bull. Rev. 21, 1157-1164. doi: 10.3758/s13423-013-0572-3.
21. Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Med. 2:e124. doi: 10.1371/journal.pmed.0020124.
22. John, L. K., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychol. Sci. 23, 524-532. doi: 10.1177/0956797611430953.
23. Krantz, D. H. (1999). The null hypothesis testing controversy in psychology. J. Am. Stat. Assoc. 44, 1372-1381. doi: 10.1080/01621459.1999.10473888.
24. Krueger, J. (2001). Null hypothesis significance testing: on the survival of a flawed method. Am. Psychol. 56, 16-26. doi: 10.1037/0003-066X.56.1.16.
25. Kruschke, J. K. (2011). Bayesian assessment of null values via parameter estimation and model comparison. Perspect. Psychol. Sci. 6, 299-312. doi: 10.1177/1745691611406925.
26. Meehl, P. E. (1990). Why summaries of research on psychological theories are often uninterpretable. Psychol. Rep. 66, 195-244. doi: 10.2466/pr0.1990.66.1.195.
27. Morey, R. D., Rouder, J. N., Verhagen, J., and Wagenmakers, E.-J. (2014). Why hypothesis tests are essential for psychological science: a comment on Cumming. Psychol. Sci. 25, 1289-1290. doi: 10.1177/0956797614525969.
28. Mulaik, S. A., Raju, N. S., and Harshman, R. A. (1997). "There is a time and place for significance testing," in What If There Were No Significance Tests?, eds L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Mahwah, NJ: Lawrence Erlbaum), 65-116.
29. Nickerson, R. S. (2000). Null hypothesis significance testing: a review of an old and continuing controversy. Psychol. Methods 5, 241-301. doi: 10.1037/1082-989X.5.2.241.
30. Nosek, B. A., Spies, J. R., and Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631. doi: 10.1177/1745691612459058.
31. Parker, S. (1995). The "Difference of means" may not be the "effect size." Am. Psychol. 50, 1101-1102. doi: 10.1037/0003-066X.50.12.1101.b.
32. Pashler, H., and Wagenmakers, E.-J. (2012). Editors' introduction to the special section on replicability in psychological science: a crisis of confidence? Perspect. Psychol. Sci. 7, 528-530. doi: 10.1177/1745691612465253.
33. Poitevineau, J., and Lecoutre, B. (2001). Interpretation of significance levels by psychological researchers: the.05 cliff effect may be overstated. Psychon. Bull. Rev. 8, 847-850. doi: 10.3758/BF03196227.
34. Reichardt, C. S., and Gollob, H. F. (1997). "When confidence intervals should be used instead of statistical significance tests, and vice versa," in What If There Were No Significance Tests?, eds L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Mahwah, NJ: Lawrence Erlbaum), 259-286.
35. Rouder, J. N. (2014). Optional stopping: no problem for Bayesians. Psychon. Bull. Rev. 21, 301-308. doi: 10.3758/s13423-014-0595-4.
36. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., and Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychon. Bull. Rev. 16, 225-237. doi: 10.3758/PBR.16.2.225.
37. Sanborn, A. N., and Hills, T. T. (2014). The frequentist implications of optional stopping on Bayesian hypothesis tests. Psychon. Bull. Rev. 21, 283-300. doi: 10.3758/s13423-013-0518-9.
38. Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: implications for training of researchers. Psychol. Methods 1, 115-129. doi: 10.1037/1082-989X.1.2.115.
39. Schmidt, F. L., and Hunter, J. E. (1997). "Eight common but false objections to the discontinuation of significance testing in the analysis of research data," in What If There Were No Significance Tests?, eds L. L. Harlow, S. A. Mulaik, and J. H. Steiger (Mahwah, NJ: Erlbaum), 37-64.
40. Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359-1366. doi: 10.1177/0956797611417632.
41. Simonsohn, U. (2014). Posterior-Hacking: Selective Reporting Invalidates Bayesian Results Also. Available online at: SSRN: http://ssrn.com/abstract=2374040.
42. Vankov, I., Bowers, J., and Munafo, M. R. (2014). On the persistence of low power in psychological science. Q. J. Exp. Psychol. 67, 1037-1040. doi: 10.1080/17470218.2014.885986.
43. Wagenmakers, E. J. (2007). A practical solution to the pervasive problems of p values. Psychon. Bull. Rev. 14, 779-804. doi: 10.3758/BF03194105.
44. Wagenmakers, E. J., Wetzels, R., Borsboom, D., and van der Maas, H. (2011). Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). J. Pers. Soc. Psychol. 100, 426-432. doi: 10.1037/a0022790.
45. Wetzels, R., Matzke, D., Lee, M. D., Rouder, J. N., Iverson, G. J., and Wagenmakers, E.-J. (2011). Statistical evidence in experimental psychology an empirical comparison using 855 t-tests. Perspect. Psychol. Sci. 6, 291-298. doi: 10.1177/1745691611406923.
46. Yu, E. C., Sprenger, A. M., Thomas, R. P., and Dougherty, M. R. (2014). When decision heuristics and science collide. Psychon. Bull. Rev. 21, 268-282. doi: 10.3758/s13423-013-0495-z.