Blinded by the Light: How a Focus on Statistical “Significance” May Cause p-Value Misreporting and an Excess of p-Values Just Below.05 in Communication Science

Communication Methods and Measures

Volumn 9, Issue 4, 2015, Pages 253-279

  Vermeulen, Ivar ^a   Beukeboom, Camiel J ^a   Batenburg, Anika ^a   Avramiea, Arthur ^a   Stoyanov, Dimo ^a   van de Velde, Bob ^a   Oegema, Dirk ^a  

^a VU UNIVERSITY AMSTERDAM   (Netherlands)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84948665787     PISSN: 19312458     EISSN: 19312466     Source Type: Journal    
DOI: 10.1080/19312458.2015.1096333     Document Type: Article

References (44)

1. J.B.Asendorpf,, M.Conner,, F.De Fruyt,, J.De Houwer,, J.J.A.Denissen,, K.Fiedler, … J.M.Wicherts, (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. doi:10.1002/per.1919
2. M.Bakker,, & J.M.Wicherts, (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43(3), 666–678. doi:10.3758/s13428-011-0089-5
3. M.Bakker,, & J.M.Wicherts, (2014). Outlier removal and the relation with reporting errors and quality of psychological research. Plos One, 9(7), e103360. doi:10.1371/journal.pone.0103360
4. M.J.Brandt,, H.IJzerman,, A.Dijksterhuis,, F.J.Farach,, J.Geller,, R.Giner-Sorolla, … A.Van’t Veer, (2014). The replication recipe: What makes for a convincing replication? Journal of Experimental Social Psychology, 50, 217–224. doi:10.1016/j.jesp.2013.10.005
5. J.Cohen, (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York, NY: Lawrence Erlbaum Associates.
6. G.Cumming, (2008). Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science, 3, 286–300. doi:10.1111/ppsc.2008.3.issue-4
7. E.Dawson,, T.Gilovich,, & D.T.Regan, (2002). Motivated reasoning and performance on the Wason selection task. Personality and Social Psychology Bulletin, 28, 1379–1387. doi:10.1177/014616702236869
8. K.Dickersin, (1990). The existence of publication bias and risk-factors for its occurrence. JAMA: The Journal of the American Medical Association, 263(10), 1385–1389. doi:10.1001/jama.1990.03440100097014
9. D.Fanelli, (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5(4), e10068. doi:10.1371/journal.pone.0010068
10. A.Franco,, N.Malhotra,, & G.Simonovits, (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 1255484. doi:10.1126/science.1255484
11. E.Garcia-Berthou,, & C.Alcaraz, (2004). Incongruence between test statistics and P values in medical papers. BMC Medical Research Methodology, 4(1), 13. doi:10.1186/1471-2288-4-13
12. M.L.Head,, L.Holman,, R.Lanfear,, A.T.Kahn,, & M.D.Jennions, (2015). The extent and consequences of p-hacking in science. Plos Biology, 13(3), e1002106–e1002106. doi:10.1371/journal.pbio.1002106
13. J.P.A.Ioannidis, (2005). Why most published research findings are false. Plos Medicine, 2.8, e124. doi:10.1371/journal.pmed.0020124
14. J.P.A.Ioannidis, (2008). Why most discovered true associations are inflated. Epidemiology, 19(5), 640–648. doi:10.1097/EDE.0b013e31818131e7
15. L.K.John,, G.Loewenstein,, & D.Prelec, (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
16. Kahneman, D. (2012). Open letter: A proposal to deal with questions about priming effects. Retrieved from http://www.decisionsciencenews.com/2012/10/05/kahneman-on-the-storm-of-doubts-surrounding-social-priming-research/
17. N.L.Kerr, (1998). HARKing: hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. doi:10.1207/s15327957pspr0203_4
18. S.L.Koole,, & D.Lakens, (2012). Rewarding replications: A sure and simple way to improve psychological science. Perspectives on Psychological Science, 7(6), 608–614. doi:10.1177/1745691612462586
19. A.Kühberger,, A.Fritz,, & T.Scherndl, (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. Plos One, 9(9), e105825. doi:10.1371/journal.pone.0105825
20. Z.Kunda, (1990). The case for motivated reasoning. Psychological Bulletin, 108(3), 480–498. doi:10.1037/0033-2909.108.3.480
21. D.Lakens, (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, 863. doi:10.3389/fpsyg.2013.00863
22. D.Lakens, (2015). What p-hacking really looks like: A comment on Masicampo and LaLande (2012). The Quarterly Journal of Experimental Psychology, 68(4), 829–832.
23. N.C.Leggett,, N.A.Thomas,, T.Loetscher,, & M.E.R.Nicholls, (2013). The life of p: “Just significant” results are on the rise. The Quarterly Journal of Experimental Psychology, 66(12), 2303–2309. doi:10.1080/17470218.2013.863371
24. W.J.M.Levelt,, P.Drenth,, & E.Noort, (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Tilburg, The Netherlands: Commissioned by the Tilburg University, University of Amsterdam and the University of Groningen.
25. M.C.Makel,, J.A.Plucker,, & B.Hegarty, (2012). Replications in psychology research: How often do they really occur? Perspectives on Psychological Science, 7(6), 537–542. doi:10.1177/1745691612460688
26. E.J.Masicampo,, & D.R.Lalande, (2012). A peculiar prevalence of p values just below.05. The Quarterly Journal of Experimental Psychology, 65(11), 2271–2279. doi:10.1080/17470218.2012.711335
27. N.Mazar,, O.Amir,, & D.Ariely, (2008). The dishonesty of honest people: A theory of self-concept maintenance. Journal of Marketing Research, 45(6), 633–644. doi:10.1509/jmkr.45.6.633
28. J.Mervis, (2014). Why null results rarely see the light of day. Science, 345(6200), 992–992. doi:10.1126/science.345.6200.992
29. R.S.Nickerson, (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5(2), 241–301. doi:10.1037/1082-989X.5.2.241
30. B.A.Nosek,, & Y.Bar-Anan, (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217–243. doi:10.1080/1047840X.2012.692215
31. B.A.Nosek,, & D.Lakens, (2014). Registered reports: A method to increase the credibility of published results. Social Psychology, 45(3), 137–141. doi:10.1027/1864-9335/a000192
32. B.A.Nosek,, J.R.Spies,, & M.Motyl, (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615–631. doi:10.1177/1745691612459058
33. H.Pashler,, & E.-J.Wagenmakers, (2012). Editors’ Introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530. doi:10.1177/1745691612465253
34. R.Rosenthal, (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. doi:10.1037/0033-2909.86.3.638
35. R.L.Rosnow,, & R.Rosenthal, (1989). Statistical procedures and the justification of knowledge in psychological science. American Psychologist, 44, 1276–1284. doi:10.1037/0003-066X.44.10.1276
36. J.Sakaluk,, A.Williams,, & M.Biernat, (2014). Analytic review as a solution to the misreporting of statistical results in psychological science. Perspectives on Psychological Science, 9(6), 652–660. doi:10.1177/1745691614549257
37. T.Sellke,, M.J.Bayarri,, & J.O.Berger, (2001). Calibration of ρ values for testing precise null hypotheses. The American Statistician, 55(1), 62–71. doi:10.1198/000313001300339950
38. J.P.Simmons,, L.D.Nelson,, & U.Simonsohn, (2011). False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632
39. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2013). Life after p-hacking (SSRN Scholarly Paper No. ID 2205186). Rochester, NY: Social Science Research Network. Retrieved from http://papers.ssrn.com/abstract=2205186
40. U.Simonsohn,, L.Nelson,, & J.Simmons, (2014). P-curve: A key to the file drawer. Journal of Experimental Psychology: General, 143(2), 534–547. doi:10.1037/a0033242
41. M.Stathern, (1997). ‘Improving ratings’: Audit in the British University system. European Review, 5(3), 305–321.
42. R.Van De Schoot,, D.Kaplan,, J.Denissen,, J.B.Asendorpf,, F.J.Neyer,, & M.A.G.Van Aken, (2014). A gentle introduction to Bayesian analysis: Applications to research in child development. Child Development, 85(3), 842–860. doi:10.1111/cdev.12169
43. E.-J.Wagenmakers, (2007). A practical solution to the pervasive problems ofp values. Psychonomic Bulletin & Review, 14(5), 779–804. doi:10.3758/BF03194105
44. J.M.Wicherts,, M.Bakker,, & D.Molenaar, (2011). Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PLoS ONE, 6(11), e26828. doi:10.1371/journal.pone.0026828