The search for significance: A few peculiarities in the distribution of p values in experimental psychology literature

PLoS ONE

Volumn 10, Issue 6, 2015, Pages

  Krawczyk, Michał ^a  

^a UNIVERSITY OF WARSAW   (Poland)

Author keywords

[No Author keywords available]

Indexed keywords

ARTICLE; CALCULATION; EMPIRICAL RESEARCH; EXPERIMENTAL PSYCHOLOGY; HISTOGRAM; HYPOTHESIS; KERNEL METHOD; MEDICAL LITERATURE; PROBABILITY; PSYCHOLOGY; PUBLISHING; REIMBURSEMENT; SELECTION BIAS; STATISTICAL DISTRIBUTION; STATISTICAL SIGNIFICANCE; HUMAN; PROCEDURES; PUBLICATION; STATISTICAL ANALYSIS; STATISTICS AND NUMERICAL DATA;

DATA INTERPRETATION, STATISTICAL; HUMANS; PROBABILITY; PSYCHOLOGY, EXPERIMENTAL; PUBLICATION BIAS; PUBLICATIONS;

EID: 84936817477     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0127872     Document Type: Article

References (39)

1. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359-1366. doi: 10.1177/0956797611417632 PMID: 22006061
2. Nosek BA, Spies JR, Motyl M (2012) Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science 7(6): 615-631.
3. Sterling TD, Rosenbaum WL, Weinkam JJ (1995) Publication decisions revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa. American Statistician 49(1): 108-112.
4. Stanley TD (2005) Beyond Publication Bias. Journal of Economic Surveys 19(3): 309-345.
5. Dwan K, Altman D G, Arnaiz J A, Bloom J, Chan A W, Cronin E, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PloS one, 3(8), e3081. doi: 10.1371/journal.pone.0003081 PMID: 18769481
6. Ferguson C, Heene M (2012) A Vast Graveyard of Undead Theories Publication Bias and Psychological Science's Aversion to the Null. Perspectives on Psychological Science 7(6): 555-561.
7. Iyengar S, Greenhouse JB (1988) Selection models and the file drawer problem. Statistical Science 3: 109-117.
8. Wolf PK (1986) Pressure to publish and fraud in science. Annals of Internal Medicine 104(2): 254-6. PMID: 3946955
9. Steneck N (2006) Fostering Integrity in Research: Definitions, Current Knowledge, and Future Directions. Science and Engineering Ethics 12: 53-74. PMID: 16501647
10. Fanelli D (2009) How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLOS one 4: e5738. doi: 10.1371/journal.pone.0005738 PMID: 19478950
11. Martinson BC, Anderson MS, de Vries R (2005) Scientists behaving badly. Nature 435: 737-8. PMID: 15944677
12. John LK, Loewenstein G, Prelec D (2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological science 23(5): 524-532. doi: 10.1177/0956797611430953 PMID: 22508865
13. Wagenmakers E-J (2007) A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 14(5): 779-804.
14. Kerr NL (1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2: 196-217. PMID: 15647155
15. Leamer EE (1983) Let's take the con out of econometrics. American Economic Review 73(1): 31-43.
16. Roth A (1994) Let's Keep the Con out of Experimental Economics. A Methodological Note. Empirical Economics 19: 279-289. PMID: 7813966
17. Gerber A S, Malhotra N, Dowling C M, Doherty D (2010) Publication bias in two political behavior literatures. American politics research, 38(4), 591-613.
18. Ridley J, Kolm N, Freckelton RP, Gage MJG (2007) An unexpected influence of widely used significance thresholds on the distribution of reported P-values. Journal of Evolutionary Biology 20: 1082-1089. PMID: 17465918
19. Ioannidis JPA, Trikalinos TA (2007) An exploratory test for an excess of significant findings. Clinical Trials 4: 245-253. PMID: 17715249
20. Gerber AS, Malhotra N (2008a) Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals. Quarterly Journal of Political Science 3: 313-326.
21. Gerber AS, Malhotra N (2008b) Publication bias in empirical sociological research - Do arbitrary significance levels distort published results? Sociological Methods & Research 37: 3-30.
22. Brodeur A, Lé M, Sangnier M, Zylberberg Y (2013) Star wars: The empirics strike back (No. 7268). Discussion Paper Series, Forschungsinstitut zur Zukunft der Arbeit.
23. Bakker M, Wicherts JM (2011) The (mis)reporting of statistical results in psychology. Behavior Research Methods 43(3): 666-678. doi: 10.3758/s13428-011-0089-5 PMID: 21494917
24. Leggett N C, Thomas N A, Loetscher T, Nicholls M E (2013) The life of p:"Just significant" results are on the rise. The Quarterly Journal of Experimental Psychology, 66(12), 2303-2309. doi: 10.1080/17470218.2013.863371 PMID: 24205936
25. Masicampo EJ, Lalande DR (2012) A peculiar prevalence of p values just below .05. The Quarterly Journal of Experimental Psychology 65(11): 2271-2279. doi: 10.1080/17470218.2012.711335 PMID: 22853650
26. Fanelli D (2010) "Positive" results increase down the Hierarchy of the Sciences. PLoS ONE 5(3): e10068 doi: 10.1371/journal.pone.0010068
27. Hung H J, O'Neill R T, Bauer P, Kohne K (1997) The behavior of the P-value when the alternative hypothesis is true. Biometrics, 11-22.
28. Simonsohn U, Nelson L D, Simmons, J P (2013) P-Curve: A Key to the File-Drawer. Journal of Experimental Psychology: General, Jul 15.
29. Cox DR (1966) Notes on the analysis of mixed frequency distributions. British Journal of Mathematical Statistical Psychology 19: 39-47.
30. Good IJ, Gaskins RA (1980) Density estimation and bump-hunting by the penalized likelihood method exemplified by scattering and meteorite data. Journal of the American Statistical Association 75(369): 42-56. PMID: 12263159
31. Fanelli D (2013) Redefine misconduct as distorted reporting. Nature 494(149) doi: 10.1038/494149a PMID: 23407504
32. Pashler H, Wagenmakers E J (2012) Editors' Introduction to the Special Section on Replicability in Psychological Science A Crisis of Confidence?. Perspectives on Psychological Science, 7(6), 528-530.
33. Dewald W G, Thursby J G, Anderson RG (1986) "Replication in empirical economics: The journal of money, credit and banking project." The American Economic Review Vol. 76, No. 4: 587-603.
34. Humphreys M, de la Sierra R S, van der Windt P (2013) Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Political Analysis, 21(1), 1-20.
35. Cohen J (1994) The Earth Is Round (p < .05). American Psychologist 49(12): 997-1003.
36. Dixon P (2003) The p value Fallacy and How to Avoid It. Canadian Journal of Experimental Psychology 57(3): 189-202. PMID: 14596477
37. Davies J B, Ross A (2012) Sorry everyone, but it didn't work (p = 0.06). Addiction Research & Theory, 21(4), 348-355.
38. Sterne JAC, Smith GD (2001) Sifting the evidence - what's wrong with significance tests? British Medical Journal 322: 226-31. PMID: 11159626
39. Goodman SN (1999) Toward Evidence-Based Medical Statistics .1: The P Value fallacy. Annals of Internal Medicine 130: 995-1004. PMID: 10383371