Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size

PLoS ONE

Volumn 9, Issue 9, 2014, Pages

  Kühberger, Anton ^a   Fritz, Astrid ^b   Scherndl, Thomas ^a  

^a UNIVERSITY OF SALZBURG   (Austria)

^b Österreichisches Zentrum für Begabtenförderung und Begabungsforschung   (Austria)

Author keywords

[No Author keywords available]

Indexed keywords

DIAGNOSIS; EFFECT SIZE; HUMAN; HUMAN EXPERIMENT; POWER ANALYSIS; PSYCHOLOGY; PUBLISHING; SAMPLE SIZE; STATISTICAL SIGNIFICANCE; EPIDEMIOLOGY; MENTAL DISORDERS; RESEARCH;

EPIDEMIOLOGIC RESEARCH DESIGN; HUMANS; MENTAL DISORDERS; PSYCHOLOGY; PUBLICATION BIAS; RESEARCH; SAMPLE SIZE;

EID: 84951193475     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0105825     Document Type: Article

References (61)

1. Nickerson RS (2000) Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy. Psychological Methods 5: 241-301.
2. Levine TR, Weber R, Hullett C, Park HS, Lindsey LLM (2008) A Critical Assessment of Null Hypothesis Significance Testing in Quantitative Communication Research. Human Communication Research 34: 171-187.
3. Gliner JA, Leech NL, Morgan GA (2002) Problems with Null Hypothesis Significance Testing (NHST): What Do the Textbooks Say? The Journal of Experimental Education 71: 83-92.
4. Silva-Aycaguer LC, Suarez-Gil P, Fernandez-Somoano A (2010) The Null Hypothesis Significance Test in Health Sciences Research (1995-2006): Statistical Analysis and Interpretation. BMC Medical Research Methodology 10: 44.
5. American Educational Research Association (2006) Standards for Reporting on Empirical Social Science Research in AERA Publications. Educational Researcher 35: 33-40.
6. American Psychological Association (2010) Publication Manual of the American Psychological Association. 6th ed. Washington, DC: Author.
7. Thompson B (2009) A Brief Primer on Effect Sizes. Journal of Teaching in Physical Education 28: 251-254.
8. Fritz A, Scherndl T, Kuhberger A (2012) A Comprehensive Review of Reporting Practices in Psychological Journals - Are Effect Sizes Really Enough? Theory & Psychology 23: 98-122. doi:10.1177/0959354312436870.
9. Morris PE, Fritz CO (2013) Methods: Why are effect sizes still neglected? The Psychologist 26: 580-583.
10. Brand A, Bradley MT, Best LA, Stoica G (2011) Accuracy of Effect Size Estimates from Published Psychological Experiments Involving Multiple Trials. The Journal of General Psychology 138: 281-291.
11. Lecoutre M-P, Poitevineau J, Lecoutre B (2003) Even Statisticians Are Not Immune to Misinterpretations of Null Hypothesis Significance Tests. International Journal of Psychology 38: 37-45.
12. John LK, Loewenstein G, Prelec D (2012) Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling. Psychological Science 23: 524-532.
13. Germine L, Nakayama K, Duchaine BC, Chabris CF, Chatterjee G, et al. (2012) Is the Web as good as the lab? Comparable performance from Web and lab in cognitive/perceptual experiments. Psychonomic bulletin & review 19: 847-857.
14. Aguinis H, Beaty JC, Boik RJ, Pierce CA (2005) Effect size and power in assessing moderating effects of categorical variables using multiple regression: A 30-year review. Journal of Applied Psychology 90: 94.
15. Bakker M, Dijk A van, Wicherts JM (2012) The Rules of the Game Called Psychological Science. Perspectives on Psychological Science 7: 543-554. doi:10.1177/1745691612459060.
16. Fanelli D, Ioannidis JP (2013) US studies may overestimate effect sizes in softer research. Proceedings of the National Academy of Sciences 110: 15031-15036.
17. Ferguson CJ, Brannick MT (2012) Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 17: 120.
18. Moller AP, Jennions MD (2001) Testing and Adjusting for Publication Bias. Trends in Ecology & Evolution 16: 580-586.
19. Cooper H, DeNeve K, Charlton K (1997) Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods 2: 447.
20. Reysen S (2006) Publication of Nonsignificant Results: A Survey of Psychologists' Opinions. Psychological Reports 98: 169-175.
21. Rotton J, Foos PW, Van Meek L, Levitt M (1995) Publication Practices and the File Drawer Problem: A Survey of Published Authors. Journal of Social Behavior and Personality 10: 1-13.
22. Greenwald AG (1975) Consequences of Prejudice against the Null Hypothesis. Psychological Bulletin 82: 1-20.
23. Gigerenzer G, Murray DJ (1987) Cognition as Intuitive Statistics. Hillsdale, NJ: Erlbaum. Available: http://psycnet.apa.org/psycinfo/1987-97295-000. Accessed 2013 Nov 21.
24. Atkinson DR, Furlong MJ, Wampold BE (1982) Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship? Journal of Counseling Psychology 29: 189-194.
25. Coursol A, Wagner EE (1986) Effect of Positive Findings on Submission and Acceptance Rates: A Note on Meta-analysis Bias. Professional Psychology 17: 136-137.
26. Simmons JP, Nelson LD, Simonsohn U (2011) False-positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science 22: 1359-1366.
27. Ferguson CJ, Heene M (2012) A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null. Perspectives on Psychological Science 7: 555-561.
28. Allen M, D'Alessio D, Brezgel K (1995) A Meta-Analysis Summarizing the Effects of Pornography II Aggression after Exposure. Human Communication Research 22: 258-283.
29. Gerber AS, Green DP, Nickerson D (2001) Testing for Publication Bias in Political Science. Political Analysis 9: 385-392.
30. La France BH, Heisel AD, Beatty MJ (2004) Is there Empirical Evidence for a Nonverbal Profile of Extraversion? A Meta-analysis and Critique of the Literature. Communication Monographs 71: 28-48.
31. Levine TR, Asada KJ, Carpenter C (2009) Sample Sizes and Effect Sizes Are Negatively Correlated in Meta-analyses: Evidence and Implications of a Publication Bias against Nonsignificant Findings. Communication Monographs 76: 286-302.
32. Slavin R, Smith D (2009) The Relationship between Sample Sizes and Effect Sizes in Systematic Reviews in Education. Educational Evaluation and Policy Analysis 31: 500-506.
33. Wood W, Wong FY, Chachere JG (1991) Effects of Media Violence on Viewers' Aggression in Unconstrained Social Interaction. Psychological Bulletin 109: 371-383.
34. Dubben H-H, Beck-Bornholdt H-P (2005) Systematic review of publication bias in studies on publication bias. BMJ 331: 433-434.
35. Odgaard EC, Fowler RL (2010) Confidence Intervals for Effect Sizes: Compliance and Clinical Significance in the Journal of Consulting and Clinical Psychology. Journal of Consulting and Clinical Psychology 78: 287-297.
36. Shadish WR, Robinson L, Lu C (1999) ES: A Computer Program and Manual for Effect Size Calculation. St. Paul, MN: Assessment Systems Corporation.
37. Peterson RA, Brown SP (2005) On the Use of Beta Coefficients in Meta-analysis. Journal of Applied Psychology 90: 175-181.
38. Thompson B (2002) "Statistical", "Practical", and "Clinical": How Many Kinds of Significance Do Counselors Need To Consider? Journal of Counseling and Development 80: 64-71.
39. Gerber AS, Malhotra N (2008) Publication Bias in Empirical Sociological Research Do Arbitrary Significance Levels Distort Published Results? Sociological Methods & Research 37: 3-30.
40. Gerber AS, Malhotra N (2008) Do Statistical Reporting Standards Affect What Is Published? Publication Bias in Two Leading Political Science Journals. Quarterly Journal of Political Science 3: 313-326.
41. Wicherts JM, Bakker M, Molenaar D (2011) Willingness to share research data is related to the strength of the evidence and the quality of reporting of statistical results. PloS one 6: E26828.
42. Ridley J, Kolm N, Freckelton RP, Gage MJG (2007) An unexpected influence of widely used significance thresholds on the distribution of reported P-values. Journal of evolutionary biology 20: 1082-1089.
43. Masicampo EJ, Lalande DR (2012) A peculiar prevalence of p values just below. 05. The Quarterly Journal of Experimental Psychology 65: 2271-2279.
44. Leggett NC, Thomas NA, Loetscher T, Nicholls ME (2013) The life of p:"Just significant" results are on the rise. The Quarterly Journal of Experimental Psychology 66: 2303-2309.
45. Stern JM, Simes RJ (1997) Publication Bias: Evidence of Delayed Publication in a Cohort Study of Clinical Research Projects. British Medical Journal 315: 640.
46. Francis G (2014) The frequency of excess success for articles in Psychological Science. Psychonomic bulletin & review: 1-8.
47. Howard GS, Hill TL, Maxwell SE, Baptista TM, Farias MH, et al. (2009) What's Wrong with Research Literatures? And How to Make Them Right. Review of General Psychology 13: 146-166.
48. Ioannidis JP (2005) Why Most Published Research Findings are False. PLoS Medicine 2: 696-701.
49. Macleod M (2011) Why Animal Research Needs to Improve. Nature 477: 511.
50. Ioannidis JP (2005) Contradicted and Initially Stronger Effects in Highly Cited Clinical Research. JAMA 294: 218-228.
51. Schooler J (2011) Unpublished Results Hide the Decline Effect. Nature 470: 437.
52. Fanelli D (2010) "Positive" Results Increase Down the Hierarchy of the Sciences. PLoS One 5: E10068.
53. Stanley TD, Jarrell SB, Doucouliagos H (2010) Could it Be Better to Discard 90% of the Data? A Statistical Paradox. The American Statistician 64: 70-77.
54. Charles P, Giraudeau B, Dechartres A, Baron G, Ravaud P (2009) Reporting of Sample Size Calculation in Randomised Controlled Trials: Review. British Medical Journal 338: b1732.
55. Schulz KF, Grimes DA (2005) Sample Size Calculations in Fandomised Trials: Mandatory and Mystical. The Lancet 365: 1348-1353.
56. De Angelis C, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2004) Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors. New England Journal of Medicine 351: 1250-1251.
57. Maxmen A (2013) Preserving Research. The Scientist. Available: http://www.the-scientist.com/?articles.view/articleNo/36695/title/Preserving-Research/Accessed 30 June 2014.
58. Nosek BA, Spies JR, Motyl M (2012) Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability. Perspectives on Psychological Science 7: 615-631.
59. Makel MC, Plucker JA, Hegarty B (2012) Replications in Psychology Research How Often Do They Really Occur? Perspectives on Psychological Science 7: 537-542.
60. Cumming G, Fidler F (2009) Confidence Intervals. Better Answers to Better Questions. Journal of Psychology 217: 15-26.
61. Cumming G (2012) Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-analysis. New York, NY: Routledge.