A Vast Graveyard of Undead Theories: Publication Bias and Psychological Science's Aversion to the Null

Perspectives on Psychological Science

Volumn 7, Issue 6, 2012, Pages 555-561

  Ferguson, Christopher J ^a   Heene, Moritz ^b  

^a TEXAS A AND M INTERNATIONAL UNIVERSITY   (United States)

^b UNIVERSITY OF MUNICH   (Germany)

Author keywords

fail safe number; falsification; meta analyses; null hypothesis significance testing; publication bias

Indexed keywords

EID: 84869028702     PISSN: 17456916     EISSN: 17456924     Source Type: Journal    
DOI: 10.1177/1745691612459059     Document Type: Article

References (63)

1. Anderson C. A.,Shibuya A.,Ihori N.,Swing E. L.,Bushman B. J.,Sakamoto A.,Saleem M.Violent video game effects on aggression, empathy, and prosocial behavior in Eastern and Western countries: A meta-analytic review.Psychological Bulletin. 2010;136:151-173.
2. Atkinson D. R.,Furlong M. J.,Wampold B. E.Statistical significance, reviewer evaluations, and the scientific process: Is there a (statistically) significant relationship?.Journal of Counseling Psychology. 1982;29:189-194.
3. Baumeister R. F.,DeWall C.,Vohs K. D.Social rejection, control, numbness, and emotion: How not to be fooled by Gerber and Wheeler (2009).Perspectives On Psychological Science. 2009;4:489-493.
4. Begley C. G.,Ellis L. M.Drug development: Raise standards for preclinical cancer research.Nature. 2012;483:531-533.
5. Bem D. J.Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect.Journal of Personality and Social Psychology. 2011;100:407-425.
6. Cohen J."The earth is round (p <.05)".American Psychologist. 1994;49:997-1003.
7. Coursol A.,Wagner E. E.Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias.Professional Psychology: Research and Practice. 1986;17:136-137.
8. Cumming G.Replication and p intervals: P values predict the future only vaguely, but confidence intervals do much better.Perspectives on Psychological Science. 2008;3:286-300.
9. Cumming G.,Maillardet R.Confidence intervals and replication: Where will the next mean fall?.Psychological Methods. 2006;11:217-227.
10. Dallam S. J.,Gleaves D. H.,Cepeda-Benito A.,Silberg J. L.,Kraemer H. C.,Spiegel D.The effects of child sexual abuse: Comment on Rind, Tromovitch, and Bauserman (1998).Psychological Bulletin. 2001;127:715-733.
11. Dickersin K.,Min Y. I.Publication bias: The problem that won't go away.Annals of the New York Academy of Sciences. 1993;703:135-148.
12. Drotar D.Editorial: A call for replications of research in pediatric psychology and guidance for authors.Journal of Pediatric Psychology. 2010;35:801-805.
13. Elsahoff J. D.Commentary.Behavioral and Brain Sciences. 1978;1:392.
14. Fanelli D.Do pressures to publish increase scientists' bias? An empirical support from US States data.PLoS ONE. 2010a;5:e10271.
15. Fanelli D."Positive" results increase down the hierarchy of the sciences.PLoS ONE. 2010b;5:e10068.
16. Fanelli D.Negative results are disappearing from most disciplines and countries.Scientometrics. 2012;90:891-904.
17. Ferguson C. J.An effect size primer: A guide for clinicians and researchers.Professional Psychology: Research and Practice. 2009;40:532-538.
18. Ferguson C. J.,Brannick M. T.Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses.Psychological Methods. 2012;17:120-128.
19. Ferguson C. J.,Kilburn J.The public health risks of media violence: A meta-analytic review.Journal of Pediatrics. 2009;154:759-763.
20. Ferguson C. J.,Kilburn J.Much ado about nothing: The misestimation and over interpretation of violent video game effects in Eastern and Western nations: Comment on Andersonet al.(2010).Psychological Bulletin. 2010;136:174-178.
21. Ferguson C. J.,Munoz M. E.,Contreras C.,Velasquez K.Mirror, mirror on the wall: Peer competition, television influences and body image dissatisfaction.Journal of Social & Clinical Psychology. 2011;30:458-483.
22. Ferguson C. J.,San Miguel C.,Garza A.,Jerabeck J.A longitudinal test of video game violence effects on dating violence, aggression and bullying: A 3-year longitudinal study of adolescents.Journal of Psychiatric Research. 2012;46:141-146.
23. Ferguson E.,Bibby P. A.Openness to experience and all-cause mortality: A meta-analysis and requivalent from risk ratios and odds ratios.British Journal of Health Psychology. 2011;:.
24. Fritz A.,Scherndl T.,Kühberger A. Paper presented at the 10th Conference of the Austrian Psychological Society; 2012Graz, Austria; 2012.
25. Gerber J.,Wheeler L.On being rejected: A meta-analysis of experimental research on rejection.Perspectives on Psychological Science. 2009;4:468-488.
26. Greenwald A. G.Consequences of prejudice against the null hypothesis.Psychological Bulletin. 1975;82:1-12.
27. Greenwald A. G.There is nothing so theoretical as a good method.Perspectives on Psychological Science. 2012;7:99-108.
28. HeeneM. (2010). A brief history of the fail safe number in applied research. Arxiv preprint arXiv:1010.2326, 1-8. Retrieved from http://arxiv.org/ftp/arxiv/papers/1010/1010.2326.pdf.
29. HeeneM.LorenziP.SchonemannP. H. (2010). How valid is the General Record Examination really? Retrieved from http://demonstrations.wolfram.com/HowValidIsTheGeneralRecordExaminationReally/.
30. Hsu L. M.Fail-safe ns for one- versus two-tailed tests lead to different conclusions about publication bias.Understanding Statistics. 2002;1:85-100.
31. Ioannidis J. P.Why most published research findings are false.PLoS Medicine. 2005;2:e124.
32. IoannidisJ. P.TrikalinosT. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey. Canadian Medical Association Journal, 176(8). Retrieved from http://www.cmaj.ca/content/176/8/1091.full.
33. Iyengar S.,Greenhouse J.Rejoinder.Statistical Science. 1988a;3:133-135.
34. Iyengar S.,Greenhouse J. B.Selection models and the file drawer problem.Statistical Science. 1988b;3:109-117.
35. LeBel E. P.,Peters K. R.Fearing the future of empirical psychology: Bem's (2011) evidence of psi as a case study of deficiencies in modal research practice.Review of General Psychology. 2011;15:371-379.
36. Levine T. R.,Asada K. J.,Carpenter C.Sample sizes and effect sizes are negatively correlated in meta-analyses: Evidence and implications of a publication bias against nonsignificant findings.Communication Monographs. 2009;76:286-302.
37. Levine T. R.,Weber R.,Park H.,Hullett C.A communication researchers' guide to null hypothesis significance testing and alternatives.Human Communication Research. 2008;34:188-209.
38. Lilienfeld S. O.When worlds collide: Social science, politics, and the Rindet al.(1998) child sexual abuse meta-analysis.American Psychologist. 2002;57:176-188.
39. Martins A.,Ramalho N.,Morin E.A comprehensive meta-analysis of the relationship between emotional intelligence and health.Personality and Individual Differences. 2010;49:554-564.
40. Meyer G. J.,Finn S. E.,Eyde L. D.,Kay G. G.,Moreland K. L.,Dies R. R.,Reed G. M.Psychological testing and psychological assessment: A review of evidence and issues.American Psychologist. 2001;56:128-165.
41. Prinzie P.,Stams G. J. J.,Deković M.,Reijntjes A. H.,Belsky J.The relations between parents' Big Five personality factors and parenting: A meta-analytic review.Journal of Personality and Social Psychology. 2009;97:351-362.
42. Rind B.,Tromovitch P.,Bauserman R.A meta-analytic examination of assumed properties of child sexual abuse using college samples.Psychological Bulletin. 1998;124:22-53.
43. Rosenthal R.The file drawer problem and tolerance for null results.Psychological Bulletin. 1979;86:638-641.
44. Rosenthal R.,Rubin D. B.Interpersonal expectancy effects: The first 345 studies.Behavioral and Brain Sciences. 1978;1:377-386.
45. Rosnow R.,Rosenthal R.Effect sizes for experimenting psychologists.Canadian Journal of Experimental Psychology. 2003;57:221-237.
46. Rothstein H. R.,Bushman B. J.Publication bias in psychological science: Comment on Ferguson and Brannick (2012).Psychological Methods. 2012;17:129-136.
47. ScargleJ. D. (2000). Publication bias: The "File-Drawer" problem in scientific inference. Journal of Scientific Exploration, 14, 91-106. Retrieved from http://www.scientificexploration.org/journal/jse_14_1_scargle.pdf.
48. Scherer R. W.,Langenberg P.,Von Elm E.Full publication of results initially presented in abstracts.Cochrane Database of Systematic Reviews. 2007;2:158-162.
49. SchonemannP. H.HeeneM. (2009). Predictive validities: Figures of merit or veils of deception?Psychology Science Quarterly, 51, 195-215. Retrieved from http://www.psychologie-aktuell.com/fileadmin/download/PschologyScience/2-2009/06_Heene.pdf.
50. SchonemannP. H.ScargleJ. D. (2008). A generalized publication bias model. Chinese Journal of Psychology, 50, 21-29. Retrieved from http://www.schonemann.de/pdf/91_Schonemann_Scargle.pdf.
51. SheaC. (2011). Fraud scandal fuels debate over practices of social psychology: Even legitimate researchers cut corners, some admit. Chronicle of Higher Education. Retrieved from http://chronicle.com/article/As-Dutch-Research-Scandal/129746/.
52. Sherry J.Mass media effects research: Advances through meta-analysis. Preiss R.Gayle B.Burrell N.Allen M.Bryant J., ed. Mahwah, NJ: Erlbaum; 2007:231-248.
53. Simmons J. P.,Nelson L. D.,Simonsohn U.False- positive psychology.Psychological Science. 2011;22:1359-1366.
54. Sterling T. D.,Rosenbaum W. L.,Weinkam J. J.Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa.American Statistician. 1995;49:108-112.
55. Sterne J.,Gravaghan D.,Egger M.Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature.Journal of Clinical Epidemiology. 2000;53:1119-1129.
56. Thornton A.,Lee P.Publication bias in meta-analysis: Its causes and consequences.Journal of Clinical Epidemiology. 2000;53:207-216.
57. Trafimow D.The theory of reasoned action: A case study of falsification in psychology.Theory & Psychology. 2009;19:501-518.
58. Trzesniewski K. H.,Donnellan M.,Robins R. W.Do today's young people really think they are so extraordinary? An examination of secular trends in narcissism and self-enhancement.Psychological Science. 2008;19:181-188.
59. Twenge J. M.,Konrath S.,Foster J. D.,Campbell W.,Bushman B. J.Egos inflating over time: A cross-temporal meta-analysis of the Narcissistic Personality Inventory.Journal of Personality. 2008;76:875-902.
60. Voyer D.Time limits and gender differences on paper- and-pencil tests of mental rotation: A meta-analysis.Psychonomic Bulletin & Review. 2011;18:1-11.
61. Wagenmakers E.,Wetzels R.,Borsboom D.,van der Maas H. J.Why psychologists must change the way they analyze their data: The case of psi: Comment on Bem (2011).Journal of Personality and Social Psychology. 2011;100:426-432.
62. Wallach L.,Wallach M. A.Some theories are unfalsifiable: A comment on Trafimow.Theory & Psychology. 2010;20:703-706.
63. Statistical methods in psychological journals: Guidelines and explanations.American Psychologist. 1999;54:594-604.