Clarifications on the application and interpretation of the test for excess significance and its extensions

Journal of Mathematical Psychology

Volumn 57, Issue 5, 2013, Pages 184-187

  Ioannidis, John P A ^a,b  

^a STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b STANFORD UNIVERSITY   (United States)

Author keywords

Bias; Publication bias; Selective reporting; Statistical significance

Indexed keywords

EID: 84886718148     PISSN: 00222496     EISSN: 10960880     Source Type: Journal    
DOI: 10.1016/j.jmp.2013.03.002     Document Type: Note

References (33)

1. Bakker M., van Dijk A., Wicherts J.M. The rules of the game called psychological science. Perspectives in Psychological Science 2012, 7:543-554.
2. Boutron I., Dutton S., Ravaud P., Altman D.G. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA 2010, 303:2058-2064.
3. Chanock S.J., Manolio T., Boehnke M., Boerwinkle E., Hunter D.J., Thomas G., et al. Replicating genotype-phenotype associations. Nature 2007, 447:655-660.
4. Chatzinasiou F., Lill C.M., Kypreou K., Stefanaki I., Nicolaou V., Spyrou G., et al. Comprehensive field synopsis and systematic meta-analyses of genetic association studies in cutaneous melanoma. Journal of the National Cancer Institute 2011, 103:1227-1235.
5. Dwan K., Altman D.G., Cresswell L., Blundell M., Gamble C.L., Williamson P.R. Comparison of protocols and registry entries to published reports for randomised controlled trials. Cochrane Database of Systematic Reviews 2011, 19. 10.1002/14651858.MR000031.pub2.
6. Evangelou E., Siontis K.C., Pfeiffer T., Ioannidis J.P. Perceived information gain from randomized trials correlates with publication in high-impact factor journals. Journal of Clinical Epidemiology 2012, 265:1274-1281.
7. Fanelli D. "Positive" results increase down the hierarchy of the sciences. PLoS ONE 2010, 5:e10068.
8. Ferguson C.J., Brannick M.T. Publication bias in psychological science: revalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods 2012, 17:120-128.
9. Francis G. The psychology of replication and replication in psychology. Perspectives in Psychological Science 2012, 7:585-594.
10. Francis G. Too good to be true: publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review 2012, 19:151-156.
11. Francis G. Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 2013, 57:153-169.
12. Gadbury G.L., Allison D.B. Inappropriate fiddling with statistical analyses to obtain a desirable p-value: tests to detect its presence in published literature. PLoS ONE 2012, 7:e46363.
13. Galak J., Meyvis T. You could have just asked: reply to Francis (2012). Perspectives in Psychological Science 2012, 7:595-596.
14. Higgins J.P., Thompson S.G., Deeks J.J., Altman D.G. Measuring inconsistency in meta-analyses. British Medical Journal 2003, 327:557-560.
15. Ioannidis J.P.A. Excess significance bias in the literature on brain volume abnormalities. Archives of General Psychiatry 2011, 68:773-780.
16. Ioannidis J.P.A. Scientific communication is down at the moment, please check again later. Psychological Inquiry 2012, 23:267-270.
17. Ioannidis J.P.A., Trikalinos T.A. Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials. Journal of Clinical Epidemiology 2005, 58:543-549.
18. Ioannidis J.P.A., Trikalinos T.A. An exploratory test for an excess of significant findings. Clinical Trials 2007, 4:245-253.
19. John L.K., Loewenstein G., Prelec D. Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science 2012, 23:524-532.
20. Johnson V., Yuan Y. Comments on an exploratory test for an excess of significant findings. Clinical Trials 2007, 4:254-255.
21. Kavvoura F.K., McQueen M.B., Khoury M.J., Tanzi R.E., Bertram L., Ioannidis J.P. Evaluation of the potential excess of statistically significant findings in reported genetic association studies: application to Alzheimer's disease. American Journal of Epidemiology 2008, 168:855-865.
22. Makel M., Plucker J., Hegarty B. Replications in psychology research: how often do they really occur?. Perspectives in Psychological Science 2012, 7:537-542.
23. Murphy, S.E., Norbury, R., Godlewska, B.R., Cowen, P.J., Mannie, Z.M., & Harmer, C.J. et al. 2012. The effect of the serotonin transporter polymorphism (5-HTTLPR) on amygdala function: a meta-analysis. Molecular Psychiatry, (in press). http://dx.doi.org/10.1038/mp.2012.19.
24. Nosek B.A., Spies J.R., Motyl M. Scientific utopia: II. restructuring incentives and practices to promote truth over publishability. Perspectives in Psychological Science 2012, 7:615-631.
25. Pereira T.V., Patsopoulos N.A., Salanti G., Ioannidis J.P.A. Critical interpretation of Cochran's Q test depends on power and prior assumptions about heterogeneity. Research Synthesis Methods 2010, 1:149-161.
26. Pereira T.V., Ioannidis J.P.A. Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology 2011, 64:1060-1069.
27. Pfeiffer T., Bertram L., Ioannidis J.P. Quantifying selective reporting and the Proteus phenomenon for multiple datasets with similar bias. PLoS One 2011, 6:e18362.
28. Sena E.S., van der Worp H.B., Bath P.M., Howells D.W., Macleod M.R. Publication bias in reports of animal stroke studies leads to major overstatement of efficacy. PLoS Biology 2010, 8:e1000344.
29. Simmons J.P., Nelson L.D., Simonsohn U. Falsepositive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 2011, 22:1359-1366.
30. Staines-Urias E., Paez M.C., Doyle P., Dudbridge F., Serrano N.C., Ioannidis J.P., et al. Genetic association studies in pre-eclampsia: systematic meta-analyses and field synopsis. International Journal of Epidemiology 2012, 41:1764-1775.
31. Sterne J.A., Sutton A.J., Ioannidis J.P., Terrin N., Jones D.R., Lau J., et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomized controlled trials. British Medical Journal 2011, 343:d4002.
32. Trinquart, L., Ioannidis, J. P. A., Chatellier, G., & Ravaud, P. A generalized test for reporting bias in trial networks and their meta-analyses: simulation and case studies (submitted for publication).
33. Tsilidis K.K., Papatheodorou S.I., Evangelou E., Ioannidis J.P. Evaluation of excess statistical significance in meta-analyses of 98 biomarker associations with cancer risk. Journal of the National Cancer Institute 2012, 104:1867-1878.