Replication, statistical consistency, and publication bias

Journal of Mathematical Psychology

Volumn 57, Issue 5, 2013, Pages 153-169

  Francis, Gregory ^a  

^a PURDUE UNIVERSITY   (United States)

Author keywords

Hypothesis testing; Publication bias; Scientific publishing; Statistics

Indexed keywords

EID: 84886719811     PISSN: 00222496     EISSN: 10960880     Source Type: Journal    
DOI: 10.1016/j.jmp.2013.02.003     Document Type: Article

References (70)

1. Anscombe F.J. Fixed-sample-size analysis of sequential observations. Biometrics 1954, 10(1):89-100.
2. Observation of a new particle in the search for the standard model Higgs boson with the ATLAS detector at the LHC. Physics Letters B 2012, 716(1):1-29. ATLAS collaboration, arXiv:1207.7214. 10.1016/j.physletb.2012.08.020.
3. Balcetis E., Dunning D. Wishful seeing: more desired objects are seen as closer. Psychological Science 2010, 21(1):147-152.
4. Balcetis E., Dunning D. A false-positive error in search in selective reporting: a refutation of Francis. i-Perception 2012, 3. Author response.
5. Begg C.B., Mazumdar M. Operating characteristics of a rank correlation test for publication bias. Biometrics 1994, 50:1088-1101.
6. Bem D.J. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology 2011, 100:407-425.
7. Bones A.K. We knew the future all along. Perspectives on Psychological Science 2012, 7(3):307-309. 10.1177/1745691612441216.
8. Borenstein M., Hedges L.V., Higgins J.P.T., Rothstein H.R. Introduction to meta-analysis 2009, Wiley, Chichester, UK.
9. Champely, S. (2009). Pwr: Basic functions for power analysis. R package version 1.1.1. http://CRAN.R-project.org/package%3Dpwr.
10. Observation of a new boson at a mass of 125 GeV with the CMS experiment at the LHC. Physics Letters B 2012, 716(1):30-61. CMS collaboration, arXiv:1207.7235. 10.1016/j.physletb.2012.08.021.
11. Cohen J. Statistical power analysis for the behavioral sciences 1988, Erlbaum, Hillsdale, NJ. 2nd ed.
12. Cumming G. Replication and p intervals: p values predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science 2008, 3:286-300.
13. Cumming G. Understanding the new statistics: effect sizes, confidence intervals, and meta-analysis 2012, Routledge, New York.
14. Duval S., Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000, 56:455-463.
15. Enserink M. Dutch university sacks social psychologist over faked data. Science 2011, http://news.sciencemag.org/scienceinsider/2011/09/dutch-university-sacks-social.html.
16. Fanelli D. Positive results increase down the hierarchy of the sciences. PLoS ONE 2010, 5(4):e10068. 10.1371/journal.pone.0010068.
17. Francis G. Too good to be true: publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review 2012, 19:151-156.
18. Francis G. The same old new look: publication bias in a study of wishful seeing. i-Perception 2012, 3:176-178.
19. Francis G. Response to author: some clarity about publication bias and wishful seeing. i-Perception 2012, 3. 10.1068/i0519ic.
20. Francis G. Evidence that publication bias contaminated studies relating social class and unethical behavior. Proceedings of the National Academy of Sciences 2012, 109:E1587.
21. Francis, G. (2012e). Checking the counterarguments confirms that publication bias contaminated studies relating social class and unethical behavior.Downloaded from http://www1.psych.purdue.edu/~gfrancis/Publications/FrancisRebuttal2012.pdf.
22. Francis G. Publication bias and the failure of replication in experimental psychology. Psychonomic Bulletin & Review 2012, 10.3758/s13423-012-0322-y.
23. Francis G. The psychology of replication and replication in psychology. Perspectives on Psychological Science 2012, 7(6):580-589. 10.1177/1745691612459520.
24. Francis G. Publication bias in red, rank, and romance in women viewing men by Elliot et al. (2010). Journal of Experimental Psychology: General 2013, 142(1):292-296.
25. Galak J., LeBoeuf R.A., Nelson L.D., Simmons J.P. Correcting the past: failures to replicate Psi. Journal of Personality and Social Psychology 2012, 103(6):933-948.
26. Galak J., Meyvis T. The pain was greater if it will happen again: the effect of anticipated continuation on retrospective discomfort. Journal of Experimental Psychology: General 2011, 140:63-75.
27. Galak J., Meyvis T. You could have just asked: reply to Francis (2012). Perspectives on Psychological Science 2012, 7(6):595-596.
28. Gillett R. Post hoc power analysis. Journal of Applied Psychology 1994, 79:783-785.
29. Hedges L.V. Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics 1981, 6:107-128.
30. Hedges L.V., Olkin I. Statistical methods for meta-analysis 1985, Academic Press, New York.
31. Hoenig J.M., Heisey D.M. The abuse of power: the pervasive fallacy of power calculations for data analysis. American Statistician 2001, 55:19-24.
32. Ioannidis J.P.A. Why most discovered true associations are inflated. Epidemiology 2008, 19:640-648.
33. Ioannidis J.P.A., Trikalinos T.A. An exploratory test for an excess of significant findings. Clinical Trials 2007, 4:245-253.
34. John L.K., Loewenstein G., Prelec D. Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science 2012, 23:524-532.
35. Johnson C.S., Smeesters D., Wheeler S.C. Retraction of Johnson, Smeesters, and Wheeler (2012). Journal of Personality and Social Psychology 2012, 103(4):605.
36. Johnson V., Yuan Y. Comments on an exploratory test for an excess of significant findings by JPA loannidis and TA Trikalinos. Clinical Trials 2007, 4:254-255.
37. Kelley K. Confidence intervals for standardized effect sizes: theory, application, and implementation. Journal of Statistical Software 2007, 20. http://www.jstatsoft.org/v20/a08/.
38. Kerr N.L. HARKing: hypothesizing after the results are known. Personality and Social Psychology Review 1998, 2:196-217.
39. Kim J., Francis G. Color selection, color capture, and afterimage filling-in. Journal of Vision 2011, 11(3):22. http://www.journalofvision.org/content/11/3/23/, 10.1167/11.3.23.
40. Kline R.B. Beyond significance testing: reforming data analysis methods in behavioral research 2004, American Psychological Association, Washington, DC.
41. Kruschke J.K. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science 2010, 1(5):658-676. 10.1002/wcs.72.
42. Kruschke J.K. Doing Bayesian data analysis: a tutorial with R and BUGS 2010, Academic Press/Elsevier Science.
43. Lane D.M., Dunlap W.P. Estimating effect size: bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical, Psychology 1978, 31:107-112.
44. Meehl P.E. Theoretical risks and tabular asterisks: sir Karl, sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology 1978, 46:806-834.
45. Miller J. What is the probability of replicating a statistically significant effect?. Psychonomic Bulletin & Review 2009, 16:617-640.
46. Pashler H., Wagenmakers E.-J. Editors' introduction to the special section on replicability in psychological science: a crisis of confidence?. Perspectives on Psychological Science 2012, 7:528-530.
47. Peters J.L., Sutton A.J., Jones D.R., Abrams K.R., Rushton L. Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine 2007, 26:4544-4562.
48. Phillips C.V. Publication bias in situ. BMC Medical Research Methodology 2004, 4(20). 10.1186/1471-2288-4-20.
49. Piff P.K., Stancato D.M., Côté S., Mendoza-Denton R., Keltner D. Higher social class predicts increased unethical behavior. Proceedings of the National Academy of Sciences USA 2012, 109:4086-4091. 10.1073/pnas.1118373109.
50. Piff P.K., Stancato D.M., Côté S., Mendoza-Denton R., Keltner D. Reply to Francis: cumulative power calculations are faulty when based on observed power and a small sample of studies. Proceedings of the National Academy of Sciences USA 2012, 10.1073/pnas.1205367109.
51. R Development Core Team. (2011) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. Austria. ISBN 3-900051-07-0. http://www.R-project.org/.
52. Ritchie S.J., Wiseman R., French C.C. Failing the future: three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect. PLoS ONE 2012, 7(3):e33423. 10.1371/journal.pone.0033423.
53. Roediger H.L. Psychology's woes and a partial cure: the value of replication. APS Observer 2012, 25(2). http://www.psychologicalscience.org/index.php/publications/observer/2012/february-11-2012-observer-publications/psychologys-woes-and-a-partial-cure-the-value-of-replication.html.
54. Rosenthal R. Applied social research methods series, vol. 6, Meta-analytic procedures for social research 1984, Sage Publications, Newbury Park, CA.
55. Sagan C. The demon-haunted world: science as a candle in the dark 1997, Ballantine Books, New York.
56. Scargle J.D. Publication bias: the file-drawer problem in scientific inference. Journal of Scientific Exploration 2000, 14(1):91-106.
57. Schimmack U. The ironic effect of significant results on the credibility of multiple study articles. Psychological Methods 2012, Advance online publication. 10.1037/a0029487.
58. Shea, C. (2011). Fraud scandal fuels debate over practices of social psychology. In The Chronicle of Higher Education. Downloaded from http://chronicle.com/article/As-Dutch-Research-Scandal/129746/.
59. Simmons J.P., Nelson L.D., Simonsohn U. False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 2011, 22:1359-1366.
60. Simonsohn U. Weather to go to college. The Economic Journal 2010, 120(543):270-280.
61. Simonsohn U. It does not follow: evaluating the one-off publication bias critiques by Francis (2012a, b, c, d, e, f). Perspectives on Psychological Science 2012, 7(6):597-599.
62. Sterling T.D. Publication decisions and the possible effects on inferences drawn from test of significance-or vice versa. Journal of the American Statistical Association 1959, 54:30-34.
63. Sterne J.A., Gavaghan D., Egger M. Publication and related bias in meta-analysis: power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology 2000, 53:1119-1129.
64. Thompson B. Foundations of behavioral statistics: an insight-based approach 2006, Guilford, New York.
65. Topolinski S., Sparenberg P. Turning the hands of time: clockwise movements increase preference for novelty. Social Psychological and Personality Science 2012, 3:308-314.
66. Wagenmakers E.-J. A practical solution to the pervasive problems of p values. Psychonomic Bulletin & Review 2007, 14:779-804.
67. Wagenmakers E.-J., Wetzels R., Borsboom D., van der Maas H.L.J. Why psychologists must change the way they analyze their data: the case of psi: comment on Bem (2011). Journal of Personality and Social Psychology 2011, 100:426-432.
68. Yong E. Bad copy. Nature 2012, 485:298-300.
69. Yong E. The data detective. Nature 2012, 487:18-19. 10.1038/487018a.
70. Yuan K.H., Maxwell S. On the post hoc power in testing mean differences. Journal of Educational and Behavioral Statistics 2005, 30:141-167.