Publication bias and the failure of replication in experimental psychology

Psychonomic Bulletin and Review

Volumn 19, Issue 6, 2012, Pages 975-991

  Francis, Gregory ^a  

^a PURDUE UNIVERSITY   (United States)

Author keywords

Bayesian methods; Hypothesis testing; Meta analysis; Publication bias; Replication

Indexed keywords

EID: 84869068553     PISSN: 10699384     EISSN: 15315320     Source Type: Journal    
DOI: 10.3758/s13423-012-0322-y     Document Type: Review

References (47)

1. Abrams, R. A., Davoli, C. C., Du, F., Knapp, W. H., III, & Paull, D. (2008). Altered vision near the hands. Cognition, 107, 1035-1047.
2. Bakker, M., & Wicherts, J. (2011). The (mis)reporting of statistical results in psychology journals. Behavior Research Methods, 43, 666-678.
3. Banks, G. C., & McDaniel, M. (2011). The kryptonite of evidence-based I-O psychology. Industrial and Organizational Psychology, 4, 40-44.
4. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101.
5. Bem, D. J. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100, 407-425.
6. Berger, J., & Berry, D. (1988). The relevance of stopping rules in statistical inference (with discussion). In S. S. Gupta & J. Berger (Eds.), Statistical decision theory and related topics, 1 (Vol. 4, pp. 29-72). New York: Springer.
7. Champely, S. (2009). pwr: Basic functions for power analysis. R package version 1. 1. 1. http://CRAN. R-project. org/package=pwr.
8. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
9. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
10. Davoli, C. C., & Abrams, R. A. (2009). Reaching out with the imagination. Psychological Science, 20(3), 293-295.
11. Del Re, A. C. (2010). compute. es: Compute Effect Sizes. R package version 0. 2. http://CRAN. R-project. org/web/packages/compute. es/.
12. Dienes, Z. (2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6, 274-290.
13. Fayard, J. V., Bassi, A. K., Bernstein, D. M., & Roberts, B. W. (2009). Is cleanliness next to godliness? Dispelling old wives' tales: Failure to replicate Zhong and Liljenquist (2006). Journal of Articles in Support of the Null Hypothesis, 6, 21-28.
14. Fischer, P., Krueger, J. I., Greitemeyer, T., Vogrincic, C., Kastenmüller, A., Frey, D., Heene, M., Wicher, M., & Kainbacher, M. (2011). The bystander-effect: A meta-analytic review on bystander intervention in dangerous and non-dangerous emergencies. Psychological Bulletin, 137, 517-537.
15. Francis, G. (2012a). The same old New Look: Publication bias in a study of wishful seeing. Perception, 3(3), 176-178.
16. Francis, G. (2012b). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, 19, 151-156.
17. Francis, G. (in press). Publication bias in "Red, Rank, and Romance in Women Viewing Men" by Elliot et al. (2010). Journal of Experimental Psychology: General.
18. Gelman, A., & Weakliem, D. (2009). Of beauty, sex and power. American Scientist, 97, 310-316.
19. Greenwald, A. G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
20. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational Statistics, 9, 61-85.
21. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press.
22. Ioannidis, J. P. A. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640-648.
23. Ioannidis, J. P. A., & Trikalinos, T. A. (2007). An exploratory test for an excess of significant findings. Clinical Trials, 4, 245-253.
24. John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth-telling. Psychological Science, 23, 524-532.
25. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. American Statistical Association, 90, 773-7935.
26. Kerridge, D. (1963). Bounds for the frequency of misleading Bayes inferences. Annals of Mathematical Statistics, 34, 1109-1110.
27. Kruschke, J. K. (2010a). Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5), 658-676. doi: 10. 1002/wcs. 72.
28. Kruschke, J. K. (2010b). Doing Bayesian Data Analysis: A Tutorial with R and BUGS. Academic Press/Elsevier Science.
29. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112.
30. Munafò, M. R., & Flint, J. (2010). How reliable are scientific studies? The British Journal of Psychiatry, 197, 257-258.
31. Nairne, J. S. (2009). Psychology (5th ed.). Belmont, CA: Thomson Learning.
32. R Development Core Team. (2011). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0, http://www. R-project. org/.
33. Renkewitz, F., Fuchs, H. M., & Fiedler, S. (2011). Is there evidence of publication biases in JDM research? Judgment and Decision Making, 6, 870-881.
34. Rosenthal, R. (1984). Applied Social Research Methods Series, Vol. 6. Meta-analytic procedures for social research. Newbury Park, CA: Sage Publications.
35. Rouder, J. N., & Morey, R. D. (2011). A Bayes fator meta-analysis of Bem's ESP claim. Psychonomic Bulletin & Review, 18, 682-689.
36. Rouder, J. N., Morey R. D., Speckman P. L., & Province J. M. (in press). Default Bayes factors for ANOVA designs. Journal of Mathematical Psychology.
37. Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16, 225-237.
38. Scargle, J. D. (2000). Publication bias: The "File-Drawer" problem in scientific inference. Journal of Scientific Exploration, 14(1), 91-106.
39. Schimmack, U. (in press). The ironic effect of significant results on the credibility of multiple study articles. Psychological Methods.
40. Schooler, J. (2011). Unpublished results hide the decline effect. Nature, 470, 437.
41. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366.
42. Sterling, T. D. (1959). Publication decisions and the possible effects on inferences drawn from test of significance-or vice versa. Journal of the American Statistical Association, 54, 30-34.
43. Sterling, T. D., Rosenbaum, W. L., & Weinkam, J. J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish and vice versa. The American Statistician, 49, 108-112.
44. Sterne, J. A., Gavaghan, D., & Egger, M. (2000). Publication and related bias in meta-analysis: Power of statistical tests and prevalence in the literature. Journal of Clinical Epidemiology, 53, 1119-1129.
45. Strube, M. J. (2006). SNOOP: A program for demonstrating the consequences of premature and repeated null hypothesis testing. Behavior Research Methods, 38, 24-27.
46. Wagenmakers, E.-J. (2007). A practical solution to the pervasive problems of p values. Psychonomic Bulleting & Review, 14, 779-804.
47. Zhong, C., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313, 1451-1452.