Publication bias in psychological science: Comment on Ferguson and Brannick (2012)

Psychological Methods

Volumn 17, Issue 1, 2012, Pages 129-136

  Rothstein, Hannah R ^a   Bushman, Brad J ^b,c  

^a CITY UNIVERSITY OF NEW YORK   (United States)

^b OHIO STATE UNIVERSITY   (United States)

^c VU UNIVERSITY AMSTERDAM   (Netherlands)

Author keywords

Bias; Gray literature; Meta analysis; Meta analytic review; Publication bias

Indexed keywords

HUMAN; META ANALYSIS (TOPIC); NOTE; PSYCHOLOGY; PUBLISHING; QUALITY CONTROL; STATISTICS;

HUMANS; META-ANALYSIS AS TOPIC; PSYCHOLOGY; PUBLICATION BIAS; QUALITY CONTROL;

EID: 84869018002     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/a0027128     Document Type: Article

References (56)

1. American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2. Anderson, C. A., Shibuya, A., Ihori, N., Swing, E. L., Bushman, B. J., Sakamoto, A., Barlett, C. P. (2010). Violent video game effects on aggression, empathy, and prosocial behavior in Eastern and Western countries: A meta-analytic review. Psychological Bulletin, 136, 151-173. doi:10.1037/a0018251
3. Aytug, Z., Rothstein, H. R., Zhou, W., & Kern, M. (2011). Revealed or concealed? Transparency of procedures, decisions, and judgment calls in meta-analysis. Organizational Research Methods, 15, 103-133. doi: 10.1177/1094428111403495
4. Baumeister, R., DeWall, C., & Vohs, K. (2009). Social rejection, control, numbness, and emotion: How not to be fooled by Gerber and Wheeler (2009). Perspectives on Psychological Science, 4, 489-493. doi: 10.1111/j.1745-6924.2009.01159.x
5. Becker, B. J. (2005). Failsafe N or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds), Publication bias in meta-analysis: Prevention, assessment, and adjustments (pp. 111-125). Chichester, England: Wiley.
6. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101. doi: 10.2307/2533446
7. Bonett, D. G. (2008). Meta-analytic interval estimation for bivariate correlations. Psychological Methods, 13, 173-181. doi:10.1037/a0012868
8. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. New York, NY: Wiley. doi:10.1002/ 9780470743386
9. Card, N. A. (2012). Applied meta-analysis for social science research. New York, NY: Guilford Press.
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
11. Cook, D. J., Guyatt, G. H., Ryan, G., Clifton, J., Buckingham, L., Willan, A., & Oxman, A. D. (1993). Should unpublished data be included in meta-analyses? Current convictions and controversies. Journal of the American Medical Association, 269, 2749-2753.
12. Cook, T. D., Cooper, H., Cordray, D. S., Hartmann, H., Hedges, L. V., Light, R. J.,... Mosteller, F. (1992). Meta-analysis for explanation: A casebook. New York, NY: Russell Sage Foundation.
13. Cooper, H. (2009). Research synthesis and meta-analysis: A step-by-step approach (4th ed.). Thousand Oaks, CA: Sage.
14. Cooper, H., & Dent, A. (2011). Ethical issues in the conduct and reporting of meta-analysis. In A. T. Panter & S. K. Sterba (Eds.), Handbook of ethics in quantitative psychology (pp. 417-444). New York, NY: Routledge.
15. Cooper, H. M., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis (2nd ed.). New York, NY: Russell Sage Foundation.
16. Duval, S. J., & Tweedie, R. L. (1998). Practical estimates of the effect of publication bias in meta-analysis. Australasian Epidemiologist, 5, 14-17.
17. Duval, S. J., & Tweedie, R. L. (2000a). A nonparametric "trim-and-fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-99. doi:10.2307/2669529
18. Duval, S. J., & Tweedie, R. L. (2000b). Trim and fill: A simple funnelplot-based method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. doi:10.1111/j.0006-341X.2000 .00455.x
19. Egger, M., & Davey Smith, G. (1998). Meta-analysis: Bias in location and selection of studies. British Medical Journal, 316, 61-66. Retrieved from http://www.BritishMedicalJournal.com/content/316/7124/61.full doi:10.1136/bmj.316.7124.61
20. Egger, M., Davey Smith, G., & Altman, D. G. (2001). Systematic reviews in health care: Meta-analysis in context (2nd ed.). London, England: BMJ Publishing Group. doi:10.1002/9780470693926
21. Egger, M., Davey Smith, G., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple graphical test. British Medical Journal, 315, 629-634. doi:10.1136/bmj.315.7109.629
22. Egger, M., Dickersin, K., & Smith, G. (2001). Problems and limitations in conducting systematic reviews. In M. Egger, G. Davey-Smith, & D. Altman (Eds), Systematic reviews in health care: Meta-analysis in context (pp. 43-68). London, England: BMJ Publishing Group.
23. Ellis, P. (2010). The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results. Cambridge, England: Cambridge University Press.
24. Eysenck, H. J. (1978). An exercise in mega-silliness [Letter to the editor]. American Psychologist, 33, 517. doi:10.1037/0003-066X.33.5.517.a
25. Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17, 120-128. doi:10.1037/a0024445
26. Frattaroli, J. (2006). Experimental disclosure and its moderators: A metaanalysis. Psychological Bulletin, 132, 823-865. doi:10.1037/0033-2909.132.6.823
27. Gerber, J., & Wheeler, L. (2009). On being rejected: A meta-analysis of experimental research on rejection. Perspectives on Psychological Science, 4, 468-488. doi:10.1111/j.1745-6924.2009.01158.x
28. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
29. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research. Thousand Oaks, CA. Sage.
30. Greenland, S., & O'Rourke, K. (2001). On the bias produced by quality scores in meta-analysis, and a hierarchical view of proposed solutions. Biostatistics, 2, 463-471.
31. Hammerstrøm, L., Wade, A., & Jørgensen, A. M. K. (2010). Searching for studies: A guide to informational retrieval for Campbell Systematic Reviews. Campbell Systematic Reviews 2010 [Suppl. 1]. doi: 10.4073/ csrs.2010.1.
32. Hartung, J., Knapp, G., & Sinha, B. K. (2008). Statistical meta-analysis with applications. Chichester, England: Wiley.
33. Higgins, J. P. T., & Green, S. (Eds.). (2011). Cochrane handbook for systematic reviews of interventions (Version 5.1.0, updated March 2011]. London, England: The Cochrane Collaboration. Available from www.cochrane-handbook.org
34. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
35. Ioannidis, J. P. A., & Trikalinos, T. (2007). The appropriateness of asymmetry tests for publication bias in meta-analyses: A large survey. Canadian Medical Association Journal, 176, 1091-1096. doi:10.1503/ cmaj.060410
36. Jüni, P., Witschi, A., Bloch, R., & Egger, M. (1999). The hazards of scoring the quality of clinical trials for meta-analysis. Journal of the American Medical Association, 282, 1054-1060. doi:10.1001/ jama.282.11.1054
37. Kromrey, J., & Rendina-Gobioff, G. (2006). On knowing what we do not know: An empirical comparison of methods to detect publication bias in meta-analysis. Educational and Psychological Measurement, 66, 357-373. doi:10.1177/0013164405278585
38. Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
39. Lipsey, M. W. (1990). Design sensitivity: Statistical power for experimental research. Newbury Park, CA: Sage.
40. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Newbury Park, CA: Sage.
41. Littell, J. H., Corcoran, J., & Pillai, V. (2008). Systematic reviews and meta-analysis. Oxford, England: Oxford University Press.
42. Macaskill, P., Walter, S. D., & Irwig, L. (2001). A comparison of methods to detect publication bias in meta-analysis. Statistics in Medicine, 20, 641-654. doi:10.1002/sim.698
43. Moher, D., Cook, D. J., Jadad, A. R., Tugwell, P., Moher, M., Jones, A.,... Klassen, T. P. (1999). Assessing the quality of reports of randomised trials: Implications for the conduct of meta-analyses. Health Technology Assessment, 3, 1-98.
44. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2006). Comparison of two methods to detect publication bias in metaanalysis. Journal of the American Medical Association, 295, 676-680. doi:10.1001/jama.295.6.676
45. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2007). Performance of the trim and fill method in the presence of publication bias and between-study heterogeneity. Statistics in Medicine, 26, 4544-4562. doi:10.1002/sim.2889
46. Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Oxford, England: Blackwell. doi:10.1002/ 9780470754887
47. Rosenthal, R. (1991). Meta-analysis: A review. Psychosomatic Medicine, 53, 247-271.
48. Shadish, W., & Myers, D. (2004). Research design policy brief. Oslo, Norway: Campbell Collaboration. Retrieved from http://www.campbellcollaboration .org/artman2/uploads/1/C2_Research_Design_Policy_Brief-2.pdf
49. Shea, B. J., Grimshaw, J. M., Wells, G. A., Boers, M., Andersson, N., Hamel, C.,... Bouter, L. M. (2007). Development of AMSTAR: A measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology, 7, 10-17. doi:10.1186/ 1471-2288-7-10
50. Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046-1055. doi:10.1016/S0895-4356(01)00377-8
51. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity, Statistics in Medicine, 22, 2113-2126. doi:10.1002/sim.1461
52. Tetzlaff, J., Moher, D., Pham, B., & Altman, D (2006, October). Survey of views on including gray literature in systematic reviews. Paper presented at the 14th Cochrane Colloquium, Dublin, Ireland.
53. Wachter, K. W., & Straf, M. L. (Eds). (1990). The future of meta-analysis. New York, NY: Russell Sage Foundation.
54. Wang, M. C., & Bushman, B. J. (in press). Integrating results through meta-analytic review using SAS software (2nd ed.). Cary, NC: SAS Institute.
55. Whitehead, A. (2002). Meta-analysis of controlled clinical trials. Chichester, England: Wiley. doi:10.1002/0470854200
56. Wolf, F. (1986). Meta-analysis: Quantitative methods for research synthesis. Newbury Park, CA: Sage.