Publication bias as a function of study characteristics

Psychological Methods

Volumn 20, Issue 3, 2015, Pages 310-330

  Coburn, Kathleen M ^a   Vevea, Jack L ^a  

^a UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Cumulative meta analysis; Funnel plot; Publication bias; Trim and fill; Weight function model

Indexed keywords

HUMAN; MEDICAL RESEARCH; META ANALYSIS (TOPIC); PUBLISHING; STANDARDS; STATISTICAL ANALYSIS;

BIOMEDICAL RESEARCH; DATA INTERPRETATION, STATISTICAL; HUMANS; META-ANALYSIS AS TOPIC; PUBLICATION BIAS;

EID: 84941206223     PISSN: 1082989X     EISSN: None     Source Type: Journal    
DOI: 10.1037/met0000046     Document Type: Article

References (71)

1. Abramowitz, S. I., Gomes, B., & Abramowitz, C. V. (1975). Publish or politic: Referee bias in manuscript review. Journal of Applied Social Psychology, 5, 187-200. http://dx.doi.org/10.1111/j.1559-1816.1975.tb00675.x
2. American Psychological Association. (2009). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
3. Armstrong, J. S. (1996). Publication of research on controversial topics: The early acceptance procedure. International Journal of Forecasting, 12, 299-302. http://dx.doi.org/10.1016/0169-2070(95)00626-5
4. Banks, G. C., Kepes, S., & McDaniel, M. A. (2015). Publication bias: Understanding the myths concerning threats to the advancement of science. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 36-64). New York, NY: Routledge.
5. Barnes, D. E., & Bero, L. A. (1998). Why review articles on the health effects of passive smoking reach different conclusions. Journal of the American Medical Association, 279, 1566-1570. http://dx.doi.org/10.1001/jama.279.19.1566
6. Becker, B. J. (2005). Fail-safeN or file-drawer number. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 111-125). Chichester, England: Wiley.
7. Begg, C. B., & Berlin, J. A. (1988). Publication bias: A problem in interpreting medical data. Journal of the Royal Statistical Society Series A, 151, 419-463. http://dx.doi.org/10.2307/2982993
8. Begg, C. B., & Berlin, J. A. (1989). Publication bias and dissemination of clinical research. Journal of the National Cancer Institute, 81, 107-115. http://dx.doi.org/10.1093/jnci/81.2.107
9. Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088-1101. http://dx.doi.org/10.2307/2533446
10. Bekelman, J. E., Li, Y., & Gross, C. P. (2003). Scope and impact of financial conflicts of interest in biomedical research: A systematic review. Journal of the American Medical Association, 289, 454-465. http://dx.doi.org/10.1001/jama.289.4.454
11. Berlin, J. A., Begg, C. B., & Louis, T. A. (1989). An assessment of publication bias using a sample of published clinical trials. Journal of the American Statistical Association, 84, 381-392. http://dx.doi.org/10.1080/01621459.1989.10478782
12. Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2009). Introduction to meta-analysis. West Sussex, United Kingdom: Wiley. http://dx.doi.org/10.1002/9780470743386
13. Cooper, H., DeNeve, K., & Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447-452. http://dx.doi.org/10.1037/1082-989X.2.4.447
14. Copas, J., & Shi, J. Q. (2000). Meta-analysis, funnel plots and sensitivity analysis. Biostatistics, 1, 247-262. http://dx.doi.org/10.1093/biostatistics/1.3.247
15. Dear, K. B. G., & Begg, C. B. (1992). An approach for assessing publication bias prior to performing a meta-analysis. Statistical Science, 7, 237-245. http://dx.doi.org/10.1214/ss/1177011363
16. Dickersin, K., & Min, Y. I. (1993). Publication bias: The problem that won't go away. Annals of the New York Academy of Sciences, 703, 135-148. http://dx.doi.org/10.1111/j.1749-6632.1993.tb26343.x
17. Doucet, M., & Sismondo, S. (2008). Evaluating solutions to sponsorship bias. Journal of Medical Ethics, 34, 627-630. http://dx.doi.org/10.1136/jme.2007.022467
18. Duval, S. (2005). The trim and fill method. In H. R. Rothstein, A. J. Sutton, & M. Borenstein (Eds.), Publication bias in meta-analysis: Prevention, assessment and adjustments (pp. 127-144). Chichester, England: Wiley.
19. Duval, S., & Tweedie, R. (2000a). A nonparametric "trim and fill" method of accounting for publication bias in meta-analysis. Journal of the American Statistical Association, 95, 89-98.
20. Duval, S., & Tweedie, R. (2000b). Trim and fill: A simple funnel-plotbased method of testing and adjusting for publication bias in metaanalysis. Biometrics, 56, 455-463. http://dx.doi.org/10.1111/j.0006-341X.2000.00455.x
21. Duval, S., & Weinhandl, E. (2011). Correcting for publication bias in the presence of covariates (AHRQ Publication No. 11-EHC041-EF). Rockville, MD: Agency for Healthcare Research and Quality. Retrieved from www.effectivehealthcare.ahrq.gov/reports/final.cfm
22. Eagly, A. H., Johannesen-Schmidt, M. C., & van Engen, M. L. (2003). Transformational, transactional, and laissez-faire leadership styles: A meta-analysis comparing women and men. Psychological Bulletin, 129, 569-591. http://dx.doi.org/10.1037/0033-2909.129.4.569
23. Easterbrook, P. J., Gopalan, R., Berlin, J. A., & Matthews, D. R. (1991. April). Publication bias in clinical research. Lancet, 337, 867-872. http://dx.doi.org/10.1016/0140-6736(91)90201-Y
24. Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 315, 629-634. http://dx.doi.org/10.1136/bmj.315.7109.629
25. Felson, D. T. (1992). Bias in meta-analytic research. Journal of Clinical Epidemiology, 45, 885-892. http://dx.doi.org/10.1016/0895-4356(92)90072-U
26. Freese, J. (2007). Replication standards for quantitative social science: Why not sociology? Sociological Methods & Research, 36, 153-172. http://dx.doi.org/10.1177/0049124107306659
27. Glass, G. V., McGaw, B., & Smith, M. L. (1981). Meta-analysis in social research (Vol. 56). Beverly Hills, CA: Sage.
28. Goodstein, L. D., & Brazis, K. L. (1970). Psychology of scientist: XXX. Credibility of psychologists: An empirical study. Psychological Reports, 27, 835-838. http://dx.doi.org/10.2466/pr0.1970.27.3.835
29. Heckman, J. J. (1995). Lessons from the bell curve. Journal of Political Economy, 103, 1091-1120. http://dx.doi.org/10.1086/262014
30. Hedges, L. V. (1984). Estimation of effect size under nonrandom sampling: The effects of censoring studies yielding statistically insignificant mean differences. Journal of Educational and Behavioral Statistics, 9, 61-85. http://dx.doi.org/10.3102/10769986009001061
31. Hedges, L. V. (1992). Modeling publication selection effects in metaanalysis. Statistical Science, 7, 246-255. http://dx.doi.org/10.1214/ss/1177011364
32. Herrnstein, R. J., & Murray, C. (1994). Bell curve: Intelligence and class structure in American life. New York, NY: Simon & Schuster.
33. Higgins, J. P. (Ed.). (2008). Cochrane handbook for systematic reviews of interventions (Vol. 5). Chichester, England: Wiley-Blackwell. http://dx .doi.org/10.1002/9780470712184
34. Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis. Psychological Bulletin, 104, 53-69. http://dx.doi.org/10.1037/0033-2909.104.1.53
35. Iyengar, S., & Greenhouse, J. B. (1988). Selection models and the file drawer problem. Statistical Science, 3, 109-117. http://dx.doi.org/10.1214/ss/1177013012
36. Jennions, M. D., & Møller, A. P. (2002). Publication bias in ecology and evolution: An empirical assessment using the "trim and fill" method. Biological Reviews of the Cambridge Philosophical Society, 77, 211-222. http://dx.doi.org/10.1017/S1464793101005875
37. Jick, T. D. (1979). Mixing qualitative and quantitative methods: Triangulation in action. Administrative Science Quarterly, 24, 602-611. http://dx.doi.org/10.2307/2392366
38. Johnson, D. H. (2001). Sharing data: It's time to end psychology's guild approach. APS Observer, 14, 38-39. Retrieved from http://www .psychologicalscience.org/observer/1001/data.html
39. Kepes, S., Banks, G. C., McDaniel, M., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662. http://dx.doi.org/10.1177/1094428112452760
40. Kepes, S., Banks, G. C., & Oh, I. S. (2014). Avoiding bias in publication bias research: The value of "null" findings. Journal of Business and Psychology, 29, 183-203. http://dx.doi.org/10.1007/s10869-012-9279-0
41. King, G. (1995). Replication, replication. PS: Political Science and Politics, 28, 444-452.
42. Kling, K. C., Hyde, J. S., Showers, C. J., & Buswell, B. N. (1999). Gender differences in self-esteem: A meta-analysis. Psychological Bulletin, 125, 470-500. http://dx.doi.org/10.1037/0033-2909.125.4.470
43. Lane, D. M., & Dunlap, W. P. (1978). Estimating effect size: Bias resulting from the significance criterion in editorial decisions. British Journal of Mathematical and Statistical Psychology, 31, 107-112. http://dx.doi .org/10.1111/j.2044-8317.1978.tb00578.x
44. Light, R. J., & Pillemer, D. B. (1984). Summing up: The science of reviewing research. Cambridge, MA: Harvard University Press.
45. Maccoby, E. E., & Jacklin, C. N. (Eds.). (1974). The psychology of sex differences (Vol. 1). Stanford, CA: Stanford University Press.
46. Mahoney, M. J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy and Research, 1, 161-175. http://dx.doi.org/10.1007/BF01173636
47. McDaniel, M. A. (2009, April). Cumulative meta-analysis as a publication bias method. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology, New Orleans, LA.
48. McDaniel, M. A., McKay, P., & Rothstein, H. R. (2006, May). Publication bias and racial effects on job performance: The elephant in the room. Paper presented at the 21st Annual Conference of the Society for Industrial and Organizational Psychology, Dallas, TX. http://dx.doi.org/10.1037/0021-9010.91.3.538
49. McDaniel, M. A., Rothstein, H. R., & Whetzel, D. L. (2006). Publication bias: A case study of four test vendors. Personnel Psychology, 59, 927-953. http://dx.doi.org/10.1111/j.1744-6570.2006.00059.x
50. McKay, P. F., & McDaniel, M. A. (2006). A reexamination of black-white mean differences in work performance: More data, more moderators. Journal of Applied Psychology, 91, 538-554. http://dx.doi.org/10.1037/0021-9010.91.3.538
51. Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F., & The QUOROM Group. (1999, November). Improving the quality of reports of meta-analyses of randomised controlled trials: The QUOROM statement. Lancet, 354, 1896-1900. http://dx.doi.org/10.1016/S0140-6736(99)04149-5
52. Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G., & The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. Annals of Internal Medicine, 151, 264-269, W64. http://dx.doi.org/10.7326/0003-4819-151-4-200908180-00135
53. Peters, J. L., Sutton, A. J., Jones, D. R., Abrams, K. R., & Rushton, L. (2008). Contour-enhanced meta-analysis funnel plots help distinguish publication bias from other causes of asymmetry. Journal of Clinical Epidemiology, 61, 991-996. http://dx.doi.org/10.1016/j.jclinepi.2007.11.010
54. R Core Team. (2013). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from http://www.R-project.org/
55. Roseman, M., Milette, K., Bero, L. A., Coyne, J. C., Lexchin, J., Turner, E. H., & Thombs, B. D. (2011). Reporting of conflicts of interest in meta-analyses of trials of pharmacological treatments. JAMA: Journal of the American Medical Association, 305, 1008-1017. http://dx.doi.org/10.1001/jama.2011.257
56. Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638-641. http://dx.doi.org/10.1037/0033-2909.86.3.638
57. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in meta-analysis: Prevention, assessment, and adjustments. Chichester, England: Wiley. http://dx.doi.org/10.1002/0470870168
58. RStudio. (2012). RStudio: Integrated development environment for R (Version 0.96.122) [Computer software]. Boston, MA: Author. Available from http://www.rstudio.org/
59. Rufibach, K. (2011). Selection models with monotone weight functions in meta analysis. Biom. J., 53, 689-704.
60. Schneider, B. (2004). Building a scientific community: The need for replication. Teachers College Record, 106, 1471-1483. http://dx.doi .org/10.1111/j.1467-9620.2004.00386.x
61. Schwarzer, G. (2015). meta: Meta-analysis with R [Computer software]. R package, version 4.0-3.
62. Smith, M. L. (1980). Sex bias in counseling and psychotherapy. Psychological Bulletin, 87, 392-407. http://dx.doi.org/10.1037/0033-2909.87.2.392
63. Stanley, T. D., & Doucouliagos, H. (2014). Meta-regression approximations to reduce publication selection bias. Research Synthesis Methods, 5, 60-78. http://dx.doi.org/10.1002/jrsm.1095
64. Sterne, J. A., Becker, B. J., & Egger, M. (2005). The funnel plot. Publication bias in meta-analysis: Prevention, assessment, and adjustments, 75-98. http://dx.doi.org/10.1002/0470870168.ch5
65. Sterne, J. A. C., & Egger, M. (2001). Funnel plots for detecting bias in meta-analysis: Guidelines on choice of axis. Journal of Clinical Epidemiology, 54, 1046-1055. http://dx.doi.org/10.1016/S0895-4356(01)00377-8
66. Sterne, J. A., Egger, M., & Smith, G. D. (2001). Systematic reviews in health care: Investigating and dealing with publication and other biases in meta-analysis. British Medical Journal, 323, 101-105. http://dx.doi .org/10.1136/bmj.323.7304.101
67. Terrin, N., Schmid, C. H., Lau, J., & Olkin, I. (2003). Adjusting for publication bias in the presence of heterogeneity. Statistics in Medicine, 22, 2113-2126. http://dx.doi.org/10.1002/sim.1461
68. Vevea, J. L., & Hedges, L. V. (1995). A general linear model for estimating effect size in the presence of publication bias. Psychometrika, 60, 419-435. http://dx.doi.org/10.1007/BF02294384
69. Vevea, J. L., & Woods, C. M. (2005). Publication bias in research synthesis: Sensitivity analysis using a priori weight functions. Psychological Methods, 10, 428-443. http://dx.doi.org/10.1037/1082-989X.10.4.428
70. Viechtbauer, W. (2010). Conducting meta-analyses in R with the metafor package. Journal of Statistical Software, 36, 1-48.
71. Weinhandl, E. D., & Duval, S. (2012). Generalization of trim and fill for application in meta-regression. Research Synthesis Methods, 3, 51-67. http://dx.doi.org/10.1002/jrsm.1042