Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm

Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments

Volumn , Issue , 2006, Pages 11-33

  Dickersin, Kay ^a  

^a JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

Author keywords

Biased reporting of outcomes; From study start to dissemination of results; JAMA negative results; Publication bias and recognition; Publication bias in meta analysis; Publication bias significance; Systematic review problem method; Systematic reviews and meta analysis

Indexed keywords

EID: 84863732058     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/0470870168.ch2     Document Type: Chapter

References (113)

1. Ad Hoc Working Party of the International Collaborative Group on Clinical Trials Registries. International Collaborative Group on Clinical Trial Registries (1993). Position paper and consensus recommendations on clinical trial registries. Clinical Trials Metaanalysis, 28, 255-266.
2. Alanson, E. (1782). Practical observations on amputation, and the after-treatment, 2nd edition. London: Joseph Johnson.
3. Anderson, D., Costa, I. and Dickersin, K. (2001). Building Trials Central, an online register of clinical trials registers. Controlled Clinical Trials, 22(Suppl. 2), 40S.
4. Angell, M. (1989). Negative studies. New England Journal of Medicine, 321, 454-466.
5. Antes, G. and Chalmers, I. (2003). Under-reporting of clinical trials is unethical. Lancet, 361, 978-979.
6. Balk, E.M., Bonis, P.A., Moskowitz, H., Schmidt, C.H., Ioannidis, J.P.A., Wang, C. and Lau, J. (2002). Correlation of quality measures with estimates of treatment effect in metaanalyses of randomized controlled trials. Journal of the American Medical Association, 287, 2973-2982.
7. Bero, L.A. and Rennie, D. (1996). Influences on the quality of published drug studies. International Journal of Technology Assessment in Health Care, 12, 209-237.
8. Bero, L.A., Glantz, S.A. and Rennie, D. (1994). Publication bias and public health policy on environmental tobacco smoke. Journal of the American Medical Association, 272, 133-136.
9. Bradford Hill, A. (1959). Discussion of a paper by D. J. Finney. Journal of the Royal Statistical Society, Series A, 119, 19-20.
10. Callaham, M., Wears, R.L. and Weber, E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. Journal of the American Medical Association, 287, 2847-2850.
11. Chalmers I. (1990). Underreporting research is scientific misconduct. Journal of the American Medical Association, 263, 1405-1408.
12. Chalmers, I. (2000). Current Controlled Trials: An opportunity to help improve the quality of clinical research. Current Controlled Trials, 1, 3-8.
13. Chalmers, I. (2001). Using systematic reviews and registers of ongoing trials for scientific and ethical trial design, monitoring and reporting. In: M. Egger, D. Smith and D.G. Altman (eds), Systematic Reviews in Health Care: Meta-analysis in Context, 2nd edition. London: BMJ Books.
14. Chalmers, T.C. (1977). Randomize the first patient! New England Journal of Medicine, 296, 107.
15. Chalmers, T.C., Koff, R.S. and Grady, G.F. (1965). A note on fatality in serum hepatitis. Gastroenterology, 49, 22-26.
16. Cho, M.K. and Bero, L.A. (1996). The quality of drug studies published in symposium proceedings. Annals of Internal Medicine, 124, 485-489.
17. Christensen-Szalanski, J.J.J. and Beach, L.R. (1984). The citation bias: Fad and fashion in the judgment and decision literature. American Journal of Psychology, 39, 75-78.
18. Cooper, H., DeNeve, K. and Charlton, K. (1997). Finding the missing science: The fate of studies submitted for review by a human subjects committee. Psychological Methods, 2, 447-452.
19. Coursol, A. and Wagner, E.E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology Research & Practice, 17, 136-137.
20. Davidson, R.A. (1986). Source of funding and outcome of clinical trials. Journal of General Internal Medicine, 1, 155-158.
21. Dickersin, K. (1988). Report from the Panel on the Case for Registers of Clinical Trials at the Eighth Annual Meeting of the Society for Clinical Trials. Controlled Clinical Trials, 9, 76-81.
22. Dickersin, K. (1997). How important is publication bias? A synthesis of available data. AIDS Education and Prevention, 9(1 Suppl), 15-21.
23. Dickersin, K. and Min, Y.I. (1993). NIH clinical trials and publication bias. Online Journal of Current Clinical Trials, Doc. No. 50.
24. Dickersin, K. and Rennie, D. (2003). Registering clinical trials. Journal of the American Medical Association, 290, 516-523.
25. Dickersin, K., Chan, S., Chalmers, T.C., Sacks, H.S. and Smith, H. Jr. (1987). Publication bias and clinical trials. Controlled Clinical Trials, 8, 343-353.
26. Dickersin, K., Min, Y.I. and Meinert, C.L. (1992). Factors influencing publication of research results. Follow-up of applications submitted to two institutional review boards. Journal of the American Medical Association, 267, 374-378.
27. Dickersin, K., Scherer, R. and Lefebvre, C. (1994). Identifying relevant studies for systematic reviews. British Medical Journal, 309, 1286-1291.
28. Dickersin, K., Olson, C.M., Rennie, D., Cook, D., Flanagin, A., Zhu, Q., Reiling, J. and Pace, B. (2002). Association between time interval to publication and statistical significance. Journal of the American Medical Association, 287, 2829-2831.
29. Dimick, J.B., Diener-West, M. and Lipsett, P.A. (2001). Negative results of randomized clinical trials published in the surgical literature: Equivalency or error? Archives of Surgery, 136, 796-800.
30. Djulbegovic, B., Lacevic, M., Cantor, A., Fields, K.K., Bennett, C.L., Adams, J.R., Kuderer, N.M. and Lyman, G.H. (2000). The uncertainty principle and industrysponsored research. Lancet, 356, 635-638
31. Dudley, H.A.F. (1978). Surgical research: Master or servant. American Journal of Surgery, 135, 458-460.
32. Easterbrook, P.J., Berlin, J.A., Gopalan, R. and Matthews, D.R. (1991). Publication bias in clinical research. Lancet, 337, 867-872.
33. Editors (1909). The reporting of unsuccessful cases (editorial). The Boston Medical and Surgical Journal, 161, 263-264.
34. Egger, M., Zellweger-Zahner, T., Schneider, M., Junker, C., Lengeler, C. and Antes, G. (1997). Language bias in randomized controlled trials published in English and German. Lancet, 350, 326-329.
35. Einstein, A. (1930). What I believe. Forum and Century, 84, 183-194.
36. Eloubeidi, M.A., Wade, S.B. and Provenzale, D. (2001). Factors associated with acceptance and full publication of GI endoscopic research originally published in abstract form. Gastrointestinal Endoscopy, 53, 275-282.
37. Emerson, J.D., Burdick, E., Hoaglin, D.C., Mosteller, F. and Chalmers, T.C. (1990). An empirical study of the possible relation of treatment differences to quality scores in controlled randomized clinical trials. Controlled Clinical Trials, 11, 339-352.
38. Epstein, W.M. (1990). Confirmational response bias among social work journals. Science, Technology, and Human Values, 15, 9-37.
39. Ernst, E. and Pittler, M.H. (1997). Alternative therapy bias [letter]. Nature, 385, 480.
40. Evers, J.L. (2000). Publication bias in reproductive research. Human Reproduction, 15, 2063-2066.
41. Ferriar, J. (1792). Medical Histories and Reflexions (Vol. 1). London: Cadell and Davies.
42. Feynman, R.P. (1985). Surely You’re Joking Mr. Feynman. New York: Norton.
43. Glass, G.V., McGaw, B. and Smith, M.L. (1981). Meta-analysis in Social Research. Beverly Hills, CA: Sage Publications.
44. Gøtzsche, P.C. (1987). Reference bias in reports of drug trials. British Medical Journal (Clinical Research Edition), 195, 654-656.
45. Gould, S.J. (1987).Urchin in the Storm. Essays about Books and Ideas, New York: Norton.
46. Greenwald, A.G. (1975). Consequences of prejudice against the null hypothesis. Psychological Bulletin, 82, 1-20.
47. Grégoire, G., Derderian, F. and Le Lorier, J. (1995). Selecting the language of the publications included in a meta-analysis: Is there a Tower of Babel bias? Journal of Clinical Epidemiology, 48, 159-163.
48. Hahn, S., Williamson, P.R. and Hutton, J.L. (2002). Investigation of within-study selective reporting in clinical research: Follow-up of applications submitted to a local research ethics committee. Journal of Evaluation in Clinical Practice, 8, 353-359.
49. Hall, M.B. (1965). In defense of experimental essays. In: Robert Boyle on Natural Philosophy (pp. 119-131). Bloomington: Indiana University Press.
50. Hemminki, E. (1980). Study of information submitted by drug companies to licensing authorities. British Medical Journal, 280, 833-836.
51. Hetherington, J., Dickersin, K., Chalmers, I. and Meinert, C.L. (1989). Retrospective and prospective identification of unpublished controlled trials: Lessons from a survey of obstetricians and pediatricians. Pediatrics, 84, 374-380.
52. Hopewell, S., Clarke, M., Stewart, L. and Tierney, J. (2003). Time to publication for results of clinical trials. The Cochrane Database of Methodology Reviews 2001, Issue 3. Art No.: MR000011. DOI 10.1002/14651858.MR000011. John Wiley & Sons, Ltd.
53. Horton, R. (1997). Medical editor trial amnesty. Lancet, 350, 756.
54. Hutchison, B.G. and Lloyd, S. (1995). Comprehensiveness and bias in reporting clinical trials: Study of reviews of pneumococcal vaccine effectiveness. Canadian Family Physician, 41, 1356-1360.
55. Hutton, J.L. and Williamson, P.R. (2000). Bias in meta-analysis due to outcome variable selection within studies. Applied Statistics, 49, 359-370.
56. Institute of Medicine Committee on Assessing the System for Protecting Human Research Participants. Responsible Research (2002). A Systems Approach to Protecting Research Participants. Washington, DC: National Academy Press.
57. Institute of Medicine Committee on Routine Patient Care Costs in Clinical Trials for Medicare Beneficiaries (2000). Extending Medicare Reimbursement in Clinical Trials. Washington, DC: National Academy Press.
58. Ioannidis, J.P. (1998). Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. Journal of the Americal Medical Association, 279, 281-286.
59. Ioannidis, J.P.A., Cappelleri, J.C., Sacks, H.S. and Lau, J. (1997). The relationship between study design, results, and reporting of randomized clinical trials of HIV infection. Controlled Clinical Trials, 18, 431-444.
60. Ioannidis, J.P.A., Trikalinos, T.A., Ntzani, E.E. and Contopoulos-Ioannidis, D.G. (2003). Genetic associations in large versus small studies: An empiric assessment. Lancet, 361, 567-571.
61. Jennions, M.D. & Møller, A.P. (2002). Relationships fade with time: A meta-analysis of temporal trends in publication in ecology and evolution. Proceedings of the Royal Society of London, Series B: Biological Sciences, 269, 43-48.
62. Jüni, P., Holstein, F., Sterne, J., Bartlett, C. and Egger, M. (2002). Direction and impact of language bias in meta-analyses of controlled trials: An empirical study. International Journal of Epidemiology, 31, 115-123.
63. Kjaergard, L.L. and Gluud, C. (2002). Citation bias of hepato-biliary randomized clinical trials. Journal of Clinical Epidemiology, 55, 407-410.
64. Kjaergard, L.L. and Als-Nielsen, B. (2002). Association between competing interests and authors’ conclusions: Epidemiological study of randomized clinical trials published in the British Medical Journal. British Medical Journal, 325, 249-252.
65. Knight, J. (2003). Null and void. Nature, 422, 554-555.
66. Kunz, R. and Oxman, A.D. (1998). The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials. British Medical Journal, 317, 1185-1190.
67. MacLean, C.H., Morton, S.C., Ofman, J.J., Roth, E.A. and Shekelle, P.G. for the Southern California Evidence-based Practice Center (2003). How useful are unpublished data from the Food and Drug Administration in meta-analysis? Journal of Clinincal Epidemiology, 56, 44-51.
68. Mahoney, M.J. (1977). Publication prejudices: An experimental study of confirmatory bias in the peer review system. Cognitive Therapy & Research, 1, 161-175.
69. Mallett, S., Hopewell, S. and Clarke, M. (2002). The use of the grey literature in the first 1000 Cochrane reviews. In 4th Symposium on Systematic Reviews: Abstract no. 5. Pushing the Boundaries. Oxford, UK.
70. Manheimer, E. and Anderson, D. (2002). Survey of public information about ongoing clinical trials funded by industry: Evaluation of completeness and accessibility. British Medical Journal, 325, 528-531.
71. Mann, H. and Djulbegovic, B. (2004) Biases due to differences in the treatments selected for comparison (comparator bias). James Lind Library, Retrieved on January 14, 2004 from www.jameslindlibrary.org.
72. McAuley, L., Pham, B., Tugwell, P. and Moher, D. (2000). Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analysis? Lancet, 356, 1228-1231.
73. McCray, A.T. (2001). Better access to information about clinical trials. Annals of Internal Medicine, 133, 609-614.
74. McManus, R.J., Wilson, S., Delaney, B.C., Fitzmaurice, D.A., Hyde, C.J., Tobias, R.S., Jowett, S. and Hobbs, F.R. (1998). Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. British Medical Journal, 317, 1562-1563.
75. Misakian, A.L. and Bero, L.A. (1998). Publication bias and research on passive smoking. Comparison of published and unpublished studies. Journal of the American Medical Association, 280, 250-253.
76. Moscati, R., Jehle, D., Ellis, D., Fiorello, A. and Landi, M. (1994). Positive-outcome bias: Comparison of emergency medicine and general medicine literatures. Academic Emergency Medicine, 1, 267-271.
77. Olson, C.M., Rennie, D., Cook, D., Dickersin, K., Flanagin, A., Hogan, J., Zhu, Q., Reiling, J. and Pace, B. (2002). Publication bias in editorial decision making. Journal of the American Medical Association, 287, 2825-2828.
78. Ottenbacher, K. and Difabio, R. P. (1985). Efficacy of spinal manipulation/mobilization therapy: A meta-analysis. Spine, 10, 833-837.
79. Petticrew, M., Gilbody, S. and Song, F. (1999). Lost information? The fate of papers presented at the 40th Society for Social Medicine Conference. Journal of Epidemiology and Community Health, 53, 442-443.
80. Pich, J., Carné, X., Arniaz, J.A., Gómez, B., Trilla, A. and Rodés, J. (2003). Role of a research ethics committee in follow-up and publication of results. Lancet, 361, 1015-1016.
81. Radford, D.R., Smillie, L., Wilson, R.F. and Grace, A.M. (1999). The criteria used by editors of scientific dental journals in the assessment of manuscripts submitted for publication. British Dental Journal, 187, 376-379.
82. Ravnskov, U. (1992). Cholesterol lowering trials in coronary heart disease: Frequency of citation and outcome. British Medical Journal, 305, 15-19.
83. Ravnskov, U. (1995). Quotation bias in reviews of the diet heart idea. Journal of Clinical Epidemiology, 48, 713-719.
84. Rennie, D. (1997). Thyroid storm. Journal of the American Medical Association, 277, 1238-1243.
85. Rennie, D. (2002). Fourth International Congress on peer review in biomedical publication. Journal of the American Medical Association, 287, 2759-2760.
86. Rochon, P.A., Gurwitz, J.H., Simms, R.W., Fortin, P.R., Felson, D.T., Minaker, K.L. and Chalmers, T.C. (1994). A study of manufacturer supported trials of non-steroidal anti-inflammatory drugs in the treatment of arthritis. Archives of Internal Medicine, 154, 157-163.
87. Savulescu, J., Chalmers, I. and Blunt, J. (1996). Are research ethics committees behaving unethically? Some suggestions for improving performance and accountability. British Medical Journal, 313, 1390-1393.
88. Scherer, R.W. and Langenberg, P. The Cochrane Library (2005). Full publication of results initially presented in abstracts. The Cochrane Database of Methodology Reviews 2005, Issue 2. Art No.: MR000005. DOI: 10.1002/14651858.MR000005.pub2. John Wiley & Sons, Ltd.
89. Shadish, W.R., Doherty, M. and Montgomery, L.M. (1989). How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature. Clinical Psychology Review, 9, 589-603.
90. Simes, R.J. (1986). Publication bias: The case for an international registry of clinical trials. Journal of Clinical Oncology, 4, 1529-1541.
91. Smart, R.G. (1964). The importance of negative results in psychological research. Canadian Psychologist, 5, 225-232.
92. Smith, M.L. (1980a). Publication bias and meta-analysis. Evaluating Education, 4, 22-24.
93. Smith, M.L. (1980b). Sex bias in counseling and psychotherapy. Psychological Bulletin, 87, 392-407.
94. Sommer, B. (1987). The file drawer effect and publication rates in menstrual cycle research. Psychology of Women Quarterly, 11, 233-242.
95. Sterling, T.D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance -or vice versa. Journal of the American Statistical Association, 54, 30-34.
96. Sterling, T.D., Rosenbaum, W.L. and Weinkam, J.J. (1995). Publication decisions revisited: The effect of the outcome of statistical tests on the decision to publish. American Statistician, 49, 108-112.
97. Stern, J.M. and Simes, R.J. (1997). Publication bias: Evidence of delayed publication in a cohort study of clinical research projects. British Medical Journal, 315, 640-645.
98. Sterne, J.A.C., Jüni, P., Schulz, K.F., Altman, D.G., Bartlett, C. and Egger, M. (2002). Statistical methods for assessing the influence of study characteristics on treatment effects in ‘meta-epidemiological’ research. Statistics in Medicine, 21, 1513-1524.
99. Swaen, G.G., Teggeler, O. and van Amelsvoort, L.G. (2001). False positive outcomes and design characteristics in occupational cancer epidemiology studies. International Journal of Epidemiology, 30, 948-954.
100. Tannock, I.F. (1996). False-positive results in clinical trials: Multiple significance tests and the problems of unreported comparisons. Journal of the National Cancer Institute, 88, 206-207.
101. Tierney, J.F., Clarke, M. and Stewart, L.A. (2000). Is there bias in the publication of individual patient data meta-analyses? International Journal of Technology Assessment in Health Care, 16, 657-667.
102. Timmer, A., Hilsden, R.J., Cole, J., Hailey, D. and Sutherland, L.R. (2002). Publication bias in gastroenterological research -a retrospective cohort study based on abstracts submitted to a scientific meeting. BMC Medical Research Methodology, 2, 7.
103. Tonks, A. (2002). A clinical trials register for Europe. British Medical Journal, 325, 1314-1315.
104. Tröhler, U. (2000). ‘To improve the evidence of medicine’. The 18th century British origins of a critical approach. Edinburgh: Royal College of Physicians of Edinburgh.
105. Tröhler, U. (2003a) Withering’s 1795 appeal for caution when reporting on a new medicine. James Lind Library. Retrieved on April 7, 2003, from www.jameslindlibrary.org.
106. Tröhler, U. (2003b) John Clark 1780 & 1792: Learning from properly kept records. James Lind Library. Retrieved on April 7, 2003, from www.jameslindlibrary.org.
107. Vickers, A., Goyal, N., Harland, R. and Rees, R. (1998). Do certain countries produce only positive results? A systematic review of controlled trials. Controlled Clinical Trials, 19, 159-166.
108. Vogel, U. and Windeler, J. (2000). E in flu ßfaktoren auf die publikation shaiifigkeit klinischer forschung sergebnisse am Beispiel medizinischer Dissertationen. [Factors modifying the frequency of publications of clinical research results exemplified by medical dissertations]. Deutsche Medizinische Wochenschrift, 125, 110-113.
109. White, K.R. (1982). The relationship between socioeconomic status and academic achievement. Psychological Bulletin, 91, 461-481.
110. Withering, W. (1785). An account of the foxglove and some of its medical uses: With practical remarks on dropsy and other diseases. London: J. and J. Robinson.
111. Wormald, R., Bloom, J., Evans, J. and Oldfield, K. (1997). Publication bias in eye trials. Paper presented at the Fifth Annual Cochrane Colloquium, Amsterdam.
112. Yaphe, J., Edman, R., Knishkowy, B. and Herman, J. (2001). The association between funding by commercial interests and study outcome in randomized controlled drug trials. Family Practice, 18, 565-568.
113. Zimpel, T. and Windeler, J. (2000). Veröffentlichungen von Dissertationen zu unkonventionellen medizinischen Therapie-und Diagnoseverfahren -ein Beitrag zum ‘publication bias’. Forschung zu Komplementärmedizin und Klassischer Naturheilkunde, 7, 71-74.