Bias in Meta-Analytic Estimates of the Absolute Efficacy of Psychotherapy

Review of General Psychology

Volumn 11, Issue 4, 2007, Pages 329-347

  Staines, Graham L ^a   Cleland, Charles M ^b  

^a Highland Park Medical Associates   (United States)

^b NATIONAL DEVELOPMENT AND RESEARCH INSTITUTES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 38349145619     PISSN: 10892680     EISSN: None     Source Type: Journal    
DOI: 10.1037/1089-2680.11.4.329     Document Type: Article

References (73)

1. Abelson R.P. Statistics as principled argument (1995), Erlbaum, Hillsdale, NJ
2. Andrews G., and Harvey R. Does psychotherapy benefit neurotic patients? A reanalysis of the Smith, Glass, & Miller data. Archives of General Psychiatry 38 (1981) 1203-1208
3. Basham R.B. Scientific and practical advantages of comparative design in psychotherapy outcome research. Journal of Consulting and Clinical Psychology 54 (1986) 88-94
4. Beck A.T., Ward C.H., Mendelson M., Mock J., and Erbaugh J. An inventory for measuring depression. Archives of General Psychiatry 4 (1961) 561-571
5. Berman J.S., Miller R.C., and Massman P.J. Cognitive therapy versus systematic desensitization: Is one treatment superior?. Psychological Bulletin 97 (1985) 451-461
6. Cappelleri J.C., Ioannidis J.P., Schmid C.H., de Ferranti S.D., Aubert M., Chalmers T.C., and Lau J. Large trials vs. meta-analyses of small trials: How do their results compare?. Journal of the American Medical Association 276 (1996) 1332-1338
7. Cochran W.G. The combination of estimates from different experiments. Biometrics 10 (1954) 101-129
8. Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. (1988), Erlbaum, Hillsdale, NJ
9. Contopoulos-Ioannidis D.G., Gilbody S.M., Trikalinos T.A., Churchill R., Wahlbeck K., and Ioannidis J.P.A. Comparison of large versus smaller randomized trials for mental health-related interventions. American Journal of Psychiatry 162 (2005) 578-584
10. Cronbach L.J. Essentials of psychological testing. 5th ed. (1990), Harper & Row, New York
11. Duval S., and Tweedie R. Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56 (2000) 455-463
12. Elliott R., Stiles W.B., and Shapiro D.A. "Are some psychotherapies more equivalent than others?". In: Giles T.R. (Ed). Handbook of effective psychotherapy (1993), Plenum Press, New York 455-479
13. Eysenck H.J. Meta-analysis: An abuse of research integration. The Journal of Special Education 18 (1984) 41-59
14. Eysenck H.J. Forty years on: The outcome problem in psychotherapy revisited. In: Giles T.R. (Ed). Handbook of effective psychotherapy (1993), Plenum Press, New York 3-20
15. Gaffan E.A., Tsaousis I., and Kemp-Wheeler S.M. Researcher allegiance and meta-analysis: The case of cognitive therapy for depression. Journal of Consulting and Clinical Psychology 63 (1995) 966-980
16. Gilbody S.M., Song F., Eastwood A.J., and Sutton A. The causes, consequences and detection of publication bias in psychiatry. Acta Psychiatrica Scandinavica 102 (2000) 241-249
17. Greenwald A.G. Consequences of prejudice against the null hypothesis. Psychological Bulletin 82 (1975) 1-20
18. Grissom R.J. The magical number .7 ± .2: Meta-meta-analysis of the probability of superior outcome in comparisons involving therapy, placebo, and control. Journal of Consulting and Clinical Psychology 64 (1996) 973-982
19. Hedges L.V. Distribution theory for Glass's estimator of effect size and related estimators. Journal of Educational Statistics 6 (1981) 107-128
20. Hedges L.V. Estimation of effect size from a series of independent experiments. Psychological Bulletin 92 (1982) 490-499
21. Hedges L.V., and Olkin I. Statistical methods for meta-analysis (1985), Academic Press, San Diego, CA
22. Hedges L.V., and Vevea J.L. Estimating effect size under publication bias: Small sample properties and robustness of a random effects selection model. Journal of Educational and Behavioral Statistics 21 (1996) 299-332
23. Hedges L.V., and Vevea J.L. Fixed- and random-effects models in meta-analysis. Psychological Methods 3 (1998) 486-504
24. Hsu L.M. Fail-safe Ns for one- versus two-tailed tests lead to different conclusions about publication bias. Understanding Statistics 1 (2002) 85-100
25. Huedo-Medina T.B., Sanchez-Meca J., Botella F., and Marin-Martinez J. Assessing heterogeneity in meta-analysis: Q statistic or I2 index?. Psychological Methods 11 (2006) 193-206
26. Hunter J.E., and Schmidt F.L. Methods of meta-analysis: Correcting error and bias in research findings (1990), Sage, Beverly Hills, CA
27. Hunter J.E., and Schmidt F.L. Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy, and Law 2 (1996) 324-347
28. Hunter J.E., Schmidt F.L., and Le H. Implications of direct and indirect range restriction for meta-analysis methods and findings. Journal of Applied Psychology 91 (2006) 594-612
29. Johnson B.T., and Eagly A.H. Quantitative synthesis of social psychological research. In: Reis H.T., and Judd C.M. (Eds). Handbook of research methods in social and personality psychology (2000), Cambridge University Press, London 496-528
30. Kalaian H.A., and Raudenbush S.W. A multivariate mixed linear model for meta-analysis. Psychological Methods 1 (1996) 227-235
31. Kendall P.C., Holmbeck G., and Verduin T. Methodology, design, and evaluation in psychotherapy research. In: Lambert M.J. (Ed). Bergin and Garfield's handbook of psychotherapy and behavior change. 5th ed. (2004), Wiley, New York 16-43
32. Klein D.F. Listening to meta-analysis but hearing bias. Prevention & Treatment 1 (1998) article 6c
33. Kulinskaya E., Dollinger M.B., Knight E., and Gao H. A Welch-type test for homogeneity of contrasts under heteroscedasticity with application to meta-analysis. Statistics in Medicine 23 (2004) 3655-3670
34. Lambert M.J., and Bergin A.E. The effectiveness of psychotherapy. In: Bergin A.E., and Garfield S.L. (Eds). Handbook of psychotherapy and behavior change. 4th ed. (1994), Wiley, New York
35. Lambert M.J., and Ogles B.M. The efficacy and effectiveness of psychotherapy. In: Lambert M.J. (Ed). Bergin and Garfield's handbook of psychotherapy and behavior change. 5th ed. (2004), Wiley, New York 139-193
36. Landman J.T., and Dawes R.M. Psychotherapy outcome: Smith and Glass' conclusions stand up under scrutiny. American Psychologist 37 (1982) 504-516
37. LeLorier J., Grégoire G., Benhaddad A., Lapierre J., and Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials. New England Journal of Medicine 337 (1997) 536-542
38. Lipsey M.W., and Wilson D.B. The efficacy of psychological, educational, and behavioral treatment: Confirmation from meta-analysis. American Psychologist 48 (1993) 1181-1209
39. Luborsky L., Diguer L., Seligman D.A., Rosenthal R., Krause E.D., Johnson S., et al. The researcher's own therapy allegiances: A "wild card" in comparisons of treatment efficacy. Clinical Psychology: Science & Practice 6 (1999) 95-106
40. Matt G.E., and Navarro A. What meta-analyses have and have not taught us about psychotherapy effects: A review and future directions. Clinical Psychology Review 17 (1997) 1-32
41. Moses L.E., Mosteller F., and Buehler J.H. Comparing results of large clinical trials to those of meta-analyses. Statistics in Medicine 21 (2002) 793-800
42. Munafò M.R., Clark T.G., and Flint J. Assessing publication bias in genetic association studies: Evidence from a recent meta-analysis. Psychiatry Research 129 (2004) 39-44
43. Poole C., and Greenland S. Random-effects meta-analyses are not always conservative. American Journal of Epidemiology 150 (1999) 469-475
44. Preston C., Ashby D., and Smyth R. Adjusting for publication bias: Modelling the selection process. Journal of Evaluation in Clinical Practice 10 (2004) 313-322
45. Richy F., and Reginster J.Y. A simple method for detecting and adjusting meta-analyses for publication bias. The Internet Journal of Epidemiology 3 2 (2006). http://www.ispub.com/ostia/index.php?xmlFilePath=journals/ije/vol3n2/meta.xml Retrieved August 27, 2007, from http://www.ispub.com/ostia/index.php?xmlFilePath=journals/ije/vol3n2/meta.xml
46. Rosenthal R. On the social psychology of the psychological experiment: The experimenter's hypothesis as unintended determinant of experimental results. American Scientist 51 (1963) 268-283
47. Rosenthal R. The file drawer problem and tolerance for null results. Psychological Bulletin 86 (1979) 638-641
48. Rosenthal R. Effect size estimation, significance testing, and the file-drawer problem. Journal of Parapsychology 56 (1992) 57-58
49. Rosenthal R., and Rubin D.B. Meta-analytic procedures for combining studies with multiple effect sizes. Psychological Bulletin 99 (1986) 400-406
50. Sanchez Meca J., Olivares Rodríguez J., and Rosa Alcázar A.I. The problem of tobacco addiction: Meta-analysis of behavioral treatments in Spain. Psychology in Spain 3 (1999) 36-45
51. Scargle J.D. Publication bias: The "file-drawer" problem in scientific inference. Journal of Scientific Exploration 14 (2000) 91-106
52. Shadish W.R., and Baldwin S. Effects of behavioral marital therapy: A meta-analysis of randomized controlled trials. Journal of Consulting and Clinical Psychology 73 (2005) 6-14
53. Shadish W.R., Cook T.D., and Campbell D.T. Experimental and quasi-experimental designs for generalized causal inference (2002), Houghton Mifflin, Boston
54. Shadish W.R., Doherty M., and Montgomery L.M. How many studies are in the file drawer? An estimate from the family/marital psychotherapy literature. Clinical Psychology Review 9 (1989) 589-603
55. Shadish W.R., Matt G.E., Navarro A.M., Siegle G., Crits-Christoph P., Hazelrigg M.D., et al. Evidence that therapy works in clinically representative conditions. Journal of Consulting and Clinical Psychology 65 (1997) 355-365
56. Shapiro D.A., Barkham M., Rees A., Hardy G.E., Reynolds S., and Startup M. Effects of treatment duration and severity of depression on the effectiveness of cognitive-behavioral and psychodynamic-interpersonal psychotherapy. Journal of Consulting and Clinical Psychology 62 (1994) 522-534
57. Shapiro D.A., and Shapiro D. Meta-analysis of comparative therapy outcome studies: A replication and refinement. Psychological Bulletin 92 (1982) 581-604
58. Sharpe D. Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review 17 (1997) 881-901
59. Smith M.L., and Glass G.V. Meta-analysis of psychotherapy outcome studies. American Psychologist 32 (1977) 752-760
60. Smith M.L., Glass G.V., and Miller T.I. The benefits of psychotherapy (1980), Johns Hopkins University, Baltimore, MD
61. Soeken K.L., and Sripusanapan A. Assessing publication bias in meta-analysis. Nursing Research 52 (2003) 57-60
62. Stanton M.D., and Shadish W.R. Outcome, attrition, and family-couples treatment for drug abuse: A meta-analysis and review of the controlled, comparative studies. Psychological Bulletin 122 (1997) 170-191
63. Stephens R.S., Roffman R.A., and Curtin L. Comparison of extended versus brief treatments for marijuana use. Journal of Consulting and Clinical Psychology 68 (2000) 898-908
64. Strupp H.H., and Hadley S.W. Specific versus nonspecific factors in psychotherapy: A controlled study of outcome. Archives of General Psychiatry 36 (1979) 1125-1136
65. Thase M. What is the investigator allegiance effect and what should we do about it?. Clinical Psychology: Science and Practice 6 (1999) 113-115
66. Thornton A., and Lee P. Publication bias in meta-analysis: Its causes and consequences. Journal of Clinical Epidemiology 53 (2000) 207-216
67. Vevea J.L., and Hedges L.V. A general linear model for estimating effect size in the presence of publication bias. Psychometrika 60 (1995) 419-435
68. Wampold B.E. The great psychotherapy debate: Model, methods, and findings (2001), Erlbaum, Mahwah, NJ
69. Watson J.C., Gordon L.B., Stermac L., Kalogerakos F., and Steckley P. Comparing the effectiveness of process-experiential with cognitive-behavioral psychotherapy in the treatment of depression. Journal of Consulting and Clinical Psychology 71 (2003) 773-781
70. Weisz J.R., Jensen-Doss A., and Hawley K.M. Evidence-based youth treatments versus usual clinical care: A meta-analysis of direct comparisons. American Psychologist 61 (2006) 671-689
71. Weisz J.R., McCarty C.A., and Valeri S.M. Effects of psychotherapy for depression in children and adolescents: A meta-analysis. Psychological Bulletin 132 (2006) 132-149
72. Westen D., Novotny C.M., and Thompson-Brenner H. The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin 130 (2004) 631-663
73. Wierzbicki M., and Pekarik G. A meta-analysis of psychotherapy dropout. Professional Psychology: Research and Practice 24 (1993) 190-195