Some limitations on the external validity of psychotherapy efficacy studies and suggestions for future research

American Journal of Psychotherapy

Volumn 66, Issue 3, 2012, Pages 227-242

  Shean, Glenn D ^a  

^a The College of William and Mary   (United States)

Author keywords

Empirically supported treatments; Outcome; Psychotherapy effectiveness; Psychotherapy efficacy

Indexed keywords

BOOK; DOCTOR PATIENT RELATION; EMPIRICAL RESEARCH; EVIDENCE BASED PRACTICE; HUMAN; INDIVIDUALITY; PSYCHOTHERAPY; RANDOMIZED CONTROLLED TRIAL (TOPIC); REVIEW; STANDARD; TREATMENT OUTCOME;

EMPIRICAL RESEARCH; EVIDENCE-BASED PRACTICE; HUMANS; INDIVIDUALITY; MANUALS AS TOPIC; PHYSICIAN-PATIENT RELATIONS; PSYCHOTHERAPY; RANDOMIZED CONTROLLED TRIALS AS TOPIC; TREATMENT OUTCOME;

EID: 84867374574     PISSN: 00029564     EISSN: None     Source Type: Journal    
DOI: 10.1176/appi.psychotherapy.2012.66.3.227     Document Type: Article

References (92)

1. Ablon, J.S., & Jones, E.E. (1998). How expert clinicians' prototypes of an ideal treatment correlate with outcome in psychodynamic and cognitive-behavioral therapy. Psychotherapy Research, 8, 71-83. (Pubitemid 128548229)
2. Ablon, J.S., & Jones, E.E. (2002). Validity of controlled clinical trials of psychotherapy: Findings from the NIMH Treatment of Depression Collaborative Research Program. American Journal of Psychiatry, 159, 775-783. (Pubitemid 34465051)
3. Ablon, J.S., Levy, R.A., & Katzenstein, T. (2006). Beyond brand names of psychotherapy: Identifying empirically supported change processes. Psychotherapy: Theory, Research, Practice, Training, 43, 216-231. (Pubitemid 44128251)
4. American Psychological Association Task Force on Psychological Intervention Guidelines. (1995). Template for developing guidelines: Interventions for mental disorders and psychosocial aspects of physical disorders. Washington, DC: American Psychological Association.
5. American Psychological Association (2005). Evidence-based practice in psychology: APA presidential task force on evidence-based practice. American Psychologist, 61, 271-285.
6. Asay, T.P., & Lambert, M.J. (1999). The empirical case for the common factors in therapy: Quantitative findings. In M.A. Hubble, B.L. Duncan, & S.D. Miller (Eds.). The heart and soul of change: What works in therapy (pp. 23-55). Washington, DC: American Psychological Association.
7. Baker, T., McFall, R., & Shoram, V. (2009, November 15). Is your therapist a little behind the times? Washington D.C.: Washington Post.
8. Baldwin, S.A., Berkeljon, P., Atkins, J., Olsen, A., & Nielsen, T. (2009). Rates of change in naturalistic psychotherapy: Contrasting dose-effect and good-enough level models of change. Journal of Consulting and Clinical Psychology, 77, 203-211.
9. Baldwin, S.A., Wampold, B.E., & Imel, Z.E. (2007). Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75, 842-852.
10. Barlow, D.H., Gorman, J.M., Shear, M.K., & Woods, S.W. (2000). Cognitive-behavioral therapy, imipramine, or their combination for panic disorder: A randomized controlled trial. Journal of the American Medical Association, 283, 2529-2536. (Pubitemid 30324195)
11. Begley, S. (2009). Ignoring the evidence: Why do psychologists reject science? Newsweek, October 12, 2009.
12. Benish, S.G., Imel, Z.E., & Wampold, B.E. (2008). The relative efficacy of bona fide psychotherapies for treating post-traumatic stress disorder: A meta-analysis of direct comparisons. Clinical Psychology Review, 28, 746-758. (Pubitemid 351551466)
13. Beutler, L.E. (1998). Identifying empirically supported treatments: what if we didn't? Journal of Consulting and Clinical Psychology, 66, 113-120. (Pubitemid 28086812)
14. Beutler, L.E., Malik, M., Alimohamed, S., Harwood, T.M., Talbi, H., Noble, S., & Wong, E. (2004). Therapist variables. In M.J. Lambert (Ed.) Handbook of psychotherapy and behavior change (5th ed. Pp. 194-226). New York, NY: Wiley.
15. Beutler, L.E., Harwood, T.M., Michelson, A. & Song, X. (2011). Reistance/Reactance level. Journal of Clinical Psychology, 67, 133-142.
16. Blatt, S., & Zuroff, D. (1992). Interpersonal relatedness and self-definition: Two prototypes for depression. Clinical Psychology Review, 12, 527-562.
17. Calhoun, K.S., Moras, K., Pilkonis, P.A., & Rehm, L. (1998). Empirically supported treatments: Implications for training. Journal of Consulting and Clinical Psychology, 66, 151-162. (Pubitemid 28086817)
18. Carpinello, S.E., Rosenberg, L., Stone, J., Schwager, M., & Felton, C.J. (2002). New York State's campaign to implement evidence based practice practices for people with serious mental disorders. Psychiatric Services, 53, 153-155. (Pubitemid 34111493)
19. Chambless, D.L., Sanderson, W.C., Shohan, V., Bennet, B.E., Johnson, S., & Pope, K.S. (1996). An update on empirically validated therapies. The Clinical Psychologist, 49, 5-18.
20. Chambless, D.L., & Hollon, S.D. (1998. Defining empirically supported therapies. Journal of Consulting and Clinical Psychology, 66, 7-18. (Pubitemid 28086807)
21. Chambless, D.L., & Ollendick, T.H. (2001). Empirically supported psychological interventions: Controversies and evidence. Annual Review of Psychology, 52, 685-716. (Pubitemid 33705589)
22. Clarkin, J.F., Levy, K.N., Lenzenweger, M.F., & Kernberg, O.F. (2007). Evaluating three treatments for borderline personality disorder: A multiwave study. American Journal of Psychiatry, 164, 922-928. (Pubitemid 46972908)
23. Cuijpers, J., van Straten, O., Andersson, H., & van Oppen, C. (2008). Psychotherapy for depression in adults: A meta-analysis of comparative outcome studies. Journal of Consulting and Clinical Psychology, 76, 909-922.
24. Davison, G.C. (2000). Stepped care: Doing more with less? Journal of Consulting and Clinical Psychology, 68, 580-923.
25. Elkin, I., Shea, T., Watkins, J. Imber, S., Sotsky, S. & Collins, J. (1989). National Institute of Mental Health treatment of Depression Colaborative Research Program. Archives of General Psychiatry, 46, 971-982. (Pubitemid 19278225)
26. Hammen, C., Ellicott, A., Gitlin, M., & Jamison, K.R. (1989). Sociotropy/autonomy and vulnerability to specific life events in patients with unipolar depression and bipolar disorders. Journal of Abnormal Psychology, 98, 154-160. (Pubitemid 19134403)
27. Hill, C.E., Sim, W., Spangler, P., Stahl, J., Sullivan, C., & Teyber, E. (2008). Therapist immediacy in brief psychotherapy therapy: Case Study II. Psychotherapy: Theory, Research, Practice, Training, 45, 298-316.
28. Horvath, A.O., & Bedi, R.P. (2002). The alliance. In J.C. Norcross (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 37-70). New York, NY: Oxford University Press.
29. Howard, K.I., Kopta, S.M., Krause, M.S., & Orlinsky, D.E. (1986). The dose-effect relationship in psychotherapy. American Psychologist, 41, 159-164.
30. Howard, K.I., Lueger, R., Maling, M., Martinovich, Z. (1993). A phase model of psychotherapy: Causal mediation of outcome. Journal of Consulting and Clinical Psychology, 54, 106-110.
31. Howard, K.L., Comille, T.A., Lyons, J.S., Vessey, J.T., Lueger, R.J., & Saunders, S.M. (1996). Patterns of mental health service utilization. Archives of General Psychiatry, 53, 696-703. (Pubitemid 26269330)
32. Hox, J. (2002). Mutilevel analysis: Techniques and applications. Mahway, NJ: Erlbaum.
33. Huppert, J.D., Bufka, L.F., Barlow, D.H., Gorman, J.M., Shear, M.K., & Woods, S.W. (2001). Therapists, therapist variables, and cognitive-behavioral therapy outcomes in a muticenter trial for panic disorder. Journal of Consulting and Clinical Psychology, 69, 747-755. (Pubitemid 32973250)
34. Jacobson, N.S. (1999). The role of the Allegiance Effect in psychotherapy research: Controlling and accounting for it. Clinical Psychology: Science and Practice, 6, 116-119.
35. Jacobson, N.S. & Addis, M.E. (1993). Research on couples and couple therapy: What do we know? Where are we going? Journal of Consulting and Clinical Psychology, 61, 85-93. (Pubitemid 23142102)
36. Kasper, L., Hill, C.E., & Kivlighan, D.M. Jr. (2008). Therapist immediacy in brief psychotherapy therapy: Case Study I. Psychotherapy: Theory, Research, Practice, Training, 45, 281-297.
37. Kazdin, A.E. (2007). Mediators and mechanisms of change in psychotherapy research. Annual Review of Clinical Psychology, 3, 1-27. (Pubitemid 46651119)
38. Kazdin, A.E. (2008). Evidence-based treatment and practice: New opportunities to bridge clinical research and practice, enhance the knowledge base, and improve patient care. American Psychologist, 63, 146-159.
39. Kessler, R.C., Stang, P., Wittchen, H.U., Stein, M., & Walters, E.E. (1999). Lifetime comorbidities between social phobia and mood disorders in the U.S. Comorbidity Survey. Psychological Medicine, 29, 555-567. (Pubitemid 29261344)
40. Kim, D.M., Wampold, B.E., & Bolt, D.M. (2006). Therapist effects in psychotherapy: A random effects modeling of the NIMH TDCRP data. Psychotherapy Research, 16, 161-172.
41. Klein, D.N., Schwartz, J.E., Santiago, N.J., Vivian, D., Vocisano, C., Castonguay, L.G., & Keller, M.B. (2003). Therapeutic alliance in depression treatment: Controlling for prior change and patient characteristics. Journal of Consulting and Clinical Psychology, 71, 997-1006. (Pubitemid 37448890)
42. Kopta, S., Howard, K., Lowry, J., & Beutler, L. (1994). Patterns of symptomatic recovery in psychotherapy. Journal of Consulting and Clinical Psychology, 62, 1009-1016. (Pubitemid 24328508)
43. Kreuger, R.F., (2002). The structure of common mental disorders. Archives of General Psychiatry, 59, 570-571.
44. Kwon, P., & Whisman, M.A. (1998). Sociotropy and autonomy as vulnerabilities to specific life events: Issues in life event categorization. Cognitive Therapy and Research, 22, 353-362. (Pubitemid 28425759)
45. Lambert, M.J., & Ogles, B.M. (2004). The efficacy and effectiveness of psychotherapy. In M. Lambert (Ed.), Handbook of psychotherapy and behavior change (5th ed., pp. 139-193) New York, NY: Wiley.
46. LaRoche, M.J. & Christopher, M.S. (2009). Changing paradigms from empirically supported treatment to evidence-based practice: A clinical perspective. Professional Psychology: Research and Practice, 40, 396-402.
47. Leichsenring, F. (2001). Comparative effects of short-term psychodynamic psychotherapy and Cognitive- behavioral therapy in depression: A meta-analytic approach. Clinical Psychology Review, 21, 401-419. (Pubitemid 32220073)
48. Leichsenring, F. (2005). Are psychodynamic and psychoanalytic therapies effective? A review of empirical data. International Journal of Psychoanalysis, 86, 841-868. (Pubitemid 40872553)
49. Leichsenring, F. & Leibing, E. (2003). The effectiveness of psychodynamic psychotherapy and cognitive behavior therapy in the treatment of personality disorders: A meta-analysis. American Journal of Psychiatry, 160, 1223-1232. (Pubitemid 41070937)
50. Leichsenring, F., Rabung, S., & Leibing, E. (2004). The efficacy of short-term psychodynamic psychotherapy in specific psychiatric disorders: A meta-analysis. Archives of General Psychiatry, 61, 1208-1216. (Pubitemid 39602837)
51. Leichsenring, F., & Rabung, S. (2008). Effectiveness of long-term psychodynamic psychotherapy: A meta-analysis. Journal of the American Medical Association, 300, 1551-1565.
52. Levy, R., & Ablon, S.J. (Eds.). (2008). Handbook of evidence-based psychodynamic psychotherapy: Bridging the gap between science and practice. New York: Humana Press.
53. Lewis, A.J., Dennerstein, M., & Gibbs, P.M. (2008). Short-term psychodynamic psychotherapy: review of recent process and outcome studies. Australian and New Zealand Journal of Psychiatry, 42, 445-455. (Pubitemid 351891123)
54. Luborsky, L., Diguer, L., Seligman, D., Rosenthal, R., Krause, E.D., Johnson, S., et al. (1999). The researcher's own therapy allegiances: A "wild card" in comparisons of treatment efficacy. Clinical Psychology: Science and Practice, 6, 95-106.
55. Luborsky, L., Rosenthal, R., Diguer, L., Andrusyna, T.P., & Berman, J.S. (2002). The dodo bird verdict is alive and well-mostly. Clinical Psychology Science and Practice, 9, 2-12. (Pubitemid 41288903)
56. Mineka, S., Watson, D., & Clark, L.A. (1998). Comorbidity of anxiety and unipolar mood disorders. Annual Review of Psychology, 49, 377-412. (Pubitemid 128491779)
57. Morrison, C., Bradley, R., & Westen, D. (2003). The external validity of efficacy trials for depression and anxiety: A naturalistic study. Psychology and Psychotherapy: Theory, Research and Practice, 76, 109-132. (Pubitemid 36775574)
58. National Institutes of Mental Health (2008). NIMH strategic plan. Rockville, MD: U.S. Department of Health and Human Services, NIH Publication No. 08-6368.
59. Norcross, J.C. (2001). Purposes, processes and products of the task force on empirically supported therapy relationships. Psychotherapy: Theory, Research, Practice, Training, 38, 345-356. (Pubitemid 34521342)
60. Norcross, J. (Ed.). (2002). Psychotherapy relationships that work: Therapists contributions and responsiveness to patients. New York: Oxford University Press.
61. Norcross, J.C., Koocher, G.P., & Garofalo, A. (2006). Discredited psychological treatments and tests: A Delphi poll. Professional Psychology: Research and Practice, 37, 515-522. (Pubitemid 44520286)
62. Norcross, J.C., & Wampold, B.E. (2011). What works for whom: Tailoring psychotherapy to the person. Journal of Clinical Psychology, 67, 127-132.
63. Orlinsky, D.E. (2009). The "generic model of psychotherapy" after 25 years: Evolution of a researchbased metatheory. Journal of Psychotherapy Integration, 19, 319-339.
64. Prochaska, J.O., & Norcross, J.C. (2002). Stages of change. In J.C. Norcorss (Ed.), Psychotherapy relationships that work: Therapist contributions and responsiveness to patients (pp. 37-70). New York, NY: Oxford University Press.
65. Rupert, P., & Baird, R. (2004). Managed care and the independent practice of psychology. Professional Psychology: Research and Practice, 35, 185-193. (Pubitemid 38491953)
66. Seligman, M.E. P. (1995). The effectiveness of psychotherapy. American Psychologist, 50, 965-974.
67. Seligman, M.E.P. (1996) Science as an ally of practice. American Psychologist, 51, 1072-1079. (Pubitemid 126418004)
68. Seligman, M.E.P., & Levant, R. (1998). Managed care policies rely on inadequate science. Professional Psychology: Research and Practice, 29, 211-212. (Pubitemid 128075302)
69. Serlin, R.C., Wampold, B.E., & Levin, J.R. (2003). Should providers of treatment be regarded as a random factor? If it ain't broke, don't "fix" it: A comment on Siemer and Joorman (2003). Psychological Methods, 8, 524-534. (Pubitemid 38061523)
70. Shea, M., Elkin, I., Imber, S., Sotsky, S., Watkins, J., Collins, J, et al. (1992). Course of depressive symptoms over follow-up: Findings from the National Institute of Mental Health Treatment of Depression Collaborative Research Program. Archives of General Psychiatry, 49, 782-787.
71. Shedler, J. (2010). The efficacy of psychodynamic psychotherapy. American Psychologist, 65, 98-109.
72. Sternberg, R.J. (2006). Evidence-based practice: Gold standard, gold plated, or fool's gold? In C.D. Goodheart, A.E. Kazdin, & R.J. Sternberg (Eds.), Evidence-based psychotherapy: Where practice and research meet (pp. 261-271). Washington, DC: American Psychological Association.
73. Stuart, G.L., Treat, T.A., & Wade, W.A. (2000). Effectiveness of an empirically based treatment for panic disorder delivered in a service clinic setting: 1-year follow-up. Journal of Consulting and Clinical Psychology, 68, 506-512.
74. Sue, S., Zane, N., Levant, R.R., Silverstein, L.B., Brown, L.S., & Olkin, R., et al. (2006). How well do both evidence-based practices and treatment as usual satisfactorily address the various dimensions of diversity? In J.C., Norcross, L.E. Beutler, & R.F. Levant (Eds.). Evidence-based practices in mental health: Debate and dialogue on the fundamental questions (pp. 329-337). Washington, D.C.: American Psychological Association.
75. Thompson-Brenner, H., & Westen, D. (2004a). Accumulating evidence for personality subtypes in eating disorders: Differences in comorbidity, adaptive functioning, treatment response, and treatment interventions in a naturalistic sample. Unpublished manuscript, Boston University, Boston.
76. Thompson-Brenner, H., & Westen, D. (2004b). A naturalistic study of psychotherapy for bulimia nervosa: Comorbidity, outcome, and therapeutic interventions in the community. Unpublished manuscript. Boston University, Boston.
77. Wachtel, P.L. (2010). Beyond "ESTs": Problematic assumptions in the pursuit of evidence-based practice. Psychoanalytic Psychology, 27, 251-272.
78. Wampold, B.E. (2001). The great psychotherapy debate: Models, methods, and findings. Mahwah, N.J: Lawrence Erlbaum Associates.
79. Wampold, B.E. (2007). Psychotherapy: The humanist (and effective) treatment. American Psychologist, 62, 857-873.
80. Wampold, B.E., & Serlin, R.C. (2000). The consequences of ignoring a nested factor on measures of effect size in analysis of variance. Psychological Methods, 5, 425-433.
81. Wampold, B.E., Minami, T., Baskin, T.W., & Callen, T., & Tierney, S. (2002). A meta- (re)analysis of the effects of cognitive therapy versus "other therapies" for depression. Journal of Affective Disorders, 68, 159-165. (Pubitemid 34628323)
82. Wampold, B.E., & Brown, G.S. (2005). Estimating therapist variability: A naturalistic study of outcomes in managed care. Journal of Consulting and Clinical Psychology, 73, 914-923. (Pubitemid 41643241)
83. Wampold, B.E., Hollon, S.D., & Hill, C.E. (2011). Unresolved questions and future directions in psychotherapy research. Chapter 11 in C.J. Gelso, (Ed.). The real relationship in psychotherapy: The hidden foundation of change (pp. 333-355).Washington, DC.: American Psychological Association.
84. Watson, D., & Clark, L.A. (1992). Affects separable and inseparable: On the hierarchical arrangement of the negative affects. Journal of Personality and Social Psychology, 62, 489-505.
85. Westen, D., & Shedler, J. (1999a). Revising and assessing Axis II, Part 1: Developing a clinically and empirically valid assessment method. American Journal of Psychiatry, 156, 258-272. (Pubitemid 29077118)
86. Westen, D., & Shedler, J. (1999b). Revising and assessing Axis II, Part2: Toward an empirically based and clinically useful classification of personality disorders. American Journal of Psychiatry, 156, 273-285. (Pubitemid 29077119)
87. Westen, D., & Harnden-Fischer, J. (2001). Personality profiles in eating disorders: Rethinking the distinction between Axis I and Axis II. American Journal of Psychiatry, 165, 547-562. (Pubitemid 32289817)
88. Westen, D., & Morrison, K. (2001). A multidimensional meta-analysis of treatments for depression, panic, and generalized anxiety disorder: An empirical examination of the status of empirically supported therapies. Journal of Consulting and Clinical Psychology, 69, 875-899. (Pubitemid 33140488)
89. Westen, D., Novotny, C.M., & Thompson-Brenner, H. (2004). The empirical status of empirically supported psychotherapies: Assumptions, findings, and reporting in controlled clinical trials. Psychological Bulletin, 130, 631-663. (Pubitemid 38882639)
90. Westen, D., & Shedler, J. (2007). Personality diagnosis with the Shedler-Westen Assessment Procedure (SWAP): Integrating clinical and statistical measurement and prediction. Journal of Abnormal Psychology, 116, 810-822.
91. Zimmerman, M, McDermut, W., & Mattia, J. (2000). Frequency of anxiety disorders in psychiatric outpatients with major depressive disorder. American Journal of Psychiatry, 157, 1337-1340. (Pubitemid 30659823)
92. Zinbarg, R.E., & Barlow, D.H. (1996). Structure of anxiety and the anxiety disorders: A hierarchical model. Journal of Abnormal Psychology, 105, 181-193. (Pubitemid 26134142)