Validity of Therapist Self-Report Ratings of Fidelity to Evidence-Based Practices for Adolescent Behavior Problems: Correspondence between Therapists and Observers

Administration and Policy in Mental Health and Mental Health Services Research

Volumn 42, Issue 2, 2015, Pages 229-243

  Hogue, Aaron ^a   Dauber, Sarah ^a   Lichvar, Emily ^a   Bobek, Molly ^a   Henderson, Craig E ^b  

^a Yale School of Medicine   (United States)

^b SAM HOUSTON STATE UNIVERSITY   (United States)

Author keywords

Adolescent behavior problems; Cognitive behavioral therapy; Family therapy; Motivational interviewing; Observational fidelity ratings; Therapist report fidelity; Usual care

Indexed keywords

ADOLESCENT; ANXIETY DISORDERS; ATTENTION DEFICIT AND DISRUPTIVE BEHAVIOR DISORDERS; ATTENTION DEFICIT DISORDER WITH HYPERACTIVITY; CLINICAL PROTOCOL; COGNITIVE THERAPY; CONDUCT DISORDER; CONTROLLED STUDY; EVIDENCE BASED PRACTICE; FAMILY THERAPY; FEMALE; HUMAN; MALE; MOTIVATIONAL INTERVIEWING; PROTOCOL COMPLIANCE; PSYCHOTHERAPY; RANDOMIZED CONTROLLED TRIAL; REPRODUCIBILITY; SELF REPORT; STANDARDS; SUBSTANCE-RELATED DISORDERS;

ADOLESCENT; ANXIETY DISORDERS; ATTENTION DEFICIT AND DISRUPTIVE BEHAVIOR DISORDERS; ATTENTION DEFICIT DISORDER WITH HYPERACTIVITY; CLINICAL PROTOCOLS; COGNITIVE THERAPY; CONDUCT DISORDER; EVIDENCE-BASED PRACTICE; FAMILY THERAPY; FEMALE; GUIDELINE ADHERENCE; HUMANS; MALE; MOTIVATIONAL INTERVIEWING; PSYCHOTHERAPY; REPRODUCIBILITY OF RESULTS; SELF REPORT; SUBSTANCE-RELATED DISORDERS;

EID: 84897362441     PISSN: 0894587X     EISSN: 15733289     Source Type: Journal    
DOI: 10.1007/s10488-014-0548-2     Document Type: Article

References (65)

1. Aarons, G. A., Hurlburt, M., & Horowitz, S. (2011). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research,38, 4–23.
2. American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (4th ed., text revision). Washington, DC: APA.
3. Baldwin, S. A., Christian, S., Berkeljon, A., Shadish, W., & Bean, R. (2012). The effects of family therapies for adolescent delinquency and substance abuse: A meta-analysis. Journal of Marital and Family Therapy,38, 281–304.
4. Ball, S. A., Martino, S., Nich, C., Frankforter, T. L., Van Horn, D., Crits-Christoph, P., et al. (2007). Site matters: Multisite randomized trial of motivational enhancement therapy in community drug abuse clinics. Journal of Consulting and Clinical Psychology,75, 556–567.
5. Barth, R. P., Kolivoski, K. M., Lindsey, M. A., Lee, B. R., & Collins, K. S. (2014). Translating the common elements approach: Social work’s experiences in education, practice, and research. Journal of Clinical Child and Adolescent Psychology,43(2), 301–311.
6. Bearman, S. K., Weisz, J. R., Chorpita, B. F., Hoagwood, K., Ward, A., Ugueto, A. M., et al. (2013). More practice, less preach? The role of supervision processes and therapist characteristics in EBP implementation. Administration and Policy in Mental Health and Mental Health Services Research,40, 518–529.
7. Bearsley-Smith, C., Sellick, K., Chesters, J., & Francis, K. (2008). Treatment content in child and adolescent mental health services: Development of the Treatment Recording Sheet. Administration and Policy in Mental Health and Mental Health Services Research,35, 423–435.
8. Becker, S., & Curry, J. (2008). Outpatient interventions for adolescent substance abuse: A quality of evidence review. Journal of Consulting and Clinical Psychology,76, 531–543.
9. Brosan, L., Reynolds, S., & Moore, R. G. (2008). Self-evaluation of cognitive therapy performance: Do therapists know how competent they are? Behavioural and Cognitive Psychotherapy,36, 581–587.
10. Carroll, K. M., Ball, S. A., Nich, C., Martino, S., Frankforter, T. L., Farentinos, C., et al. (2006). Motivational interviewing to improve treatment engagement and outcome in individuals seeking treatment for substance abuse: A multisite effectiveness study. Drug and Alcohol Dependence,81, 301–312.
11. Carroll, K. M., Nich, C., & Rounsaville, B. J. (1998). Utility of therapist session checklists to monitor delivery of coping skills treatment for cocaine abusers. Psychotherapy Research,8, 307–320.
12. Carroll, K. M., Nich, C., Sifry, R., Nuro, K. F., Frankforter, T. L., Ball, S. A., et al. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug & Alcohol Dependence,57, 225–238.
13. Chapman, J. E., McCart, M. R., Letourneau, E. J., & Sheidow, A. J. (2013). Comparison of youth, caregiver, therapist, trained, and treatment expert raters of therapist adherence to a substance abuse treatment protocol. Journal of Consulting and Clinical Psychology,81, 674–680.
14. Chorpita, B. F. (2007). Modular cognitive-behavioral therapy for childhood anxiety disorders. New York: Guilford.
15. Chorpita, B. F., Becker, K. D., & Daleiden, E. L. (2007). Understanding the common elements of evidence-based practice: Misconceptions and clinical examples. Journal of the American Academy of Child and Adolescent Psychiatry,46, 647–652.
16. Chorpita, B., Daleiden, E., Ebesutani, C., Young, J., Becker, K., Nakamura, B., et al. (2011). Evidence-based treatments for children and adolescents: An updated review of indicators of efficacy and effectiveness. Clinical Psychology: Science and Practice,18, 154–172.
17. Chorpita, B. F., Daleiden, E. L., & Weisz, J. R. (2005). Modularity in the design and application of therapeutic interventions. Applied & Preventive Psychology,11, 141–156.
18. Chorpita, B. F., Weisz, J. R., Daleiden, E. L., Schoenwald, S. K., Palinkas, L. A., Miranda, J., et al. (2013). Long-term outcomes for the Child STEPs randomized effectiveness trial: A comparison of modular and standard treatment designs with usual care. Journal of Consulting and Clinical Psychology,81, 999–1009.
19. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment,6, 284–290.
20. Cook, J. M., Biyanova, T., Elhai, J., Schnurr, P. P., & Coyne, J. C. (2010). What do psychotherapists really do in practice? An internet study of over 2,000 practitioners. Psychotherapy: Theory Research, Practice, Training,47, 260–267.
21. Deming, W. E. (1986). Out of the crisis. Cambridge, MA: MIT Press.
22. Dennis, M. L., Godley, S. H., Diamond, G., Tims, F. M., Babor, T., Donaldson, J., et al. (2004). The Cannabis Youth Treatment (CYT) study: Main findings from two randomized trials. Journal of Substance Abuse Treatment,27, 197–213.
23. Diamond, G. M., Diamond, G. S., & Hogue, A. (2007). Attachment-based family therapy: Adherence and differentiation. Journal of Marital and Family Therapy,33, 177–191.
24. Embry, D. D., & Biglan, A. (2008). Evidence-based kernels: Fundamental units of behavioral influence. Clinical Child and Family Psychology Review,11, 75–113.
25. Fixsen, D. L., Naoom, S. F., Blasé, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).
26. Fleiss, J. L. (1981). Balanced incomplete block designs for inter-rater reliability studies. Applied Psychological Measurement,5, 105–112.
27. Gallo, K. P., & Barlow, D. H. (2012). Factors involved in clinician adoption and nonadoption of evidence-based interventions in mental health. Clinical Psychology: Science and Practice,19, 93–106.
28. Garland, A. F., Bickman, L., & Chorpita, B. F. (2010a). Change what? Identifying quality improvement targets by investigating usual mental health care. Administration and Policy in Mental Health and Mental Health Services Research,37, 15–26.
29. Garland, A. F., Haine-Schlagel, R., Brookman-Frazee, L., Baker-Ericzen, M., Trask, E., & Fawley-King, K. (2013). Improving community-based mental health care for children: Translating knowledge into action. Administration and Policy in Mental Health and Mental Health Services Research,40, 6–22.
30. Garland, A. F., Hawley, K. M., Brookman-Frazee, L. I., & Hurlburt, M. (2008). Identifying common, core elements of evidence-based practice for children with disruptive behavior disorder. Journal of the American Academy of Child and Adolescent Psychiatry,47, 505–514.
31. Garland, A. F., Hurlburt, M. S., Brookman-Frazee, L., Taylor, R. M., & Accurso, E. C. (2010b). Methodological challenges of characterizing usual care psychotherapeutic practice. Administration and Policy in Mental Health and Mental Health Services Research,37, 208–220.
32. Gifford, E. V., Tavakoli, S., Weingardt, K. R., Finney, J. W., Pierson, H. M., et al. (2012). How do components of evidence-based psychological treatment cluster in practice? A survey and cluster analysis. Journal of Substance Abuse Treatment,42, 45–55.
33. Hawkins, E. H. (2009). A tale of two systems: Co-occurring mental health and substance abuse disorders treatment for adolescents. Annual Review of Psychology,60, 197–227.
34. Hepner, K. A., Azocar, F., Greenwood, G. L., Miranda, J., & Burnam, M. A. (2010). Development of a clinician report measure to assess psychotherapy for depression in usual care settings. Administration and Policy in Mental Health and Mental Health Services Research,37, 221–229.
35. Hoagwood, K., Atkins, M., & Ialongo, N. (2013). Unpacking the black box of implementation: The next generation of policy, research and practice. Administration and Policy in Mental Health and Mental Health Services Research,40, 451–455.
36. Hogue, A., & Dauber, S. (2013). Assessing fidelity to evidence-based practices in usual care: The example of family therapy for adolescent behavior problems. Evaluation and Program Planning,37, 21–30.
37. Hogue, A., Dauber, S., Chinchilla, P., Fried, A., Henderson, C. E., Inclan, J., et al. (2008a). Assessing fidelity in individual and family therapy for adolescent substance abuse. Journal of Substance Abuse Treatment,35, 137–147.
38. Hogue, A., Dauber, S., & Henderson, C. E. (2012). Therapist self-report of evidence-based practices in usual care for adolescent behavior problems: Factor and construct validity. Administration and Policy in Mental Health and Mental Health Services Research,41(1), 126–139. doi:10.1007/s10488-012-0442-8.
39. Hogue, A., Dauber, S., Samuolis, J., & Liddle, H. A. (2006). Treatment techniques and outcomes in multidimensional family therapy for adolescent behavior problems. Journal of Family Psychology,20, 535–543.
40. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P., Fried, A., & Liddle, H. A. (2008b). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting and Clinical Psychology,76, 544–555.
41. Hogue, A., & Liddle, H. A. (2009). Family-based treatment for adolescent substance abuse: Controlled trials and new horizons in services research. Journal of Family Therapy,31, 126–154.
42. Hogue, A., Liddle, H., Rowe, C., Turner, R., Dakof, G. A., & LaPann, K. (1998). Treatment adherence and differentiation in individual versus family therapy for adolescent drug abuse. Journal of Counseling Psychology,45, 104–114.
43. Hogue, A., Ozechowski, T. J., Robbins, M. R., & Waldron, H. B. (2013). Making fidelity an intramural game: Localizing quality assurance procedures to promote sustainability of evidence-based practices in usual care. Clinical Psychology: Science and Practice,20, 60–77.
44. Hurlburt, M. S., Garland, A. F., Nguyen, K., & Brookman-Frazee, L. (2010). Child and family therapy process: Concordance of therapist and observational perspectives. Administration and Policy in Mental Health and Mental Health Services Research,37, 230–244.
45. Kazak, A., Hoagwood, K., Weisz, J. R., Hood, K., Kratochwill, T., Vargas, L., et al. (2010). A meta-systems approach for evidence-based practice for children and adolescents. American Psychologist,65, 85–97.
46. Korathu-Larson, P., Park, A., Fierstein, M., & Chorpita, B. F. (2011). Therapist satisfaction with multiple designs of evidence-based treatments: Did therapists match with MATCH?. Toronto, Canada: The Association of Behavioral and Cognitive Therapies (ABCT).
47. Martino, S., Ball, S., Nich, C., Frankforter, T. L., & Carroll, K. M. (2009). Correspondence of motivational enhancement treatment integrity ratings among therapists, supervisors, and observers. Psychotherapy Research,19, 181–193.
48. McHugh, R., & Barlow, D. H. (2010). The dissemination and implementation of evidence-based psychological treatment: A review of current efforts. American Psychologist,65, 73–84.
49. McLeod, B. D., Southam-Gerow, M. A., Tully, C. B., Rodriguez, A., & Smith, M. M. (2013). Making a case for treatment integrity as a psychosocial treatment quality indicator for youth mental health care. Clinical Psychology: Science and Practice,20, 14–32.
50. Miller, W. R., Yahne, C. E., Moyers, T. B., Martinez, J., & Pirritano, M. (2004). A randomized trial of methods to help clinicians learn motivational interviewing. Journal of Consulting and Clinical Psychology,72, 1050–1062.
51. O’Neill, S. M., Hempel, S., Lim, Y. W., Danz, M. S., Foy, R., Suttorp, M. J., et al. (2011). Identifying continuous quality improvement publications: What makes an improvement intervention ‘CQI’? British Medical Journal Quality & Safety,20, 1011–1019.
52. Perepletchikova, F. (2011). On the topic of treatment integrity. Clinical Psychology: Science and Practice,18, 148–153.
53. Regan, J., Daleiden, E. L., & Chorpita, B. F. (2013). Integrity in mental health systems: An expanded framework for managing uncertainty in clinical care. Clinical Psychology: Science and Practice,20, 78–98.
54. Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s…fidelity measurement in the real world. Clinical Psychology: Science and Practice,18, 142–147.
55. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research,38, 32–43.
56. Schoenwald, S. K., Letourneau, E. J., & Halliday-Boykins, C. (2005). Predicting therapist adherence to a transported family-based treatment for youth. Journal of Clinical Child and Adolescent Psychology,34(4), 658–670.
57. Schoenwald, S. K., Sheidow, A. J., & Chapman, J. E. (2009). Clinical supervision in treatment transport: Effects on adherence and outcomes. Journal of Consulting and Clinical Psychology,77, 410–421.
58. Sheehan, D. V., Lecrubier, Y., Sheehan, K. H., Amorim, P., Janavs, J., Weiller, E., et al. (1998). The Mini International Neuropsychiatric Interview (MINI): The development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. Journal of Clinical Psychiatry,59, 22–33.
59. Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin,86, 420–428.
60. Thorndike, R. (2007). Fisher’s Z transformation. In N. Salkind (Ed.), Encyclopedia of measurement and statistics (pp. 361–365). Thousand Oaks, CA: Sage.
61. Ward, A. M., Regan, J., Chorpita, B. F., Starace, N., Rodriguez, A., Okamura, K., et al. (2013). Tracking evidence based practice with youth: Validity of MATCH and standard manual consultation records. Journal of Clinical Child and Adolescent Psychology,42, 44–55.
62. Weersing, R. V., Weisz, J. R., & Donenberg, G. R. (2002). Development of the Therapy Procedures Checklist: A therapist-report measure of technique use in child and adolescent treatment. Journal of Clinical Child Psychology,31, 168–180.
63. Weisz, J. R., Chorpita, B. F., Palinkas, L. A., Schoenwald, S. K., Miranda, J., Bearman, S. K., et al. (2012). Testing standard and modular designs for psychotherapy treating depression, anxiety, and conduct problems in youth: A randomized effectiveness trial. Archives of General Psychiatry,69, 274–282.
64. Westen, D., Novotny, C. M., & Thompson-Brenner, H. (2005). EBP does not equal EST: Reply to Crits-Christoph et al. (2005) and Weisz et al. (2005). Psychological Bulletin,131, 427–433.
65. Winters, K. C., Stinchfield, R., Latimer, W. W., & Lee, S. (2007). Long-term outcome of substance-dependent youth following 12-step treatment. Journal of Substance Abuse Treatment,33, 61–69.