Measuring Implementation of Evidence-Based Programs Targeting Young Children at Risk for Emotional/Behavioral Disorders: Conceptual Issues and Recommendations

Journal of Early Intervention

Volumn 35, Issue 2, 2013, Pages 129-149

  Sutherland, Kevin S ^a   McLeod, Bryce D ^a   Conroy, Maureen A ^b   Cox, Julia R ^a  

^a VIRGINIA COMMONWEALTH UNIVERSITY   (United States)

^b UNIVERSITY OF FLORIDA   (United States)

Author keywords

prevention; treatment implementation; treatment integrity

Indexed keywords

EID: 84891501195     PISSN: 10538151     EISSN: None     Source Type: Journal    
DOI: 10.1177/1053815113515025     Document Type: Article

References (76)

1. Aarons G. A.,Hurlburt M.,Horwitz S. M.Advancing a conceptual model of evidence-based practice implementation in public service sectors.Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:4-23.
2. Baker C. N.,Kupersmidt J. B.,Voegler-Lee M. E.,Arnold D.,Willoughby M. T.Predicting teacher participation in a classroom-based, integrated preventive intervention for preschoolers.Early Childhood Research Quarterly. 2010;25:270-283.
3. Barber J. P.,Sharpless B. A.,Klostermann S.,McCarthy K.Assessing intervention competence and its relation to therapy outcome: A selected review derived from the outcome literature.Professional Psychology: Science and Practice. 2007;38:493-500.
4. Barnett W. S.,Jung K.,Yarosz D. J.,Thomas J.,Hornbeck A.,Stechuk R.,Burns S.Educational effects of the Tools of the Mind curriculum: A randomized trial.Early Childhood Research Quarterly. 2008;23:233-313.
5. Becker K. D.,Domitrovich C. E.The conceptualization, integration, and support of evidence-based interventions in the schools.School Psychology Review. 2011;40:582-589.
6. Berlin L. J.,Brooks-Gunn J.,McCarton C.,McCormick M. C.The effectiveness of early intervention: Examining risk factors and pathways to enhance development.Preventive Medicine. 1998;27:238-245.
7. Beutler L. E.,Baker M.Empirically supported therapies. Dobson K. S.Craig K. D., ed. Thousand Oaks, CA: SAGE; 1998:43-65.
8. Bierman K. L.,Nix R. L.,Greenberg M. T.,Blair C.,Domitrovich C. E.Executive functions and school readiness intervention: Impact, moderation, and mediation in the Head Start REDI program.Development and Psychopathology. 2008;20:821-843.
9. BurchinalM.HowesC.KontosS. (2002). Structural predictors of child care quality in child care homes. Early Childhood Research Quarterly, 17, 87-105. doi:http://dx.doi.org/10.1016/S0885-2006(02)00132-1.
10. Carroll K. M.,Nich C.,Sifty R. L.,Nuro K. F.,Frankforter T. L.,Ball S. A.,Rounsaville B. J.A general system for evaluating therapist adherence and competence in psychotherapy research.Drug and Alcohol Dependence. 2000;57:225-238.
11. Carroll K. M.,Nuro K. F.One size cannot fit all: A stage model for psychotherapy manual development.Clinical Psychology: Science and Practice. 2002;9:396-406.
12. Chorpita B. F.,Daleiden E. L.Mapping evidence-based treatments for children and adolescents: Application of the distillation and matching model to 615 treatments from 322 randomized trials.Journal of Consulting and Clinical Psychology. 2009;77:566-579.
13. ChuB. C.ChoudhuryM. S.ShorttA. L.PincusD. B.CreedT. A.KendallP. C. (2004). Alliance, technology, and outcome in the treatment of anxious youth. Cognitive and Behavioral Practice, 11, 44-55. doi:http://dx.doi.org/10.1016/S1077-7229(04)80006-3.
14. Chu B. C.,Kendall P. C.Positive association of child involvement and treatment outcome within a manual-based cognitive-behavioral treatment for children with anxiety.Journal of Consulting and Clinical Psychology. 2004;72:821-829.
15. Conroy M. A.,Sutherland K. S.,Vo A. K.,Carr S. E.,Ogston P.Early childhood teachers' use of effective instructional practices and the collateral effects on young children's behavior.Journal of Positive Behavior Interventions. 2013;:.
16. Dane A. V.,Schneider B. H.Program integrity in primary and early secondary prevention: Are implementation effects out of control?.Clinical Psychology Review. 1998;18:23-45.
17. Domitrovich C. E.,Cortes R. C.,Greenberg M. T.Improving young children's social and emotional competence: A randomized trial of the preschool "PATHS" curriculum.The Journal of Primary Prevention. 2007;28:67-91.
18. Domitrovich C. E.,Gest S. D.,Jones D.,Gill S.,DeRousie R. M. S.Implementation quality: Lessons learned in the context of the Head Start REDI trial.Early Childhood Research Quarterly. 2010;25:284-298.
19. Domitrovich C. E.,Moore J. E.,Greenberg M. T.Handbook of implementation science for psychology in education. Kelly B.Perkins D. F., ed. Cambridge, UK: Cambridge University Press; 2012:207-229.
20. Donabedian A.The quality of care: How can it be assessed?.Journal of the American Medical Association. 1988;260:1743-1748.
21. Doss B. D.Changing the way we study change in psychotherapy.Clinical Psychology: Science and Practice. 2004;11:368-386.
22. Driscoll K. C.,Pianta R. C.Banking time in Head Start: Early efficacy of an intervention designed to promote supportive teacher-child relationships.Early Education and Development. 2010;21:38-64.
23. Durlak J. A.The importance of doing well in whatever you do: A commentary on the special section, "Implementation research in early childhood education.".Early Childhood Research Quarterly. 2010;25:348-357.
24. Eccles M. P.,Mittman B. S.Welcome to implementation science.Implementation Science. 2006;1:.
25. Embry D. D.,Biglan A.Evidence-based kernels: Fundamental units of behavioral influence.Clinical Child and Family Psychology Review. 2008;11:75-113.
26. Fixsen D. L.,Naoom S. F.,Blasé K. A.,Friedman R. M.,Wallace F.Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network; 2005:.
27. Garland A. F.,Bickman L.,Chorpita B. F.Change what? Identifying quality improvement targets by investigating usual mental health care.Administration and Policy in Mental Health. 2010;37:15-26.
28. Goldfried M. R.,Padawer W.Converging themes in psychotherapy: Trends in psychodynamic, humanistic, and behavioral practice. Goldfried M. R., ed. New York, NY: Springer; 1982:3-49.
29. Hamre B. K.,Justice L.,Pianta R. C.,Kilday C.,Sweeny B.,Downer J.,Leach A.Implementation fidelity of the My Teaching Partner literacy and language activities: Associations with preschoolers' language and literacy growth.Early Childhood Research Quarterly. 2010;25:329-347.
30. Hamre B. K.,Pianta R. C.,Burchinal M.,Field S.,LoCasale-Crouch J.,Downer J. T.,Scott-Little C.A course on effective teacher-child interactions: Effects on teacher beliefs, knowledge, and observed practice.American Educational Research Journal. 2012;49:88-123.
31. Han S. S.,Catron T.,Weiss B.,Marciel K. K.A teacher-consultation approach to social skills training for pre-kindergarten children: Treatment model and short-term outcome effects.Journal of Abnormal Child Psychology. 2005;33:681-693.
32. Han S. S.,Weiss B.Sustainability of teacher implementation of school-based mental health programs.Journal of Abnormal Child Psychology. 2005;33:665-679.
33. Harn B.,Parisi D.,Stoolmiller M.Balancing fidelity with flexibility and fit: What do we really know about fidelity of implementation in schools?.Exceptional Children. 2013;79:181-193.
34. Hill C. E.,O'Grady K. E.,Elkin I.Applying the Collaborative Study Psychotherapy Rating Scale to rate therapist adherence in cognitive-behavior therapy, interpersonal therapy, and clinical management.Journal of Consulting and Clinical Psychology. 1992;60:73-79.
35. Hogue A.,Dauber S.,Chinchilla P.,Fried A.,Henderson C.,Inclan J.,Liddle H. A.Assessing fidelity in individual and family therapy for adolescent substance abuse.Journal of Substance Abuse Treatment. 2008;35:137-147.
36. Hogue A.,Dauber S.,Henderson C. E.Therapist self-report of evidence-based practices in usual care for adolescent problems: Factor and construct validity.Administration and Policy in Mental Health and Mental Health Services Research. 2013;:.
37. Hogue A.,Henderson C. E.,Dauber S.,Barajas P. C.,Fried A.,Liddle H. A.Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems.Journal of Consulting and Clinical Psychology. 2008;76:544-555.
38. Hogue A.,Liddle H. A.,Rowe C.Treatment adherence process research in family therapy: A rationale and some practical guidelines.Psychotherapy: Theory, Research, Practice, Training. 1996;33:332-345.
39. JonesH. A.ClarkeA. T.PowerT. J. (2008, Spring). Expanding the concept of intervention integrity: A multidimensional model of participant engagement. In Balance: Society of Clinical Child and Adolescent Psychology Newsletter, 23(1), 4-5. Retrieved from http://enewsletter.clinicalchildpsychology.org/InBalance/sites/default/files/Spring%202008.pdf.
40. Joyce B.,Showers B.Student achievement through staff development. 2002). Student achievement through staff development (3rd ed.). Alexandria, VA: Association for Supervision and Curriculum Development.Alexandria, VA: Association for Supervision and Curriculum Development; 2002:.
41. Karver M. S.,Handelsman J. B.,Fields S.,Bickman L.Meta-analysis of therapeutic relationship variables in youth and family therapy: The evidence for different relationship variables in the child and adolescent treatment outcome literature.Clinical Psychology Review. 2006;26:50-65.
42. Knox L. M.,Aspy C. B.Quality improvement as a tool for translating evidence based interventions into practice: What the youth violence prevention community can learn from healthcare.American Journal of Community Psychology. 2011;48:56-64.
43. Kratochwill T. R.,Hoagwood K. E.,Kazak A. E.,Weisz J. R.,Hood K.,Vargas L. A.,Banez G. A.Practice-based evidence for children and adolescents: Advancing the research agenda in schools.School Psychology Review. 2012;41:215-235.
44. McLeod B. D.The relation of the alliance with outcomes in youth psychotherapy: A meta-analysis.Clinical Psychology Review. 2011;31:603-616.
45. McLeod B. D.,Islam N. Y.,Wheat E.The Oxford handbook of research strategies for clinical psychology. Comer J.Kendall P., ed. New York, NY: Oxford University Press; 2013:142-164.
46. McLeod B. D.,Southam-Gerow M. A.,Tully C. B.,Rodriguez A.,Smith M. M.Making a case for treatment integrity as a psychological treatment quality indicator.Clinical Psychology: Science and Practice. 2013;20:14-32.
47. McLeod B. D.,Southam-Gerow M. A.,Weisz J. R.Conceptual and methodological issues in treatment integrity measurement.School Psychology Review. 2009;38:541-546.
48. McLeod B. D.,Weisz J. R.The Therapy Process Observational Coding System for Child Psychotherapy Strategies Scale.Journal of Clinical Child and Adolescent Psychology. 2010;39:436-443.
49. Mendel P.,Meredith L. S.,Schoenbaum M.,Sherbourne C. D.,Wells K. B.Interventions in organizational and community context: A framework for building evidence on dissemination and implementation in health services research.Administration and Policy in Mental Health and Mental Health Services Research. 2008;35:21-37.
50. Nelson J. R.,Stage S.,Duppong-Hurley D.,Synhorst L.,Epstein M. H.Risk factors predictive of the problem behavior of children at risk for emotional and behavioral disorders.Exceptional Children. 2007;73:367-379.
51. Odom S. L.,Fleming K.,Diamond K.,Lieber J.,Hanson M.,Butera G.,Marquis J.Examining different forms of implementation and in early childhood curriculum research.Early Child Research Quarterly. 2010;25:314-328.
52. Perepletchikova F.,Kazdin A. E.Treatment integrity and therapeutic change: Issues and research recommendations.Clinical Psychology Science and Practice. 2005;12:365-383.
53. Pianta R. C.,Rimm-Kaufman S.Blackwell handbooks of developmental psychology. McCartney K.Phillips D., ed. Malden, MA: Blackwell Publishing; 2006:490-507.
54. Pianta R. C.,Stuhlman M. W.Conceptualizing risk in relational terms: Associations among the quality of child-adult relationships prior to school entry and children's developmental outcomes in first grade.Educational and Child Psychology. 2004;21:32-45.
55. Qi C. H.,Kaiser A. P.Behavior problems of preschool children from low-income families: Review of the literature.Topics in Early Childhood Special Education. 2003;23:188-216.
56. Quesenberry A.,Hemmeter M. L.,Ostrosky M.Addressing challenging behavior in Head Start: A closer look at program policies and procedures.Topics in Early Childhood Special Education. 2011;30:209-220.
57. Raver C. C.,Jones S. M.,Li-Grining C.,Zhai F.,Metzger M. W.,Solomon B.Targeting children's behavior problems in preschool classrooms: A cluster-randomized controlled trial.Journal of Consulting and Clinical Psychology. 2009;77:302-316.
58. Sanetti L. M.,DiGennaro Reed F. D.Barriers to implementing treatment integrity procedures in school psychology research: Survey of treatment outcome researchers.Assessment for Effective Intervention. 2012;38:1-8.
59. Sanetti L. M.,Dobey L. M.,Gritter K. L.Treatment integrity of interventions with children in the Journal of Positive Behavior Interventions from 1999 to 2009.Journal of Positive Behavior Interventions. 2012;14:29-46.
60. Sanetti L. M.,Gritter K. L.,Dobey L. M.Treatment integrity of interventions with children in the school psychology literature from 1995 to 2008.School Psychology Review. 2011;40:72-84.
61. Sanetti L. M.,Kratochwill T. R.Toward developing a science of treatment integrity: Introduction to the special series.School Psychology Review. 2009;38:445-459.
62. Schoenwald S. K.,Garland A. F.,Chapman J. E.,Frazier S. L.,Sheidow A. J.,Southam-Gerow M. A.Toward the effective and efficient measurement of implementation fidelity.Administration and Policy in Mental Health and Mental Health Services Research. 2011;38:32-43.
63. Schoenwald S. K.,Hoagwood K.Effectiveness, transportability, and dissemination of interventions: What matters when?.Psychiatric Services. 2001;52:1190-1197.
64. Southam-Gerow M. A.,McLeod B. D.Advances in applying treatment integrity research for dissemination and implementation science: Introduction to special issue.Clinical Psychology: Science and Practice. 2013;20:1-13.
65. Southam-Gerow M. A.,Weisz J. R.,Chu B. C.,McLeod B. D.,Gordis E. B.,Connor-Smith J. K.Does cognitive behavioral therapy for youth anxiety outperform usual care in community clinics? An initial effectiveness test.Journal of the American Academy of Child and Adolescent Psychiatry. 2010;49:1043-1052.
66. Sutherland K. S.,Lewis-Palmer T.,Stichter J.,Morgan P.Examining the influence of teacher behavior and classroom context on the behavioral and academic outcomes for students with emotional or behavioral disorders.Journal of Special Education. 2008;41:223-233.
67. Sutherland K. S.,McLeod B. D.,Conroy M. A.,Abrams L.,Smith M. M.Preliminary psychometric properties of the BEST in CLASS Adherence and Competence Scale.Journal of Emotional and Behavioral Disorders. 2013;:.
68. Swanson E.,Wanzek J.,Haring C.,Ciullo S.,McCulley L.Intervention fidelity in special and general education research journals.The Journal of Special Education. 2011;45:1-11.
69. Vo A. K.,Sutherland K. S.,Conroy M. A.Best in class: A classroom-based model for ameliorating problem behavior in early childhood settings.Psychology in the Schools. 2012;49:402-415.
70. Waltz J.,Addis M. E.,Koerner K.,Jacobson N. E.Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence.Journal of Consulting and Clinical Psychology. 1993;61:620-630.
71. Webster-Stratton C.,Reid M. J.,Hammond M.Preventing conduct problems, promoting social competence: A parent and teacher training partnership in Head Start.Journal of Clinical Child Psychology. 2001;30:283-302.
72. Webster-Stratton C.,Reid M. J.,Hammond M.Treating children with early-conduct problems: Intervention outcomes for parent, child, and teacher training.Journal of Clinical Child and Adolescent Psychology. 2004;33:105-124.
73. Webster-Stratton C.,Reid M. J.,Stoolmiller M.Preventing conduct problems and improving school readiness: Evaluation of the Incredible Years Teacher and Child Training Programs in high-risk schools.Journal of Child Psychology and Psychiatry. 2008;49:471-488.
74. WeilandC.UlvestadK.SachsJ.YoshikawaH. (2013). Associations between classroom quality and children's vocabulary and executive function skills in an urban public prekindergarten program. Early Childhood Research Quarterly, 28, 199-209. doi:http://dx.doi.org.proxy.library.vcu.edu/10.1016/j.ecresq.2012.12.002.
75. Wolery M.Intervention research: The importance of fidelity measurement.Topics in Early Childhood Education. 2011;31:155-157.
76. Yoder P.,Symons F.Observational measurement of behavior. New York, NY: Springer; 2010:.