Observational Measures of Implementer Fidelity for a School-Based Preventive Intervention: Development, Reliability, and Validity

Prevention Science

Volumn 16, Issue 1, 2015, Pages 122-132

  Cross, Wendi ^a   West, Jennifer ^a   Wyman, Peter A ^a   Schmeelk Cone, Karen ^a   Xia, Yinglin ^a   Tu, Xin ^a   Teisl, Michael ^a   Brown, C Hendricks ^b   Forgatch, Marion ^c  

^a UNIVERSITY OF ROCHESTER SCHOOL OF MEDICINE AND DENTISTRY   (United States)

^b NORTHWESTERN UNIVERSITY   (United States)

^c OREGON SOCIAL LEARNING CENTER   (United States)

Author keywords

Adherence; Competence; Implementer fidelity; Measurement; Observation

Indexed keywords

ADULT; CHILD; CHILD BEHAVIOR DISORDERS; CONTROLLED STUDY; FEMALE; HUMAN; MALE; MIDDLE AGED; ORGANIZATION AND MANAGEMENT; PROGRAM DEVELOPMENT; PROGRAM EVALUATION; RANDOMIZED CONTROLLED TRIAL; REPRODUCIBILITY; SCHOOL HEALTH SERVICE; VALIDATION STUDY; VIDEORECORDING;

ADULT; CHILD; CHILD BEHAVIOR DISORDERS; FEMALE; HUMANS; MALE; MIDDLE AGED; PROGRAM DEVELOPMENT; PROGRAM EVALUATION; REPRODUCIBILITY OF RESULTS; SCHOOL HEALTH SERVICES; VIDEOTAPE RECORDING;

EID: 84939886926     PISSN: 13894986     EISSN: 15736695     Source Type: Journal    
DOI: 10.1007/s11121-014-0488-9     Document Type: Article

References (50)

1. Aarons, G. A., Hurlburt, M., & Horwitz, S. M. (2010). Advancing a conceptual model of evidence-based practice implementation in public service sectors. Administration and Policy in Mental Health and Mental Health Services Research, 38, 4–23. doi:10.1007/s10488-010-0327-7.
2. Aarons, G. A., Green, A. E., Palinkas, L. A., Self-Brown, S., Whitaker, D. J., Lutzker, J. R., Silovsky, J. F., Hecht, D. B., & Chaffin, M. J. (2012). Dynamic adaptation process to implement an evidence-based child maltreatment intervention. Implementation Science, 7, 32. doi:10.1186/1748-5908-7-32.
3. American Psychological Association (1994). Diagnostic and statistical manual of mental disorders (5th ed.) Washington, DC: Author.
4. Breitenstein, S. M., Fogg, L., Garvey, C., Hill, C., Resnick, B., & Gross, D. (2010). Measuring implementation fidelity in a community-based parenting intervention. Nursing Research, 59, 158–165. doi:10.1097/NNR.0b013e3181dbb2e2.
5. Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural models. Newbury Park: Sage.
6. Brownson, R. C., Colditz, G. A., & Proctor, E. K. (2012). Dissemination and implementation research in health: Translating science to practice. New York: Oxford University.
7. Byrnes, H. F., Miller, B. A., Aalborg, A. E., Plasencia, A. V., & Keagy, C. D. (2010). Implementation fidelity in adolescent family-based prevention programs: Relationship to family engagement. Health Education Research, 25, 531–541. doi:10.1093/her/cyq006.
8. Carroll, K. M., Nich, C., Sifry, R. L., Nuro, K. F., Frankforter, T. L., Ball, S. A., Fenton, L., & Rounsaville, B. J. (2000). A general system for evaluating therapist adherence and competence in psychotherapy research in the addictions. Drug and Alcohol Dependence, 57, 225–238. doi:10.1016/S0376-8716(99)00049-6.
9. Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychological Assessment, 6, 284–290. doi:10.1037/1040-3590.6.4.284.
10. Cross, W., & West, J. (2011). Examining implementer fidelity: Conceptualising and measuring adherence and competence. Journal of Children’s Services, 6, 18–33. doi:10.5042/jcs.2011.0123.
11. Cross, W. & Wyman, P. A. (2004). Promoting resilient children initiative (PRCI): Mentor’s manual. Unpublished manuscript. University of Rochester: Authors.
12. Cross, W., Pisani, A, Schmeelk-Cone, K., Xia, Y., Tu, X., McMahon, M., Funkfah, J., & Gould. (2014). Fidelity assessment of the transfer of training in a suicide prevention program for crisis hotlines. Crisis: The Journal of Crisis Intervention and Suicide Prevention. In press.
13. Curran, G. M., Mukherjee, S., Allee, E., & Owen, R. R. (2008). A process for developing an implementation intervention: QUERI Series. Implementation Science, 3, 17. doi:10.1186/1748-5908-3-17.
14. Damschroder, L. J., Aron, D. C., Keith, R. E., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation. Implementation Science, 4, 50. doi:10.1186/1748-5908-4-50.
15. Dumas, J. E., Lynch, A. M., Laughlin, J. E., Smith, E. P., & Prinz, R. J. (2001). Promoting intervention fidelity: Conceptual issues, methods, and preliminary results from the EARLY ALLIANCE prevention trial. American Journal of Preventative Medicine, 20, 38–47. doi:10.1016/S0749-3797(00)00272-5.
16. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. doi:10.1007/s10464-008-9165-0.
17. Fixsen, D. L., Naoom, S. F., Blase, K. A., Freidman, R. M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network.
18. Floyd, F., & Widaman, K. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286–299. doi:10.1037/1040-3590.7.3.286.
19. Forgatch, M. S., & DeGarmo, D. S. (2011). Sustaining fidelity following the nationwide PMTO implementation in Norway. Prevention Science, 12, 235–246. doi:10.1007/s11121-011-0225-6.
20. Forgatch, M. S., DeGarmo, D. S., & Beldavs, Z. G. (2005). An efficacious theory-based intervention for stepfamilies. Behavior Therapy, 36, 357–365. doi:10.1016/S0005-7894(05)80117-0.
21. Glasgow, R. E., Vogt, T. M., & Boles, S. M. (1999). Evaluating the public health impact of health promotion interventions: The RE-AIM framework. American Journal of Public Health, 89, 1322–1327. doi:10.2105/AJPH.89.9.1322.
22. Glasgow, R. E., Vinson, C., Chambers, D., Khoury, M. J., Kaplan, R. M., & Hunter, C. (2012). National Institutes of Health approaches to dissemination and implementation science: Current and future directions. American Journal of Public Health, 102, 1274–1281. doi:10.2105/AJPH.2012.300755.
23. Gould, M. S., Cross, W., Pisani, A. R., Munfakh, J. L., & Kleinman, M. (2013). Impact of Applied Suicide Intervention Skills Training (ASIST) on national suicide prevention Lifeline counselor interventions and suicidal caller outcomes. Suicide and Life Threatening Behaviors, 43, 676–691. doi:10.1111/sltb.12049.
24. Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., & Leach, A. (2010). Implementation fidelity of MyTeachingPartner literacy and language activities: Association with preschoolers’ language and literacy growth. Early Childhood Research Quarterly, 25, 329–347. doi:10.1016/j.ecresq.2009.07.002.
25. Hepner, K. A., Hunter, S. B., Paddock, S. M., Zhou, A. J., & Watkins, K. E. (2011). Training addiction counselors to implement CBT for depression. Administration and Policy in Mental Health, 38, 313–323. doi:10.1007/s10488-011-0359-7.
26. Hogue, A., Henderson, C. E., Dauber, S., Barajas, P. C., Fried, A., & Liddle, H. A. (2008). Treatment adherence, competence, and outcome in individual and family therapy for adolescent behavior problems. Journal of Consulting & Clinical Psychology, 76, 544–555. doi:10.1037/0022-006X.76.4.544.
27. Hu, L., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariate structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 6, 1–55. doi:10.1080/10705519909540118.
28. James, I. A., Blackburn, I. M., Milne, D. L., & Reichfelt, F. K. (2001). Moderators of trainee therapists’ competence in cognitive therapy. British Journal of Clinical Psychology 40, 131–141.
29. Kellam, S. G., Ling, X., Merisca, R., Brown, C. H., & Ialongo, N. (1998). The effect of the level of aggression in the first grade classroom on the course and malleability of aggressive behavior into middle school. Development and Psychopathology, 10, 165–186. doi:10.1017/S0954579498001564.
30. Liber, J. M., van Widenfelt, B. M., van der Leeden, A. J., Goedhart, A. W., Utens, E. M., & Treffers, P. D. (2010). The relation of severity and comorbidity to treatment outcome with Cognitive Behavioral Therapy for childhood anxiety disorders. Journal of Abnormal Child Psychology, 38, 683–694. doi:10.1007/s10802-010-9394-1.
31. Lillehoj, C. J., Griffin, K. W., & Spoth, R. (2004). Program provider and observer ratings of school-based preventive intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242–257. doi:10.1177/1090198103260514.
32. Moore, J. E., Beck, T. C., Sylvertsen, A., & Domitrovich, C. (2009). Making sense of implementation: Multiple dimensions, multiple sources, multiple methods. Washington, DC: Paper presented to the 17th Annual Meeting of the Society for Prevention Research.
33. Muthén, L. K., & Muthén, B. O. (1998–2010). Mplus Version 6. Los Angeles: Muthén & Muthén.
34. Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011). Outcomes for implementation research: Conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health, 38, 65–76. doi:10.1007/s10488-010-0319-7.
35. SAS. (2008). SAS, Version 9.2. Cary: SAS Institute Inc.
36. Schoenwald, S. K. (2011). It’s a bird, it’s a plane, it’s fidelity measurement in the real world. Clinical Psychology: Science and Practice, 18, 142–147. doi:10.1111%2Fj.1468-2850.2011.01245.x.
37. Schoenwald, S. K., & Garland, A. F. (2013). A review of treatment adherence measurement methods. Psychological Assessment, 25, 146–156. doi:10.1037/a0029715.
38. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. J., & Southam-Gerow, M. A. (2011). Toward effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. doi:10.1007/s10488-010-0321-0.
39. Smith, G. T., & McCarthy, D. M. (1995). Methodological considerations in the refinement of clinical assessment instruments. Psychological Assessment, 7, 300–308. doi:10.1037/1040-3590.7.3.300.
40. Snyder, J., Reid, J., Stoolmiller, M., Howe, G., Brown, H., Dagne, G., & Cross, W. (2006). The role of behavior observation in measurement systems for randomized prevention trials. Prevention Science, 7, 43–56. doi:10.1007/s11121-005-0020-3.
41. Stetler, C. B., McQueen, L., Demakis, J., & Mittman, B. S. (2008). An organizational framework and strategic implementation for systems-level change to enhance research-based practice: QUERI series. Implementation Science, 3, 30. doi:10.1186/1748-5908-3-30.
42. Streiner, D. L. (2003). Being inconsistent about consistency: When coefficient alpha does and doesn’t matter. Journal of Personality Assessment, 80, 217–222. doi:10.1207/S15327752JPA8003_01.
43. Tabachnick, B. G., & Fidell, L. S. (2007). Using multivariate statistics (Vol. 5). Boston: Pearson Education, Inc.
44. Valla, J. P., Bergeron, L., & Smolla, N. (2000). The Dominic-R: A pictorial interview for 6-to 11-year old children. Journal of the American Academy of Child and Adolescent Psychiatry, 39, 85–93. doi:10.1097/00004583-200001000-00020.
45. Waltz, J., Addis, M. E., Koerner, K., & Jacobson, N. S. (1993). Testing the integrity of a psychotherapy protocol: Assessment of adherence and competence. Journal of Consulting & Clinical Psychology, 61, 620–630. doi:10.1037/0022-006X.61.4.620.
46. Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78, 200–211. doi:10.1037/a0018912.
47. Wells, M. G., Burlingame, G. M., & Lambert, M. J. (1996). Conceptualization and measurement of patient change during psychotherapy: Development of the Outcome Questionnaire and Youth Outcome Questionnaire. Psychotherapy: Theory, Research, Practice, Training, 33, 275–283. doi:10.1037/0033-3204.33.2.275.
48. Werthamer-Larsson, L., Kellam, S., & Wheeler, L. (1991). Effect of first-grade classroom environment on shy behavior, aggressive problems, and concentration problems. American Journal of Community Psychology, 19, 585–601. doi:10.1007/BF00937993.
49. Wyman, P. A., Cross, W., Hendricks Brown, C., Yu, Q., Tu, X., & Eberly, S. (2010). Intervention to strengthen emotional self-regulation in children with emerging mental health problems: Proximal impact on school behavior. Journal of Abnormal Child Psychology, 38, 707–720. doi:10.1007/s10802-010-9398-x.
50. Wyman, P. A., Cross, W., Wu, P., Schmeelk-Cone, K. West, J., Tu, X., Lu, Naiji & Brown, C. H. (2013). Intervention to strengthen emotion self-regulation in young urban children: Impact on internalizing and externalizing problems and mediating effects of self-regulation skills. (in review).