Treatment fidelity in multisite evaluation: A multilevel longitudinal examination of provider adherence status and change

American Journal of Evaluation

Volumn 30, Issue 1, 2009, Pages 44-61

  Zvoch, Keith ^a  

^a UNIVERSITY OF OREGON   (United States)

Author keywords

Early childhood literacy; Multilevel growth models; Multisite evaluation; Treatment fidelity

Indexed keywords

EID: 61349095396     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214008329523     Document Type: Article

References (58)

1. Bauman, L.J., Stein, R.E., & Ireys, H.T. (1991). Reinventing fidelity: The transfer of social technology among settings. American Journal of Community Psychology, 19, 619-639.
2. Bellg, A.J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D.S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations for the NIH Behavior Change Consortium. Health Psychology, 23, 443-451.
3. Berman, P. (1980). Thinking about programmed and adaptive implementation: Matching strategies to situations. In H. M. Ingram & D. E. Mann (Eds.), Why policies succeed or fail (pp. 205-227). Beverly Hills, CA: Sage.
4. Berman, P. (1981). Educational change: An implementation paradigm. In R. Lehming & D. E. Mann (Eds.), Improving schools: Using what we know (pp. 253-291). Beverly Hills, CA: Sage.
5. Bickman, L. (1987). Using program theory in evaluation. In L. Bickman (Ed.), Using program theory in evaluation (New Directions for Program Evaluation, No. 33, pp. 5-18). San Francisco, CA: Jossey-Bass.
6. Blakely, C.H., Mayer, J.P., Gottschalk, R.G., Schmitt, N., Davidson, W.S., Roitman, D.B., et al. (1987). The fidelity-adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253-268.
7. Bond, G.R., Evans, L., Salyers, M.P., Williams, J., & Kim, H.W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2, 75-87.
8. Borrelli, B., Sepinwall, D., Ernst, D., Bellg, A.J., Czajkowski, S., Breger., R., et al. (2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73, 852-860.
9. Breakthrough to Literacy. (2004). Professional development guide: Essential practices. Chicago, IL: Wright Group/McGraw-Hill.
10. Breakthrough to Literacy. (2008). About breakthrough to literacy. Retrieved July 13, 2008, from http://www.breakthroughtoliteracy.com/.
11. Chen, H.T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
12. Crits-Christoph, P., Siqueland, L., Chittams, J., Barber, J.P., Beck, A.T., Frank, A., et al. (1998). Training in cognitive, supportive-expressive, and drug counseling therapies for cocaine dependence. Journal of Consulting and Clinical Psychology, 66, 484-492.
13. Dane, A.V., & Schneider, B.H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control ? Clinical Psychology Review, 18, 23-45.
14. Donaldson, S.I. (2003). Theory-driven program evaluation in the new millennium. In S. I. Donaldson & M. Scriven (Eds.), Evaluating social programs and problems: Visions for the new millennium (pp. 111-142). Mahwah, NJ: Erlbaum.
15. Dumas, J.E., Lynch, A.M., Laughlin, J.E., Smith, E.P., & Prinz, R.J. (2001). Promoting implementation fidelity: Conceptual issues, methods, and preliminary results from the Early Alliance prevention trial. American Journal of Preventive Medicine, 20(Suppl. 1), 38-47.
16. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W.B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237-256.
17. Fullan, M., & Pomfret, A. (1977). Research on curriculum and instruction. Review of Educational Research, 47, 335-397.
18. Good, R.H., & Kaminski, R.A. (2002). Dynamic indicators of basic early literacy skills (6th ed.). Eugene, OR: Institute for the Development of Education Achievement.
19. Good, R.H., Kaminski, R.A., Simmons, D., & Kame'enui, E.J. (2001). Using dynamic indicators of basic early literacy skills (DIBELS) in an outcomes-driven model: Steps to reading outcomes. Eugene, OR: University of Oregon.
20. Gottfredson, D.C. (1984). A theory-ridden approach to program evaluation: A method for stimulating researcherimplementer collaboration. American Psychologist, 39, 1101-1112.
21. Hall, G.E., & Hord, S.M. (1987). Change in schools: Facilitating the process. Albany, NY: State University of New York Press.
22. Hall, G.E., & Hord, S.M. (2006). Implementing change: Patterns, principles, and potholes (2nd ed.). Boston, MA: Allyn & Bacon.
23. Hall, G.E., & Loucks, S.F. (1977). A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14, 263-276.
24. Hall, G.E., Loucks, S.F., Rutherford, W.L., & Newlove, B.W. (1975). Levels of use of the innovation: A framework for analyzing innovation adoption. Journal of Teacher Education, 26, 52-56.
25. Kalafat, J., Illback, R.J., & Sanders, D. (2007). The relationship between implementation fidelity and educational outcomes in a school-based family support program: Development of a model for evaluating multidimensional full-service programs. Evaluation and Program Planning, 30, 136-148.
26. Kaminski, R., & Cummings, K.D. (2007). DIBELS: Myths and facts. Eugene, OR: Dynamic Measurement Group. Retrieved July 16, 2007, from http://www.dibels.org/papers/Myths03_web.pdf.
27. Layzer, C., Boulay, B., & Hurtig, R. (2005). The CLIMBERs study: The effects of one year of Breakthrough to Literacy in preschool on classrooms and students. Washington, DC: Institute of Education Sciences, U.S. Department of Education.
28. Lillehoj, C.J., Griffin, K.W., & Spoth, R. (2004). Program provider and observer ratings of school-based prevention intervention implementation: Agreement and relation to youth outcomes. Health Education & Behavior, 31, 242-257.
29. Lipsey, M.W. (1993). Theory as method. Small theories of treatments. In L. B. Sechrest & A. G. Scott (Eds.), Understanding causes and generalizing about them (New Directions for Program Evaluation, No. 57, pp. 5 - 38). San Francisco, CA: Jossey-Bass.
30. McGrew, J.H., Bond, G.R., Dietzen, L., & Salyers, M. (1994). Measuring the fidelity of implementation of a mental health program model. Journal of Consulting and Clinical Psychology, 62, 670-678.
31. McHugo, G.J., Drake, R.E., Teague, G.B., & Xie, H. (1999). The relationship between model fidelity and client outcomes in the New Hampshire dual disorders study. Psychiatric Services, 50, 818-824.
32. Melde, C., Esbensen, F.A., & Tusinski, K. (2006). Addressing program fidelity using onsite observations and program provider descriptions of program delivery. Evaluation Review, 30, 714-740.
33. Miller, S.J., & Binder, J.L. (2002). The effects of manual-based training on treatment fidelity and outcome: A review of the literature on adult individual psychotherapy. Psychotherapy: Theory/Research/ Practice/Training, 39, 184-198.
34. Moncher, F.J., & Prinz, R.J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247-266.
35. Mowbray, C.T., & Herman, S.E. (1991). Using multiple sites in mental health evaluations: Focus on program theory and implementation issues. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 45 - 57). San Francisco, CA: Jossey-Bass.
36. Mowbray, C.T., Holter, M.C., Teague, G.B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24, 315-340.
37. National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Rockville, MD: National Institutes of Health (NIH Pub. No. 00-4754).
38. No Child Left Behind Act of 2001, Pub. L. No. 107-110 (2002).
39. Nunnery, J.A., Ross, S.M., & McDonald, A. (2006). A randomized experimental evaluation of the impact of accelerated reader/reading renaissance implementation on reading achievement in grades 3 to 6. Journal of Education for Students Placed At Risk, 11, 1-18.
40. Orwin, R.G. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95, S309-S327.
41. Parcel, G.S., Ross, J.G., Lavin, A.T., Portnoy, B., Nelson, G.D., & Winters, F. (1991). Enhancing implementation of the teenage health teaching modules. Journal of School Health, 61, 35-38.
42. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.
43. Raudenbush, S.W., Bryk, A.S., Cheong, Y.F., & Congdon, R.T. (2004). HLM 6: Hierarchical linear and nonlinear modeling. Chicago, IL: Scientific Software International.
44. Ringwalt, C.L., Ennett, S., Johnson, R. Rohrbach, L.A., Simons-Rudolph, A., Vincus, A., et al. (2003). Factors associated with fidelity to substance use prevention curriculum guides in the nation's middle schools. Health Education & Behavior, 30, 375-391.
45. Robinson, C., & Grek, M.L. (2004). Breakthrough to literacy. Tallahassee, FL: Florida Center for Reading Research. Retrieved July 13, 2008, from http://www.fcrr.org/FCRRreports/PDF/BTLReport.pdf.
46. Rogosa, D. (1995). Myths and methods: "Myths about longitudinal research" plus supplemental questions. In J. M. Gottman (Ed.), The analysis of change (pp. 3-67). Mahwah, NJ: Erlbaum.
47. Rohrbach, L.A., Graham, J.W., & Hansen, W.B. (1993). Diffusion of a school-based substance abuse prevention program: Predictors of program implementation. Preventive Medicine, 22, 237-260.
48. Rossi, P.H., Lipsey, M.W., & Freeman, H.E. (2004). Evaluation: A systematic approach (7th ed.). Thousand Oaks, CA: Sage.
49. Scheirer, M.A. (1981). Program implementation: The organizational context. Beverly Hills, CA: Sage.
50. Scheirer, M.A. (1987). Program theory and program implementation: Implications for evaluators. In L. Bickman (Ed.), Using program theory in evaluation (New Directions for Program Evaluation, No. 33, pp. 59 - 76). San Francisco, CA: Jossey-Bass.
51. Sechrest, L., West, S.G., Phillips, M.A., Redner, R., & Yeaton, W. (1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner & W. Yeaton (Eds.), Evaluation studies review annual (Vol. 4, pp. 15-35). Thousand Oaks, CA: Sage.
52. Smylie, M.A. (1988). The enhancement function of staff development: Organizational and psychological antecedents to individual teacher change. American Education Research Journal, 25, 1-30.
53. Voyager Expanded Learning (2006). Research foundations and design: Voyager Universal Literacy System. Author. Retrieved July 15, 2006, from www.voyagerlearning.com.
54. Weiss, C.H. (1997). How can theory-based evaluation make greater headway ? Evaluation Review, 21, 501-524.
55. What Works Clearinghouse. (2007). Voyager Universal Literacy System. Washington, DC: Institute of Education Sciences, U.S. Department of Education. Retrieved July 13, 2008, from http://ies.ed.gov/ncee/wwc/pdf/ WWC_Voyager_072307.pdf.
56. Willett, J.B. (1988). Questions and answers in the measurement of change. In E. Rothkopf (Ed.), Review of research in education 1988-89 (pp. 345 - 422). Washington, DC: American Educational Research Association.
57. Yeaton, W.H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49, 156-167.
58. Zvoch, K., Letourneau, L.E., & Parker, R.P. (2007). A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model. American Journal of Evaluation, 28, 132-150.