How Does Fidelity of Implementation Matter? Using Multilevel Models to Detect Relationships Between Participant Outcomes and the Delivery and Receipt of Treatment

American Journal of Evaluation

Volumn 33, Issue 4, 2012, Pages 547-565

  Zvoch, Keith ^a  

^a UNIVERSITY OF OREGON   (United States)

Author keywords

early childhood literacy; summer learning; summer school; treatment fidelity

Indexed keywords

EID: 84869015844     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214012452715     Document Type: Article

References (54)

1. Alexander K. L.,Entwisle D. R.,Olson L. S.Schools, achievement, and inequality: A seasonal perspective.Educational Evaluation and Policy Analysis. 2001;23:171-191.
2. Bellg A. J.,Borrelli B.,Resnick B.,Hecht J.,Minicucci D. S.,Ory M.,Czajkowski S.Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations for the NIH Behavior Change Consortium.Health Psychology. 2004;23:443-451.
3. Berman P.,McLaughlin M. W.Implementation of educational innovation.Educational Forum. 1976;40:345-370.
4. Bloom H. S.,Hill C. J.,Riccio J. A.Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments.Journal of Policy Analysis and Management. 2003;22:551-575.
5. Borman G. D.,Dowling N. M.The longitudinal achievement effects of multiyear summer school: Evidence from the Teach Baltimore randomized field trial.Educational Evaluation and Policy Analysis. 2006;28:25-48.
6. Borrelli B.,Sepinwall D.,Ernst D.,Bellg A. J.,Czajkowski S.,Breger R.,Orwig D.A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research.Journal of Consulting and Clinical Psychology. 2005;73:852-860.
7. Century J.,Rudnick M.,Freeman C.A framework for measuring fidelity of implementation. A foundation for shared language and accumulation of knowledge.American Journal of Evaluation. 2010;31:199-218.
8. Chen H. T.Theory-driven evaluations. Newbury Park, CA: Sage; 1990:.
9. Test of Oral Reading Fluency (TORF). Eden Prairie, MN: Author; 1987:.
10. Cooper H.,Charlton K.,Valentine J. C.,Muhlenbruck L.Making the most of summer school: A meta-analytic and narrative review.Monographs of the Society for Research in Child Development, 65. 2000;:.
11. Cooper H.,Nye B.,Charlton K.,Lindsay J.,Greathouse S.The effects of summer vacation on achievement test scores: A narrative and meta-analytic review.Review of Educational Research. 1996;66:227-268.
12. Dane A. V.,Schneider B. H.Program integrity in primary and early secondary prevention: Are implementation effects out of control?.Clinical Psychology Review. 1998;18:23-45.
13. Dedrick R. F.,Ferron J. M,Hess M. R.,Hogarty K. Y.,Kromrey J. D.,Lang T. R.,Lee R. S.Multilevel modeling: A review of methodological issues and applications.Review of Educational Research. 2009;79:69-102.
14. Donaldson S. I.Program Theory-Driven Evaluation Science: Strategies and Applications. Mahwah, NJ: Lawrence Erlbaum; 2007:.
15. Durlack J. A.,DuPre E. P.Implementation matters: A review on the influence of implementation on program outcomes and the factors affecting implementation.American Journal of Community Psychology. 2008;41:327-350.
16. Dusenbury L.,Brannigan R.,Falco M.,Hansen W. B.A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings.Health Education Research. 2003;18:237-256.
17. Entwisle D. R.,Alexander K. L.,Olson L. S.Children, schools, and inequality. Boulder, CO: Westview Press; 1997:.
18. Fullan M.,Pomfret A.Research on curriculum and instruction.Review of Educational Research. 1977;47:335-397.
19. Greenwood C. R.Treatment integrity: Revisiting some big ideas.School Psychology Review. 2009;38:547-553.
20. Gresham F. M.Evolution of the treatment integrity concept: Current status and future directions.School Psychology Review. 2009;38:533-540.
21. Hagermoser Sanetti L. M.,Kratochwill T. R.Treatment integrity in behavioral consultation: Measurement, promotion, and outcomes.International Journal of Behavioral Consultation and Therapy. 2008;4:95-114.
22. Hagermoser Sanetti L. M.,Kratochwill T. R.Toward developing a science of treatment integrity: Introduction to the special series.School Psychology Review. 2009;38:445-459.
23. Hall G. E.,Loucks S. F.A developmental model for determining whether the treatment is actually implemented.American Educational Research Journal. 1977;14:263-276.
24. Hall G. E.,Loucks S. F.,Rutherford W. L.,Newlove B. W.Levels of use of the innovation: A framework for analyzing innovation adoption.Journal of Teacher Education. 1975;26:52-56.
25. Heyns B.Summer learning and the effects of schooling. New York, NY: Academic Press; 1978:.
26. Heyns B.Schooling and cognitive development: Is there a season for learning?.Child Development. 1987;58:1151-1160.
27. Hulleman C. S.,Cordray D. S.Moving from the lab to the field: The role of fidelity and achieved relative intervention strength.Journal of Research on Educational Effectiveness. 2009;2:88-110.
28. Institute for Education Sciences. (2009). Request for Applications: Education Research Grants. Retrieved fromhttp://ies.ed.gov/funding/pdf/2009-84305A.pdf.
29. Jo B.Estimation of intervention effects with noncompliance: Alternative model specifications.Journal of Educational and Behavioral Statistics. 2002;27:385-409.
30. Jo B.,Muthén B. O.New developments and techniques in structural equation modeling. Marcoulides G. A.Schumacker R. E., ed. Mahwah, NJ: Lawrence Erlbaum; 2001:57-87.
31. Jones H. A.,Clark A. T.,Power T. J.Expanding the concept of intervention integrity: A multidimensional model of participant engagement.Balance. 2008;23:4-5.
32. McGrew J. H.,Bond G. R.,Dietzen L.,Salyers M.Measuring the fidelity of implementation of a mental health program model.Journal of Consulting and Clinical Psychology. 1994;62:670-678.
33. McKenna S. A.,Rosenfield S.,Gravois T. A.Evaluating the validity of instructional consultation: Level of Implementation Scale-Revised.School Psychology Review. 2009;38:496-509.
34. Moncher F. J.,Prinz R. J.Treatment fidelity in outcome studies.Clinical Psychology Review. 1991;11:247-266.
35. Mowbray C. T.,Bybee D.,Holter M.,Lewandowski L.Validation of a fidelity rating instrument for consumer-operated services.American Journal of Evaluation. 2006;27:9-27.
36. Mowbray C. T.,Holter M. C.,Teague G. B.,Bybee D.Fidelity criteria: Development, measurement, and validation.American Journal of Evaluation. 2003;24:315-340.
37. Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Rockville, MD: National Institutes of Health; 2000:.
38. On evaluating curricular effectiveness: Judging the quality of K-12 mathematics evaluations. Washington, DC: National Academies Press; 2004:.
39. Noell G. H.Handbook of research in school consultation: Empirical foundations for the field. Erchul W. P.Sheridan S. M., ed. Mahwah, NJ: Lawrence Erlbaum; 2008:315-334.
40. O'Donnell C. L.Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research.Review of Educational Research. 2008;78:33-84.
41. Peterson L.,Homer A. L.,Wonderlich S. A.The integrity of independent variables in behavior analysis.Journal of Applied Behavior Analysis. 1982;15:477-492.
42. Raudenbush S. W.,Bryk A. S.Hierarchical linear models: Applications and data analysis methods. 2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousand Oaks, CA: Sage.Thousand Oaks, CA: Sage; 2002:.
43. Raudenbush S. W.,Bryk A. S.,Cheong Y. F.,Congdon R. T.HLM 6: Hierarchical linear and nonlinear modeling. Chicago, IL: Scientific Software International; 2004:.
44. Raudenbush S. W.,Xiaofeng L.Statistical power and optimal design for multisite randomized trials.Psychological Methods. 2000;5:199-213.
45. Raudenbush S. W.,Xiaofeng L.Effects of study duration, frequency of observation, and sample size on power in studies of group differences in polynomial change.Psychological Methods. 2001;6:387-401.
46. Scheirer M. A.Using program theory in evaluation. Bickman L., ed. San Francisco, CA: Jossey-Bass; 1987:.
47. Schulte A. C.,Parker J.,Easton J.Advances in treatment integrity research: Multidisciplinary perspectives on the conceptualization, measurement, and enhancement of treatment integrity.School Psychology Review. 2009;38:460-475.
48. Sechrest L.,West S. G.,Phillips M. A.,Redner R.,Yeaton W.Evaluation studies review annual. Sechrest L.West S. G.Phillips M. A.Redner R.Yeaton W., ed. Thousand Oaks, CA: Sage; 1979:15-35.
49. Shadish W. R.,Cook T. D.,Campbell D. T.Experimental and quasi-experimental designs for generalized causal inference. Boston. MA: Houghton Mifflin; 2002:.
50. Tindal G.,Marston D.,Deno S. L.The reliability of direct and repeated measurement. Minneapolis: University of Minnesota Institute for Research on Learning Disabilities; 1983:.
51. Yeaton W. H.,Sechrest L.Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness.Journal of Consulting and Clinical Psychology. 1981;49:156-167.
52. Zvoch K.Treatment fidelity in multisite evaluation: A multilevel longitudinal examination of provider adherence status and change.American Journal of Evaluation. 2009;30:44-61.
53. Zvoch K.,Letourneau L. E.,Parker R. P.A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model.American Journal of Evaluation. 2007;28:132-150.
54. Zvoch K.,Stevens J. J.Summer school and summer learning: An examination of the short and longer term changes in student literacy.Early Education and Development. 2011;22:649-675.