A multilevel multisite outcomes-by-implementation evaluation of an early childhood literacy model

American Journal of Evaluation

Volumn 28, Issue 2, 2007, Pages 132-150

  Zvoch, Keith ^a   Letourneau, Lawrence E ^a   Parker, Robert P ^a  

^a UNIVERSITY OF NEVADA   (United States)

Author keywords

Multilevel growth models; Multisite evaluation; Treatment fidelity

Indexed keywords

EID: 34249092273     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214007301138     Document Type: Article

References (55)

1. Aitkin, M., & Longford, N. (1986). Statistical modeling issues in school effectiveness studies. Journal of the Royal Statistical Society, Series A, 149, 1-43.
2. Babor, T., Steinberg, K., McRee, B., Vendetti, J., & Carroll, K. (2002). Treating marijuana dependence in adults: A multisite randomized clinical trial. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 17-30). San Francisco: Jossey-Bass.
3. Banks, S. (2002). A prospective meta-analytic approach in a multisite study of homelessness prevention. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 45-60). San Francisco: Jossey-Bass.
4. Bellg, A.J., Borrelli, B., Resnick, B., Hecht, J., Minicucci, D.S., Ory, M., et al. (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations for the NIH Behavior Change Consortium. Health Psychology, 23(5), 443-451.
5. Berman, P. (1980). Thinking about programmed and adaptive implementation: Matching strategies to situations. In H. M. Ingram & D. E. Mann (Eds.), Why policies succeed or fail (pp. 205-227). Beverly Hills, CA: Sage.
6. Berman, P. (1981). Educational change: An implementation paradigm. In R. Lehming & D. E. Mann (Eds.), Improving schools: Using what we know (pp. 253-291). Beverly Hills, CA: Sage.
7. Bickman, L. (1987). Using program theory in evaluation. In L. Bickman (Ed.), Using program theory in evaluation (New Directions for Program Evaluation, No. 33, pp. 5-18). San Francisco: Jossey-Bass.
8. Blakely, C.H., Mayer, J.P., Gottschalk, R.G., Schmitt, N., Davidson, W.S., Roitman, D.B., et al. (1987). The fidelity-adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15(3), 253-268.
9. Bond, G.R., Evans, L., Salyers, M.P., Williams, J., & Kim, H.W. (2000). Measurement of fidelity in psychiatric rehabilitation. Mental Health Services Research, 2(2), 75-87.
10. Borrelli, B., Sepinwall, D., Ernst, D., Bellg, A.J., Czajkowski, S., Breger, R., et al. (2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73(5), 852-860.
11. Boyle, M.H., & Willms, J.D. (2001). Multilevel modeling of hierarchical data in developmental studies. Journal of Child Psychology and Psychiatry, 42, 141-162.
12. Bryk, A.S., & Raudenbush, S.W. (1987). Application of hierarchical linear models to assessing change. Psychological Bulletin, 101, 147-158.
13. Bryk, A.S., & Raudenbush, S.W. (1988). Toward a more appropriate conceptualization of research on school effects: A three-level hierarchical linear model. In R. D. Bock (Ed.), Multilevel analysis of educational data (pp. 159-204). San Diego: CA: Academic Press.
14. Burstein, L. (1980). The analysis of multilevel data in educational research and evaluation. In D. C. Berliner (Ed.), Review of research in education (Vol. 8, pp. 158-233). Washington, DC: American Educational Research Association.
15. Chen, H.T. (1990). Theory-driven evaluations. Newbury Park, CA: Sage.
16. Chen, H.T., & Rossi, P.H. (1983). Evaluating with sense. The theory-driven approach. Evaluation Review, 7, 283-302.
17. Cook, A., Carey, M., Razzano, L., Burke, J., & Blyler, C. (2002). The pioneer: The employment intervention demonstration program. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 31-44). San Francisco: Jossey-Bass.
18. Cottingham, P. (1991). Unexpected lessons: Evaluation of job-training programs for single mothers. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 59-70). San Francisco: Jossey-Bass.
19. Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of state policy evidence. Educational Policy Analysis Archives, 8(1). Available at http://epaa.asu.edu/epaa/v8n1/
20. Finn, J.D., Gerber, S.B., Achilles, C.M., & Boyd-Zaharias, J. (2001). The enduring effects of small classes. Teachers College Record, 103(2), 145-183.
21. Frechtling, J.A., Zhang, X., & Silverstein, G. (2006). The Voyager Universal Literacy System: Results from a study of kindergarten students in inner-city schools. Journal of Education for Students Placed at Risk, 11(1), 75-95.
22. Good, R.H., & Kaminski, R.A. (2002). Dynamic indicators of basic early literacy skills (6 th ed.). Eugene, OR: Institute for the Development of Education Achievement.
23. Good, R.H., Kaminski, R.A., Simmons, D., & Kame'enui, E.J. (2001). Using dynamic indicators of basic early literacy skills (DIBELS) in an outcomes-driven model: Steps to reading outcomes. Eugene: University of Oregon.
24. Greenwald, R., Hedges, L.V., & Laine, R.D. (1996). The effect of school resources on student achievement. Review of Educational Research, 66, 361-391.
25. Grissmer, D. (1999). Class size effects: Assessing the evidence, its policy implications, and future research agenda. Educational Evaluation and Policy Analysis, 21(2), 231-248.
26. Hall, G.E., & Loucks, S.F. (1977). A developmental model for determining whether the treatment is actually implemented. American Educational Research Journal, 14(3), 263-276.
27. Lee, V.E., & Smith, J.B. (2001). Restructuring high schools for equity and excellence: What works. New York: Teachers College Press.
28. Leff, H.S., & Mulkern, V. (2002). Lessons learned about science and participation from multisite evaluations. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 89-100). San Francisco: Jossey-Bass.
29. Lipsey, M.W. (1993). Theory as method. Small theories of treatments. In L. B. Sechrest & A. G. Scott (Eds.), Understanding causes and generalizing about them (New Directions for Program Evaluation, No. 57, pp. 5-38). San Francisco: Jossey-Bass.
30. Miller, S.J., & Binder, J.L. (2002). The effects of manual-based training on treatment fidelity and outcome: A review of the literature on adult individual psychotherapy. Psychotherapy: Theory/Research/ Practice/Training, 39(2), 184-198.
31. Moncher, F.J., & Prinz, R.J. (1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11, 247-266.
32. Mowbray, C.T., Bybee, D., Holter, D., & Lewandowski, L. (2006). Validation of a fidelity rating instrument for consumer-operated services. American Journal of Evaluation, 27(1), 9-27.
33. Mowbray, C.T., & Herman, S.E. (1991). Using multiple sites in mental health evaluations: Focus on program theory and implementation issues. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 45-57). San Francisco: Jossey-Bass.
34. Mowbray, C.T., Holter, M.C., Teague, G.B., & Bybee, D. (2003). Fidelity criteria: Development, measurement, and validation. American Journal of Evaluation, 24(3), 315-340.
35. Nye, B., Hedges, L.V., & Konstantopoulos, S. (1999). The long term effects of small classes: A five year follow-up of the Tennessee class size experiment. Educational Evaluation and Policy Analysis, 21(2), 127-142.
36. Orwin, R.G. (2000). Assessing program fidelity in substance abuse health services research. Addiction, 95(3), S309-S327.
37. Raudenbush, S.W., & Bryk, A.S. (2002). Hierarchical linear models: Applications and data analysis methods (2 nd ed.). Thousand Oaks, CA: Sage.
38. Raudenbush, S.W., Bryk, A.S., Cheong, Y.F., & Congdon, R.T. (2004). HLM 6: Hierarchical linear and nonlinear modeling. Chicago: Scientific Software International.
39. Rawson, R., Marinelli-Casey, P., & Huber, A. (2002). A multisite evaluation of treatment of methamphetamine dependence in adults. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 73-88). San Francisco: Jossey-Bass.
40. Reiss, A., & Boruch, R. (1991). The program review team approach and multisite experiments: The spouse assault replication program. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 33-44). San Francisco: Jossey-Bass.
41. Rog, D., & Randolph, F. (2002). A multisite evaluation of supported housing: Lessons learned from cross-site collaboration. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 61-72). San Francisco: Jossey-Bass.
42. Rowan, B., Correnti, R., & Miller, R.J. (2002). What large-scale survey research tells us about teacher effects on student achievement: Insights from the Prospects study of elementary schools. Teachers College Record, 104, 1525-1567.
43. Scheirer, M.A. (1987). Program theory and program implementation: Implications for evaluators. In L. Bickman (Ed.), Using program theory in evaluation (New Directions for Program Evaluation, No. 33, pp. 59-76). San Francisco: Jossey-Bass.
44. Sechrest, L., West, S.G., Phillips, M.A., Redner, R., & Yeaton, W. (1979). Some neglected problems in evaluation research: Strength and integrity of treatments. In L. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual (Vol. 4, pp. 15-35). Beverly Hills, CA: Sage.
45. Sinacore, J.M., & Turpin, R.S. (1991). Multiple sites in evaluation research: A survey of organizational and methodological issues. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 5-18). San Francisco: Jossey-Bass.
46. Skogan, W., & Lurgio, A. (1991). Multisite evaluations in criminal justice settings: Structural obstacles to success. In R. S. Turpin & J. M. Sinacore (Eds.), Multisite evaluations (New Directions for Program Evaluation, No. 50, pp. 83-96). San Francisco: Jossey-Bass.
47. Snidjers, T., & Bosker, R. (1999). Multilevel analysis. Thousand Oaks, CA: Sage.
48. Straw, R.B., & Herrell, J.M. (2002). A framework for understanding and improving multisite evaluations. In J. M. Herrell & R. B. Straw (Eds.), Conducting multiple site evaluations in real-world settings (New Directions for Program Evaluation, No. 94, pp. 5-15). San Francisco: Jossey-Bass.
49. Teddlie, C., & Reynolds, D. (2000). The international handbook of school effectiveness research. New York: Falmer.
50. Voyager Expanded Learning. (2006). Research foundations and design: Voyager Universal Literacy System. Retrieved July 15, 2006, from www.voyagerlearning.com
51. Wayne, A.J., & Youngs, P. (2003). Teacher characteristics and student achievement gains: A review. Review of Educational Research, 73(1), 89-122.
52. Weiss, C.H. (1997). How can theory-based evaluation make greater headway? Evaluation Review, 21(4), 501-524.
53. Wright, S.W., Horn, S.P., & Sanders, W.L. (1997). Teacher and classroom context effects on student achievement: Implications for teacher evaluation. Journal of Personnel Evaluation in Education, 11, 57-67.
54. Yeaton, W.H., & Sechrest, L. (1981). Critical dimensions in the choice and maintenance of successful treatments: Strength, integrity, and effectiveness. Journal of Consulting and Clinical Psychology, 49(2), 156-167.
55. Zvoch, K., & Stevens, J.J. (2006). Longitudinal effects of school context and practice on middle school mathematics achievement. Journal of Educational Research, 99(6), 347-356.