Opening the black box: Intervention fidelity in a randomized trial of a preschool teacher professional development program

Journal of Educational Psychology

Volumn 108, Issue 1, 2016, Pages 130-145

  Mendive, Susana ^a   Weiland, Christina ^b   Yoshikawa, Hirokazu ^c   Snow, Catherine ^d  

^a PONTIFICIA UNIVERSIDAD CATÓLICA DE CHILE   (Chile)

^b UNIVERSITY OF MICHIGAN   (United States)

^c NEW YORK UNIVERSITY   (United States)

^d HARVARD GRADUATE SCHOOL OF EDUCATION   (United States)

Author keywords

Early childhood education; Early literacy; Intervention fidelity; Randomized controlled trials; Teacher professional development

Indexed keywords

EID: 84954370931     PISSN: 00220663     EISSN: 19392176     Source Type: Journal    
DOI: 10.1037/edu0000047     Document Type: Article

References (80)

1. Alvarado, C. G., Ruef, M. L., Schrank, F. A. (2005) Comprehensive manual: Woodcock-Muñoz Language Survey-Revised. Itasca, IL: Riverside.
2. Araujo, M. C., Carneiro, P., Cruz-Aguayo, Y., Schady, N. (2014) A helping hand? Teacher quality and learning outcomes in kindergarten. Unpublished manuscript. Washington, DC: Inter-American Development Bank.
3. Araujo, M. C., López-Boo, F., Puyana, J. M. (2013) Overview of early childhood development services in the Latin American and Caribbean region. Washington, DC: Inter-American Development Bank.
4. Aro, M., Wimmer, H. (2003) Learning to read: English in comparison to six more regular orthographies. Applied Psycholinguistics, 24, 621-635. http://dx.doi.org/10.1017/S0142716403000316
5. Bakeman, R., Quera, V. (2011) Sequential analysis and observational methods for the behavioral sciences. Cambridge, UK: Cambridge University Press. http://dx.doi.org/10.1017/CBO9781139017343
6. Beck, I. L., McKeown, M. G. (2007) Increasing young low-income children's oral vocabulary repertoires through rich and focused instruction. The Elementary School Journal, 107, 251-271. http://dx.doi.org/ 10.1086/511706
7. Beck, I., McKeown, M., Kucan, L. (2002) Bringing words to life. Robust vocabulary instruction. London: The Guilford Press.
8. Bierman, K. L. (July, 2014). Partnering with Head Start to incorporate evidence-based approaches: The REDI experience. Paper presented at the Head Start Research Conference, Washington, DC.
9. Bloom, H. S., Weiss, M. J., Brock, T. (2013) A conceptual framework for studying the sources of variation in program effects. New York, NY: MDRC Working Papers on Research Methodology.
10. Borko, H. (2004) Professional development and teacher learning: Mapping the terrain. Educational Researcher, 33, 3-15. http://dx.doi.org/10.3102/ 0013189X033008003
11. Bronfenbrenner, U., Morris, P. A. (2006) The bioecological model of human development. In R. Lerner & W. Damon (Eds.), Handbook of child psychology (6th ed.). Vol. 1: Theoretical models of human development (pp. 793-828). Hoboken, NJ: Wiley.
12. Bus, A. G., van IJzendoorn, M. H. (1988) Mother-child interactions, attachment, and emergent literacy: A cross-sectional study. Child Development, 59, 1262-1272. http://dx.doi.org/10.2307/1130489
13. Century, J., Rudnick, M., Freeman, C. A. (2010) Framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. The American Journal of Evaluation, 31, 199-218. http://dx.doi.org/10.1177/1098214010366173
14. Clements, D. H., Sarama, J. (2008) Experimental evaluation of the effects of a research-based preschool mathematics curriculum. American Educational Research Journal, 45, 443-494. http://dx.doi.org/10.3102/ 0002831207312908
15. Clements, D. H., Sarama, J., Spitler, M. E., Lange, A. A., Wolfe, C. B. (2011) Mathematics learned by young children in an intervention based on learning trajectories: A large-scale cluster randomized trial. Journal for Research in Mathematics Education, 4, 127-166.
16. Connor, C. M., Morrison, F. J., Slominski, L. (2006) Preschool instruction and children's emergent literacy growth. Journal of Educational Psychology, 98, 665-689. http://dx.doi.org/10.1037/0022-0663.98.4 .665
17. Dane, A., Schneider, B. H. (1998) Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23-45. http://dx.doi.org/10.1016/ S0272-7358(97)00043-3
18. Darrow, C. (March, 2010). Measuring Fidelity in Preschool Interventions: A Microanalysis of Fidelity Instruments Used in Curriculum Interventions. Paper presented at the Conference of the Society for Research in Educational Effectiveness, Washington, DC.
19. DeBaryshe, B. D., Gorecki, D. M. (2007) An experimental validation of a preschool literacy curriculum. Early Education and Development, 18, 93-110. http://dx.doi.org/10.1080/10409280701274741
20. Dickinson, D. K. (2011) Teachers' language practices and academic outcomes of preschool children. Science, 333, 964-967. http://dx.doi .org/10.1126/science.1204526
21. Dickinson, D. K., Freiberg, J. B., Barnes, E. (2011) Why are so few interventions really effective? A call for fine-grained research methodology. In S. B. Neuman & D. K. Dickinson (Eds.), Handbook of early literacy research (Vol. 3, pp. 337-357). New York, NY: Guilford Press.
22. Dickinson, D. K., Golinkoff, R. M., Hirsh-Pasek, K. K. (2010) Speaking out for language: Why language is central to reading development. Educational Researcher, 39, 305-310. http://dx.doi.org/10.3102/ 0013189X10370204
23. Domitrovich, C. E., Gest, S. D., Jones, D., Gill, S., Sanford Derousie, R. M. (2010) Implementation quality: Lessons learned in the context of the head start REDI trial. Early Childhood Research Quarterly, 25, 284-298. http://dx.doi.org/10.1016/j.ecresq.2010.04.001
24. Downer, J. T., Locasale-Crouch, J., Hamre, B., Pianta, R. (2009) Teacher characteristics associated with responsiveness and exposure to consultation and online professional development resources. Early Education and Development, 20, 431-455. http://dx.doi.org/10.1080/ 10409280802688626
25. Dunst, C. J., Trivette, C. M., Raab, M. (2013) An implementation science framework for conceptualizing and operationalizing fidelity in early childhood intervention studies. Journal of Early Intervention, 35, 85-101. http://dx.doi.org/10.1177/1053815113502235
26. Durlak, J. A. (2010) The importance of doing well in whatever you do: A commentary on the special section "Implementation Research in Early Childhood Education." Early Childhood Research Quarterly, 25, 348-357. http://dx.doi.org/10.1016/j.ecresq.2010.03.003
27. Durlak, J. A., DuPre, E. P. (2008) Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350. http://dx.doi.org/10.1007/s10464-008-9165-0
28. Dusenbury, L., Brannigan, R., Falco, M., Hansen, W. B. (2003) A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237-256. http://dx.doi.org/10.1093/her/18.2.237
29. Facultad de Educación, UDP. (2010) Seguimiento y evaluación de la implementación de los Programas Pedagógicos para el primer y segundo nivel de transición en escuelas municipales del país Santiago de Chile [Implementation and follow-up study of prekindergarten and kindergarten curricular programs in public schools in the country]. Retrieved from http://w3app.mineduc.cl/mineduc/ded/documentos/ Evaluacion_Implementacion_Programas_Pedagogicos_NT1_NT2.pdf
30. Facultad de Educación PUC/MINEDUC. (2011) Informe final: Alfabetización en establecimientos chilenos subvencionados [Final report: Early literacy in subsidized schools]. Santiago, Chile: Facultad de Educación PUC. Retrieved from http://ded.mineduc.cl/mineduc/ded/documentos/ Alfabetizacion%20-%20Informe%20final.pdf
31. Farran, D. C., Wilson, S., Lipsey, M. (2013, April). Effects of a curricular attempt to improve self-regulation and achievement in prekindergarten children. Presentation at the biannual meeting of the Society for Research in Child Development, Seattle, WA.
32. Franks, R., Schroeder, J. (2013) Implementation science and early childhood and education: What do we know and where do we go from here? In T. Halle, A. Metz, I. Martinez-Beck (Eds.), Applying implementation science in early childhood programs and systems (pp. 5-19). Baltimore, MD: Brookes.
33. Goldenberg, C., Reese, L., Gallimore, R. (1992) Effects of literacy materials from school on Latino children's home experiences and early reading achievement. American Journal of Education, 100, 497-537. http://dx.doi.org/10.1086/444026
34. Greenwood, C., Tapia, Y., Abbott, M., Walton, C. (2003) A buildingbased case study of evidence-based literacy practices: Implementation, reading behavior, and growth in reading fluency, K-4. The Journal of Special Education, 37, 95-110. http://dx.doi.org/10.1177/ 00224669030370020401
35. Hamre, B. K., Justice, L. M., Pianta, R. C., Kilday, C., Sweeney, B., Downer, J. T., Leach, A. (2010) Implementation fidelity of MyTeachingPartner literacy and language activities: Association with preschoolers' language and literacy growth. Early Childhood Research Quarterly, 25, 329-347. http://dx.doi.org/10.1016/j.ecresq.2009.07.002
36. Hindman, A. H., Wasik, B. A. (2013) Vocabulary learning in Head Start: Nature and extent of classroom instruction and its contributions to children's learning. Journal of School Psychology, 51, 387-405. http:// dx.doi.org/10.1016/j.jsp.2013.01.001
37. Howes, C., Burchinal, M., Pianta, R., Bryant, D., Early, D., Clifford, R., Barbarin, O. (2008) Ready to learn? Children's pre-academic achievement in pre-Kindergarten programs. Early Childhood Research Quarterly, 23, 27-50. http://dx.doi.org/10.1016/j.ecresq.2007.05.002
38. Hulleman, C. S., Rimm-Kaufman, S. E., Abry, T. D. S. (2013) Innovative methodologies to explore implementation: Whole-part-whole- Construct validity, measurement, and analytical issues for intervention fidelity assessment in education research. In T. G. Halle, A. J. Metz, I. Martinez-Beck (Eds.), Applying implementation science in early childhood programs and systems (pp. 65-93). Baltimore, MD: Brookes.
39. Juel, C., Biancarosa, G., Coker, D., Deffes, R. (2003) Walking with Rosie: A cautionary tale of early reading instruction. Educational Leadership, 60, 12-18.
40. Justice, L. M., Mashburn, A., Pence, K. L., Wiggins, A. (2008) Experimental evaluation of a preschool language curriculum: Influence on children's expressive language skills. Journal of Speech, Language, and Hearing Research, 51, 983-1001. http://dx.doi.org/10.1044/1092- 4388(2008/072)
41. Justice, L. M., McGinty, A. (2012) Early literacy intervention intensity and its relation to child outcomes. In C. Howes, B. Hamre, R. Pianta (Eds.), Effective early childhood professional development: Improving teacher practice and child outcomes (pp. 89-110). Baltimore, MD: Brookes.
42. Leyva, D., Weiland, C., Barata, M., Yoshikawa, H., Snow, C., Treviño, E., Rolla, A. (2015) Teacher-child interactions in Chile and their associations with prekindergarten outcomes. Child Development. Advance online publication. http://dx.doi.org/10.1111/cdev.12342
43. Maldonado-Carreño, C., Votruba-Drzal, E. (2013, March). Children's early educational experiences in private settings: Evidence from Bogotá, Colombia. Presentation at the ProLEER Annual Meeting, Cambridge, MA.
44. Mascareño, M. (2012) Coding scheme for the analysis of moment-bymoment teacher-child(ren) interactions. Unpublished document. The Netherlands: GION, University of Groningen.
45. Mashburn, A. J., Pianta, R. C., Hamre, B. K., Downer, J. T., Barbarin, O. A., Bryant, D., . . . Howes, C. (2008) Measures of classroom quality in prekindergarten and children's development of academic, language, and social skills. Child Development, 79, 732-749. http://dx.doi.org/ 10.1111/j.1467-8624.2008.01154.x
46. Mattera, S., Lloyd, C., Fishman, M. (2013, April). Dosage and quality of implementation in the context of scale-up: The Head Start CARES trial. Presentation at the biannual meeting of the Society for Research in Child Development, Seattle, WA.
47. McDonald, J. F., Moffitt, R. A. (1980) The uses of Tobit analysis. The Review of Economics and Statistics, 62, 318-321. http://dx.doi.org/ 10.2307/1924766
48. McGinty, A. S., Breit-Smith, A., Fan, X., Justice, L. M., Kaderavek, J. N. (2011) Does intensity matter? Preschoolers' print knowledge development within a classroom-based intervention. Early Childhood Research Quarterly, 26, 255-267. http://dx.doi.org/10.1016/j.ecresq .2011.02.002
49. Ministerio de Educación. (2014) Estado del arte de educación parvularia en Chile [State of the art of early childhood education in Chile]. Santiago, Chile: Author.
50. Murnane, R., Willett, J. (2011) Methods matter: Improving causal inference in educational research. New York, NY: Oxford University Press.
51. National Early Literacy Panel. (2008) Developing early literacy: Report of the National Early Literacy Panel. Washington, DC: National Institute for Literacy.
52. Neuman, S. B. (2011) The challenge of teaching vocabulary in early education. In S. B. Neuman & D. K. Dickinson (Eds.), Handbook of early literacy research (Vol. 3, pp. 358-372). New York, NY: Guilford Press.
53. Odom, S. L., Fleming, K., Diamond, K., Lieber, J., Hanson, M., Butera, G., . . . Children's School Success Project. (2010) Examining different forms of implementation and in early childhood curriculum research. Early Childhood Research Quarterly, 25, 314-328. http://dx.doi.org/ 10.1016/j.ecresq.2010.03.001
54. O'Donnell, C. (2008) Defining, conceptualizing, and measuring fidelity of implementation and its relationship to outcomes in K-12 curriculum intervention research. Review of Educational Research, 78, 33-84. http://dx.doi.org/10.3102/0034654307313793
55. Organisation for Economic Co-operation and Development. (2013) Education at a Glance 2013: OECD Indicators, OECD Publishing. http://dx .doi.org/10.1787/eag-2013-en
56. Paris, S. (2011) Developmental differences in early reading skills. In S. B. Neuman & D. K. Dickinson (Eds.), Handbook of early literacy research (Vol. 3, pp. 228-241). New York, NY: Guilford Press.
57. Pianta, R. C., La Paro, K. M., Hamre, B. K. (2008) The classroom assessment manual, pre-K. Baltimore, MD: Brookes.
58. Piza, E. (2012) Using Poisson and negative binomial regression models to measure the influence of risk on crime incident counts. Newark, NJ: Rutgers Center on Public Security.
59. Powell, D. R., Diamond, K. E. (2012) Promoting literacy and language development. In R. C. Pianta, W. S. Barnett, L. M. Justice, S. M. Sheridan (Eds.), Handbook of early childhood education (pp. 194-216). New York, NY: Guilford Press.
60. Powell, D. R., Diamond, K. E., Burchinal, M. R., Koehler, M. J. (2010) Effects of an early literacy professional development intervention on Head Start teachers and children. Journal of Educational Psychology, 102, 299-312. http://dx.doi.org/10.1037/a0017763
61. Rimmele, R. (2004) Videograph. Multimedia-Player zur Kodierung von Videos. Kiel, Germany: 437 IPN.
62. Roth, F. P., Speece, D. L., Cooper, D. H. (2002) A longitudinal analysis of the connection between oral language and early reading. The Journal of Educational Research, 95, 259-272. http://dx.doi.org/10.1080/ 00220670209596600
63. Schady, N. (2013, October). Early childhood development in the Latin America/Caribbean region. Presentation at the Inter-American Development Bank, Washington, DC.
64. Schulte, A. C., Easton, J. E., Parker, J. (2009) Advances in Treatment Integrity Research: Multidisciplinary Perspectives on the Conceptualization, Measurement, and Enhancement of Treatment Integrity. School Psychology Review, 38, 460-475.
65. Simonstein, S., Romo, V., Reveco, O., Peralta, V., Mayorga, L., Berstein, M. (2009) Estudio sobre las necesidades de especialización para la formación inicial y continua de los/las educadores de párvulos que atienden a niños y niñas de 4 a 6 años [Study of the need for specialization in pre- and in-service training for preschool teachers of 4 to 6 year olds]. Report submitted to CPEIP MINEDUC, Santiago, Chile.
66. Snow, C. E., Burns, M. S., Griffin, P. (Eds.). (1998) Preventing reading difficulties in young children. Washington, DC: National Academy Press.
67. Staub, F. C., West, L., Bickel, D. D. (2003) What is content-focused coaching? In L. West & F. C. Staub (Eds.), Content-focused coaching: Transforming mathematics lessons (pp. 1-17). Portsmouth, NH: Heinemann.
68. Stein, M., Berends, M., Fuchs, D., McMaster, K., Saenz, L., Yen, L., . . . Compton, D. L. (2008) Scaling up an early reading program: Relationships among teacher support, fidelity of implementation, and student performance across different sites and years. Educational Evaluation and Policy Analysis, 30, 368-388. http://dx.doi.org/10.3102/ 0162373708322738
69. Strasser, K., Lissi, M. R. (2009) Home and instruction effects on emergent literacy in a sample of Chilean kindergarten children. Scientific Studies of Reading, 13, 175-204. http://dx.doi.org/10.1080/ 10888430902769525
70. Strasser, K., Lissi, M. R., Silva, M. (2009) Gestión del tiempo en 12 salas Chilenas de kindergarten: Recreo, colación y algo de instrucción [Time management in 12 Chilean kindergarten classrooms: Recess, snack and a little teaching]. Psykhe: Revista de la Escuela de Psicología, 18, 85-96. http://dx.doi.org/10.4067/S0718-22282009000100008
71. Tobin, J. (1958) Estimation of relationships for limited dependent variables. Econometrica, 26, 24-36. http://dx.doi.org/10.2307/1907382 Ukrainetz, T. A., Ross, C. L., Harm, H. M. (2009) An investigation of treatment scheduling for phonemic awareness with kindergartners who are at risk for reading difficulties. Language, Speech, and Hearing Services in Schools, 40, 86-100. http://dx.doi.org/10.1044/0161- 1461(2008/07-0077)
72. Ukrainetz, T. A., Ross, C. L., Harm, H. M. (2009) An investigation of treatment scheduling for phonemic awareness with kindergartners who are at risk for reading difficulties. Language, Speech, and Hearing Services in Schools, 40, 86-100. http://dx.doi.org/10.1044/0161- 1461(2008/07-0077)
73. Wasik, B. A., Bond, M. A., Hindman, A. H. (2006) The effects of a language and literacy intervention on Head Start children and teachers. Journal of Educational Psychology, 98, 63-74. http://dx.doi.org/ 10.1037/0022-0663.98.1.63
74. Wasik, B. A., Hindman, A. (2011) Improving vocabulary and preliteracy skills of at-risk preschoolers through teacher professional development. Journal of Educational Psychology, 103, 455-469. http://dx .doi.org/10.1037/a0023067
75. Weiland, C., Yoshikawa, H. (2013) Impacts of a prekindergarten program on children's mathematics, language, literacy, executive function, and emotional skills. Child Development, 84, 2112-2130. http://dx .doi.org/10.1111/cdev.12099
76. Wilson, D. B., Lipsey, M. W. (2001) The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods, 6, 413-429. http://dx.doi.org/10.1037/1082-989X.6.4.413
77. Wilson, S. M., Berne, J. (1999) Teacher learning and the acquisition of professional knowledge: An examination of research on contemporary professional development. Review of Research in Education, 24, 173-209.
78. Woodcock, R. W., Muñoz-Sandoval, A. F., Ruef, M., Alvarado, C. G. (2005) Woodcock Muñoz Language Survey-Revised. Itasca, IL: Riverside.
79. Yoon, K. S., Duncan, T., Lee, S. W.-Y., Scarloss, B., Shapley, K. (2007) Reviewing the evidence on how teacher professional development affects student achievement (Issues & Answers Report, REL 2007-No. 033). Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, Regional Educational Laboratory Southwest. Retrieved from http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/ REL_2007033.pdf
80. Yoshikawa, H., Leyva, D., Snow, C. E., Treviño, E., Barata, M. C., Weiland, C., . . . Arbour, M. C. (2015) Impacts on preschool classroom and child outcomes of a teacher professional development program in Chile. Developmental Psychology, 51, 309-322. http://dx.doi.org/ 10.1037/a0038785