Characteristics of School Districts That Participate in Rigorous National Educational Evaluations

Journal of Research on Educational Effectiveness

Volumn 10, Issue 1, 2017, Pages 168-206

  Stuart, Elizabeth A ^a   Bell, Stephen H ^b   Ebnesajjad, Cyrus ^a   Olsen, Robert B ^b   Orr, Larry L ^a  

^a JOHNS HOPKINS UNIVERSITY   (United States)

^b ABT ASSOCIATES   (United States)

Author keywords

external validity; generalizability; randomized experiment

Indexed keywords

EID: 84989822187     PISSN: 19345747     EISSN: 19345739     Source Type: Journal    
DOI: 10.1080/19345747.2016.1205160     Document Type: Article

References (35)

1. Agodini, R., Harris, B., Thomas, M., Murphy, R., & Gallagher, L., (2010). Achievement effects of four early elementary school math curricula:Findings for first and second graders (NCEE 2011-4001). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
2. Bell, S. H., Olsen, R. B., Orr, L. L, & Stuart, E. A., (2016). Estimates of external validity bias when impact evaluations select sites non-randomly. Educational Evaluation and Policy Analysis. Advance online publication. doi:10.3102/0162373715617549
3. Bernstein, L., Dun Rappaport, C., Olsho, L., Hunt, D., & Levin, M., (2009). Impact evaluation of the U.S. Department of Education's Student Mentoring Program:Final report (NCEE 2009-4047). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
4. Betts, J., Kitmitto, S., Levin, J., Bos, J., & Eaton, M., (2015). What happens when schools become magnet schools? A longitudinal study of diversity and achievement (ED-04-CO-0025/0013). Washington, DC:American Institutes for Research.
5. Bifulco, R., & Ladd, H. F., (2006). The impacts of charter schools on student achievement:Evidence from North Carolina. Education, 1(1), 50–90.
6. Black, A. R., Somers, M.-A., Doolittle, F., Unterman, R., & Grossman, J. B., (2009). The evaluation of enhanced academic instruction in after-school programs:Final report (NCEE 2009-4077). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
7. Booker, K., Gilpatric, S. M., Gronberg, T., & Jansen, D., (2007). The impact of charter school attendance on student performance. Journal of Public Economics, 91(5), 849–876.
8. Campuzano, L., Dynarski, M., Agodini, R., & Rall, K., (2009). Effectiveness of reading and mathematics software products:Findings from two student cohorts (NCEE 2009-4041). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
9. Clark, M. A., Chiang, H. S., Silva, T., McConnell, S., Sonnenfeld, K., Erbe, A., & Puma, M., (2013). The effectiveness of secondary math teachers from Teach For America and the Teaching Fellows Programs (NCEE 2013-4015). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
10. Constantine, J., Player, D., Silva, T., Hallgren, K., Grider, M., Deke, J., (2009). An evaluation of teachers trained through different routes to certification:Final report (NCEE 2009-4043). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
11. Cook, T. D., (2014). Generalizing causal knowledge in the policy sciences:External validity as a task of both multi-attribute representation and multi-attribute extrapolation. Journal of Policy Analysis and Management, 33(2), 527–536.
12. Deke, J., Dragoset, L., Bogen, K., & Gill, B., (2012). Impacts of Title I Supplemental Educational Services on student achievement (NCEE 2012-4053). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
13. Dynarski, M., Agodini, R., Heaviside, S., Novak, T., Carey, N., Campuzano, L., … Sussex, W., (2007). Effectiveness of reading and mathematics software products:Findings from the first student cohort (NCEE 2007-4005). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
14. Dynarski, M., James-Burdumy, S., Moore, M., Rosenberg, L., Deke, J., & Mansfield, W., (2004). When schools stay open late:The National Evaluation of the 21st Century Community Learning Centers Program:New findings (ED-99-CO-0134). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
15. Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F., (2008). Reading First Impact Study final report (NCEE 2009-4038). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
16. Glazerman, S., Dolfin, S., Bleeker, M., Johnson, A., Isenberg, E., Lugo-Gil, J., … Ali, M., (2008). Impacts of comprehensive teacher induction:Results from the first year of a randomized controlled study (NCEE 2009-4034). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
17. Glazerman, S., Isenberg, E., Dolfin, S., Bleeker, M., Johnson, A., Grider, M., & Jacobus, M., (2010). Impacts of comprehensive teacher induction:Final results from a randomized controlled study (NCEE 2010-4027). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
18. Glazerman, S., Protik, A., Teh, B., Bruch, J., & Max, J., (2013). Transfer incentives for high-performing teachers:Final results from a multisite experiment (NCEE 2014-4003). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
19. Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E., (2010). The evaluation of charter school impacts:Final report (NCEE 2010-4029). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
20. Isenberg, E., Glazerman, S., Bleeker, M., Johnson, A., Lugo-Gil, J., Grider, M., … Britton, E., (2009). Impacts of comprehensive teacher induction:Results from the second year of a randomized controlled study (NCEE 2009-4072). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
21. James-Burdumy, S., Deke, J., Lugo-Gil, J., Carey, N., Hershey, A., Gersten, R., … Faddis, B., (2010). Effectiveness of selected supplemental reading comprehension interventions:Findings from two student cohorts (NCEE 2010-4015). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
22. James-Burdumy, S., Goesling, B., Deke, J., & Einspruch, E., (2010). The effectiveness of mandatory-random student drug testing (NCEE 2010-4025). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
23. Olsen, R. B., & Orr, L. L. (in press). On the “where” of social experiments:Selecting more representative samples to inform policy. New Directions for Evaluation, 2016(152).
24. Olsen, R. B., Orr, L. L., Bell, S. H., & Stuart, E. A., (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32(1), 107–121.
25. Puma, M., Bell, S., Cook, R., Heid, C., Shapiro, G., Broene, P., … Spier, E., (2010). Head Start Impact Study:Final report. Washington, DC:Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
26. Schochet, P. Z., Puma, M., & Deke, J., (2014). Understanding variation in treatment effects in education impact evaluations:An overview of quantitative methods (NCEE 2014-4017). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
27. Seftor, N. S., Mamun, A., & Schirm, A., (2009). The impacts of regular Upward Bound on postsecondary outcomes seven to nine years after scheduled high school graduation:Final report. Princeton, NJ:Mathematica Policy Research.
28. Silvia, S., Blitstein, J., Williams, J., Ringwalt, C., Dusenbury, L., & Hansen, W., (2011). Impacts of a violence prevention program for middle schools:Findings after 3 years of implementation (NCEE 2011-4017). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
29. Somers, M. A., Corrin, W., Sepanik, S., Salinger, T., Levin, J., & Zmach, C., (2010). The enhanced reading opportunities study final report:The impact of supplemental literacy courses for struggling ninth-grade readers (NCEE 2010-4021). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
30. Stuart, E. A., Bradshaw, C. P., & Leaf, P. J., (2015). Assessing the generalizability of randomized trial results to target populations. Prevention Science, 16(3), 475–485.
31. Tipton, E., (2014). How generalizable is your experiment? An index for comparing experimental samples and populations. Journal of Educational and Behavioral Statistics, 39(6), 478–501.
32. Tipton, E., Hedges, L., Vaden-Kiernan, M., Borman, G., Sullivan, K., & Caverly, S., (2014). Sample selection in randomized experiments:A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7(1), 114–135.
33. Torgesen, J., Schirm, A., Castner, L., Vartivarian, S., Mansfield, W., Myers, D., … Haan, C., (2007). National assessment of Title I final report—Volume II:Closing the reading gap, Findings from a randomized trial of four reading interventions for striving readers (NCEE 2008-4013). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
34. Weiss, M. J., Bloom, H. S., & Brock, T., (2014). A conceptual framework for studying the sources of variation in program effects. Journal of Policy Analysis and Management, 33(3), 778–808.
35. Wolf, P., Gutmann, B., Puma, M., Kisida, B., Rizzo, L., Eissa, N., & Carr, M., (2010). Evaluation of the DC Opportunity Scholarship Program:Final report (NCEE 2010-4018). Washington, DC:National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.