Can Broad Inferences Be Drawn from Lottery Analyses of School Choice Programs? An Exploration of Appropriate Sensitivity Analyses

Journal of School Choice

Volumn 10, Issue 1, 2016, Pages 48-72

  Zimmer, Ron ^a   Engberg, John ^b  

^a VANDERBILT UNIVERSITY   (United States)

^b RAND CORPORATION   (United States)

Author keywords

inferences; school choice

Indexed keywords

EID: 84961212268     PISSN: 15582159     EISSN: 15582167     Source Type: Journal    
DOI: 10.1080/15582159.2015.1132936     Document Type: Article

References (38)

1. A.Abdulkadiroglu,, J.Angrist,, S.Cohodes,, S.Dynarski,, J.Fullerton,, T.Kane,, & P.Pathak, (2009). Informing the debate: Comparing Boston’s charter, pilot and traditional schools. Boston, MA: The Boston Foundation.
2. Alliance for School Choice. (2012). School choice now: The year of school choice school choice yearbook 2011–12. Washington, DC: Alliance for School Choice.
3. J.G.Altonji,, T.E.Elder,, & C.R.Taber, (2005). Selection on observed and unobserved variables: Assessing the effectiveness of Catholic Schools. Journal of Political Economy, 113, 151–184. doi:10.1086/426036
4. J.Angrist,, E.Bettinger,, E.Bloom,, M.Kremer,, & E.King, (2002). The effects of school vouchers on students: Evidence from Colombia. American Economic Review, 92(5), 1535–1558. doi:10.1257/000282802762024629
5. J.Angrist,, E.Bettinger,, & M.Kremer, (2006). Long-term educational consequences of secondary school vouchers: Evidence from administrative records in Colombia. American Economic Review, 96(3), 847–862. doi:10.1257/aer.96.3.847
6. D.Ballou,, E.Goldring,, & K.Liu, (2006). Magnet schools and student achievement (Working Paper). Vanderbilt University. Retrieved from http://ncspe.org/publications_files/OP123.pdf
7. E.Bettinger,, M.Kremer,, & J.Saavedra, (2006). How educational vouchers affect students—The role of vocational high schools in Colombia. Cleveland, OH: Case Western Reserve mimeo.
8. E.Bettinger,, & R.Slonim, (2006). Using experimental economics to measure the effects of a natural educational experiment on Altruism. Journal of Public Economics, 90(8–9), 1625–1648. doi:10.1016/j.jpubeco.2005.10.006
9. R.Bifulco, (2012). Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management, 31(3), 729–751. doi:10.1002/pam.20637
10. R.Bifulco,, C.Cobb,, & C.Bell, (2009). Can interdistrict choice boost achievement: Evidence from Connecticut’s interdistrict magnets. Educational Evaluation and Policy Analysis, 323–345. doi:10.3102/0162373709340917
11. H.Bloom,, & R.Unterman, (2014). Can small high schools of choice improve educational prospects for disadvantaged students? Journal Policy Analysis and Management, 33(2), 290–319. doi:10.1002/pam.21748
12. J.B.Cullen,, & B.A.Jacob, (2009). Is gaining access to selective elementary schools gaining ground? Evidence from randomized lotteries. In J.Gruber (Ed), The problems of disadvantage youth: An economic perspective (pp. 43–84). Chicago, IL: University of Chicago Press.
13. J.B.Cullen,, B.A.Jacob,, & S.Levitt, (2006). The effect of school choice on participants: Evidence from randomized lotteries. Econometrica, 74(5), 1191–1230. doi:10.1111/ecta.2006.74.issue-5
14. D.Deming,, J.Hastings,, T.Kane,, & D.Staiger, (2011). School choice, school quality and academic achievement. Retrieved from http://aefpweb.org/sites/default/files/webform/DHKS_March2011.pdf
15. W.Dobbie,, & R.G.Fryer, (2011). Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem Children’s Zone. American Economic Journal: Applied Economics, 3(3), 158–187. doi:10.1257/app.3.3.158
16. J.Engberg,, D.Epple,, J.Imbrogno,, H.Sieg,, & R.Zimmer, (2014). Identification and estimating of treatment effects when program participation is partially determined by lotteries: The case of magnet programs. Journal of Labor Economics, 32(1), 2763.
17. K.Frank, (2000). Impact of a confounding variable on a regression coefficient. Sociological Methods & Research, 29(2), 147–194. doi:10.1177/0049124100029002001
18. P.Gleason,, M.Clark,, M.C.Tuttle,, & E.Dwoyer, (2010, June). The evaluation of charter school impacts: Final Report (NCEE 2010–4029). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
19. J.P.Greene, (2000). The effect of school choice: An evaluation of the Charlotte children’s scholarship fund. New York, NY: Center for Civic Innovation at the Manhattan Institute.
20. J.S.Hastings,, T.J.Kane,, & D.O.Staiger, (2006a). Gender and performance: Evidence from school assignment by randomized lottery. American Economic Review Papers and Proceedings, 96(2), 232–236. doi:10.1257/000282806777211919
21. J.S.Hastings,, T.J.Kane,, & D.O.Staiger, (2006b). Preferences and heterogeneous treatment of effects in a public school choice lottery (NBER Working Paper 12145). Cambridge, MA: National Bureau of Economic Research.
22. J.S.Hastings,, & J.M.Weinstein, (2008). Information, school choice, and academic achievement: Evidence from two experiments. Quarterly Journal of Economics, 123(4), 1373–1414. doi:10.1162/qjec.2008.123.issue-4
23. L.V.Hedges,, & C.A.O’Muircheartaigh, (2011). Improving generalizations from designed experiments. Evanston, IL: Northwestern University.
24. W.G.Howell,, & P.E.Peterson, (2002). The education gap: Vouchers and urban schools. Washington, DC: Brookings Institute.
25. C.Hoxby,, J.Kang,, & S.Murarka, (2009). Technical report: How New York City’s charter schools affect achievement. New York City Charter Schools Evaluation Project. Retrieved from http://users.nber.org/~schools/charterschoolseval/how_nyc_charter_schools_affect_achievem ent_technical_report_2009.pdf
26. C.M.Hoxby,, & S.Murarka, (2006). Methods of assessing the achievement of students in charter schools. National Conference on Charter School Research, Vanderbilt University, Nashville, TN. Retrieved from http://www.vanderbilt.edu/schoolchoice/conference/papers/Hoxby-Murarka_2006DRAFT.pdf
27. C.M.Hoxby,, & J.Rockoff, (2004). The impact of charter schools on student achievement (Working Paper). Cambridge, MA: Harvard University.
28. Illinois State Board of Education. (2012). 2009–2010 and 2010–2011 Illinois Charter School Biennial Report. Retrieved from http://www.isbe.state.il.us/charter/pdf/biennial_rpt_09-10_10-11.pdf
29. A.B.Krueger,, & P.Zhu, (2003). Another look at the New York City school voucher experiment (NBER Working Paper 9418). Cambridge, MA: National Bureau of Economic Research.
30. H.Levin,, & C.Belfield, (2002). “The effects of competition on educational outcomes: A review of U.S. evidence,” with C. Belfield. Review of Educational Research, 72(2), 279–341. doi:10.3102/00346543072002279
31. D.Meyers,, P.Peterson,, D.Mayer,, J.Chou,, & W.G.Howell, (2000). School choice in New York City after two years: An evaluation of the school choice scholarship program. Washington, DC: Mathematica Policy Research.
32. E.A.Stuart,, S.R.Cole,, C.P.Bradshaw,, & P.J.Leaf, (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society: Series A (Statistics in Society), 174, 369–386. doi:10.1111/j.1467985X.2010.00673.x
33. E.Tipton, (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
34. C.Tuttle,, B.Gill,, P.Gleason,, V.Knechtel,, I.Nichols-Barrer,, & A.Resch, (2013). KIPP middle schools: Impacts on achievement and other outcomes. Washington, DC: Mathematica Policy Research.
35. C.Tuttle,, P.Gleason,, & M.Clark, (2012). Using lotteries to evaluate schools of choice: Evidence from a national study of charter schools. Economics of Education Review, 31, 237–253. doi:10.1016/j.econedurev.2011.07.002
36. P.Wolf,, B.Gutmann,, M.Puma,, B.Kisida,, L.Rizzo,, N.Eissa,, & M.Carr, (2010). Evaluation of the DC opportunity scholarship program: Final report (NCEE 2010-4018). U.S. Department of Education, Institute for Education Sciences, National Center for Education Evaluation and Regional Assistance. Washington, DC: U.S. Government Printing Office. Retrieved from http://ies.ed.gov/ncee/pubs/20104018/pdf/20104018.pdf
37. P.Wolf,, B.Kisida,, B.Gutmann,, M.Puma,, N.Eissa,, & L.Rizzo, (2013). School vouchers and student outcomes: Experimental evidence from Washington, DC. Journal of Policy Analysis and Management, 32(2), 246–270. doi:10.1002/pam.21691
38. R.Zimmer,, C.Guarino,, & R.Buddin, (2010). School choice: Options and outcomes. In N.Pindus & H.Wolman (Eds), Urban and regional policy and its effect (Vol. 3), pp. 64–107. Washington, DC: Brookings Institution.