Horseshoes, hand grenades, and treatment effects? Reassessing whether nonexperimental estimators are biased

Economics of Education Review

Volumn 44, Issue , 2015, Pages 100-113

  Fortson, Kenneth ^a   Gleason, Philip ^a   Kopa, Emma ^a   Verbitsky Savitz, Natalya ^a  

^a MATHEMATICA POLICY RESEARCH   (United States)

Author keywords

Nonexperimental methods; Randomized controlled trials; Treatment effects; Within study comparison

Indexed keywords

ESTIMATION METHOD; EXPERIMENTAL STUDY;

EID: 84917706955     PISSN: 02727757     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.econedurev.2014.11.001     Document Type: Article

References (30)

1. Abadie A., Imbens G. On the failure of the bootstrap for matching estimators. Econometrica 2008, 76(6):1537-1557.
2. Abdulkadiroglu A., Angrist J., Dynarski S., Kane T., Pathak P. Accountability and flexibility in public schools: Evidence from Boston's charters and pilots. Quarterly Journal of Economics 2011, 126:699-748.
3. Agodini R., Dynarski M. Are experiments the only option? A look at dropout prevention programs. Review of Economics and Statistics 2004, 86(1):180-194.
4. Angrist J., Cohodes S., Dynarski S., Fullerton J., Kane T., Pathak P., et al. Student achievement in Massachusetts' charter schools 2011, Center for Education Policy Research, Harvard University, Cambridge, MA.
5. Bifulco R. Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison. Journal of Policy Analysis and Management 2012, 31(3):729-751.
6. Bifulco R., Ladd H. The impact of charter schools on student achievement: Evidence from North Carolina. Journal of Education Finance and Policy 2006, 1(1):50-90.
7. Bloom H., Michalopoulos C., Hill C. Using experiments to assess nonexperimental comparison group methods for measuring program effects. Learning more from social experiments 2005, 172-235. Russell Sage Foundation, New York. H. Bloom (Ed.).
8. Cook T., Shadish W., Wong V. Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management 2008, 27(4):724-750.
9. Dehejia R., Wahba S. Causal effects in nonexperimental studies: Reevaluating the evaluation of training programs. Journal of the American Statistical Association 1999, 94:1053-1062.
10. Dobbie W., Fryer R. Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem Children's Zone. American Ecnonomic Journal: Applied Economics 2011, 3:158-187.
11. Fraker T., Maynard R. The adequacy of comparison group designs for evaluations of employment-related programs. Journal of Human Resources 1987, 41:194-227.
12. Friedlander D., Robins P. Evaluating program evaluations: New evidence on commonly used nonexperimental methods. American Economic Review 1995, 85(4):923-937.
13. Glazerman S., Levy D., Myers D. Nonexperimental versus experimental estimates of earnings impacts. Annals of the American Academy 2003, 589:63-93.
14. Gleason P., Clark M., Tuttle C., Dwoyer E. The evaluation of charter school impacts. 2010, National Center for Education Evaluation and Regional Assistance 2010-4029, NCEE, Institute of Education Sciences, U.S. Department of Education, Washington, DC.
15. Hastings, J., Neilson, C., & Zimmerman, S. (2012). The effect of school choice on intrinsic motivation and academic outcomes. National Bureau of Economic Research Working Paper 18324. Cambridge, MA: NBER.
16. Heckman, J., & Todd, P. (2009). A note on adapting propensity score matching and selection models to choice based samples. Institute for Study of Labor Discussion Paper 4304.
17. Hoxby C., Murarka S. Methods of assessing the achievement of students in charter schools: Their growth and outcomes 2007, Lawrence Erlbaum Associates, Mahwah, NJ.
18. Hoxby, C., Murarka, S., & Kang, J. (2009). How New York City's charter schools affect achievement, August 2009 report. Second report in series. Cambridge, MA: New York City Charter Schools Evaluation Project.
19. Imbens G., Wooldridge J. Recent developments in the econometrics of program evaluation. Journal of Economic Literature 2009, 47(1):5-82.
20. Lalonde R. Evaluating the econometric evaluations of training with experimental data. American Economic Review 1986, 76:604-620.
21. Lechner M. An evaluaton of public-sector-sponsored continuous vocational training programs in East Germany. Journal of Human Resources 1999, 35(2):347-375.
22. Little R.J., Rubin D.B. Causal effects in clinical and epidemiological studies via potential outcomes: Concepts and analytical approaches. Annual Review of Public Health 2000, 21:121-145.
23. Peikes D., Moreno L., Orzol S. Propensity score matching: A note of caution for evaluators of social programs. American Statistician 2008, 62(3):222-231.
24. Puma M., Olsen R., Bell S., Price C. What to do when data are missing in group randomized controlled trials. National Center for Education Evaluation and Regional Assistance 2009-0049 2009, NCEE, Institute of Education Sciences, U.S. Department of Education, Washington, DC.
25. Rosenbaum P.R., Rubin D.B. The central role of propensity score in observational studies for causal effects. Biometrika 1983, 70(1):41-55.
26. Schafer J.L., Kang J. Average causal effects from nonrandomized studies: A practical guide and simulated example. Psychological Methods 2008, 13(4):279-313.
27. Shadish W., Clark M., Steiner P. Can nonrandomized experiments yield accurate answers? A randomized experiment comparing random and nonrandom assignments. Journal of the American Statistical Association 2008, 103:1334-1356.
28. Smith J., Todd P. Does matching overcome LaLonde's critique of nonexperimental estimators?. Journal of Econometrics 2005, 125:305-353.
29. Tuttle C., Teh B., Nichols-Barrer I., Gill B., Gleason P. Student characteristics and achievement in 22 KIPP Middle Schools 2010, KIPP Foundation, San Francisco, CA.
30. Wilde E., Hollister R. How close is close enough? Evaluating propensity score matching using data from a class size reduction experiment. Journal of Policy Analysis and Management 2007, 26:455-477.