External Validity in Policy Evaluations That Choose Sites Purposively

Journal of Policy Analysis and Management

Volumn 32, Issue 1, 2013, Pages 107-121

  Olsen, Robert B ^a   Orr, Larry L ^b   Bell, Stephen H ^a   Stuart, Elizabeth A ^c  

^a ABT ASSOCIATES   (United States)

^b JOHNS HOPKINS UNIVERSITY   (United States)

^c JOHNS HOPKINS BLOOMBERG SCHOOL OF PUBLIC HEALTH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84871975599     PISSN: 02768739     EISSN: 15206688     Source Type: Journal    
DOI: 10.1002/pam.21660     Document Type: Article

References (37)

1. Bloom, H. S., Michalopoulos, C., & Hill, C. J. (2005). Using experiments to assess nonexperimental comparison-group methods for measuring program effects. In H. S. Bloom (Ed.), Learning more from social experiments (pp. 173-235). New York: Russell Sage Foundation.
2. Bos, J. M., & Fellerath, V. (1997). Final report on Ohio's welfare initiative to improve school attendance among teenage parents: Ohio's Learning, Earning, and Parenting Program. New York: Manpower Demonstrative Research Corporation.
3. Brick, J. M., & Jones, M. E. (2008). Propensity to respond and nonresponse bias. Metron-International Journal of Statistics, 6, 51-73.
4. Cole, S. R., & Stuart, E. A. (2010). Generalizing evidence from randomized clinical trials to target populations: The ACTG-320 trial. American Journal of Epidemiology, 172, 107-115.
5. Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which experiments and observational studies produce comparable causal estimates: New findings from within-study comparisons. Journal of Policy Analysis and Management, 27, 724-750.
6. Corson, W., Decker, P., Dunstan, S., & Kerachsky, S. (1991). Pennsylvania reemployment bonus demonstration. Princeton, NJ: Mathematica Policy Research.
7. Fraker, T., & Maynard, R. (1987). The adequacy of comparison group designs for evaluations of employment-related programs. Journal of Human Resources, 22, 194-227.
8. Freedman, S., Friedlander, D., Hamilton, G., Rock, J., Mitchell, M., Nudelman, J., Schweder, A., & Storto, L. (2000). Evaluating alternative welfare-to-work approaches: Two-year impacts for eleven programs. Washington, DC: U.S. Department of Health and Human Services and the U.S. Department of Education.
9. Glazerman, S., Levy, D. M., & Myers, D. (2003). Nonexperimental versus experimental estimates of earnings impacts. The Annals of the American Academy of Political and Social Science, 589, 63-93.
10. Gleason, P., Clark, M., Tuttle, C. C., & Dwoyer, E. (2010). The evaluation of charter school impacts: Final report NCEE 2010-4029. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
11. Greenberg, D., & Shroder, M. (2004). The digest of social experiments. Washington, DC: The Urban Institute Press.
12. Greenberg, D., Meyer, R., Michaelopoulos, C., & Wiseman, M. (2003). Explaining variation in the effects of welfare-to-work programs. Evaluation Review, 27, 359-394.
13. Hamilton, G., & Brock, T. (1994). Early lessons from seven sites. Washington, DC: U.S. Department of Health and Human Services and the U.S. Department of Education.
14. Haneuse, S., Schildcrout, J., Crane, P., Sonnen, J., Breitner, J., & Larson, E. (2009). Adjustment for selection bias in observational studies with application to the analysis of autopsy data. Neuroepidemiology, 32, 229-239.
15. Heckman, J. J. (1976). The common structure of statistical models of truncation, sample selection, and limited independent variables. Annals of Economic and Social Measurement, 5, 120-137.
16. Imai, K., King, G., & Stuart, E. A. (2008). Misunderstandings between experimentalists and observationalists about causal inference. Journal of the Royal Statistical Society, Series A, 171, 481-502.
17. Kauff, J., Fraker, T., & Milliner-Waddell, J. (2002). Iowa families that left TANF: How are they faring two years later? Washington, DC: Mathematica Policy Research, Inc.
18. Kauff, J., Olsen, R., & Fraker, T. (2002). Nonrespondents and nonresponse bias: Evidence from a survey of former welfare recipients in Iowa. Washington, DC: Mathematica Policy Research, Inc.
19. Kemple, J. J., Friedlander, D., & Fellerath, V. (1995). Florida's project independence: Benefits, costs, and two-year impacts of Florida's JOBS program. New York: Manpower Demonstrative Research Corporation.
20. LaLonde, R. (1986). Evaluating the econometric evaluations of training with experimental data. American Economic Review, 76, 604-620.
21. Little, R. J. A. (1986). Survey nonresponse adjustments. International Statistical Review, 54, 139-157.
22. Nisar, H. (2010). Do charter schools improve student achievement? Unpublished Working Paper. Retrieved June 21, 2012, from http://www.ssc.wisc.edu/~scholz/Seminar/Charter_School_MPS.pdf.
23. Oh, H. L., & Scheuren, F. J. (1983). Weighting adjustment for unit nonresponse. Incomplete Data in Sample Surveys, 2, 143-184.
24. Orr, L., Feins, J. D. Jacob, R., Beecroft, E., Sanbonmatsu, L., Katz, L. F., Liebman, J. B., & Kling, J. R. (2003). Moving to opportunity interim impacts evaluation: Final report. Washington, DC: U.S. Department of Housing and Urban Development.
25. Pan, Q., & Schaubel, D. E. (2009). Evaluating bias correction in weighted proportional hazards regression. Lifetime Data Analysis, 15, 120-146.
26. Puma, M. J., Burstein, N. R., Merrill, K., & Silverstein, G. (1990). Evaluation of the Food Stamp Employment and Training Program. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service, Office of Analysis and Evaluation.
27. Puma, M., Bell, S., Cook, R., & Heid, C. (2010). Head Start Impact Study final report. Washington, DC: Administration for Children and Families, U.S. Department of Health and Human Services.
28. Riccio, J., Friedlander, D., & Freedman, S. (1994). GAIN: Benefits, costs, and three-year impacts of a welfare-to-work program. New York: MDRC.
29. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581-592.
30. Rubin, D. B. (2001). Using propensity scores to help design observational studies: Application to the tobacco litigation. Health Services & Outcomes Research Methodology, 2, 169-188.
31. Seftor, N. S., Mamun, A., & Schirm, A. (2009). The impacts of regular Upward Bound on postsecondary outcomes 7-9 years after scheduled high school graduation: Final report. Washington, DC: Mathematica Policy Research.
32. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin Company.
33. Stapleton, D., Bell, S., Wittenberg, D., Sokol, S., & McInnis, D. (2010). BOND final design report. Bethesda, MD: Abt Associates Inc.
34. Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science, 25, 1-21.
35. Stuart, E. A., Cole, S. R., Bradshaw, C. P., & Leaf, P. J. (2011). The use of propensity scores to assess the generalizability of results from randomized trials. The Journal of the Royal Statistical Society, Series A, 174, 368-386.
36. Watts, H. W., Peck, J. K., & Taussig, M. (1977). Site selection, representativeness of the sample, and possible attrition bias. In H. W. Watts & A. Rees (Eds.), The New Jersey income maintenance experiment (Vol. 3, pp. 441-66). New York: Academic Press.
37. Weisberg, H., Hayden, V., & Pontes, V. (2009). Selection criteria and generalizability within the counterfactual framework: Explaining the paradox of antidepressant-induced suicidality? Clinical Trials, 6, 109-118.