A Design-Based Approach to Improve External Validity in Welfare Policy Evaluations

Evaluation Review

Volumn 41, Issue 4, 2017, Pages 326-356

  Tipton, Elizabeth ^a   Peck, Laura R ^b  

^a Howard Hughes Medical Institute   (United States)

^b ABT ASSOCIATES   (United States)

Author keywords

content area; job training; methodological development

Indexed keywords

CLUSTER ANALYSIS; EVALUATION STUDY; METHODOLOGY; PUBLIC POLICY; REPRODUCIBILITY; SOCIAL WELFARE;

CLUSTER ANALYSIS; EVALUATION STUDIES AS TOPIC; PUBLIC POLICY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SOCIAL WELFARE;

EID: 85025451379     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X16655656     Document Type: Article

References (31)

1. Aldenderfer M. S., Blashfield R. K., (1984). Cluster analysis. Beverly Hills, CA: Sage.
2. Bell S. H., Olsen R. B., Orr L. L., Stuart E. A., (2016). Estimates of external validity bias when impact evaluations select sites purposively. Educational Evaluation and Policy Analysis, 38, 318–335.
3. Bureau of Labor Statistics. (2012a). Employment Projections [data file and code book]. Retrieved from http://data.bls.gov/projections/occupationProj
4. Bureau of Labor Statistics. (2012b). Unemployment Estimate [data file and code book]. Retrieved from http://www.bls.gov/lau/lastrk13.htm
5. Bureau of Labor Statistics. (2014). May 2013 State Occupational Employment and Wage Estimates [data file]. Retrieved from http://www.bls.gov/oes/current/oessrcst.htm
6. Card D., Kluve J., Weber A., (2010). Active labour market policy evaluations: A meta-analysis. The Economic Journal, 120, F452–F477. doi:10.1111/j.1468-0297.2010.02387.x
7. Cook T. D., (2014). Generalizing causal knowledge in the policy sciences: External validity as a task of both multiattribute representation and multiattribute extrapolation. Journal of Policy Analysis and Management, 33, 527–536.
8. Gower J. C., (1971). A general coefficient of similarity and some of its properties. Biometrics, 27, 857–872.
9. Greenberg D., Shroder M., (2004). The digest of social experiments (3rd ed.). Washington, DC: The Urban Institute Press.
10. Hamilton G., (2002). Moving people from welfare to work: Lessons from the National Evaluation of Welfare-to-Work Strategies. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning and Evaluation and U.S. Department of Education Office of the Under Secretary, Office of Vocational and Adult Education.
11. Hamilton G., Brock T., Farkas J., (1994). The JOBS evaluation: Early lessons from seven sites. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of the Assistant Secretary for Planning Evaluation and U.S. Department of Education, Office of the Under Secretary, Office of Vocational and Adult.
12. Job search assistance (JSA) strategies evaluation; Notice of proposed information collection activity; Comment request. (2015, 7 15). Federal Register, 80, 41505–41506.
13. Kassabian D., Huber E., Cohen E., Giannarelli L., (2013). Welfare rules databook: State TANF policies as of July 2012: Final Report, Table I.A.2 (pp.42-44) & Table L7 (pp. 214–220). Washington, DC: The Urban Institute.
14. Liu S., Huang J. L., Wang M., (2014). Effectiveness of job search interventions: A meta-analytic review. Psychological Bulletin, 140, 1009–1041.
15. Maechler M., Rousseeuw P., Struyf A., Hubert M., Hornik K., (2015). Cluster analysis basics and extensions. R package version 2.0.1.
16. Nisar H., Klerman J. A., Juras R., (2012). Estimation of intra class correlation in job training programs (working paper). Bethesda, MD: Abt Associates.
17. Olsen R. B., Orr L. L., Bell S. H., Stuart E. A., (2013). External validity in policy evaluations that choose sites purposively. Journal of Policy Analysis and Management, 32, 107–121.
18. O’Muircheartaigh C. A., Hedges L. V., (2014). Generalizing from unrepresentative experiments: A stratified propensity score approach. Journal of the Royal Statistical Society: Series C (Applied Statistics), 63, 195–210.
19. Orr L. L., (2015). 2014 Rossi Award lecture: Beyond internal validity. Evaluation Review, 39, 167–178. doi:10.1177/0193841X15573659
20. Peck L. R., Mayo A., (2011, 11 3). An empirical assessment of external validity. Presented at the Association for Public Policy Analysis and Management (APPAM) Conference (Panel: External Validity in Randomized Experiments), Washington, DC.
21. R Development Core Team. (2014). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. ISBN 3-900051-07-0. Retrieved from http://www.R-project.org/
22. Raudenbush S. W., Bryk A. S., (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Volume 1 of Advanced Quantitative Techniques in the Social Sciences. Thousand Oaks, CA: Sage.
23. Stuart E. A., Cole S. R., Bradshaw C. P., Leaf P. J., (2011). The use of propensity scores to assess the generalizability of results from randomized trials. Journal of the Royal Statistical Society, Series A, Part 2, 764, 369–386.
24. Tipton E., (2013). Improving generalizations from experiments using propensity score subclassification: Assumptions, properties, and contexts. Journal of Educational and Behavioral Statistics, 38, 239–266.
25. Tipton E., (2014a). Stratified sampling using cluster analysis: A balanced-sampling strategy for improved generalizations from experiments. Evaluation Review, 37, 109–139.
26. Tipton E., (2014b). How generalizable is your experiment? Comparing a sample and population through a generalizability index. Journal of Educational and Behavioral Statistics, 39, 478–501.
27. Tipton E., Hedges L. V., Vaden-Kiernan M., Borman G. D., Sullivan K., Caverly S., (2014). Sample selection in randomized experiments: A new method using propensity score stratified sampling. Journal of Research on Educational Effectiveness, 7, 114–135.
28. U.S. Census Bureau. (2010a). 2010 Urban and rural classification and urban area criteria [data file and code book]. Retrieved from https://www.census.gov/geo/reference/ua/urban-rural-2010.html
29. U.S. Census Bureau. (2010b). American Fact Finder Survey: Community facts—Education [data file]. Retrieved from http://factfinder2.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t
30. Valliant R., Dorfman A. H., Royall R. M., (2000). Finite population sampling and inference: A prediction approach (Wiley Series in Probability and Statistics). New York, NY: John Wiley & Sons.
31. Weaver R. K., (2000). Ending welfare as we know it. Washington, DC: Brookings Institution Press.