Using experiments to assess nonexperimental comparison-group methods for measuring program effects

Learning More from Social Experiments: Evolving Analytic Approaches

Volumn , Issue , 2006, Pages 173-235

  Bloom, Howard S ^a   Michalopoulos, Charles ^a   Hill, Carolyn J ^b  

^a MDRC ^*  (United States)

^b GEORGETOWN UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84903099890     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (56)

1. Agodini, Roberto, and Mark Dynarski. 2004. "Are Experiments the Only Option? A Look at Dropout Prevention Programs." Review of Economics and Statistics 86(1): 180-94.
2. Amemiya, Takeshi. 1985. Advanced Econometrics. Cambridge, Mass.: Harvard University Press.
3. Ashenfelter, Orley. 1974. "The Effect of Manpower Training on Earnings: Preliminary Results." Industrial Relations Section working paper 60. Princeton: Princeton University.
4. -. 1978. "Estimating the Effects of Training Programs on Earnings." Review of Economics and Statistics 60(1): 47-57.
5. Ashenfelter, Orley, and David Card. 1985. "Using the Longitudinal Structure of Earnings to Estimate the Effect of Training Programs." Review of Economics and Statistics 67(4): 648-60.
6. Barnow, Burt S. 1987. "The Impact of CETA Programs on Earnings: A Review of the Literature." Journal of Human Resources 22(2): 157-93.
7. Barnow, Burt S., Glen G. Cain, and Arthur S. Goldberger. 1980. "Issues in the Analysis of Selectivity Bias." In Evaluation Studies Review Annual, edited by Ernst Stromsdorfer and George Farkas. Volume 5. San Francisco: Sage Publications.
8. Bassi, Laurie J. 1984. "Estimating the Effect of Training Programs with Non-Random Selection." Review of Economics and Statistics 66(1): 36-43.
9. Bell, Stephen H., Larry L. Orr, John D. Blomquist, and Glen G. Cain. 1995. Program Applicants as a Comparison Group in Evaluating Training Programs. Kalamazoo, Mich.: W. E. Upjohn Institute for Employment Research.
10. Bloom, Howard S., and Maureen A. McLaughlin. 1982. CETA Training Programs - Do They Work for Adults? Washington: U.S. Congressional Budget Office and National Commission for Employment Policy.
11. Bloom, Howard S., Larry L. Orr, George Cave, Stephen H. Bell, Fred Doolittle, and Winston Lin. 1997. "The Benefits and Costs of JTPAPrograms: Key Findings from the National JTPA Study." Journal of Human Resources 32(3): 549-76.
12. Bloom, Howard S., Charles Michalopoulos, Carolyn J. Hill, and Ying Lei. 2002. Can Nonexperimental Comparison Group Methods Match the Findings from a Random Assignment Evaluation of Mandatory Welfare-to-Work Programs? New York: MDRC.
13. Campbell, Donald T., and Julian C. Stanley. 1963. Experimental and Quasi-Experimental Design for Research. Chicago: Rand McNally.
14. Commission on Behavioral and Social Sciences and Education. 1985. Youth Employment and Training Programs: The YEDPA Years, edited by Charles L. Betsey, Robinson G. Hollister, and Mary R. Papageorgiou. Washington, D.C.: National Academy Press.
15. Cook, Thomas D., and Donald T. Campbell. 1979. Quasi-Experimentation: Design and Analysis Issues for Field Settings. Chicago: Rand McNally.
16. Couch, Kenneth. 1992. "New Evidence on the Long-Term Effects of Employment and Training Programs." Journal of Labor Economics 10(4): 380-88.
17. Dehejia, Rajeev H., and Sadek Wahba. 1999. "Causal Effects in Nonexperimen-tal Studies: Reevaluating the Evaluation of Training Programs." Journal of the American Statistical Association 94(448): 1053-62.
18. -. 2002. "Propensity Score Matching Methods for Nonexperimental Causal Studies." Review of Economics and Statistics 84(1): 151-61.
19. Director, Steven M. 1979. "Underadjustment Bias in the Evaluation of Manpower Training." Evaluation Review 3(May): 190-218.
20. Fraker, Thomas M., and Rebecca A. Maynard. 1987. "The Adequacy of Comparison Group Designs for Evaluations of Employment-Related Programs." Journal of Human Resources 22(2): 194-227.
21. Friedlander, Daniel, and Philip K. Robins. 1995. "Evaluating Program Evaluations: New Evidence on Commonly Used Nonexperimental Methods." American Economic Review 85(4): 923-37.
22. Glass, Gene V. 1976. "Primary, Secondary, and Meta-Analysis of Research." Educational Researcher 5(10): 3-8.
23. Glazerman, Steven, Dan M. Levy, and David Myers. 2003. "Nonexperimental versus Experimental Estimates of Earnings Impacts." Annals of the American Academy of Political and Social Science 589(September): 63-93.
24. Goldstein, Jon. 1972. The Effectiveness of Manpower Training Programs: A Review of Research on the Impact on the Poor. Washington: U.S. Government Printing Office.
25. Gronau, Reuben. 1974. "Wage Comparisons - A Selectivity Bias." Journal of Political Economy 82(6): 1119-43.
26. Hamilton, Gayle, and Thomas Brock. 1994. The JOBS Evaluation: Early Lessons from Seven Sites. Washington: U.S. Department of Health and Human Services and U.S. Department of Education.
27. Hamilton, Gayle, Stephen Freedman, Lisa Gennetian, Charles Michalopoulos, Johanna Walter, Diana Adams-Ciardullo, Anna Gassman-Pines, Sharon McGroder, Martha Zaslow, Jennifer Brooks, and Surjeet Ahluwalia. 2001. How Effective Are Different Welfare-to-Work Approaches? Five-Year Adult and Child Impacts for Eleven Programs. Washington: U.S. Department of Health and Human Services and U.S. Department of Education.
28. Heckman, James J. 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models."" Annals of Economic and Social Measurement 5(4): 475-92.
29. -. 1978. "Dummy Endogenous Variables in a Simultaneous Equation System." Econometrica 46(4): 931-59.
30. Heckman, James J., and V. Joseph Hotz. 1989. "Choosing Among Alternative Nonexperimental Methods for Estimating the Impact of Social Programs: The Case of Manpower Training." Journal of the American Statistical Association 84(408): 862-74.
31. Heckman, James J., Hidehiko Ichimura, Jeffrey Smith, and Petra Todd. 1998. "Characterizing Selection Bias Using Experimental Data." Econometrica 66(5): 1017-98.
32. Heckman, James J., Hidehiko Ichimura, and Petra Todd. 1997. "Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Program." Review of Economic Studies 64(4): 605-54.
33. -. 1998. "Matching as an Econometric Evaluation Estimator." Review of Economic Studies 65(2): 261-94.
34. Heinsman, Donna T., and William R. Shadish. 1996. "Assignment Methods in Experimentation: When Do Nonrandomized Experiments Approximate Answers from Randomized Experiments?" Psychological Methods 1(2): 154-69.
35. Hirano, Keisuke, Guido W. Imbens, and Geert Ridder. 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score." Econometrica 71(4): 1161-89.
36. Hollister, Robinson G., and Jennifer Hill. 1995. "Problems in the Evaluation of Community-Wide Initiatives." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, edited by James P. Connell, Anne C. Kubisch, Lisbeth B. Schorr, and Carol H. Weiss. Washington, D.C.: Aspen Institute.
37. Hotz, V. Joseph, Guido W. Imbens, and Jacob A. Klerman. 2000. "The LongTerm Gains from GAIN: A Re-Analysis of the Impacts of the California GAIN Program." NBER working paper W8007. Cambridge, Mass.: National Bureau of Economic Research.
38. Hsiao, Cheng. 1990. Analysis of Panel Data. Cambridge: Cambridge University Press.
39. Joreskog, Karl G., and Dag Sorbom. 1988. LISREL 7: A Guide to the Program and Applications. Chicago: SPSS Inc.
40. LaLonde, Robert J. 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data." American Economic Review 76(4): 604-20.
41. Lipsey, Mark W., and David B. Wilson. 1993. "The Efficacy of Psychological, Educational, and Behavioral Treatment." American Psychologist 48(12): 1181-1209.
42. Maddala, G. S., and Lung-Fei Lee. 1976. "Recursive Models with Qualitative Endogenous Variables." Annals of Economic and Social Measurement 5(4): 525-45.
43. Michalopoulos, Charles, Howard S. Bloom, and Carolyn J. Hill. 2004. "Can Propensity-Score Methods Match the Findings from a Random Assignment Experiment of Mandatory Welfare-to-Work Programs?" Review of Economics and Statistics 86(1): 156-79.
44. Muthén, Bengt. 1994. "Multilevel Covariance Structure Analysis." Sociological Methods and Research 22(3): 376-98.
45. Muthén, Linda, and Bengt Muthén. 1998. Mplus User's Guide. Los Angeles: Muthén & Muthén.
46. Raudenbush, Stephen, and Anthony Bryk. 2002. Hierarchical Linear Models: Applications and Data Analysis. 2nd edition. Thousand Oaks, Calif.: Sage.
47. Rosenbaum, Paul R., and Donald B. Rubin. 1983. "The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika 70(1): 41-55.
48. Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.
49. Smith, Jeffrey, and Petra Todd. 2005. "Does Matching Overcome LaLonde's Critique of Nonexperimental Estimators?" Journal of Econometrics 125(1-2): 305-53
50. Smith, Mary Lee, Gene V. Glass, and Thomas I. Miller. 1980. The Benefits of Psychotherapy. Baltimore: Johns Hopkins University Press.
51. Stromsdorfer, Ernst, Howard Bloom, Robert Boruch, Michael Borus, Judith Gueron, Alan Gustman, Peter Rossi, Fritz Scheuren, Marshall Smith, and Frank Stafford. 1985. Recommendations of the Job Training Longitudinal Survey Research Advisory Panel. Washington: U.S. Department of Labor.
52. U.S. Department of Labor. Bureau of Labor Statistics. 2002. Consumer Price Index. Available at: www.bls.gov/cpi (accessed January 5, 2005).
53. Vella, Francis. 1998. "Estimating Models with Sample Selection Bias: A Survey." Journal of Human Resources 33(1): 127-69.
54. Wilde, Elizabeth Ty, and Robinson Hollister. 2002. "How Close Is Close Enough? Testing Nonexperimental Estimates of Impact Against Experimental Estimates of Impact with Education Test Scores as Outcomes." Discussion paper 1242-02. University of Wisconsin, Institute for Research on Poverty. Available at: www.ssc.wisc.edu/irp/pubs/dp124202.pdf (accessed January 5, 2005).
55. Winship, Christopher, and Stephen L. Morgan. 1999. "The Estimation of Causal Effects from Observational Data." Annual Review of Sociology 25: 659-707.
56. Zhao, Zhong. 2004. "Using Matching to Estimate Treatment Effects: Data Requirements, Matching Metrics, and Monte Carlo Evidence." Review of Economics and Statistics 86(1): 91-107.