A conceptual framework for studying the sources of variation in program effects

Journal of Policy Analysis and Management

Volumn 33, Issue 3, 2014, Pages 778-808

  Weiss, Michael J ^a   Bloom, Howard S ^a   Brock, Thomas ^b  

^a MDRC   (United States)

^b National Center for Health Research   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84902544454     PISSN: 02768739     EISSN: 15206688     Source Type: Journal    
DOI: 10.1002/pam.21760     Document Type: Article

References (82)

1. Abadie, A. K., Angrist, J. D., & Imbens, G. (2002). Instrumental variables estimates of the effect of subsidized training on the quantiles of trainee earnings. Econometrica, 70, 91-117.
2. Agodini, R., & Harris, B. (2010). An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness, 3, 199-253.
3. Angrist, J. D., Imbens, G., & Rubin, D. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-455.
4. Angrist, J. D., Dynarski, S. M., Kane, T. J., Pathak, P. A., & Walters, C. R. (2012). Who benefits from KIPP? Journal of Policy Analysis and Management, 31, 837-860.
5. Bardach, E. (1980). The implementation game: What happens after a bill becomes a law. Cambridge, MA: MIT Press.
6. Bitler, M. P., Gelbach, J. B., & Hoynes, H. W. (2006). What mean impacts miss: Distributional effects of welfare reform experiments. American Economic Review, 96, 988-1012.
7. Blakely, C. H., Mayer, J. P., Gottschalk, R. G., Schmitt, N., Davidson, W. S., Roitman, D. B., & Emshoff, J. G. (1987). The fidelity adaptation debate: Implications for the implementation of public sector social programs. American Journal of Community Psychology, 15, 253-268.
8. Bloom, H. S., Hill, C. J., & Riccio, J. A. (2003). Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments. Journal of Policy Analysis and Management, 22, 551-575.
9. Bloom, H. S., & Unterman, R. (forthcoming). Can small high schools of choice improve education prospects for disadvantaged students? Journal of Policy Analysis and Management.
10. Borghans, L., Diris, R., Heckman, J. J., Kautz, T., & Weel, B. (2012). Fostering non-cognitive skills to promote lifetime success. Unpublished manuscript.
11. Borman, G. D., Slavin, R. E., Cheung, A., Chamberlain, A., Madden, N., & Chambers, B. (2007). Final reading outcomes of the National Randomized Field Trial of Success for All. American Educational Research Journal, 44, 701-731.
12. Browne, A., & Wildavsky, A. (1983). Implementation as mutual adaptation. In J. L. Pressman & A. Wildavsky (Eds.), Implementation: How great expectations in Washington are dashed in Oakland (pp. 206-231). Berkeley, CA: University of California Press.
13. Bryk, A. S., & Raudenbush, S. W. (1988). Toward a more appropriate conceptualization of research on school effects: A three-level hierarchical linear model. American Journal of Education, 97, 65-108.
14. Bryk, A. S., Sebring, P. B., Allensworth, E., Luppescu, S., & Easton, J. Q. (2010). Organizing schools for improvement: Lessons from Chicago. Chicago, IL: University of Chicago Press.
15. Bullock, J. G., Green, D. P., & Ha, S. E. (2010). Yes, but what's the mechanism (don't expect an easy answer). Journal of Personality and Social Psychology, 98, 550-558.
16. Carroll, C., Patterson, M., Wood, S., Booth, A., Rick, J., & Balain, S. (2007). A conceptual framework for implementation fidelity. Implementation Science, 2, 40. Retrieved February 18, 2014, from http://www.biomedcentral.com/content/pdf/1748-5908-2-40.pdf.
17. Chase, G. (1979). Implementing a human services program: How hard can it be? Public Policy, 27, 385-420.
18. Chen, H.-T., & Rossi, P. (1983). Evaluating with sense: The theory-driven approach. Evaluation Review, 7, 283-302.
19. Cook, T. D., Shadish, W. R., & Wong, V. C. (2008). Three conditions under which observational studies produce the same results as experiments. Journal of Policy Analysis and Management, 274, 724-750.
20. Cordray, D. S., & Pion, G. M. (2006). Treatment strength and integrity: Models and methods. In R. R. Bootzin & P. E. McKnight (Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103-124). Washington, DC: American Psychological Association.
21. Damschroder, L. J., Aron, D. C., Rosalind, E. K., Kirsh, S. R., Alexander, J. A., & Lowery, J. C. (2009). Fostering implementation of health services research findings into practice: A consolidated framework for advancing implementation science. Implementation Science, 4, 50. Retrieved February 18, 2014, from http://www.implementationscience.com/content/pdf/1748-5908-4-50.pdf.
22. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23-45.
23. Djebbari, H., & Smith, J. (2008). Heterogeneous impacts in PROGRESA. Bonn, Germany: The Institute for the Study of Labor (IZA).
24. Dobson, K. S., & Shaw, B. F. (1984). The use of treatment manuals in cognitive therapy: Experience and issues. Journal of Consulting and Clinical Psychology, 56, 673-680.
25. Duggan, A., Caldera, D., Rodriguez, K., Burrell, L., Rohde, C., & Crowne, S. S. (2007). Impact of a statewide home visiting program to prevent child abuse. Child Abuse and Neglect, 31, 801-827.
26. Dumitrescu, A., Levy, D., Orfield, C., & Sloan, M. (2011). Impact evaluation of Niger's IMAGINE program. Washington, DC: Mathematica Policy Research.
27. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327-350.
28. Durlak, J. A., Weissberg, R. P., & Pachan, M. (2010). A meta-analysis of after-school programs that seek to promote personal and social skills in children and adolescents. American Journal of Community Psychology, 45, 294-309.
29. Dusenbury, L., Brannigan, R., Falco, M., & Hansen, W. B. (2003). A review of research on fidelity of implementation: Implications for drug abuse prevention in school settings. Health Education Research, 18, 237-256.
30. Elmore, R. F. (1985). Forward and backward mapping: Reversible logic in the analysis of public policy. In K. Hanf & T. A. J. Toonen (Eds.), Policy implementation in federal and unitary systems: Questions of analysis and design (pp. 33-70). Dordrecht, Netherlands: Martinus Nijhoff.
31. Fisher, R. A. (1935). The design of experiments. London, UK: Oliver and Boyd.
32. Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M., & Wallace, F. (2005). Implementation research: A synthesis of the literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, the National Implementation Research Network.
33. Flay, B. R., Graumlich, S., Segawa, E., Burns, J. L., Holliday, M. Y., & Aban Aya Investigators. (2004). Effects of two prevention programs on high-risk behaviors among African American youth: A randomized trial. Archives of Pediatrics and Adolescent Medicine, 158, 377-384.
34. Friedlander, D. (1993). Subgroup impacts of large-scale welfare employment programs. Review of Economics and Statistics, 75, 138-143.
35. Friedlander, D., & Robins, P. K. (1997). The distributional impacts of social programs. Evaluation Review, 21, 531-553.
36. Gamse, B. C., Jacob, R. T., Horst, M., Boulay, B., & Unlu, F. (2009). Reading First Impact Study final report. NCEE 2009-4039. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
37. Glisson, C., Landsverk, J., Schoenwald, S., Kelleher, K., Hoagwood, K. E., Mayberg, S., & Green, P. (2008). Assessing the organizational social context (OSC) of mental health services: Implications for research and practice. Administration and Policy in Mental Health and Mental Health Services Research, 35, 98-113.
38. Gomby, D. S. (2005). Home visitation in 2005: Outcomes for children and parents. Working Paper 7. Retrieved February 18, 2014, from http://www.readynation.org/docs/ivk/report_ivk_gomby_2005.pdf.
39. Gueron, J. M., & Pauly, E. (1991). From welfare to work. New York, NY: Russell Sage Foundation.
40. Hamilton, G. (2002). Moving people from welfare to work: Lessons from the National Evaluation of Welfare-to-Work Strategies. New York, NY: MDRC.
41. Heckman, J. J. (2001). Micro data, heterogeneity, and the evaluation of public policy: Nobel lecture. Journal of Political Economy, 109, 673-748.
42. Heckman, J. J. (2005). The scientific model of causality. Sociological Methodology, 35, 1-97.
43. Heckman, J. J., Smith, J., & Clements, N. (1997). Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. Review of Economic Studies, 64, 487-535.
44. Herbst, C. M. (2008). Do social policy reforms have different impacts on employment and welfare use as economic conditions change? Journal of Policy Analysis and Management, 27, 867-894.
45. Hjern, B., & Hull, C. (1982). Implementation research as empirical constitutionalism. European Journal of Political Research, 10, 105-115.
46. Holland, P. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81, 945-960.
47. Hulleman, C. S., & Cordray, D. S. (2009). Moving from the lab to the field: The role of fidelity and achieved relative intervention strength. Journal of Research on Educational Effectiveness, 2, 88-110.
48. Hulleman, C. S., Rimm-Kaufman, S. E., & Abry, T. D. S. (2013). Whole-part-whole: Construct validity, measurement, and analytical issues for fidelity assessment in education research. In T. Halle, A. Metz, & I. Martinez-Beck (Eds.), Applying implementation science in early childhood programs and systems (pp. 65-93). Baltimore, MD: Paul H. Brookes Publishing Co.
49. Imai, K., Tingley, D., & Keele, L. (2009). A general approach to causal mediation analysis. Psychological Methods, 15, 309-334.
50. Kazdin, A. E. (1986). Comparative outcome studies of psychotherapy: Methodological issues and strategies. Journal of Consulting and Clinical Psychology, 54, 95-105.
51. Kemple, J. J., Snipes, J. C., & Bloom, H. S. (2001). A regression-based strategy for defining subgroups in a social experiment. New York, NY: MDRC.
52. Kezar, A. (2011). What is the best way to achieve broader reach of improved practices in higher education? Innovations in Higher Education, 36, 235-247.
53. Kisker, E. E., Paulsell, D., Love, J. M., & Raikes, H. (2002). Pathways to quality and full implementation in early Head Start programs. Princeton, NJ: Mathematica Policy Research.
54. Klein, A., Starkey, P., Clements, D., Sarama, J., & Iyer, R. (2008). Effects of a pre-kindergarten mathematics intervention: A randomized experiment. Journal of Research on Educational Effectiveness, 1, 155-178.
55. Levy, D., Sloan, M., Linden, L., & Kazianga, H. (2009). Impact evaluation of Burkina Faso's BRIGHT program. Washington, DC: Mathematica Policy Research.
56. Lipsey, M. W., Landenberger, N. A., & Wilson, S. J. (2007). Effects of cognitive behavioral programs for criminal offenders. Nashville, TN: Center for Evaluation Research and Methodology, Vanderbilt Institute for Public Policy Studies.
57. Mazmanian, D. A., & Sabatier, P. A. (1989). Implementation and public policy. Lanham, MD: University Press of America.
58. Michalopoulos, C., & Schwartz, C. (2000). What works best for whom: Impacts of 20 welfare-to-work programs by subgroup. Washington, DC: U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation and Administration for Children and Families, and U.S. Department of Education.
59. Neyman, J. (1923). Statistical problems in agricultural experiments. Journal of the Royal Statistical Society, 2, 107-180.
60. Olds, D. L., Kitzman, H., Hanks, C., Cole, R., Anson, E., Sidora-Arcoleo, K., Luckey, D. W., Henderson, C. R., Holmberg, J., Tutt, R. A., Stevenson, A. J., & Bondy, J. (2007). Effects of nurse home visiting on maternal and child functioning: Age-9 follow-up of a randomized trial. Pediatrics, 120, e832-e845.
61. Paulsell, D., Avellar, S., Martin, E. S., & Del Grosso, P. (2010). Home visiting evidence of effectiveness review: Executive summary. Princeton, NJ: Mathematica Policy Research.
62. Pianta, R. C., Mashburn, A. J., Hamre, B. B., Downer, J. T., Barbarin, O. A., Bryant, D., & Early, D. M. (2008). Measures of classroom quality in prekindergarten and children's development of academic language and social skills. Child Development, 79, 732-749.
63. Quandt, R. (1972). A new approach to estimating switching regressions. Journal of the American Statistical Association, 67, 306-310.
64. Raudenbush, S. W., & Liu, X. (2000). Statistical power and optimal design for multisite randomized trials. Psychological Methods, 5, 199-213.
65. Riccio, J., Dechausay, N., Greenberg, D., Miller, C., Rucks, Z., & Verma, N. (2010). Toward reduced poverty across generations: Early findings from New York City's conditional cash transfer program. New York, NY: MDRC.
66. Richburg-Hayes, L., Brock, T., LeBlanc, A., Paxson, C., Rouse, C. E., & Barrow, L. (2009a). Rewarding persistence: Effects of a performance-based scholarship program for low-income parents. New York, NY: MDRC.
67. Richburg-Hayes, L., Cha, P., Cuevas, M., Grossman, A., Patel, R., & Sommo, C. (2009b). Paying for college success: An introduction to the performance-based scholarship demonstration. New York, NY: MDRC.
68. Roberts, B. W. (2009). Back to the future: Personality and assessment and personality development. Journal of Research in Personality, 43, 137-145.
69. Rothwell, P. M. (2005). Subgroup analysis in randomised control trials: Importance, indications and interpretation. Lancet, 365, 176-186.
70. Roy, A. D. (1951). Some thoughts on the distribution of earnings. Oxford Economic Papers, 3, 135-146.
71. Rubin, D. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Education Psychology, 66, 688-701.
72. Rubin, D. (1978). Bayesian Inference for causal effects: The role of randomization. Annals of Statistics, 6, 34-58.
73. Scrivener, S., Weiss, M. J., & Sommo, C. (2012). What can a multifaceted program do for community college students? Early results from an evaluation of accelerated study in associate programs (ASAP) for developmental education students. New York, NY: MDRC.
74. Sechrest, L. B., & Redner, R. (1979). Strength and integrity of treatments in evaluation studies. Washington, DC: National Criminal Justice Reference Service.
75. Sechrest, L. B., West, S. G., Phillips, M. A., Redner, R., & Yeaton, W. (1979). Some neglected problems in evaluation research: Strength and integrity of treatment. In L. B. Sechrest, S. G. West, M. A. Phillips, R. Redner, & W. Yeaton (Eds.), Evaluation studies review annual (pp. 15-35). Thousand Oaks, CA: Sage.
76. Tinto, V. (1998). Learning communities and the reconstruction of remedial education in higher education. Paper prepared for the Ford Foundation and U.S. Department of Education Conference on Replacing Remediation in Higher Education. Stanford, CA: Stanford University.
77. Trenholm, C., Devaney, B., Fortson, K., Clark, M., Bridgespan, L. Q., & Wheeler, J. (2008). Impacts of education on teen sexual activity, risk of pregnancy, and risk of sexually transmitted diseases. Journal of Policy Analysis and Management, 27, 255-276.
78. Van Meter, D. S., & Van Horn, C. E. (1975). The policy implementation process: A conceptual framework. Administration and Society, 6, 445-488.
79. Ware, M., & Patel, R. (2012). Does more money matter? An introduction to the performance-based scholarship demonstration in California. New York, NY: MDRC.
80. Weiss, C. (1997). How can theory-based evaluation make greater headway. Evaluation Review, 21, 501-524.
81. Wood, R. G., McConnell, S., Moore, Q., Clarkwest, A., & Hsueh, J. (2012). The effects of building strong families: A healthy marriage and relationship skills education program for unmarried parents. Journal of Policy Analysis and Management 31, 228-252.
82. Zachry Rutschow, E., Richburg-Hayes, L., Brock, T., Orr, G., Cerna, O., Cullinan, D., Reid, M. K., Jenkins, D., Gooden, S., & Martin, K. (2011). Turning the tide: Five years of achieving the dream in community colleges. New York, NY: MDRC.