Flaws in Evaluations of Social Programs: Illustrations From Randomized Controlled Trials

Evaluation Review

Volumn 38, Issue 5, 2014, Pages 359-387

  Greenberg, David ^a   Barnow, Burt S ^b  

^a UNIVERSITY OF MARYLAND BALTIMORE COUNTY   (United States)

^b GEORGE WASHINGTON UNIVERSITY   (United States)

Author keywords

crossover; experimental flaws; implementation; response bias; social experiments

Indexed keywords

HUMAN; METHODOLOGY; PROCEDURES; PROGRAM DEVELOPMENT; PROGRAM EVALUATION; RANDOMIZED CONTROLLED TRIAL (TOPIC); SOCIAL PROBLEM; SOCIAL WELFARE; SOCIOMETRIC STATUS; STATISTICAL BIAS; STATISTICS;

BIAS (EPIDEMIOLOGY); HUMANS; PROGRAM DEVELOPMENT; PROGRAM EVALUATION; RANDOMIZED CONTROLLED TRIALS AS TOPIC; RESEARCH DESIGN; SOCIAL PROBLEMS; SOCIAL WELFARE; SOCIOMETRIC TECHNIQUES; STATISTICS AS TOPIC;

EID: 84908137982     PISSN: 0193841X     EISSN: 15523926     Source Type: Journal    
DOI: 10.1177/0193841X14545782     Document Type: Article

References (49)

1. Angrist J. D.,Lavy V.The effect of high school matriculation awards: Evidence from randomized trials. Cambridge, MA: National Bureau of Economic Research working paper 9389; 2002:
2. Azrin N. H.,Flores T.,Kaplan S. J.Job-finding club: A group-assisted program for finding employment.Behavioral Research and Therapy. 1975;13:17-27
3. Azrin N. H.,Philip R. A.The job club method for the handicapped: A comparative outcome study.Rehabilitation Counseling Bulletin. 1979;23:144-155
4. Azrin N. H.,Philip R. A.,Thienes-Hontos P.,Besalel V. A.Comparative evaluation of the job club program with welfare recipients.Journal of Vocational Behavior. 1980;16:133-145
5. Banerjee A.,Duflo E.,Glennerster R.Explorations in the economics of aging. Wise D. A., ed. Chicago, IL: University of Chicago Press; 2011:317-344.
6. Barnow B.,Greenberg D.2013:239-252.
7. Basch C. E.,Sliepcevich E. M.,Gold R. S.,Duncan D. F.,Kolbe L. J.Avoiding type III errors in health education program evaluations: A case study.Health Education Quarterly. 1985;12:315-331
8. Birtwhistle A.,Barnes D.,Looby C.Evaluation of supportive caseloading (1-2-1) in North Norfolk. Sheffield, England: Research and Evaluation, Employment Service, Tracking Study; 1994:
9. Bloom H. S.Minimum detectable effects: A simple way to report the statistical power of experimental designs.Evaluation Review. 1995;19:547-556
10. Bloom H. S.,Hill C. J.,Riccio J.Linking program implementation and effectiveness: Lessons from a pooled sample of welfare-to-work experiments.Journal of Policy Analysis and Management. 2003;22:551-575
11. Blustein J.Toward a more public discussion of the ethics of federal social program evaluation.Journal of Policy Analysis and Management. 2005;24:824-846
12. Bruhn M.,McKenzie D.In pursuit of balance: Randomization in practice in development field experiments.American Economic Journal: Applied Economics. 2009;1:200-232
13. Cancian M.,Kaplan T.,Rothe I.Wisconsin’s self-sufficiency/pay for performance: Results and lessons from a social experiment. Madison: University of Wisconsin-Madison, Institute for Research on Poverty; 2000:
14. Cook T. D.,Wong V.The warrant for universal pre-K: Can several thin reeds make a strong policy boat?.Social Policy Review. 2007;21:14-15
15. Cook P. J.,O’Brien M.,Braga A.,Ludwig J.Lessons from a partially controlled field trial.Journal of Experimental Criminology. 2012;8:271-287
16. Davies V.,Irving P.New deal for young people: Intensive gateway trailblazers. A final report to the Employment Services. Birmingham, England: ECOTEC Research and Consulting Limited; 2000:
17. Devine J. A.,Wright J. D.,Brody C. J.Evaluating an alcohol and drug treatment program for the homeless: An econometric approach.Evaluation and Program Planning. 1997;20:205-215
18. Dunham K.,Wiegand A.,Michalopoulos C.The effort to implement the Youth Offender Demonstration Project (YODP) impact evaluation: Lessons and implications for future research, final report. Ottawa Ontario: Social Policy Research Associates; 2008:
19. Dynarski M.,Wood R.Helping high-risk youths; Results from the Alternative Schools Demonstration Program. Princeton, NJ: Mathematica Policy Research; 1997:
20. Epstein D.,Klerman J. A.When is a program ready for rigorous impact evaluation? The role of a falsifiable logic model.Evaluation Review. 2012;36:375-401
21. Glennerster R.,Takavarasha K.Running randomized evaluation: A practical guide. Princeton, NJ: Princeton University Press; 2013:
22. Greenberg D.,Ashworth K.,Cebulla A.,Walker R.Meta-analysis: Discovering what works best in welfare provision.Evaluation. 2004;10:193-216
23. Greenberg D.,Shroder M.The digest of social experiments. 2004, The digest of social experiments. 3rd ed. Washington, DC: Urban Institute Press.Washington, DC: Urban Institute Press; 2004:
24. Greenberg D.,Michalopoulos C.,Robins P. K.A meta-analysis of government-sponsored training programs.Industrial and Labor Relations Review. 2003;57:31-53
25. Greenberg D.,Shroder M.,Onstott M.The social experiment market.Journal of Economic Perspectives. 1999;13:157-172
26. Grinstein-Weiss M.,Sherraden M. W.,Gale W. G.,Rohe W.,Schreiner M.,Key C.Long-term follow-up of individual development accounts: Evidence from the ADD experiment. Chapel Hill, NC: The University of North Carolina; 2012:
27. Hahn A.Evaluation of the Quantum Opportunities Program (QOP): Did the program work?. Waltham, MA: Brandeis University Center for Human Resources; 1994:
28. Harrison G. W.,List J. A.Field experiments.Journal of Economic Literature. 2004;42:1009-1055
29. Hendra R.,Riccio James A.,Dorsett Richard,Greenberg David H.,Knight Genevieve,Phillips Joan,Robins Philip K.,Vegeris Sandra,Walter Johanna,Hill Aaron,Ray Kathryn,Smith JaredSheffield, London: Department for Work and Pensions. Research Report No 765; 2011:
30. Leiman J. M.The WIN labs: A federal/local partnership in social research. New York, NY: MDRC; 1982:
31. List J. A.Why economists should conduct field experiments and 14 tips for pulling one off.The Journal of Economic Perspectives. 2011;25:3-15
32. Manski C.Nonparametric bounds on treatment effects.American Economic Review. 1990;80:319-323
33. Maxfield M.,Castner L.,Maralani V.,Vencill M.The quantum opportunity program demonstration: Implementation findings. Princeton, NJ: Mathematica Policy Research; 2003:
34. Mowbray C. T.,Collins M. E.,Plum T. B.,Masterton T.,Mulder R.Harbinger I: The development and evaluation of the first pact replication.Administration and Policy in Mental Health. 1997;25:105-123
35. Newman J.,Pradham M.,Rawlings L. B.An impact evaluation of education, health, and water supply investments by the Bolivian Social Investment Fund.The World Bank Economic Review. 2002;16:241-271
36. Olsen R. B.,Orr L. L.,Bell S. H.,Stuart E. A.External validity in policy evaluations that choose sites purposively.Journal of Policy Analysis and Management. 2013;32:107-121
37. Orr L.Social experiments: Evaluating public programs with experimental methods. Thousand Oaks, CA: Sage; 1999:
38. Orr L. L.,Bell S.,Lam K.Long-term impacts of the new deal for disabled people: Final report. Leeds, England: DWP Research Report No. 342; 2007:
39. Orr L. L.,Bloom H. S.,Bell S. H.,Doolittle F.,Lin W.,Cave G.Does training for the disadvantaged work? Evidence from the National JTPA Study. Washington, DC: Urban Institute Press; 1996:
40. Puma M. J.,Burstein N. R.,Merrell K.,Silverstein G.Evaluation of the food stamp employment and training program: Final report. Washington, DC: U.S. Department of Agriculture, Food and Nutrition Service; 1990:
41. Rosholm M.,Skipper L.Is labour market training a curse for the unemployed? Evidence from a social experiment.Journal of Applied Econometrics. 2009;24:338-365
42. Seftor N. S.,Mamun A.,Schirm A.The impacts of regular upward bound on postsecondary outcomes 7-9 years after scheduled high school graduation: Final report. Princeton, NJ: Mathematica Policy Research; 2009:
43. Shadish W. R.,Cook T. D.,Campbell D. T.Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin; 2002:
44. Turturro C.,Benda B.,Turney H.Arkansas welfare waiver demonstration final report. Fayetteville, NC: University of Arkansas; 1997:
45. Washington, D.C: Office of Community Services, Monograph Series; 1992:100-89.
46. Washington, D.C: Office of Community Services, Monograph Series; 1994:100-90.
47. Walker R.,Hoggart L.,Hamilton G.Making random assignment happen: Evidence from the UK employment retention and advancement (ERA) demonstration. London, England: Department for Work and Pensions Research Report 330; 2006:
48. Wholey J.Handbook of practical program evaluation. 2004. Exploratory evaluation. In HatryH. P.WholeyJ. S.NewcomerK. E. (Eds.), Handbook of practical program evaluation (2nd ed, pp. 241–260). San Francisco, CA: Jossey-Bass.Hatry H. P.Wholey J. S.Newcomer K. E., ed. San Francisco, CA: Jossey-Bass; 2004:241-260.
49. Wilson L. A.,Stoker R. P.,McGrath D.Welfare bureaus as moral tutors: What do clients learn from paternalistic welfare reforms?.Social Science Quarterly. 1999;80:473-486