What Doesn't Work? Three Failures, Many Answers

Evaluation

Volumn 16, Issue 4, 2010, Pages 371-387

  Stame, Nicoletta ^a  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

Author keywords

external validity; implementation failure; methodology failure; ruling out rival explanations; theory failure

Indexed keywords

EID: 77958529168     PISSN: 13563890     EISSN: 14617153     Source Type: Journal    
DOI: 10.1177/1356389010381914     Document Type: Review

References (55)

1. Barca F. (2009) An Agenda for a Reformed Cohesion Policy, independent report prepared at the request of the Commissioner for Regional Policy, EU.
2. Baron RM, Kenny DA (1986) The mediator-moderator variable distinction in social psychological research: Conceptual, strategic and statistical considerations. Journal of Personality and Social Psychology 51(6): 1173-82.
3. Bickman L (ed.) (2000) alidity and Social Experimentation, vol. 1 of Donald Campbell's Legacy. Thousand Oaks, CA: SAGE.
4. Brodkin E. (2003) Street-level research: Policy at the front lines. In: Corbett t, Lennon MC (eds) Policy into Action: Implementation Research and Welfare Reform. Washington, DC: Urban Institute Press.
5. Campbell D. (1969) Reforms as experiments. American Psychologist 24: 409-29. Now in Campbell D., Russo J. (1999) Social Experimentation. Thousand Oaks, CA: SAGE.
6. Campbell D. (1984) Can we be scientific in applied social sciences? In: Connor RF, Altman DG, and Jackson C (eds) Evaluation Studies Review Annual, 26-48. Newbury Park, CA: SAGE.
7. Campbell D. (1986) Relabeling internal and external validity for applied social scientists. In: Trochim WMK (ed.)Advances in Quasi-Experimental Design and Analysis, New Developments in Program Evaluation, 31: 67-78.
8. Campbell D. (1994) Foreword. In: Yin R, Case Study Research, ix-xi. Thousand Oaks, CA: SAGE.
9. Campbell D., Russo MJ (1999) Social Experimentation. Thousand Oaks, CA: SAGE.
10. Campbell D., Stanley JC (1966) Experimental and Quasi-Experimental Design for Research. Chicago: Rand-Macmillan.
11. Chambers R., Karlan D., Ravallion M., and Rogers P. (2009) 'Designing Impact Evaluations: Different Perspectives. 3IE Working Paper, 4. Washington.
12. Cook TD (2000) Toward a practical theory of external validity. In: Bickman L (ed.)Validity and Social Experimentation, vol. 1 of Donald Campbell's Legacy, 3-43. Thousand Oaks, CA: SAGE.
13. Cronbach LJ (1982) Designing Evaluations of Educational and Social Programs. San Francisco: Jossey-Bass.
14. Donaldson S. (2007) Program Theory-Driven Evaluation Science. New York: Lawrence Erlbaum Associates.
15. Donaldson S, Christie C, and Mark MM (eds) (2009) What Counts as Credible Evidence in Applied Research and Evaluation Practice? Los Angeles: SAGE.
16. Donkoh C., Undershill K., and Montgomery P. (2006) Independent living programs for improving outcomes for young people leaving the care system. Campbell Systematic Reviews, Campbell Collaboration. URL:www.cochrane.org/reviews/en/ab005558.html.
17. EES (European Evaluation Society) (2007) The Importance of a Methodologically Diverse Approach to Impact Evaluation - Specifically with Respect to Development Aid and Development Interventions. URL: europeanevaluation.org.
18. European Commission, DG Regional Affairs and DG Employment (2006) Indicative Guidelines on Evaluation Methods: Evaluation during the Programming Period. URL: http://www.3kps.gr/ex-ante2007-2013/html/arxeia/WD5_Ongoing_Evaluation_E N_0906.pdf.
19. GAO (2009) A Variety of Rigorous Methods Can Help Identify Effective Interventions. Washington, DC: GAO.
20. Glouberman S., Zimmerman B. (2002), Complicated and Complex Systems: What Would Successful Reform of Medicare Look Like? Commission on the Future of Health Care in Canada, Discussion Paper, 8. URL: http://www.hc-sc.gc.ca/english/pdf/romanow/pdfs/8_Glouberman_E.pdf.
21. Greene JC (2004) The educative evaluator: an interpretation of Lee J. Cronbach's vision of evaluation. In: Alkin MC (ed.) Evaluation Roots: Tracing Theorists' Views and Influences. Thousand Oaks, CA: SAGE.
22. Greene JC, Caracelli V., and Graham WF (1989) Toward a conceptual framework for mixed-method evaluation designs. Educational Evaluation and Policy Analysis 11(3): 255-274.
23. Hood C. (1976) The Limits of Administrations. London: John Wiley.
24. Lindblom CE (1968) The Policy-Making Process. Englewood Cliffs, NJ: Prentice Hall.
25. Lipsey M. (2000) Statistical conclusion validity for intervention research: A significant (p <.05) problem. In: Bickman L (ed.) Validity and Social Experimentation, vol. 1 of Donald Campbell's Legacy. Thousand Oaks, CA: SAGE.
26. Lipsey M., Crosse S., Dunkle J., Pollard J., and Stobart G. (1985) Evaluation: The state of the art and the sorry state of the science. In: Cordray DS (ed.) Utilizing Prior Research in Evaluation Planning, New Directions for Program Evaluation, 27. San Francisco: Jossey-Bass, 7-28.
27. Lipsky M. (1980) Street-Level Bureaucracy. New York: Russel Sage Foundation.
28. McLaughlin MW (1984) Implementation realities and evaluation design. In: Shotland RL, Mark MM (eds) Social Science and Social Policy. Beverly Hills, CA: SAGE.
29. Mark M. (1986) Validity typologies and the logic and practice of quasi-experimentation. In: Trochim WMK (ed.)Advances in Quasi-Experimental Design and Analysis, New Developments in Program Evaluation, 31: 47-66. San Francisco: Jossey-Bass.
30. Martini A., Sisti M. (2009) Valutare il successo delle politiche pubbliche. Bologna: Il Mulino.
31. Mayne J. (2010) Addressing cause and effect in simple and complex settings through contribution analysis. In: Schwartz R, Forss K, and Marra M (eds) Evaluating the Complex. New Brunswick, NJ: Transactions Publishers.
32. Oecd, Dac (2002) Glossary of Key Terms in Evaluation and Results Based Management. URL: www.oecd.org/dac/evaluation.
33. Patton MQ (1989) A context and boundaries for a theory-driven approach to validity. Evaluation and Program Planning 12: 375-7.
34. Patton MQ (2008), State of the Art in Measuring Development Assistance, Mimeo.
35. Patton MQ (2010) Developmental Evaluation. New York: Guildford Press.
36. Pawson R. (2004) Would Campbell be a member of the Campbell Collaboration? The Evaluator (Winter), 13-15.
37. Pawson R. (2006) Evidence Based Policy: A Realist Perspective. London: SAGE.
38. Pawson R. (2010) Middle range theory and program theory evaluation: From provenance to practice. In: Vaessen J, Leeuw FL (eds) Mind the Gap: Perspectives on Policy Evaluation and the Social Sciences, 171-202. New Brunswick, NJ: Transactions Publishers.
39. Pawson R., Tilley N. (1997) Realistic Evaluation. London: SAGE.
40. Perrin B. (2000) Donald T. Campbell and the art of practical 'in-the-trenches' program evaluation. In: Bickman L (ed.) Validity and Social Experimentation, vol. 1 of Donald Campbell's Legacy, 262-282. Thousand Oaks, CA: SAGE.
41. Perrin, B. (2006) How evaluation can help make knowledge management real. In: Rist R, Stame N (eds) From Studies to Streams. New Brunswick, NJ: Transactions Publishers.
42. Rogers P. (1999) Review of 'Realistic evaluation'. American Journal of Evaluation 20(3): 381-3.
43. Rogers P. (2008) Using program theories to evaluate complicated and complex aspects of interventions. Evaluation 14(1): 29-48.
44. Rogers P. (2010) Implications of complicated and complex characteristics for key tasks in evaluation. In: Schwartz R, Forss K, and Marra M (eds) Evaluating the Complex. New Brunswick, NJ: Transactions Publishers.
45. Scriven M. (1981) Evaluation Thesaurus, 3rd edn. Inverness, CA: Edgepress.
46. Scriven M. (2009) Demythologizing causation and evidence. In Donaldson S, Christie C, and Mark MM (eds) What Counts as Credible Evidence in Applied Research and Evaluation Practice?, 134-152 Los Angeles: SAGE.
47. Simon HA (1947) Administrative Behaviour. New York: Macmillan.
48. Suchman EA (1969) Evaluating educational programs. Urban Review 3(4): 15-17.
49. Weiss C. (1972) Evaluation Research. Inglewood Cliffs, NJ: Prentice Hall.
50. Weiss C. (1997) Theory-based evaluation: Past, present and future. In: Rog DJ (ed.) Progress and Future Directions in Evaluation, New Directions for Evaluation, 76. San Francisco: Jossey-Bass.
51. Weiss C. (1998) Evaluation, 2nd edn. Upper Saddle River, NJ: Prentice Hall.
52. Weiss C. (2002) What to do until the random assigner comes. In: Mosteller F, Boruch R, Evidence Matters: Randomized Trials in Education Research, 198-224. Washington, DC: Brookings Institution Press.
53. White H. (2010) A contribution to current debates in impact evaluation. Evaluation16(2).
54. Yin R. (1994) Case Study Research. Thousand Oaks, CA: SAGE.
55. Yin R. (2000) Rival explanations as an alternative to reforms as 'experiments'. In: Bickman L (ed.) Validity and Social Experimentation, vol. 1 of Donald Campbell's Legacy, 239-266. Thousand Oaks, CA: SAGE.