Against excessive rhetoric in impact assessment: Overstating the case for randomised controlled experiments

Journal of Development Studies

Volumn 47, Issue 11, 2011, Pages 1619-1635

  Shaffer, Paul ^a  

^a TRENT UNIVERSITY   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

DEVELOPMENT STRATEGY; NUMERICAL MODEL; RESEARCH;

EID: 84857874137     PISSN: 00220388     EISSN: 17439140     Source Type: Journal    
DOI: 10.1080/00220388.2010.514331     Document Type: Article

References (60)

1. Adato, M. 2008. Combining survey and ethnographic methods to improve evaluation of conditional cash transfer programs. International Journal of Multiple Research Approaches, 2(2): 222-236.
2. Bamberger, M. and White, H. 2007. Using strong evaluation designs in developing countries: experience and challenges. Journal of MultiDisciplinary Evaluation, 4(8): 58-73.
3. Banerjee, A. 2007. Making Aid Work, Cambridge, MA: MIT Press.
4. Behrman, J., Todd, P. and Segupta, P. Progressing through Progresa: an impact assessment of a school subsidy experiment. Penn Institute for Economic Research Working Paper 01-033. Pennsylvania: University of Pennsylvania, Philadelphia.
5. Cartwright, N. 2007a. Hunting Causes and Using Them. Approaches in Philosophy and Economics, Cambridge: Cambridge University Press.
6. Cartwright, N. 2007b. Are RCTs the gold standard?. BioSocieties, 2: 11-20.
7. Chambers, R. 2003. "Qualitative approaches: self-criticism and what can be gained from quantitative approaches". In Q-Squared: qualitative and Quantitative Methods of Poverty Appraisal, Edited by: Kanbur, R. 28-34. Delhi: Permanent Black.
8. Collins, J., Hall, N. and Paul, L. 2004. "Counterfactuals and causation: history, problems and prospects". In Causation and Counterfactuals, Edited by: Collins, J., Hall, N. and Paul, L. 1-59. Cambridge, MA: MIT Press.
9. Cook, T. 2000. "Comment. Impact evaluation: concepts and methods". In Evaluation and Poverty Reduction, Edited by: Feinstein, O. and Picciotto, R. 76-82. Washington, DC: The World Bank.
10. Cook, T. 2006. Describing what is special about the role of experiments in contemporary educational research: putting the 'gold standard' rhetoric into perspective. Journal of MultiDisciplinary Evaluation, 3(6): 1-7.
11. Cook, T. and Campbell, D. 1979. Quasi-Experimentation. Design & Analysis for Field Settings, Chicago: Rand McNally Publishing.
12. Deaton, A. Instruments of development: randomization in the Tropics and the search for the elusive keys to economic development. National Bureau of Economic Research (NBER) Working Paper 14690. USA: New York City.
13. Duflo, E., Glennerster, R. and Kremer, M. 2008. "Using randomization in development economics research: a toolkit". In Handbook of Development Economics. Volume 4, Edited by: Schultz, T. P. and Strauss, J. 3895-3962. Amsterdam: North-Holland.
14. Elster, J. 1983. Explaining Technical Change, Cambridge: Cambridge University Press.
15. Elster, J. 1987. Sour Grapes. Studies in the Subversion of Rationality, Cambridge: Cambridge University Press.
16. Fisman, R. and Miguel, E. 2008. Economic Gangsters, Princeton: Princeton University Press.
17. George, A. and Bennett, A. 2004. Case Studies and Theory Development in the Social Sciences, Cambridge, MA: MIT Press.
18. Gilovich, T. and Griffin, D. 2002. "Introduction - heuristics and biases: then and now". In Heuristics and Biases. The Psychology of Intuitive Judgement, Edited by: Gilovich, T., Griffin, D. and Kahneman, D. 1-18. Cambridge: Cambridge University Press.
19. Glymour, C. 1986. Comment. Statistics and metaphysics. Journal of the American Statistical Association, 81(396): 964-966.
20. Heckman, J. 2000. "Comment. Impact evaluation: concepts and methods". In Evaluation and Poverty Reduction, Edited by: Feinstein, O. and Picciotto, R. 83-84. Washington, DC: The World Bank.
21. Hedström, P. and Swedberg, R. 1998. Social Mechanisms: an Analytical Approach to Social Theory, Cambridge: Cambridge University Press.
22. Holland, P. 1986. Statistics and causal inference. Journal of the American Statistical Association, 81(396): 945-960.
23. Howes, M. 1992. Linking paradigms and practice. Key issues in the appraisal, monitoring and evaluation of British NGO projects. Journal of International Development, 4(4): 375-396.
24. Hume, D. [1748] (1955). An Inquiry Concerning Human Understanding, New York: Bobbs-Merrill.
25. Hulme, D. 2000. Impact assessment methodologies for microfinance: theory, experience and better practice. World Development, 28(1): 79-93.
26. Johnson, G. n.d. "A long trail of evidence links cigarette smoking to cancer". Accessed at http://txtwriter.com/Onscience/Articles/smokingcancer2.html (retrieved 17 August 2009)
27. Kanbur, R. and Shaffer, P. 2007. Epistemology, normative theory and poverty analysis: implications for Q-squared in practice. World Development, 35(2): 183-196.
28. Leeuw, F. and Vaessen, J. 2009. Impact Evaluations and Development. NONIE Guidance on Impact Evaluation, Washington, DC: Network or Networks on Impact Evaluation (NONIE).
29. Lelorier, J., Gregoire, G., Benhaddad, A., Lapierre, J. and Derderian, F. 1997. Discrepancies between meta-analyses and subsequent large randomised controlled trials. New England Journal of Medicine, 337: 536-542.
30. Lewis, D. 1973. Counterfactuals, Oxford: Blackwell.
31. Little, D. 1991. Varieties of Social Explanation, Boulder, CO: Westview Press.
32. Little, D. 1998. Microfoundations, Method and Causation, New Brunswick: Transaction Publishers.
33. Mackie, J. L. 1980. The Cement of the Universe, Oxford: Oxford University Press.
34. Mahoney, J. 2001. Beyond correlational analysis: recent innovations in theory and method. Sociological Forum, 16(3): 575-593.
35. Marini, M. and Singer, B. 1988. Causality in the social sciences. Sociological Methodology, 18,: 347-409.
36. Mayoux, L. and Chambers, R. 2005. Reversing the paradigm: quantification, participatory methods and pro-poor impact assessment. Journal of International Development, 17: 271-298.
37. Menzies, P. 2009. "Counterfactual theories of causation". In The Stanford Encyclopedia of Philosophy Edited by: Zalta, E. Accessed at http://plato.stanford.edu/archives/fall2009/entries/causation-counterfactual
38. Pawson, R. and Tilley, N. 1997. Realistic Evaluation, London: Sage.
39. Pawson, R. 2002. Evidence-based policy: the promise of 'realistic synthesis'. Evaluation, 8(3): 340-358.
40. Pickel, A. 2004. Systems and mechanisms. A symposium on Mario Bunge's philosophy of social science. Philosophy of Social Sciences, 34(2): 169-181.
41. Rao, V. and Woolcock, M. 2003. "Integrating qualitative and quantitative approaches in program evaluation". In The Impact of Economic Policies on Poverty and Income Distribution, Edited by: Bourguignon, F. and da Silva, L. 165-190. New York City: World Bank Publications and Oxford University Press.
42. Roche, C. 1999. Impact Assessment for Development Agencies, Oxford: Oxfam Publications.
43. Rodrik, D. 21 May 2008. "The new development economics: we shall experiment but shall we learn? Revised Draft". 21 May, Harvard University.
44. Rubin, D. B. 1974. Estimating causal effects on treatments in randomised and nonrandomised studies. Journal of Educational Psychology, 66: 688-701.
45. Schaffer, J. 2008. "The metaphysics of causation". In The Stanford Encyclopedia of Philosophy Edited by: Zalta, E. Accessed at http://plato.stanford.edu/archives/fall2008/entries/causation-metaphysics/
46. Scriven, M. 1966. "Causes, connections and conditions in history". In Philosophical Analysis and History, Edited by: Dray, W. 238-264. New York and London: Harper and Row Publishers.
47. Scriven, M. 2008. A summary evaluation of RCT methodology: an alternative approach to causal research. Journal of MultiDisciplinary Evaluation, 5(9): 11-24.
48. Schwartz, D. 2005. "Thinking inside the (black) box: a scientific realist-based approach to causal inference in political science". Canada: University of Toronto. Unpublished thesis
49. Shaffer, P. 2002. Poverty naturalized: implications for gender. Feminist Economics, 8(4): 55-75.
50. Shaffer, P. August 2005. "Handbook on tools and methods of social impact assessment. Final Draft". August, Tunis: African Development Bank/Poverty and Sustainable Development Unit.
51. Shaffer, P. and IDEA Intl. 17 July 2004. "Impact assessment of the Hunger Eradication and Poverty Reduction (HEPR) program: Final Report. Ministry of Labor and Social Affairs (MOLISA)/United Nations Development Program (UNDP)". 17 July, Hanoi
52. Shaffer, P., Kanbur, R., Thang, N. and Bortei-Doku, E. Aryeetey. 2008. Introduction to Q-squared in policy: the use of qualitative and quantitative methods of poverty analysis in decision-making. International Journal of Multiple Research Approaches, 2(2): 134-144.
53. Sudman, S., Bradburn, N. and Schwartz, N. 1996. Thinking About Answers, San Francisco: Jossey-Bass Publishers.
54. Urbach, P. 1985. Randomization and the design of experiments. Philosophy of Science, 52: 256-273.
55. Weber, E. 2007. Social mechanisms, causal inference, and the policy relevance of social science. Philosophy of the Social Sciences, 37(3): 348-359.
56. Weiss, C. 2000. "Theory-based evaluation: theories of change for poverty reduction strategies". In Evaluation and Poverty Reduction, Edited by: Feinstein, O. and Picciotto, R. 103-111. Washington, DC: The World Bank.
57. White, H. 2008. Of probits and participation: the use of mixed methods in quantitative impact evaluation. IDS Bulletin, 39(11): 98-109.
58. White, H. Some reflections on current debates in impact evaluation. International Initiative for Impact Evaluation Working Paper No. 1. New Delhi, India.
59. White, H. and Bamberger, M. 2008. Impact evaluation in official development agencies. IDS Bulletin, 39(11): 1-11.
60. Worrall, J. 2002. What evidence in evidence-based medicine?. Philosophy of Science, 69: S316-S330.