Things you Wanted to Know about Bias in Evaluations but Never Dared to Think

IDS Bulletin

Volumn 45, Issue 6, 2014, Pages 49-64

  Camfield, Laura ^a   Duvendack, Maren ^a   Palmer Jones, Richard ^a  

^a NONE

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84911380963     PISSN: 02655012     EISSN: 17595436     Source Type: Journal    
DOI: 10.1111/1759-5436.12112     Document Type: Article

References (93)

1. Ackerman, F. and Heinzerling, L. (2004) Priceless: On Knowing the Price of Everything and the Value of Nothing, New York NY: The New Press
2. ADB (n.d.) Cost-Benefit Analysis for Development: A Practical Guide, Asian Development Bank, www.adb.org/sites/default/files/cost-benefit-analysis-development.pdf (accessed 4 August 2014)
3. Andersen, O.W. and Broegaard, E. (2012) 'The Political Economy of Joint-Donor Evaluations', Evaluation 18.1: 47-59
4. Babcock, L. and Loewenstein, G. (1997) 'Explaining Bargaining Impasse: The Role of Self-Serving Biases', in C. Camerer, G. Loewenstein and M. Rabin (eds), Advances in Behavioral Economics, Princeton NJ: Princeton University Press
5. Bakewell, O. (2007) 'Breaching the Borders between Research and Practice: Development NGOs and Qualitative Data', in M. Smith (ed.), Negotiating Boundaries and Borders (Studies in Qualitative Methodology, Volume 8), Bingley: Emerald Group Publishing
6. Baldwin, K.D.S. (1957) The Niger Valley Project, Oxford: Blackwell
7. Bavinck, M. (2008) 'Collective Strategies and Windfall Catches: Fisher Responses to Tsunami Relief Efforts in South India', Transforming Cultures eJournal 3.2: 76-92
8. Boaz, A. and Ashby, D. (2003) Fit for Purpose? Assessing Research Quality for Evidence Based Policy and Practice, London: ESRC UK Centre for Evidence Based Policy and Practice
9. Brandon, P.R.; Smith, N.L.; Trenholm, C. and Devaney, B. (2010) 'Evaluation Exemplar: The Critical Importance of Stakeholder Relations in a National, Experimental Abstinence Education Evaluation', American Journal of Evaluation 31.4: 517-31
10. Braudel, F. (1958) 'Histoire et Sciences Sociales: La Longue Durée', Annales, Histoire, Sciences Sociales 13.4: 725-53
11. Brown, A.N.; Cameron, D.B. and Wood, B.D.K. (2014) 'Quality Evidence for Policymaking. I'll Believe It When I See the Replication', International Initiative for Impact Evaluation, Replication Paper 1, Washington DC: International Initiative for Impact Evaluation (3ie)
12. Button, K.S.; Ioannidis, J.P.A.; Mokrysz, C.; Nosek, B.A.; Flint, J.; Robinson, E.S.J. and Munafò, M.R. (2013) 'Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience, Nature Reviews Neuroscience 14.5: 365-76
13. Cable, V. (2004) 'Evidence and UK Politics, Does Evidence Matter?', presentation as part of an ODI Meeting Series on 'Does Evidence Matter?': 11-13, www.odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/206.pdf (accessed 26 August 2014)
14. Camfield, L. and Palmer-Jones, R.W. (2013a) 'Improving the Quality of Development Research: What Could Archiving Qualitative Data for Reanalysis and Revisiting Research Sites Contribute?', Progress in Development Studies 13.4: 323-38
15. Camfield, L. and Palmer-Jones, R.W. (2013b) 'Editorial: As Well as the Subject: Additional Dimensions in Development Research Ethics', Progress in Development Studies 13.4: 255-65
16. Campbell, D.T. and Russo, M.J. (1998) Social Experimentation, 1st ed., Thousand Oaks CA: Sage Publications
17. Christley, R.M. (2010) 'Power and Error: Increased Risk of False Positive Results in Underpowered Studies', Open Epidemiology Journal 3: 16-19
18. Cochrane Collaboration (2011) Cochrane Handbook for Systematic Reviews of Interventions, version 5.1.0., http://handbook.cochrane.org/front_page.htm (accessed 4 August 2014)
19. Cohen, Stanley (2001) States of Denial: Knowing About Atrocities and Suffering, Cambridge: Polity Press
20. Cooke, Bill and Kothari, Uma (2001) Participation: The New Tyranny?, 1st ed., London and New York NY: Zed Books
21. Copestake, J. (2013) 'Credible Impact Evaluation in Complex Contexts: Confirmatory and Exploratory Approaches', Working Paper Draft, Centre for Development Studies, University of Bath, available from the author
22. Datta, L.-E. (2011) 'Politics and Evaluation: More than Methodology', American Journal of Evaluation 32.2: 273-94
23. Davis, Graham A. (2013) 'Replicating Sachs and Warner's Working Papers on the Resource Curse', Journal of Development Studies 49.12: 1615-30
24. DFID (2014) How-to-Note: Assessing the Strength of Evidence, www.gov.uk/government/uploads/system/uploads/attachment_data/file/291982/HTN-strength-evidence-march2014.pdf (accessed 4 August 2014)
25. DiMaggio, Paul J. and Powell, W.W. (1991) 'Introduction', in W.W. Powell and P.J. DiMaggio (eds), The New Institutionalism in Organisational Analysis, Chicago IL: University of Chicago Press
26. DiMaggio, Paul J. and Powell, W.W. (1983) 'The Iron Cage Revisited: Institutional Isomorphism and Collective Rationality in Organizational Fields', American Sociological Review 48.2: 147-60
27. Duflo, E.; Glennerster, R. and Kremer, M. (2007) Using Randomization in Development Economics Research: A Toolkit, Discussion Paper 6059, London: Centre for Economic Policy Research (CEPR)
28. Duvendack, M. and Palmer-Jones, R. (2013) 'Replication of Quantitative Work in Development Studies: Experiences and Suggestions', Progress in Development Studies 13.4: 307-22
29. Duvendack, M.; Hombrados, J.G.; Palmer-Jones, R. and Waddington, H. (2012) 'Assessing "What Works" in International Development: Meta-Analysis for Sophisticated Dummies', Journal of Development Effectiveness 4.3: 456-71
30. Duvendack, M.; Palmer-Jones, R.; Copestake, J.G.; Hooper, L.; Loke, Y. and Rao, N. (2011) What Is the Evidence of the Impact of Microfinance on the Well-Being of Poor People?, London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
31. Ebrahim, A. (2002) 'Implementation Struggles: the Role of Information in the Reproduction of NGO-Funder Relationships', Non-Profit and Voluntary Sector Quarterly 31.1: 84-114
32. EPPI-Centre (2010) EPPI Centre Methods for Conducting Systematic Reviews, Evidence for Policy and Practice Information and Co-ordinating Centre, http://eppi.ioe.ac.uk/cms/LinkClick.aspx?fileticket=hQBu8y4uVwI%3D&tabid=88 (accessed 4 August 2014)
33. Etzioni, Amitai (1975) A Comparative Analysis of Complex Organizations: On Power, Involvement, and their Correlates, New York NY: Free Press
34. Etzioni, Amitai (1964) Modern Organizations, Englewood Cliffs NJ: Prentice-Hall
35. Eyben, R. (2013) Uncovering the Politics of Evidence and Results. A Framing Paper for Development Practitioners, http://bigpushforward.net/wp-content/uploads/2011/01/The-politics-of-evidence-11-April-20133.pdf (accessed 4 August 2014)
36. Eyben, R. (2009) 'Hovering on the Threshold: Challenges and Opportunities for Critical and Reflexive Ethnographic Research in Support of International Aid Practice', in S. Hagberg and C. Widmark (eds), Ethnographic Practice and Public Aid. Methods and Meanings in Development Cooperation, Sweden: University of Uppsala
37. Fay, Brian (1975) Social Theory and Political Practice, London: Allen & Unwin
38. Feigenbaum, S. and Levy, D.M. (1993) 'The Market for (Ir)reproducible Econometrics', Social Epistemology 7.3: 215-32
39. Ferguson, J. (1990) The Anti Politics Machine. Development, Depoliticization and Bureaucratic Power in Lesotho, Cambridge: Cambridge University Press
40. Giddens, A. (1984) The Constitution of Society: Outline of the Theory of Structuration, Cambridge: Polity Press
41. Harding, S. (1987) 'Introduction: Is there a Feminist Method?', in S. Harding (ed.), Feminism and Methodology, Bloomington IN: University of Indiana Press
42. Heckman, J.J. and Smith, J.A. (1995) 'Assessing the Case for Social Experiments', Journal of Economic Perspectives 9.2: 85-110
43. Hogendorn, J. and Scott, K. (1981) 'The East-African Groundnut Scheme - Lessons of a Large-Scale Agricultural Failure', African Economic History 10: 81-115
44. Holdcroft, L. (1984) 'The Rise and Fall of Community Development, 1950-1965: A Critical Assessment', in C. Eicher and J. Staatz (eds), Agricultural Development in the Third World, Baltimore MD: Johns Hopkins University Press
45. House, E.R. (2008) 'Blowback: Consequences of Evaluation for Evaluation', American Journal of Evaluation 29.4: 416-26
46. House, E.R. (1973) School Evaluation: The Politics and Process, Berkeley CA: McCutchan Publishing
47. House of Commons (2011) Peer Review in Scientific Publications: Eighth Report of Session 2010-12, Vol. 1: Report, Together with Formal Minutes, Oral and Written Evidence, Vol 1, London: House of Commons Science and Technology Committee, The Stationery Office
48. Ioannidis, J.P.A. (2012) 'Why Science is Not Necessarily Self-Correcting', Perspectives on Psychological Science 7.6: 645-54
49. Ioannidis, J.P.A. (2005) 'Why Most Published Research Findings are False', PLOS Medicine 2.8: e124
50. Ioannidis, J.P.A. and Doucouliagos, C. (2013) 'What's to Know About the Credibility of Empirical Economics?: Scientific Credibility of Economics', Journal of Economic Surveys 27.5: 997-1004
51. Irwin, S. (2013) 'Qualitative Secondary Data Analysis: Ethics, Epistemology and Context', Progress in Development Studies 13.4: 295-306
52. John, Leslie K.; Loewenstein, George and Prelec, Drazen (2012) 'Measuring the Prevalence of Questionable Research Practices with Incentives for Truth Telling', Psychological Science 23.5: 524-32
53. Krueger, Anne O. (1974) 'The Political Economy of the Rent-Seeking Society', American Economic Review 64.3: 291-303
54. Kunda, Ziva (1990) 'The Case for Motivated Reasoning', Psychological Bulletin 108.3: 480-98
55. Loftus, G.R. (1991) 'On the Tyranny of Hypothesis Testing in the Social Sciences', Contemporary Psychology 36.2: 102-5
56. Maniadis, Z.; Tufano, F. and List, J.A. (2014) 'One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects, ' American Economic Review 104.1: 277-90
57. Manski, C.F. (2011) 'Policy Analysis with Incredible Certitude', Economic Journal 121.554: F261-89
58. Manski, C.F. (2007) Identification for Prediction and Decision, Cambridge MA: Harvard
59. Merleau-Ponty, M. (1942) The Structure of Behavior, trans. Alden Fisher, Boston MA: Beacon Press, 1963; London: Methuen, 1965
60. Merton, R.K. (1973) The Sociology of Science: Theoretical and Empirical Investigations, Chicago IL and London: University of Chicago Press
61. Merton, R.K. (1942) 'Science and Technology in a Democratic Order', Journal of Legal and Political Sociology 1: 115-26
62. Mirowski, P. and Sklivas, S. (1991) 'Why Econometricians Don't Replicate (Although they Do Reproduce)', Review of Political Economy 3: 146-63
63. Moonesinghe, Ramal; Khoury, Muin J. and Janssens, Cecile J.W. (2007) 'Most Published Research Findings Are False - But a Little Replication Goes a Long Way', PLOS Medicine 4.2: 218-21
64. Mosse, David (2005) Cultivating Development: An Ethnography of Aid Policy and Practice, London and Ann Arbor MI: Pluto Press
65. Mosse, D. (2001) 'People's Knowledge, Participation and Patronage', in B. Cooke and U. Kothari (eds), Participation: The New Tyranny?, London: Zed Press
66. Nosek, Brian A.; Spies, Jeffrey R. and Motyl, Matt (2012) 'Scientific Utopia II. Restructuring Incentives and Practices to Promote Truth Over Publishability', Perspectives on Psychological Science 7.6: 615-31
67. Pashler, Harold and Wagenmakers, Eric-Jan (2012) 'Editors' Introduction to the Special Section on Replicability in Psychological Science: A Crisis of Confidence?', Perspectives on Psychological Science 7.6: 528-30
68. Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use, 1st ed., New York NY: The Guilford Press
69. Patton, M.Q. (2002) Qualitative Research and Evaluation Methods, London: Sage
70. Petticrew, M. and Roberts, H. (2006) Systematic Reviews in the Social Sciences: A Practical Guide, Oxford: Blackwell Publishing
71. Phillips, B.; Ball, C.; Sackett, D.; Badenoch, D.; Straus, S.; Haynes, B. and Dawes, M. (2009) Oxford Centre for Evidence Based Medicine - Level of Evidence, www.cebm.net/index.aspx?o=1025 (accessed 4 August 2014)
72. Piketty, Thomas (2014) Capital in the Twenty-First Century, trans. Arthur Goldhammer, 1st ed., Cambridge MA: Belknap Press
73. Pritchett, Lant (2002) 'It Pays to Be Ignorant: A Simple Political Economy of Rigorous Program Evaluation', Journal of Policy Reform 5.4: 251-69
74. Pritchett, Lant; Woolcock, Michael and Andrews, Matt (2013) 'Looking Like a State: Techniques of Persistent Failure in State Capability for Implementation', Journal of Development Studies 49.1: 1-18
75. Reitbergen-McCracken, J. and Narayan, D. (1998) Participation and Social Assessment: Tools and Techniques, Washington DC: World Bank
76. Rosenthal, R. (1979) 'The "File Drawer" Problem and Tolerance for Null Results', Psychological Bulletin 86: 638-41
77. Rosenthal, R. (1966) Experimenter Effects in Behavioral Research, Vol. xiii, East Norwalk CT: Appleton-Century-Crofts
78. Savage, Mike (2005) 'Revisiting Classic Qualitative Studies', Forum Qualitative Sozialforschung / Forum: Qualitative Social Research 6.1, www.qualitative-research.net/index.php/fqs/article/view/502 (accessed 4 August 2014)
79. Savedoff, W.D.; Levine, R. and Birdsall N. (2005) When Will We Ever Learn? Improving Lives Through Impact Evaluation, Center for Global Development, Washington DC, www.cgdev.org/publication/when-will-we-ever-learn-improving-lives-through-impact-evaluation (accessed 4 August 2014)
80. Schooler, Jonathan (2011) 'Unpublished Results Hide the Decline Effect', Nature News 470.7335: 437
81. Shaffer, Paul (2011) 'Against Excessive Rhetoric in Impact Assessment: Overstating the Case for Randomised Controlled Experiments', Journal of Development Studies 47.11: 1619-35
82. Sen, Amartya (1993) 'Positional Objectivity', Philosophy and Public Affairs 22.2: 126-45
83. Simmons, J.P.; Nelson, L.D. and Simonsohn, U. (2011) 'False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant', Psychological Science 22.11: 1359-66
84. Temple, B.; Edwards, R. and Alexander, C. (2006) 'Grasping at Context: Cross Language Qualitative Research as Secondary Qualitative Data Analysis', FQS 7.4, art. 10
85. Thody, Angela (2006) Principles for Selecting Appropriate Writing and Presentation Styles, in Writing and Presenting Research, London: Sage
86. Tversky, Amos and Kahneman, Daniel (1971) 'Belief in the Law of Small Numbers', Psychological Bulletin 76.2: 105-10
87. Wacholder, Sholom; Chanock, Stephen; Garcia-Closas, Montserrat; El Ghormli, Laure and Rothman, Nathaniel (2004) 'Assessing the Probability that a Positive Report is False: An Approach for Molecular Epidemiology Studies', Journal of the National Cancer Institute 96.6: 434-42
88. Walker, D.; Bergh, G.; Page, E. and Duvendack, M. (2013) Adapting a Systematic Review for Social Research in International Development: A Case Study from the Child Protection Sector, London: Overseas Development Institute
89. Walker, Kate; Neuburger, Jenny; Groene, Oliver; Cromwell, David A. and van der Meulen, Jan (2013) 'Public Reporting of Surgeon Outcomes: Low Numbers of Procedures Lead to False Complacency', The Lancet 382.9905: 1674-7
90. Weick, K.E. (1974) 'Education Organisations and Loosely Coupled Systems', Administratics Science Quarterly 2.1: 1-19
91. White, Howard and Masset, Edoardo (2007) 'Assessing Interventions to Improve Child Nutrition: A Theory-Based Impact Evaluation of the Bangladesh Integrated Nutrition Project', Journal of International Development 19.5: 627-52
92. White, Howard and Phillips, Daniel (2012) Addressing Attribution of Cause and Effect in Small n Impact Evaluations: Towards an Integrated Framework, New Delhi: International Initiative for Impact Evaluation
93. Wilson, Timothy D.; DePaulo, Bella M.; Mook, Douglas G. and Klaaren, Kristen J. (1993) 'Scientists' Evaluations of Research: The Biasing Effects of the Importance of the Topic', Psychological Science 4.5: 322-5