Replication of quantitative work in development studies: Experiences and suggestions

Progress in Development Studies

Volumn 13, Issue 4, 2013, Pages 307-322

  Duvendack, Maren ^a   Palmer Jones, Richard ^a  

^a UNIVERSITY OF EAST ANGLIA   (United Kingdom)

Author keywords

economic methodology; incentives; professionalism; Replication; research ethics; sociology of economics

Indexed keywords

DEVELOPMENT ECONOMICS; ECONOMICS; ETHICS; INCENTIVE; QUANTITATIVE ANALYSIS; RESEARCH;

EID: 84883038976     PISSN: 14649934     EISSN: None     Source Type: Journal    
DOI: 10.1177/1464993413490480     Document Type: Article

References (124)

1. Abbott A.The system of the professions: Essay on the division of expert labour. Chicago University Press; 1988:.
2. Acemoglu D.,Johnson S.,Robinson J.A.The colonial origins of comparative development: An empirical investigation.American Economic Review. 2001;91:1369-401.
3. Acemoglu D.,Johnson S.,Robinson J.A.The colonial origins of comparative development: An empirical investigation: Reply.American Economic Review. 2012;102:3077-3110.
4. AlbouyD.Y.2004: The colonial origins of comparative development: A reexamination based on improved settler mortality data. Unpublished paper, University of California, Berkeley, July.
5. Albouy D.Y.The colonial origins of comparative development: An investigation of the settler mortality data. Cambridge, MA:; 2008:.
6. Albouy D.Y.The colonial origins of comparative development: An empirical investigation: Comment.American Economic Review. 2012;102:3059-76.
7. Anderson R.G.,Greene W.H.,McCullough B.D.,Vinod H.D.The role of data/code archives in the future of economic research.Journal of Economic Methodology. 2008;15:99-119.
8. Antonovics K.L.,Goldberger A.S.Does increasing women's schooling raise the schooling of the next generation? Comment.American Economic Review. 2005;95:1738-44.
9. The art of rhetoric. Penguin Classics; 1991:.
10. ArndtC.JonesS.TarpF.2011: Aid effectiveness: Opening the black box. World Institute for Development Economic Research (UNU-WIDER), http://ideas.repec.org/p/unu/wpaper/wp2011-44.html, last accessed 5 July 2012.
11. Banerjee A.V.'New development economics' and the challenge to theory.Economic and Political Weekly. 2005;40:4340-44.
12. Banerjee A.,Duflo E.Poor economics. New York: Public Affairs; 2011:.
13. Banerjee A.,Iyer L.Colonial land tenure, electoral competition and public goods in India.American Economic Review. 2005;95:1190-213.
14. Behrman J.R.,Rosenzweig M.R.Does increasing women's schooling raise the schooling of the next generation?.American Economic Review. 2002;92:323-34.
15. Behrman J.R.,Rosenzweig M.R.Does increasing women's schooling raise the schooling of the next generation? Reply.American Economic Review. 2005;95:1745-51.
16. Bernanke B.S.Editorial statement.American Economic Review. 2004;94:404.
17. Bourdieu P.Outline of a theory of culture. Cambridge University Press; 1977:.
18. Bourdieu P.Language and symbolic power. Polity Press; 1991:.
19. Bourdieu P.Practical reason: On the theory of action. Polity Press; 1998:.
20. Boyce J.K.,Ravallion Martin.A dynamic econometric model of agricultural wage determination in Bangladesh.Oxford Bulletin of Economics and Statistics. 1991;53:361-76.
21. Broad R.Research, knowledge, and the art of 'paradigm maintenance': The World Bank's development economics vice-presidency (DEC).Review of International Political Economy. 2006;13:387-419.
22. Burman L.E.,Reed W.R.,Alm J.A call for replication studies.Public Finance Review. 2010;38:787-93.
23. Burnside R.J.,Dollar D.Aid, policies and growth.American Economic Review. 2000;90:847-68.
24. Card D.,Katz L.F.,Krueger A.B.Comment on David Neumark and William Wascher, 'Employment effects of minimum and subminimum wages: Panel data on state minimum wage laws'.Industrial and Labor Relations Review. 1994;47:487-97.
25. Card D.,Krueger A.B.Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania: Reply.American Economic Review. 2000;90:1397-420.
26. Chandrasekar S.Truth and beauty: Aesthetics and motivations in science. University of Chicago Press; 1985:.
27. Chemin M.The benefits and costs of microfinance: Evidence from Bangladesh.Journal of Development Studies. 2008;44:463-84.
28. Chemin M.Response to 'high noon for microfinance impact evaluation'.The Journal of Development Studies. 2012a;48:1881-85.
29. CheminM.2012b: Response to 'high noon for microfinance impact evaluation', http://matthieuchemin-research.mcgill.ca/research.html, last accessed on 3 June 2013.
30. Collins H.M.Changing order: Replication and induction in scientific practice. University of Chicago Press; 1985:.
31. Csikszentmihalyi M.Finding flow: The psychology of engagement with everyday life. Harper Row; 1990:.
32. Della Vigna S.,Gentzkow M.Persuasion: Empirical evidence.Annual Review of Economics. 2010;2:643-69.
33. Dewald W.G.,Thursby J.G.,Anderson R.G.Replication in empirical economics: The journal of money, credit and banking project.American Economic Review. 1986;76:587-603.
34. Donoho D.L.An invitation to reproducible computational research.Biostatistics. 2010;11:385-88.
35. DonohueJ.LevittS.D.2001: The impact of legalized abortion on crime. Quarterly Journal of Economics116, http://works.bepress.com/john_donohue/8/, last accessed 19 June 2012.
36. DonohueJ.LevittS.D.2004: Further evidence that legalized abortion lowered crime: A reply to Joyce. Journal of Human Resources39, 29-49, http://works.bepress.com/cgi/viewcontent.cgi?article=1011&context=john_donohue, last accessed 19 June 2012.
37. Donohue J.,Levitt S.D.Measurement error, legalized abortion, and the decline in crime: A response to Foote and Goetz.The Quarterly Journal of Economics. 2008;123:425-40.
38. Duvendack M.,Palmer-Jones R.High noon for microfinance impact evaluations: Re-investigating the evidence from Bangladesh.Journal of Development Studies. 2012a;48:1864-80.
39. Duvendack M.,Palmer-Jones R.Reply to Chemin and to Pitt.The Journal of Development Studies. 2012b;48:1892-97.
40. DuvendackM.Palmer-JonesR.2012c. What Mat[t]hieu did next: Reply to Chemin, [online], http://www.uea.ac.uk/~nga07htu/Rejoinder_to_Chemin_Final.pdf.
41. DuvendackM.Palmer-JonesR.2012d. Wyatt Earp's high noon: Reply to Pitt [online], http://www.uea.ac.uk/~nga07htu/Rejoinder_to_Pitt_Final.pdf.
42. Easley R.W.,Madden C.S.,Dunn M.G.Conducting marketing science: The role of replication in the research process.Journal of Business Research. 2000;48:83-92.
43. Easterly William,Levine Ross,Roodman D.Aid, policies, and growth: Comment.The American Economic Review. 2004;94:774-80.
44. Feigenbaum S.,Levy D.M.The market for (ir)reproducible econometrics.Social Epistemology. 1993;7:215-32.
45. Feldstein M.S.Social security and private saving: Reply.Journal of Political Economy. 1982;90:630-42.
46. Foote C.L.,Goetz C.F.The impact of legalized abortion on crime: Comment.The Quarterly Journal of Economics. 2008;123:407-23.
47. Freeze J.Replication standards for quantitative social science: Why not sociology?.Sociological Methods and Research. 2007;36:153-72.
48. Freidson E.Profession of medicine: A study of the sociology of applied knowledge. Chicago University Press; 1970:.
49. Freidson E.Professionalism: The third logic. Chicago University Press; 2001:.
50. FrischR.1933: Editor's note. Econometrica1, 1-4, http://www.jstor.org/stable/1912224, last accessed on 3 June 2013.
51. Gustin B.H.Charisma, recognition and the motivation of scientists.American Journal of Sociology. 1973;78:1119-33.
52. GlandonP.2010: Report on the American Economic Review data availability compliance project, http://www.aeaweb.org/aer/2011_Data_Compliance_Report.pdf, last accessed on 3 June 2013.
53. Gleditsch P.N.,Metelits C.The replication debate.International Studies Perspectives. 2003;4:443-99.
54. Goldacre B.Bad science. Fourth Estate; 2008:.
55. Rigour, respect, responsibility: A universal ethical code for scientists. Government Office of Science; 2007:.
56. HadornD.1996: The role of cognitive errors in the drug policy debate. International Journal of Drug Policy7, 166-71, http://www.druglibrary.net/schaffer/media/hadorn1.htm.
57. Hamermesh D.S.Viewpoint: Replication in economics.Canadian Journal of Economics. 2007;40:715-33.
58. Hammersley M.Against the ethicists: On the evils of ethical regulation.International Journal of Social Research Methodology. 2009;12:211-25.
59. Hansen H.,Tarp F.Aid effectiveness disputed.Journal of International Development. 2000;12:375-98.
60. Hatton L.,Roberts A.How accurate is scientific software?.IEEE Trans. Softw. Eng. 1994;20:785-97.
61. Hendry D.F.,Krolzig H.-M.We ran one regression.Oxford Bulletin of Economics and Statistics. 2004;66:799-810.
62. HernandezJ.C.2005: Star Ec prof caught in academic feud. The Harvard crimson. http://www.thecrimson.com/article/2005/7/8/star-ec-prof-caught-in-academic/, last accessed 14 August 2012.
63. Hoxby C.M.The effects of class size on student achievement: New evidence from population variation.The Quarterly Journal of Economics. 2000;115:1239-85.
64. HoxbyC.M.2005: Competition among public schools: A reply to Rothstein (2004). National Bureau of Economic Research Working Paper Series No. 11216, http://www.nber.org/papers/w11216, last accessed on 3 June 2013.
65. Hoxby C.M.Does competition among public schools benefit students and taxpayers? A reply.American Economic Review. 2007;97:2038-55.
66. Ionnidis J.P.A.Why most published research findings are false.PLoS Medicine. 2005;28:e124.
67. Jasny B.R.,Chin G.,Chong L.,Vignieri S.Again, and Again, and Again..Science. 2011 1225;334:.
68. Johansson S.R.Food for thought: Rhetoric and reality in modern mortality history.Historical Methods. 1994;27:101-25.
69. Joyce T.Did legalized abortion lower crime?.Journal of Human Resources. 2004;39:1-28.
70. Kahneman D.,Slovic P.,Tversky A.Judgement under uncertainty: Heuristics and biases. Cambridge University Press; 1982:.
71. Kane E.J.Why journal editors should encourage the replication of applied econometric research.Quarterly Journal of Business and Economics. 1984;23:3-8.
72. Kerr N.L.HARKing: Hypothesizing after the results are known.Personality and Social Psychology Review. 1998;2:196-217.
73. Kevles D.The Baltimore case. Norton; 1998:.
74. King G.Replication, replication.PS: Political Science and Politics. 1995;28:444-52.
75. Latour B.Science in Action: How to follow scientists and engineers through society. Harvard University Press; 1987:.
76. Leamer E.E.Let's take the con out of econometrics.American Economic Review. 1983;73:31-43.
77. Leimer D.R.,Lesnoy S.D.Social Security and Private Saving: New Time-Series Evidence.Journal of Political Economy. 1982;90:606-29.
78. Levitt S.D.Using electoral cycles in police hiring to estimate the effect of police on crime.The American Economic Review. 1997;87:270-90.
79. Levitt S.D.Using electoral cycles in police hiring to estimate the effects of police on crime: Reply.The American Economic Review. 2002;92:1244-50.
80. Madden C.S.,Easley R.W.,Dunn M.G.How journal editors view replication research.Journal of Advertising. 1995;24:77-87.
81. Mann M.The hockey stick and the climate wars: Dispatches from the front lines. 2012. The hockey stick and the climate wars: Dispatches from the front lines (1st edition). Columbia University Press.Columbia University Press; 2012:.
82. Manski C.F.Policy analysis with incredible certitude.The Economic Journal. 2011;121:F261-F289.
83. McCloskey D.N.The rhetoric of economics. University of Wisconsin Press; 1985:.
84. McCrary J.Using electoral cycles in police hiring to estimate the effect of police on crime: Comment.The American Economic Review. 2002;92:1236-43.
85. McCullough B.D.Open access economics journals and the market for reproducible economic research.Economic Analysis and Policy. 2009;39:117-26.
86. McCullough B.D.,McGeary K.A.,Harrison T.D.Do economics journal archives promote replicable research? Est-ce que les archives des revues économiques promeuvent les travaux de réplication.Canadian Journal of Economics. 2008;41:1406-20.
87. McCullogh B.D.,McKitrick R.Check the numbers: The case for due diligence in policy formation. Fraser Institute; 2009:.
88. McCullough B.D.,Vinod H.D.The numerical reliability of econometric software.Journal of Economic Literature. 1999;37:633-55.
89. McCullough B.D.,Vinod H.D.Verifying the solution from a nonlinear solver: A case study.The American Economic Review. 2003;93:873-92.
90. McCullough B.,McGeary K.A.,Harrison T.D.Lessons from the JMCB archive.Journal of Money, Credit and Banking. 2006;38:1093-107.
91. McCullough B.,McGeary K.A.,Harrison T.D.Do economics journal archives promote replicable research?.Canadian Journal of Economics. 2008;41:1406-20.
92. Macdonald K.M.The sociology of the professions. SAGE Publications Ltd; 1995:.
93. Merali Z.Computational science:..Error.Nature News. 2010;467:775-77.
94. Mirowski P.,Sklivas S.Why econometricians don't replicate (although they do reproduce).Review of Political Economy. 1991;3:146-63.
95. MorduchJ.1998: Does microfinance really help the poor? New evidence from flagship programs in Bangladesh. Unpublished mimeo.
96. Mosse D.Adventures in aidland: The anthropology of professionals in international development. Berghahn Books; 2011:.
97. Nagler J.Coding style and good computing practices.Political Science & Politics. 1995;3:488-92.
98. Neumark D.,Wascher W.Employment effects of minimum and subminimum wages: A reply to Card, Katz, and Kreuger.Industrial and Labour Relations Review. 1994;47:487-97.
99. Neumark D.,Wascher W.Minimum wages and employment: A case study of the fast-food industry in New Jersey and Pennsylvania: Comment.American Economic Review. 2000;90:1362-96.
100. O'Hara R.Negative results are published.Nature. 2011;470:448.
101. Palmer-Jones R.W.Sonar Bangla? Agrarian structure and agricultural growth in Bengal. Rogaly B.Harris-White B.Bose S., ed. SAGE; 1999:92-136.
102. Peng R.D.Reproducible research in computational science.Science. 2011;334:1226-27.
103. Pesaran H.Introducing a replication section.Journal of Applied Econometrics. 2003;18:111.
104. Pesaran H.An example of replication and data archiving in journal publications.; 2012; 2012. An example of replication and data archiving in journal publications. Presentation at the Association of Learned and Professional Society Publishers (ALPSP) Seminar, Data Publishing, London, accessed on 3 June 2013
105. PittM.M.1999: Reply to Jonathan Morduch's 'Does microfinance really help the poor? New evidence from flagship programs in Bangladesh'. Unpublished mimeo.
106. Pitt M.M.Gunfight and the Not OK corral: Reply to 'High noon for microfinance'.The Journal of Development Studies. 2012a;48:1886-91.
107. PittM.M.2012b: Gunfight and the NOT OK corral: Reply to 'High noon for microfinance' by Duvendack and Palmer-Jones (Uncut Version) [online], http://www.brown.edu/research/projects/pitt/sites/brown.edu.research.projects.pitt/files/uploads/faults6.pdf, last accessed on 3 June 2012.
108. Pitt M.M.,Khandker S.R.The impact of group-based credit programs on poor households in Bangladesh: Does the gender of participants matter?.Journal of Political Economy. 1998;106:958-96.
109. Roodman D.Due diligence: An impertinent inquiry into microfinance. Center for Global Development; 2012:.
110. RoodmanD.MorduchJ.2009: The impact of microcredit on the poor in Bangladesh: Revisiting the evidence. Center for Global Development, Working Paper No. 174, June.
111. RothsteinJ.2005: Does competition among public schools benefit students and taxpayers? A comment on Hoxby (2000). National Bureau of Economic Research Working Paper Series No. 11215.
112. Rothstein J.Does competition among public schools benefit students and taxpayers? A comment on Hoxby (2000).American Economic Review. 2007a;97:2026-37.
113. RothsteinJ.2007b: Rejoinder to Hoxby. http://gsppi.berkeley.edu/faculty/jrothstein/hoxby/documentation-for-hoxby-comment/, last accessed on 3 June 2013.
114. Sala-i-MartinX.X.1997: I just ran four million regressions. National Bureau of Economic Research, http://www.nber.org/papers/w6252, last accessed 7 August 2012.
115. Schooler J.Unpublished results hide the decline effect.Nature News. 2011;470:437-37.
116. Selvin H.C.,Stuart A.Data-dredging procedures in survey analysis.The American Statistician. 1966;20:20-23.
117. Sen A.K.Description as choice.Oxford Economic Papers. 1982;32:353-69.
118. Stokes E.The English utilitarians and India. Oxford University Press; 1989:.
119. Help in the right places.The Economist. 2002;:.
120. Tversky A.,Kahneman D.Judgment under uncertainty: Heuristics and biases.Science. 1974;185:1124-31.
121. US EPA2010: Denial of petitions for reconsideration of the endangerment and cause or contribute findings for greenhouse gases under Section 202(a) of the Clean Air Act [WWW document]. Climate Change Division, http://epa.gov/climatechange/endangerment/petitions.html, last accessed on 7 May 2012.
122. Wade R.US hegemony and the World Bank: The fight over people and ideas.Review of International Political Economy. 2002;9:215-43.
123. Does aid help.Washington Post. 2002;:A26.
124. Ziliak S.T.,McCloskey D.N.Thecult of statistical significance: How the standard error cost us jobs, justice, and lives. University of Michigan Press; 2008:.