Quality evidence for policymaking: I'll believe it when I see the replication

Journal of Development Effectiveness

Volumn 6, Issue 3, 2014, Pages 215-235

  Brown, Annette N ^a   Cameron, Drew B ^a   Wood, Benjamin D K ^a  

^a London School of Hygiene and Tropical Medicine   (United States)

Author keywords

impact evaluation; policy analysis; replication; statistical methods; theory of change

Indexed keywords

DEVELOPING WORLD; DEVELOPMENT THEORY; ECONOMIC DEVELOPMENT; LOW INCOME POPULATION; POLICY ANALYSIS; POLICY MAKING; RESEARCH AND DEVELOPMENT; RESEARCH WORK; ACADEMIC RESEARCH; DESIGN; STATISTICAL ANALYSIS; THEORETICAL STUDY;

EID: 84907596523     PISSN: 19439342     EISSN: 19439407     Source Type: Journal    
DOI: 10.1080/19439342.2014.944555     Document Type: Article

References (89)

1. Abbott, A. 2007. "Notes on Replication." Sociological Methods & Research 36 (2): 210-219. doi:10.1177/0049124107306664.
2. Acemoglu, D., S. Johnson, J. A. Robinson, and P. Yared. 2008. "Income and Democracy." American Economic Review 98 (3): 808-842. doi:10.1257/aer.98.3.808.
3. Angrist, J., and J.-S. Pischke. 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con Out of Econometrics." Journal of Economic Perspectives 24 (2): 3-30. doi:10.1257/jep.24.2.3.
4. Auvert, B., D. Taljaard, E. Lagarde, J. Sobngwi-Tambekou, R. Sitta, and A. Puren. 2005. "Randomized, Controlled Intervention Trial of Male Circumcision for Reduction of HIV Infection Risk: The ANRS 1265 Trial." PLoS Medicine 2 (11): e298. doi:10.1371/journal.pmed.0020298.
5. Bailey, R. C., S. Moses, C. B. Parker, K. Agot, I. Maclean, J. N. Krieger, C. F. M. Williams, R. T. Campbell, and J. O. Ndinya-Achola. 2007. Male Circumcision for HIV Prevention in Young Men in Kisumu, Kenya: A Randomised Controlled Trial." The Lancet 369 (9562): 643-656. doi:10.1016/S0140-6736(07)60312-2.
6. Baker, D. 2013. In History of Economic Errors, Martin Feldstein Deserves Mention. Center for Economic and Policy Research, April 17. Accessed February 12, 2014. http://www.cepr.net/index.php/blogs/beat-the-press/in-history-of-economic-errors-martin-feldstein-deserves-mention
7. Banerjee, A., S. Cole, E. Duflo, and L. Linden. 2007. "Remedying Education: Evidence from Two Randomized Experiments in India." The Quarterly Journal of Economics 122 (3): 1235-1264. doi:10.1162/qjec.122.3.1235.
8. Basu, K., and J. E. Foster. 1998. "On Measuring Literacy." The Economic Journal 108 (451): 1733-1749. doi:10.1111/1468-0297.00369.
9. Bernanke, B. 2004. "Editorial Statement." American Economic Review 94 (1): 404.
10. Berthon, P., L. Pitt, M. Ewing, and C. L. Carr. 2002. "Potential Research Space in MIS: A Framework for Envisioning and Evaluating Research Replication, Extension, and Generation." Information Systems Research 13 (4): 416-427. doi:10.1287/isre.13.4.416.71.
11. Binmore, K., and A. Shaked. 2010. "Experimental Economics: Where Next?" Journal of Economic Behavior and Organization 73 (1): 87-100. doi:10.1016/j.jebo.2008.10.019.
12. Bold, T., M. Kimenyi, G. Mwabu, A. Ng'ang'a, and J. Sandefur. 2013. Scaling up What Works: Experimental Evidence on External Validity in Kenyan Education. Working Paper 321. Washington, DC: Center for Global Development.
13. Bueno De Mesquita, B. 2003. "Getting Firm on Replication." International Studies Perspectives 4 (1): 98-100.
14. Burman, L. E., W. R. Reed, and J. Alm. 2010. "A Call for Replication Studies." Public Finance Review 38 (6): 787-793. doi:10.1177/1091142110385210.
15. Carter, M. R., and C. B. Barrett. 2006. "The Economics of Poverty Traps and Persistent Poverty: An Asset-Based Approach." Journal of Development Studies 42 (2): 178-199. doi:10.1080/00220380500405261.
16. Cartwright, N. 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins." History of Political Economy 23 (1): 143-155. doi:10.1215/00182702-23-1-143.
17. Cattaneo, M. D., S. Galiani, P. J. Gertler, S. Martinez, and R. Titiunik. 2009. "Housing, Health, and Happiness." American Economic Journal: Economic Policy 1 (1): 75-105.
18. Cervellati, M., F. Jung, U. Sunde, and T. Vischer. 2014. "Income and Democracy: Comment." American Economic Review 104 (2): 707-719. doi:10.1257/aer.104.2.707.
19. Chen, S., and M. Ravallion. 2007. Absolute Poverty Measures for the Developing World, 1981-2004. Policy Research Working Paper 4211. Washington, DC: World Bank.
20. Collins, H. M. 1984. "When Do Scientists Prefer to Vary Their Experiments?" Studies in History and Philosophy of Science Part A 15 (2): 169-174. doi:10.1016/0039-3681(84)90005-0.
21. Davis, G. 2012. Replicating 'Sources of Slow Growth in African Economies'. Division of Economics and Business Working Paper Series 2012-09. Golden, CO: Colorado School of Mines.
22. Day, T. E., and S. J. Liebowitz. 1998. "Mortgage Lending to Minorities: Where's the Bias?" Economic Inquiry 36 (1): 3-28. doi:10.1111/j.1465-7295.1998.tb01692.x.
23. Deaton, A. 2010. "Instruments, Randomization, and Learning about Development." Journal of Economic Literature 48 (2): 424-455. doi:10.1257/jel.48.2.424.
24. Dewald, W. G., J. G. Thursby, and R. G. Anderson. 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project." The American Economic Review 76 (4): 587-603.
25. Donohue, J. J., and S. D. Levitt. 2001. "The Impact of Legalized Abortion on Crime." The Quarterly Journal of Economics 116 (2): 379-420. doi:10.1162/00335530151144050.
26. Donohue, J. J., and S. D. Levitt. 2008. "Measurement Error, Legalized Abortion, and the Decline in Crime: A Response to Foote and Goetz." The Quarterly Journal of Economics 123: 425-440. doi:10.1162/qjec.2008.123.1.425.
27. Duflo, E., P. Dupas, and M. Kremer. 2012. School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools. Working Paper Series 17939. Cambridge, MA: National Bureau of Economic Research.
28. Evanschitzky, H., C. Baumgarth, R. Hubbard, and J. S. Armstrong. 2007. "Replication Research's Disturbing Trend." Journal of Business Research 60 (4): 411-415. doi:10.1016/j.jbusres.2006.12.003.
29. Falk, R. 1998. "Replication-A Step in the Right Direction: Commentary on Sohn." Theory & Psychology 8 (3): 313-321. doi:10.1177/0959354398083002.
30. Fanelli, D. 2010. "'Positive' Results Increase down the Hierarchy of the Sciences." PLoS One 5 (4): e10068. doi:10.1371/journal.pone.0010068.
31. Feldstein, M. 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation." Journal of Political Economy 82 (5): 905-926. doi:10.1086/260246.
32. Feldstein, M. 1982. "Social Security and Private Saving: Reply." Journal of Political Economy 90 (3): 630-642. doi:10.1086/261078.
33. Feldstein, M. 1996. "Social Security and Saving: New Time Series Evidence." National Tax Journal 49 (2): 151-164.
34. Folbre, N. 2013. "Replicating Research: Austerity and Beyond." The New York Times, April 22. Accessed February 12, 2014. http://economix.blogs.nytimes.com/2013/04/22/replicating-research-austerity-and-beyond/?_php=true&_type=blogs&_r=0
35. Foote, C. L., and C. F. Goetz. 2008. "The Impact of Legalized Abortion on Crime: Comment." The Quarterly Journal of Economics 123 (1): 407-423. doi:10.1162/qjec.2008.123.1.407.
36. Freese, J. 2007. "Overcoming Objections to Open-Source Social Science." Sociological Methods & Research 36 (2): 220-226. doi:10.1177/0049124107306665.
37. García, F. M. 2013. "Scientific Progress in the Absence of New Data: A Procedural Replication of Ross (2006)." Available at SSRN 2256286. Accessed February 12, 2014. http://ssrn.com/abstract=2256286
38. Gerber, A., and N. Malhotra. 2008. "Publication Bias in Empirical Sociological Research: Do Arbitrary Significance Levels Distort Published Results?" Sociological Methods & Research 37 (1): 3-30. doi:10.1177/0049124108318973.
39. Gherghina, S., and A. Katsanidou. 2013. "Data Availability in Political Science Journals." European Political Science 12 (3): 333-349. doi:10.1057/eps.2013.8.
40. Gleditsch, N. P., and C. Metelits. 2003. "Replication in International Relations Journals: Policies and Practices." International Studies Perspectives 4 (1): 89-97.
41. Gleditsch, N. P., C. Metelits, and H. Strand. 2003. "Posting Your Data: Will You Be Scooped or Will You Be Famous?" International Studies Perspectives 4 (1): 89-97.
42. Gray, R. H., G. Kigozi, D. Serwadda, F. Makumbi, S. Watya, F. Nalugoda, N. Kiwanuka, et al. 2007. "Male Circumcision for HIV Prevention in Men in Rakai, Uganda: A Randomised Trial." The Lancet 369 (9562): 657-666. doi:10.1016/S0140-6736(07)60313-4.
43. Hamermesh, D. S. 2007. "Viewpoint: Replication in Economics." Canadian Journal of Economics/Revue Canadienne D'économique 40 (3): 715-733. doi:10.1111/j.1365-2966.2007.00428.x.
44. Harrison, G. W. 1998. "Mortgage Lending in Boston: A Reconsideration of the Evidence." Economic Inquiry 36 (1): 29-38. doi:10.1111/j.1465-7295.1998.tb01693.x.
45. Herndon, T., M. Ash, and R. Pollin. 2013. Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff. Working Paper Series 332. Political Economy Research Institute, University of Massachusetts, Amherst.
46. Herrnson, P. S. 1995. "Replication, Verification, Secondary Analysis, and Data Collection in Political Science." PS: Political Science and Politics 28 (3): 452-455.
47. Horne, D. K. 1994. "Evaluating the Role of Race in Mortgage Lending." FDIC Banking Review. 7 (1): 1-15.
48. Horne, D. K. 1997. "Mortgage Lending, Race, and Model Specification." Journal of Financial Services Research 11 (1-2): 43-68. doi:10.1023/A:1007927123582.
49. Center for Effective Global Action. n.d. "Housing, Health and Happiness: The Impacts of Cement Flooring in Mexico." Center for Effective Global Action E2A Cement Floors Brief. Accessed February 12, 2014. http://cega.berkeley.edu/assets/cega_events/19/E2A_Cement_Floors_Brief.pdf
50. Hubbard, R., and D. E. Vetter. 1992. "The Publication Incidence of Replications and Critical Commentary in Economics." American Economist 36 (1): 29-34.
51. Humphreys, M., R. S. De La Sierra, and P. Van Der Windt. 2013. "Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration." Political Analysis 21 (1): 1-20. doi:10.1093/pan/mps021.
52. IDEAS/RePEc. 2013. "Ideas/Repec Simple Impact Factors for Journals." Federal Reserve Bank of St. Louis. Accessed February 12, 2014. http://ideas.repec.org/top/top.journals.simple.html
53. Ioannidis, J. P. 2005. "Contradicted and Initially Stronger Effects in Highly Cited Clinical Research." JAMA: The Journal of the American Medical Association 294 (2): 218-228. doi:10.1001/jama.294.2.218.
54. Iversen, V., and R. Palmer-Jones. 2014. "TV, Female Empowerment and Fertility Decline in Rural India." 3ie Replication Paper Series #2. International Initiative for Impact Evaluation, Washington, DC.
55. Jensen, R., and E. Oster. 2009. "The Power of TV: Cable Television and Women's Status in India." Quarterly Journal of Economics 124 (3): 1057-1094. doi:10.1162/qjec.2009.124.3.1057.
56. Jensen, R., and E. Oster. 2012. Corrigendum, "The Power of TV." Emily Oster's website, 2012. Accessed March 12, 2014. http://faculty.chicagobooth.edu/emily.oster/papers/update.pdf
57. King, G. 1995. "Replication, Replication." PS: Political Science and Politics 28 (3): 444-452.
58. King, G. 2006. "Publication, Publication." PS: Political Science and Politics 39 (1): 119-125.
59. Leamer, E. E. 1983. "Let's Take the Con Out of Econometrics." American Economic Review 73 (1): 31-43.
60. Leamer, E. E. 2010. "Tantalus on the Road to Asymptopia." The Journal of Economic Perspectives 24 (2): 31-46. doi:10.1257/jep.24.2.31.
61. Leimer, D. R., and S. D. Lesnoy. 1982. "Social Security and Private Saving: New Time-Series Evidence." Journal of Political Economy 90 (3): 606-629. doi:10.1086/261077.
62. Lim, S. S., L. Dandona, J. A. Hoisington, S. L. James, M. C. Hogan, and E. Gakidou. 2010. "India's Janani Suraksha Yojana, a Conditional Cash Transfer Programme to Increase Births in Health Facilities: An Impact Evaluation." The Lancet 375 (9730): 2009-2023. doi:10.1016/S0140-6736(10)60744-1.
63. Lipset, S. M. 1959. "Some Social Requisites of Democracy: Economic Development and Political Legitimacy." American Political Science Review 53 (01): 69-105. doi:10.2307/1951731.
64. Lipsey, M. W., and D. B. Wilson. 1993. "The Efficacy of Psychological, Educational, and Behavioral Treatment: Confirmation from Meta-Analysis." The American Psychologist 48 (12): 1181-1209. doi:10.1037/0003-066X.48.12.1181.
65. Liverpool-Tasie, L. S. O., and A. Winter-Nelson. 2011. "Asset versus Consumption Poverty and Poverty Dynamics in Rural Ethiopia." Agricultural Economics 42 (2): 221-233. doi:10.1111/j.1574-0862.2010.00479.x.
66. Mazumdar, S., A. Mills, and T. Powell-Jackson. 2011. "Financial Incentives in Health: New Evidence from India's Janani Suraksha Yojana." Available at SSRN 1935442. Accessed March 12, 2014. http://ssrn.com/abstract=1935442
67. McCullough, B. 2009. "Open Access Economics Journals and the Market for Reproducible Economic Research." Economic Analysis & Policy 39 (1): 117-126.
68. McCullough, B., and R. McKitrick. 2009. Check the Numbers: The Case for Due Diligence in Policy Formation. Vancouver, BC: Fraser Institute.
69. Miguel, E., and M. Kremer. 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities." Econometrica: Journal of the Econometric Society 72 (1): 159-217. doi:10.1111/j.1468-0262.2004.00481.x.
70. Mittelstaedt, R. A., and T. S. Zorn. 1984. "Econometric Replication: Lessons from the Experimental Sciences." Quarterly Journal of Business and Economics 23 (1): 9-15.
71. Munnell, A. H., L. E. Browne, J. McEneaney, and G. M. B. Tootell. 1992. Mortgage Lending in Boston: Interpreting HMDA Data. Working Paper 92-7. Boston, MA: Federal Reserve Bank of Boston.
72. Munnell, A. H., G. M. B. Tootell, L. E. Browne, and J. McEneaney. 1996. Mortgage Lending in Boston: Interpreting HMDA Data. American Economic Review 86 (1): 25-53.
73. Muralidharan, K., and V. Sundararaman. 2011. "Teacher Performance Pay: Experimental Evidence from India." The Journal of Political Economy 119 (1): 39-77. doi:10.1086/659655.
74. Rampell, C. 2013. "A History of Oopsies in Economic Studies." The New York Times, April 17, 2013. Accessed February 12, 2014. http://economix.blogs.nytimes.com/2013/04/17/a-history-of-oopsies-in-economic-studies/
75. Reinhart, C. M., and K. S. Rogoff. 2010. "Growth in a Time of Debt." American Economic Review 100 (2): 573-578. doi:10.1257/aer.100.2.573.
76. Rokicki, S., and N. Carvalho. Forthcoming. Replication of 'India's Janani Suraksha Yojana'.
77. Rosenthal, R. 1979. "The 'File Drawer Problem' and Tolerance for Null Results." Psychological Bulletin 86 (3): 638-641. doi:10.1037/0033-2909.86.3.638.
78. Rothstein, H. R., A. J. Sutton, and M. Borenstein. 2005. Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. Chichester: John Wiley & Sons.
79. Sachs, J., and A. Warner. 1997. "Sources of Slow Growth in African Economies." Journal of African Economies 6 (3): 335-376. doi:10.1093/oxfordjournals.jae.a020932.
80. Science's Sokal moment. 2013. The Economist. October 5, p. 85.
81. Sokal, A., and J. Bricmont. 1998. Fashionable Nonsense: Postmodern Intellectuals' Abuse of Science. New York: Picador USA.
82. Toncar, M. F., and J. M. Munch. 2010. "Meaningful Replication: When is a Replication no Longer a Replication? A Rejoinder to Stella and Adam (2008)." The Journal of Marketing Theory and Practice 18 (1): 71-80. doi:10.2753/MTP1069-6679180105.
83. Tweney, R. D. 2004. "Replication and the Experimental Ethnography of Science." Journal of Cognition and Culture 4 (3): 731-758. doi:10.1163/1568537042484968.
84. Unreliable research: Trouble at the lab. 2013. The Economist. October 19, p. 26-30.
85. Valentine, J., A. Biglan, R. F. Boruch, F. G. Castro, L. M. Collins, B. R. Flay, S. Kellam, E. K. Mościcki, and S. P. Schinke. 2011. "Replication in Prevention Science." Prevention Science 12 (2): 103-117. doi:10.1007/s11121-011-0217-6.
86. Wible, J. R. 1992. "Fraud in Science an Economic Approach." Philosophy of the Social Sciences 22 (1): 5-27. doi:10.1177/004839319202200101.
87. Wohlfarth, P. 2012. Replication in the Narrow Sense of Banzhaf/Walsh (2008). Center for Statistics, Replication Working Paper No. 02/2012. Göttingen: Georg-August University.
88. Yong, E. 2012. "Replication Studies: Bad Copy." Nature 485: 298-300. doi:10.1038/485298a.
89. Zakula, B. (2012) Narrow Replication of Ashcraft (2005): Are Banks Really Special? Center for Statistics, Replication Working Paper No. 01/2012. Göttingen: Georg-August-University.