Is economics research replicable? Sixty published papers from thirteen journals say “Often Not”

Critical Finance Review

Volumn 11, Issue 1, 2022, Pages 185-206

  Chang, Andrew C ^a   Li, Phillip ^b  

^a FEDERAL RESERVE BOARD   (United States)

^b United States Department of the Treasury   (United States)

Author keywords

Data and code files; GDP; Gross domestic product; Macroeconomics; National income and product accounts; Preanalysis; Publication bias; Replication

Indexed keywords

EID: 85125475413     PISSN: 21645744     EISSN: 21645760     Source Type: Journal    
DOI: 10.1561/104.00000053     Document Type: Article

References (109)

1. Alexopoulos, M. 2011. “Read All About It!! What Happens Following a Technology Shock?” American Economic Review. 101(4): 1144–1179.
2. Alexopoulos, M. and J. Cohen. 2011. “Volumes of Evidence: Examining Technical Change in the Last Century Through a New Lens”. Canadian Journal of Economics. 44(2): 413–450.
3. Andrews, D. W. K. 1993. “Tests for Parameter Instability and Structural Change with Unknown Change Point”. Econometrica. 61(4): 821–856.
4. Ang, A., J. Boivin, S. Dong, and R. Loo-Kung. 2011. “Monetary Policy Shifts and the Term Structure”. Review of Economic Studies. 78(2): 429–457.
5. Arestis, P., G. Chortareas, and J. D. Tsoukalas. 2010. “Money and Information in a New Neoclassical Synthesis Framework”. Economic Journal. 120(542): F101–F128.
6. Auerbach, A. J. and Y. Gorodnichenko. 2012. “Measuring the Output Responses to Fiscal Policy”. American Economic Journal: Economic Policy. 4(2): 1–27.
7. Auerbach, A. J. and Y. Gorodnichenko. 2013. “Corrigendum: Measuring the Output Responses to Fiscal Policy”. American Economic Journal: Economic Policy. 5(3): 320–322.
8. Bai, J. and P. Wang. 2011. “Conditional Markov Chain and Its Application in Economic Time Series Analysis”. Journal of Applied Econometrics. 26(5): 715–734.
9. Bansak, C., M. E. Graham, and A. A. Zebedee. 2012. “Business Cycles and Gender Diversification: An Analysis of Establishment-Level Gender Dissimilarity”. American Economic Review: Papers & Proceedings. 102(3): 561–565.
10. Barro, R. J. 1991. “Economic Growth in a Cross Section of Countries”. Quarterly Journal of Economics. 106(2): 407–443.
11. Barro, R. J. and C. J. Redlick. 2011. “Macroeconomic Effects from Government Purchases and Taxes”. Quarterly Journal of Economics. 126(1): 51–102.
12. Barsky, R. B. and E. R. Sims. 2012. “Information, Animal Spirits, and the Meaning of Innovations in Consumer Confidence”. American Economic Review. 102(4): 1343–1377.
13. Baumeister, C. and G. Peersman. 2013. “Time-Varying Effects of Oil Supply Shocks on the US Economy”. American Economic Journal: Macroeconomics. 5(4): 1–28.
14. Bernanke, B. S. 2004. “Editorial Statement”. American Economic Review. 94(1): 404.
15. Bhattarai, S., J. W. Lee, and W. Y. Park. 2012. “Monetary-Fiscal Policy Interactions and Indeterminacy in Postwar US Data”. American Economic Review: Papers & Proceedings. 102(3): 173–178.
16. Bianchi, F. 2012. “Evolving Monetary/Fiscal Policy Mix in the United States”. American Economic Review: Papers & Proceedings. 102(3): 167–172.
17. Bianchi, F. 2013. “Regime Switches, Agents’ Beliefs, and Post-World War II U.S. Macroeconomic Dynamics”. Review of Economic Studies. 80(2): 463–490.
18. Bilbiie, F. O. and R. Straub. 2013. “Asset Market Participation, Monetary Policy Rules, and the Great Inflation”. Review of Economics and Statistics. 95(2): 377–392.
19. Broockman, D., J. Kalla, and P. Aronow. 2015. “Irregularities in LaCour (2014)”. Working Paper.
20. Card, D. and A. B. Krueger. 1994. “Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania”. American Economic Review. 84(4): 772–793.
21. Card, D. and A. B. Krueger. 2000. “Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania: Reply”. American Economic Review. 90(5): 1397–1420.
22. Canova, F. and L. Gambetti. 2010. “Do Expectations Matter? The Great Moderation Revisited”. American Economic Journal: Macroeconomics. 2(3): 183–205.
23. Carey, C. and S. H. Shore. 2013. “From the Peaks to the Valleys: Cross-State Evidence on Income Volatility Over the Business Cycle”. Review of Economics and Statistics. 95(2): 549–562.
24. Castelnuovo, E. and P. Surico. 2010. “Monetary Policy, Inflation Expectations and the Price Puzzle”. Economic Journal. 120(549): 1262–1283.
25. Chang, A. C. and P. Li. 2015a. “Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say ‘Usually Not’”. Finance and Economics Discussion Series. Washington: Board of Governors of the Federal Reserve System: 2015–083.
26. Chang, A. C. and P. Li. 2015b. “Measurement Error in Macroeconomic Data and Economics Research: Data Revisions, Gross Domestic Product, and Gross Domestic Income”. Finance and Economics Discussion Series. Washington: Board of Governors of the Federal Reserve System: 2015–102.
27. Chang, A. C. and P. Li. 2017. “A Preanalysis Plan to Replicate Sixty Economics Papers That Worked Half of the Time”. American Economic Review. 107(5): 60–64.
28. Chen, H., V. Curdia, and A. Ferrero. 2012. “The Macroeconomic Effects of Large-Scale Asset Purchase Programmes”. Economic Journal. 122(564): F289–F315.
29. Clark, T. E. and M. W. McCracken. 2010. “Averaging Forecasts from VARs with Uncertain Instabilities”. Journal of Applied Econometrics. 25(1): 5–29.
30. Clemens, M. A. 2015. “The Meaning of Failed Replications: A Review and Proposal”. Working Paper 399. Center for Global Development.
31. Clements, M. P. and A. B. Galvão. 2009. “Forecasting US Output Growth Using Leading Indicators: An Appraisal Using MIDAS Models”. Journal of Applied Econometrics. 24(7): 1187–1206.
32. Clements, M. P. and A. B. Galvão. 2013. “Real-Time Forecasting of Inflation and Output Growth With Autoregressive Models in the Presence of Data Revisions”. Journal of Applied Econometrics. 28(3): 458–477.
33. Corsetti, G., A. Meier, and G. J. Maller. 2012. “Fiscal Stimulus with Spending Reversals”. Review of Economics and Statistics. 94(4): 878–895.
34. Croushore, D. and T. Stark. 2003. “A Real-Time Data Set for Macroeconomists: Does the Data Vintage Matter?” Review of Economics and Statistics. 85(3): 605–617.
35. D’Agostino, A. and P. Surico. 2012. “A Century of Inflation Forecasts”. Review of Economics and Statistics. 94(4): 1097–1106.
36. Del Negro, M. and F. Schorfheide. 2009. “Monetary Policy Analysis With Potentially Misspecified Models”. American Economic Review. 99(4): 1415–1450.
37. Den Haan, W. J. and V. Sterk. 2011. “The Myth of Financial Innovation and the Great Moderation”. Economic Journal. 121(553): 707–739.
38. Dewald, W. G., J. G. Thursby, and R. G. Anderson. 1986. “Replication in Empirical Economics: The Journal of Money, Credit and Banking Project”. American Economic Review. 76(4): 587–603.
39. Duvendack, M., R. W. Palmer-Jones, and W. R. Reed. 2015. “Replication in Economics: A Progress Report”. Econ Journal Watch. 12(2): 164–191.
40. Favero, C. and F. Giavazzi. 2012. “Measuring Tax Multipliers: The Narrative Method in Fiscal VARs”. American Economic Journal: Economic Policy. 4(2): 69–94.
41. Fève, P. and A. Guay. 2010. “Identification of Technology Shocks in Structural VARs”. Economic Journal. 120(549): 1284–1318.
42. Fisher, J. D. M. and R. Peters. 2010. “Using Stock Returns to Identify Government Spending Shocks”. Economic Journal. 120(544): 414–436.
43. Gabaix, X. 2011. “The Granular Origins of Aggregate Fluctuations”. Econometrica. 79(3): 733–772.
44. Glandon, P. 2010. “Report on the American Economic Review Data Availability Compliance Project”. Working Paper.
45. Gilchrist, S. and E. Zakrajšek. 2012. “Credit Spreads and Business Cycle Fluctuations”. American Economic Review. 102(4): 1692–1720.
46. Gordon, R. J. 2010. “Okun’s Law and Productivity Innovations”. American Economic Review: Papers & Proceedings. 100(2): 11–15.
47. Hall, G. J. and T. J. Sargent. 2011. “Interest Rate Risk and Other Determinants of Post-WWII US Government Debt/GDP Dynamics”. American Economic Journal: Macroeconomics. 3(3): 192–214.
48. Hamermesh, D. S. 2007. “Viewpoint: Replication in Economics”. Canadian Journal of Economics. 40(3): 715–733.
49. Hansen, P. R., A. Lunde, and J. M. Nason. 2011. “The Model Confidence Set”. Econometrica. 79(2): 453–497.
50. Herndon, T., M. Ash, and R. Pollin. 2014. “Does High Public Debt Consistently Stifle Economic Growth? A Critique of Reinhart and Rogoff”. Cambridge Journal of Economics. 38(2): 257–279.
51. Heutel, G. 2012. “How Should Environmental Policy Respond to Business Cycles? Optimal Policy Under Persistent Productivity Shocks”. Review of Economic Dynamics. 15(2): 244–264.
52. Hodrick, R. J. and E. C. Prescott. 1997. “Postwar U.S. Business Cycles: An Empirical Investigation”. Journal of Money, Credit and Banking. 29(1): 1–16.
53. Hoxby, C. 2000. “Does Competition Among Public Schools Benefit Students and Taxpayers”. American Economic Review. 90(5): 1209–1238.
54. Hoxby, C. 2007. “Does Competition Among Public Schools Benefit Students and Taxpayers? A Reply to Rothstein”. American Economic Review. 97(5): 2038–2055.
55. Höffler, J. H. 2014. “Teaching Replication in Quantitative Empirical Economics”. Working Paper for the 68th European Meeting of the Econometric Society.
56. Ilbas, P. 2012. “Revealing the Preferences of the US Federal Reserve”. Journal of Applied Econometrics. 27(3): 440–473.
57. Inoue, A. and B. Rossi. 2011. “Identifying the Sources of Instabilities in Macroeconomic Fluctuations”. Review of Economics and Statistics. 93(4): 1186–1204.
58. Ireland, P. N. 2009. “On the Welfare Cost of Inflation and the Recent Behavior of Money Demand”. American Economic Review. 99(3): 1040–1052.
59. Jermann, U. and V. Quadrini. 2012. “Macroeconomic Effects of Financial Shocks”. American Economic Review. 102(1): 238–271.
60. Jore, A. S., J. Mitchell, and S. P. Vahey. 2010. “Combining Forecast Densities from VARs with Uncertain Instabilities”. Journal of Applied Econometrics. 25(4): 621–634.
61. Justiniano, A., G. E. Primiceri, and A. Tambalotti. 2011. “Investment Shocks and the Relative Price of Investment”. Review of Economic Dynamics. 14(1): 102–121.
62. Keen, B. D. 2009. “Output, Inflation, and Interest Rates in an Estimated Optimizing Model of Monetary Policy”. Review of Economic Dynamics. 12: 327–343.
63. Kilian, L. 2009. “Not All Oil Price Shocks Are Alike: Disentangling Demand and Supply Shocks in the Crude Oil Market”. American Economic Review. 99(3): 1053–1069.
64. Koenig, E. F., S. Dolmas, and J. Piger. 2003. “The Use and Abuse of Real-Time Data in Economic Forecasting”. Review of Economics and Statistics. 85(3): 618–628.
65. Kormilitsina, A. 2011. “Oil Price Shocks and the Optimality of Monetary Policy”. Review of Economic Dynamics. 14(1): 199–223.
66. Krishnamurthy, A. and A. Vissing-Jorgensen. 2012. “The Aggregate Demand for Treasury Debt”. Journal of Political Economy. 120(2): 233–267.
67. Kydland, F. E. and E. C. Prescott. 1982. “Time to Build and Aggregate Fluctuations”. Econometrica. 50(6): 1345–1370.
68. LaCour, M. J. and D. P. Green. 2014. “When Contact Changes Minds: An Experiment on Transmission of Support for Gay Equality”. Science. 346(6215): 1366–1369.
69. Last Week Tonight, with John Oliver, Segment on Washington D.C. Voting Rights, airdate August 2. 2015.
70. Levine, P., J. Pearlman, G. Perendia, and B. Yang. 2012. “Endogenous Persistence in an Estimated DSGE Model Under Imperfect Information”. Economic Journal. 122(565): 1287–1312.
71. Levitt, S. 1997. “Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime”. American Economic Review. 87(3): 270–290.
72. Levitt, S. 2002. “Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Reply”. American Economic Review. 92(4): 1244–1250.
73. Maheu, J. M. and S. Gordon. 2008. “Learning, Forecasting, and Structural Breaks”. Journal of Applied Econometrics. 23(5): 553–583.
74. Mavroeidis, S. 2010. “Monetary Policy Rules and Macroeconomic Stability: Some New Evidence”. American Economic Review. 100(1): 491–503.
75. McCrary, J. 2002. “Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment”. American Economic Review. 92(4): 1236–1243.
76. McCullough, B. D., K. A. McGeary, and T. D. Harrison. 2006. “Lessons From the JMCB Archive”. Journal of Money, Credit and Banking. 38(4): 1093–1107.
77. McCullough, B. D., K. A. McGeary, and T. D. Harrison. 2008. “Do Economics Journal Archives Promote Replicable Research?” Canadian Journal of Economics. 41(4): 1406–1420.
78. McCullough, B. D. and H. D. Vinod. 2003. “Verifying the Solution From a Nonlinear Solver: A Case Study”. American Economic Review. 93: 873–892.
79. McCullough, B. D. and H. D. Vinod. 2004. “Verifying the Solution From a Nonlinear Solver: Reply”. American Economic Review. 94(1): 391–396.
80. Mertens, K. and M. O. Ravn. 2010. “Measuring the Impact of Fiscal Policy in the Face of Anticipation: A Structural VAR Approach”. Economic Journal. 120(544): 393–413.
81. Mertens, K. and M. O. Ravn. 2011. “Understanding the Aggregate Effects of Anticipated and Unanticipated Tax Policy Shocks”. Review of Economic Dynamics. 14(1): 27–54.
82. Mertens, K. and M. O. Ravn. 2012. “Empirical Evidence on the Aggregate Effects of Anticipated and Unanticipated US Tax Policy Shocks”. American Economic Journal: Economic Policy. 4(2): 145–181.
83. Mertens, K. and M. O. Ravn. 2013. “The Dynamic Effects of Personal and Corporate Income Tax Changes in the United States”. American Economic Review. 103(4): 1212–1247.
84. Milani, F. 2011. “Expectation Shocks and Learning As Drivers of the Business Cycle”. Economic Journal. 121(552): 379–401.
85. Morley, J. and J. Piger. 2012. “The Asymmetric Business Cycle”. Review of Economics and Statistics. 94(1): 208–221.
86. Mountford, A. and H. Uhlig. 2009. “What Are the Effects of Fiscal Policy Shocks?” Journal of Applied Econometrics. 24(6): 960–992.
87. Nakov, A. and A. Pescatori. 2010. “Oil and the Great Moderation”. Economic Journal. 120(543): 131–156.
88. Neumark, D. and W. Wascher. 2000. “Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania: Comment”. American Economic Review. 90(5): 1362–1396.
89. Open Science Collaboration. 2015. “Estimating the Reproducibility of Psychological Science”. Science. 349(6251): aac4716.
90. Perotti, R. 2012. “The Effects of Tax Shocks on Output: Not So Large, But Not Small Either”. American Economic Journal: Economic Policy. 4(2): 214–237.
91. Piazzesi, M. and M. Schneider. 2010. “Interest Rate Risk in Credit Markets”. American Economic Review: Papers & Proceedings. 100(2): 579–584.
92. Polito, V. and M. Wickens. 2012. “Optimal Monetary Policy Using an Unrestricted VAR”. Journal of Applied Econometrics. 27(4): 525–553.
93. Ramey, V. A. 2011. “Identifying Government Spending Shocks: It’s All in the Timing”. Quarterly Journal of Economics. 126(1): 1–50.
94. Reinhart, C. M. and K. S. Rogoff. 2010. “Growth in a Time of Debt”. American Economic Review. 100(2): 573–578.
95. Reis, R. and M. W. Watson. 2010. “Relative Goods’ Prices, Pure Inflation, and The Phillips Correlation”. American Economic Journal: Macroeconomics. 2(3): 128–157.
96. Romer, C. D. and D. H. Romer. 2010. “The Macroeconomic Effects of Tax Changes: Estimates Based on a New Measure of Fiscal Shocks”. American Economic Review. 100(3): 763–801.
97. Rothstein, J. 2007. “Does Competition Among Public Schools Benefit Students and Taxpayers? A Comment on Hoxby (2000)”. American Economic Review. 97(5): 2026–2037.
98. Rudebusch, G. D. and T. Wu. 2008. “A Macro-Finance Model of the Term Structure, Monetary Policy and the Economy”. Economic Journal. 118(530): 906–926.
99. Shachar, R. and B. Nalebuff. 1999. “Follow the Leader: Theory and Evidence on Political Participation”. American Economic Review. 83(3): 525–547.
100. Stephanie, S.-G. and M. Uribe. 2011. “Business Cycles with a Common Trend in Neutral and Investment-Specific Productivity”. Review of Economic Dynamics. 14(1): 122–135.
101. Schmitt-Groche, S. and M. Uribe. 2012. “What’s News in Business Cycles”. Econometrica. 80(6): 2733–2764.
102. Senyuz, Z. 2011. “Factor Analysis of Permanent and Transitory Dynamics of the US Economy and the Stock Market”. Journal of Applied Econometrics. 26(6): 975–998.
103. Shore, S. H. 2010. “For Better, For Worse: Intrahousehold Risk-Sharing Over the Business Cycle”. Review of Economics and Statistics. 92(3): 536–548.
104. Sims, C. A. 1980. “Macroeconomics and Reality”. Econometrica. 48(1): 1–48.
105. Šustek, R. 2011. “Monetary Business Cycle Accounting”. Review of Economic Dynamics. 14(4): 592–612.
106. Taylor, J. B. 1993. “Discretion Versus Policy Rules in Practice”. Carnegie-Rochester Conference Series on Public Policy. 39: 195–214.
107. Thomson Reuters’. 2014. “Web of Science Journal Citation Reports: Social Science Edition”. https://isiknowledge.com/.
108. Vlaeminck, S. and L.-K. Herrmann. 2015. “Data policies and data archives: A new paradigm for academic publishing in economic sciences?” In: New Avenues for Electronic Publishing in the Age of Infinite Collections and Citizen Science. Edited by B. Schmidt and M. Dobreva. DOI:10.3233/-61499-562-3-145.
109. Whitlock, C. and B. Woodward. 2016. “Pentagon Buries Evidence of $125 Billion in Bureaucratic Waste”. The Washington Post: 76–9. Accessed December 6. URL: https://www.washingtonpost.com/investigations/pentagon-buriesevidence-of-125-billion-in-bureaucratic-waste/2016/12/05/e0668c76-9af6-11e6-a0ed-ab0774c1eaa5_story.html.