Improving transparency in observational social science research: A pre-analysis plan approach

Economics Letters

Volumn 168, Issue , 2018, Pages 56-60

  Burlig, Fiona ^a  

^a Saieh Hall for Economics   (United States)

Author keywords

Confidential data; Observational research; Pre registration; Transparency

Indexed keywords

EID: 85045745107     PISSN: 01651765     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.econlet.2018.03.036     Document Type: Article

References (31)

1. Anderson, M.L., Multiple inference and gender differences in the effects of early intervention: A reevaluation of the Abecedarian, Perry Preschool, and Early Training projects. J. Amer. Statist. Assoc. 103:484 (2008), 1481–1495.
2. Angrist, J.D., Pischke, J., The credibility revolution in empirical economics: How better research design is taking the con out of econometrics. J. Econ. Perspect. 24:2 (2010), 3–30.
3. BMJ, 2010. Registration of observational studies. BMJ 340:c950.
4. Brodeur, A., Lé M., Sangnier, M., Zylberberg, Y., Star wars: The empirics strike back. Am. Econ. J. Appl. Econ. 8:1 (2016), 1–32.
5. Broockman, D., Kalla, J., Aronow, P., Irregularities in LaCour. 2014.
6. Camerer, C.F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., Wu, H., Evaluating replicability of laboratory experiments in economics. Science 351:6280 (2016), 1433–1436.
7. Card, D., DellaVigna, S., Malmendier, U., The role of theory in field experiments. J. Econ. Perspect. 25:3 (2011), 39–62.
8. Casey, K., Glennerster, R., Miguel, E., 0000. Reshaping institutions: Evidence on aid impacts using a preanalysis plan. Q. J. Econ. 127(4), 1755–1812.
9. Christensen G.S. and Miguel E., Transparency, reproducibility, and the credibility of economics research. J. Econ. Lit. (forthcoming).
10. Coffman, L.C., Niederle, M., Pre-analysis plans have limited upside, especially where replications are feasible. J. Econ. Perspect. 29:3 (2015), 81–98.
11. Dal-Ré R., Ioannidis, J.P., Bracken, M.B., Buffler, P.A., Chan, A.-W., Franco, E.L., Vecchia, C.L., Weiderpass, E., Making prospective registration of observational research a reality. Sci. Transl. Med., 6(224), 2014, 224cm1.
12. Epidemiology, The registration of observational studies –when metaphors go bad. Epidemiology 21:5 (2010), 607–609.
13. Fink, G., McConnell, M., Vollmer, S., Testing for heterogeneous treatment effects in experimental data: false discovery risk and correction procedures. J. Dev. Effectiveness 6:1 (2014), 44–57.
14. Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J.P., Allen, H., Baicker, K., Group, O.H.S., The Oregon health insurance experiment: evidence from the first year. Q. J. Econ. 127:3 (2012), 1057–1106.
15. Finkelstein, A., Taubman, S., Wright, B., Bernstein, M., Gruber, J., Newhouse, J.P., Allen, H., Baicker, K., Group, T.O.H.S., 2011. The Oregon Health Insurance Experiment: Evidence from the First Year. National Bureau of Economic Research Working Paper No. 17190.
16. Franco, A., Malhotra, N., Simonovits, G., Publication bias in the social sciences: Unlocking the file drawer. Science 345:6203 (2014), 1502–1505.
17. Hendry, D.F., Econometrics - alchemy or science. Economica 47:188 (1980), 387–406.
18. Humphreys, M., de la Sierra, R., van der Windt, P., Fishing, commitment, and communication: A proposal for comprehensive nonbinding research registration. Polit. Anal. 21:1 (2013), 1–20.
19. Ioannidis, J.P.A., Stanley, T., Doucouliagos, H., The power of bias in economics research. Econ. J. 127:605 (2017), F236–F265.
20. Leamer, E.E., Let's take the con out of econometrics. Amer. Econ. Rev. 73:1 (1983), 31–43.
21. List, J.A., Shaikh, A.M., Xu, Y., 2016. Multiple Hypothesis Testing in Experimental Economics. National Bureau of Economic Research Working Paper No. 21875.
22. Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K.M., Gerber, A., Glennerster, R., Green, D.P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B.A., Petersen, M., Sedlmayr, R., Simmons, J.P., Simonsohn, U., der Laan, M.V., Promoting transparency in social science research. Science 343:6166 (2014), 30–31.
23. Neumark, D., The employment effects of minimum wages: Evidence from a prespecified research design. Ind. Relat. 40:1 (2001), 121–144.
24. Neumark, D., 1999. The Employment Effects of Recent Minimum Wage Increases: Evidence from a Pre-specified Research Design. National Bureau of Economic Research Working Paper No. 7171.
25. Nosek, B.A., Alter, G., Banks, G.C., Borsboom, D., Bowman, S.D., Breckler, S.J., Buck, S., Chambers, C.D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D.P., Hesse, B., Humphreys, M., Ishiyama, J., Karlan, D., Kraut, A., Lupia, A., Mabry, P., Madon, T., Malhotra, N., Mayo-Wilson, E., McNutt, M., Miguel, E., Paluck, E.L., Simonsohn, U., Soderberg, C., Spellman, B.A., Turitto, J., VandenBos, G., Vazire, S., Wagenmakers, E.J., Wilson, R., Yarkoni, T., Promoting an open research culture. Science 348:6242 (2015), 1422–1425.
26. Olken, B.A., Promises and perils of pre-analysis plans. J. Econ. Perspect. 29:3 (2015), 61–80.
27. Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349(6251), 2015, aa4716.
28. PLOS Medicine, Observational studies: Getting clear about transparency. PLoS Med., 11(8), 2014, e1001711.
29. Rosenthal, R., The file drawer problem and tolerance for null results. Psychol. Bull. 86:3 (1979), 638–641.
30. Simonsohn, U., Just post it: the lesson from two cases of fabricated data detected by statistics alone. Psychol. Sci. 10:24 (2013), 1875–1888.
31. The Lancet, Should protocols for observational research be registered?. The Lancet, 375(9712), 2010, 348.