Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition

Royal Society Open Science

Volumn 5, Issue 8, 2018, Pages

  Hardwicke, Tom E ^a   Mathur, Maya B ^b,d   MacDonald, Kyle ^c   Nilsonne, Gustav ^c,e,f   Banks, George C ^g   Kidwell, Mallory C ^h   Mohr, Alicia Hofelich ⁱ   Clayton, Elizabeth ^g   Yoon, Erica J ^c   Tessler, Michael Henry ^c   Lenne, Richie L ⁱ   Altman, Sara ^c   Long, Bria ^c   Frank, Michael C ^c  

^a STANFORD UNIVERSITY SCHOOL OF MEDICINE   (United States)

^b Quantitative Sciences Unit Division of Biomedical Informatics Department of Medicine   (United States)

^c STANFORD UNIVERSITY   (United States)

^d HARVARD UNIVERSITY   (United States)

^e STOCKHOLM UNIVERSITY   (Sweden)

^f KAROLINSKA INSTITUTE   (Sweden)

^g UNIVERSITY OF NORTH CAROLINA AT CHARLOTTE   (United States)

^h UNIVERSITY OF UTAH   (United States)

ⁱ UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Interrupted time series; Journal policy; Meta science; Open data; Open science; Reproducibility

Indexed keywords

EID: 85053204303     PISSN: None     EISSN: 20545703     Source Type: Journal    
DOI: 10.1098/rsos.180448     Document Type: Article

References (57)

1. Ioannidis JPA. 2014 How to make more published research true. PLoS Med. 11, e1001747-6. (doi:10.1371/journal.pmed. 1001747)
2. Munafò MR et al. 2017 A manifesto for reproducible science. Nat. Hum. Behav. 1, 1-9. (doi:10.1038/s41562-016-0021)
3. Nosek BA et al. 2015 Promoting an open research culture. Science 348, 1422-1425. (doi:10.1126/science.aab2374)
4. Goodman SN, Fanelli D, Ioannidis JPA. 2016 What does research reproducibility mean? Sci. Transl. Med. 8, 1-6. (doi:10.1126/scitranslmed. aaf5027)
5. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, Heroux MA, Ioannidis JPA, Taufer M. 2016 Enhancing reproducibility for computational methods. Science 354, 1240-1241. (doi:10.1126/science. aah6168)
6. Bollen K, Cacioppo JT, Kaplan RM, Krosnick JA, Olds JL. 2015 Social, behavioral, and economic sciences perspectives on robust and reliable science. Arlington, VA: National Science Foundation; See https://www.nsf.gov/sbe/SBE_Spring_2015_AC_Meeting_ Presentations/Bollen_Report_on_Replicability_ SubcommitteeMay_2015.pdf.
7. Alsheikh-Ali AA, Qureshi W, Al-Mallah MH, Ioannidis JPA. 2011 Public availability of published research data in high-impact journals. PLoS ONE 6, e24357-4. (doi:10.1371/journal. pone.0024357)
8. Hardwicke TE, Ioannidis JPA. In press. Populating the Data Ark: an attempt to retrieve, preserve, and liberate data from the most highly cited psychology and psychiatry articles. PLoS ONE.
9. Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA. 2016 Reproducible research practices and transparency across the biomedical literature. PLoS Biol. 14, e1002333-13. (doi:10. 1371/journal.pbio.1002333)
10. Kidwell MC et al. 2016 Badges to acknowledge open practices: a simple, low-cost, effective method for increasing transparency. PLoS Biol. 14, 1-15. (doi:10.1371/journal.pbio. 1002456)
11. Rowhani-Farid A, Barnett AG. 2016 Has open data arrived at the British Medical Journal (BMJ)? An observational study. BMJ Open 6, e011784-9. (doi:10.1136/bmjopen-2016-011784)
12. Vanpaemel W, Vermorgen M, Deriemaecker L, Storms G. 2015 Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra 1, 1-5. (doi:10.1525/ collabra.13)
13. Wicherts JM, Borsboom D, Kats J, Molenaar D. 2006 The poor availability of psychological research data for reanalysis. Am. Psychol. 61, 726-728. (doi:10.1037/0003-066X.61.7.726)
14. Womack RP. 2015 Research data in core journals in biology, chemistry, mathematics, and physics. PLoS ONE 10, e0143460. (doi:10.1371/journal. pone.0143460)
15. Vasilevsky NA, Minnier J, Haendel MA, Champieux RE. 2017 Reproducible and reusable research: are journal data sharing policies meeting the mark? PeerJ 5, e3208-18. (doi:10. 7717/peerj.3208)
16. Vines TH et al. 2014 The availability of research data declines rapidly with article age. Curr. Biol. 24, 94-97. (doi:10.1016/j.cub. 2013.11.014)
17. Pashler H, Wagenmakers E-J. 2012 Editors' introduction to the Special Section on replicability in psychological science: a crisis of confidence? Perspect. Psychol. Sci. 7, 528-530. (doi:10.1177/1745691612465253)
18. Nuijten MB, Borghuis J, Veldkamp CLS, Dominguez-Alvarez L, van Assen MALM, Wicherts JM. 2017 Journal data sharing policies and statistical reporting inconsistencies in psychology. Collabra Psychol. 3, 31. (doi:10. 1525/collabra.102)
19. Eubank N. 2016 Lessons from a decade of replications at the quarterly journal of political science. PS Polit. Sci. Polit. 49, 273-276. (doi:10.1017/S1049096516000196)
20. Chang AC, Li P. 2015 Is economics research replicable? Sixty published papers from thirteen journals say 'usually not'. In Finance and economics discussion series 2015-083. Washington, DC: Board of Governors of the Federal Reserve System.
21. Ioannidis JPA et al. 2009 Repeatability of published microarray gene expression analyses. Nat. Genet. 41, 149-155. (doi:10. 1038/ng.295)
22. Naudet F, Sakarovitch C, Janiaud P, Cristea I, Fanelli D, Moher D, Ioannidis JPA. 2018 Data sharing and reanalysis of randomized controlled trials in leading biomedical journals with a full data sharing policy: survey of studies published in The BMJ and PLOS Medicine. Br. Med. J. 360, k400-k411. (doi:10.1136/bmj.k400)
23. Sloman SA. 2015 Opening editorial: the changing face of Cognition. Cognition 135, 1-3. (doi:10.1016/j.cognition.2014.11.004)
24. Cognition. 2017 Guide for authors. See https://www.elsevier.com/journals/cognition/00100277/guide-for-authors.
25. Newcombe RG. 1998 Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat. Med. 17, 857-872. (doi:10.1002/(SICI)1097-0258(19980430)17:8,857::AID-SIM777.3.0.CO;2-E)
26. Sison CP, Glaz J. 1995 Simultaneous confidence intervals and sample size determination for multinomial proportions. J. Am. Stat. Assoc. 90, 366-369. (doi:10. 1080/01621459.1995.10476521)
27. Shadish W, Cook TD, Campbell DT. 2002 Experiment and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
28. Bernal JL, Cummins S, Gasparrini A. 2017 Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int. J. Epidemiol. 46, 348-355.
29. Zhang J, Yu KF. 1998 What's the relative risk? JAMA 280, 1690. (doi:10.1001/jama.280.19.1690)
30. Rothman KJ, Greenland S, Walker AM. 1980 Concepts of interaction. Am. J. Epidemiol. 112, 467-470. (doi:10.1093/oxfordjournals.aje.a113015)
31. Knol MJ, VanderWeele TJ. 2012 Recommendations for presenting analyses of effect modification and interaction. Int. J. Epidemiol. 41, 514 - 520. (doi:10.1093/ije/dyr218)
32. Mathur MB, VanderWeele TJ. 2018 R function for additive interaction measures. Epidemiology 29, e5-e6. (doi:10.1097/EDE.0000000000000752)
33. VanderWeele TJ. 2017 On a square-root transformation of the odds ratio for a common outcome. Epidemiology 28, e58-e60. (doi:10. 1097/EDE.0000000000000733)
34. Open Science Collaboration. 2015 Estimating the reproducibility of psychological science. Science 349, 1-8.
35. Nosek BA, Spies JR, Motyl M. 2012 Scientific utopia II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615-631. (doi:10.1177/ 1745691612459058)
36. Klein O et al. 2018 A practical guide for transparency in psychological science. Collabra: Psychol. 4, 20. (doi:10.1525/collabra.158)
37. Stodden V. 2015 Reproducing statistical results. Annu. Rev. Stat. Appl. 2, 1-19. (doi:10.1146/ annurev-statistics-010814-020127)
38. Evangelou E, Trikalinos TA, Ioannidis JPA. 2005 Unavailability of online supplementary scientific information from articles published in major journals. FASEB J. 19, 1943-1944. (doi:10. 1096/fj.05-4784lsf)
39. Field A, Miles J, Field Z. 2012 Discovering statistics using R. London, UK: Sage Publications.
40. Cumming G, Fidler F, Leonard M, Kalinowski P, Christiansen A, Kleinig A, Lo J, McMenamin N, Wilson S. 2007 Statistical reform in psychology: is anything changing? Psychol. Sci. 18, 230-232. (doi:10.1111/j.1467-9280.2007. 01881.x)
41. Veldkamp CLS, Nuijten MB, Dominguez-Alvarez L, van Assen MALM, Wicherts JM. 2014 Statistical reporting errors and collaboration on statistical analyses in psychological science. PLoS ONE 9, e114876. (doi:10.1371/journal.pone. 0114876)
42. Xie Y. 2018 Knitr: A general-purpose package for dynamic report generation in R. See https://yihui.name/knitr/.
43. Aust F, Barth M. 2017 papaja: Create APA manuscripts with R Markdown. See https://github.com/crsh/papaja.
44. Green SA, Clyburne-Sherin A. 2018 Computational reproducibility via containers in social psychology. (https://psyarxiv.com/mf82t)
45. McCullough BD, McGeary KA, Harrison TD. 2006 Lessons from the JMCB Archive. J. Money Credit Bank. 38, 1093-1107. (doi:10.1353/ mcb.2006.0061)
46. Stodden V, Guo P, Ma Z. 2013 Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS ONE 8, e67111. (doi:10.1371/journal.pone.0067111)
47. Sakaluk J, Williams A, Biernat M. 2014 Analytic review as a solution to the misreporting of statistical results in psychological science. Perspect. Psychol. Sci. 9, 652-660. (doi:10. 1177/1745691614549257)
48. Ioannidis JPA. 2012 Why science is not necessarily self-correcting. Perspect. Psychol. Sci. 7, 645-654. (doi:10.1177/1745691612464056)
49. Merton RK. 1973 The sociology of science. Theoretical and empirical investigations. Chicago, IL: University of Chicago Press.
50. Ioannidis JPA, Fanelli D, Dunne DD, Goodman SN. 2015 Meta-research: evaluation and improvement of research methods and practices. PLoS Biol. 13, e1002264-7. (doi:10.1371/journal.pbio.1002264)
51. Marwick B, Boettiger C, Mullen L. 2017 Packaging data analytical work reproducibly using R (and friends). Am. Stat. 7, 80-88.
52. Mahoney MJ. 1976 Scientist as subject: the psychological imperative. Cambridge, MA: Ballinger Publishing Company.
53. Bakker M, Wicherts JM. 2011 The (mis)reporting of statistical results in psychology journals. Behav. Res. Methods 43, 666-678. (doi:10. 3758/s13428-011-0089-5)
54. Nuijten MB, Hartgerink CHJ, van Assen MALM, Epskamp S, Wicherts JM. 2016 The prevalence of statistical reporting errors in psychology (1985-2013). Behav. Res. Methods 48, 1205-1226. (doi:10.3758/s13428-015-0664-2)
55. Rosenthal R. 1978 How often are our numbers wrong? Am. Psychol. 33, 1005-1008. (doi:10. 1037/0003-066X.33.11.1005)
56. Nelson LD, Simmons J, Simonsohn U. 2018 Psychology's renaissance. Annu. Rev. Psychol. 69, 511-534. (doi:10.1146/annurev-psych-122216-011836)
57. Spellman BA. 2015 A short (personal) future history of revolution 2.0. Perspect. Psychol. Sci. 10, 886-899. (doi:10.1177/ 1745691615609918)