Open science challenges, benefits and tips in early career and beyond

PLoS Biology

Volumn 17, Issue 5, 2019, Pages

  Allen, Christopher ^a   Mehler, David M A ^a,b  

^a CARDIFF UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF MÜNSTER   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ADOPTION; ARTICLE; CAREER; EXPLORATORY RESEARCH; HUMAN; INTERMETHOD COMPARISON; NULL RESULT; PUBLICATION; RELIABILITY; SCIENTIST; DECISION MAKING; MEDICAL RESEARCH; SCIENCE; REVIEW;

BIOMEDICAL RESEARCH; CAREER CHOICE; SCIENCE;

EID: 85066160278     PISSN: 15449173     EISSN: 15457885     Source Type: Journal    
DOI: 10.1371/journal.pbio.3000246     Document Type: Article

References (86)

1. Higginson AD, Munafò MR. Current Incentives for Scientists Lead to Underpowered Studies with Erroneous Conclusions. PLoS Biol. 2016; 14(11): E2000995. https://doi.org/10.1371/journal.pbio.2000995 PMID: 27832072 PMCID: PMC5104444
2. Munafò M, Nosek B, Bishop D, Button K, Chambers C, Percie du Sert N, et al. A manifesto for reproducible science. Nat Hum Behav. Nature Publishing Group; 2017; 1: 0021. https://doi.org/10.1038/s41562-016-0021
3. Moonesinghe R, Khoury MJ, Janssens ACJW. Most Published Research Findings Are False—But a Little Replication Goes a Long Way. PLoS Med. 2007; 4(2): E28. https://doi.org/10.1371/journal.pmed.0040028 PMID: 17326704
4. Chambers C. The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton university Press; 2017.
5. Errington TM, Iorns E, Gunn W, Tan FE Lisabet, Lomax J, Nosek BA. An open investigation of the reproducibility of cancer biology research. Elife. 2014; 3: 1–9. https://doi.org/10.7554/eLife.04333 PMID: 25490932
6. Nosek BA, Errington TM. Making sense of replications. Elife. 2017; 6: 4–7. https://doi.org/10.7554/eLife.23383 PMID: 28100398
7. Harris R. Reproducibility issues. Chem Eng News. 2017;95.
8. Wallach JD, Boyack KW, Ioannidis JPA. Reproducible research practices, transparency, and open access data in the biomedical literature, 2015–2017. PLoS Biol. 2018; 16(11): E2006930. https://doi.org/10.1371/journal.pbio.2006930 PMID: 30457984
9. Mehler D. The replication challenge: Is brain imaging next? In: Raz A, Thibault RT, editors. Casting Light on the Dark Side of Brain Imaging. 1st ed. Academic Press; 2019. pp. 84–88. https://doi.org/10.1016/B978-0-12-816179-1.00010–4
10. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013; 14: 365–376. https://doi.org/10.1038/nrn3475 PMID: 23571845
11. Iqbal SA, Wallach JD, Khoury MJ, Schully SD, Ioannidis JPA. Reproducible Research Practices and Transparency across the Biomedical Literature. PLoS Biol. 2016; 14(1): E1002333. https://doi.org/10.1371/journal.pbio.1002333 PMID: 26726926
12. Szucs D, Ioannidis JPA. Empirical assessment of published effect sizes and power in the recent cognitive neuroscience and psychology literature. PLoS Biol. 2017; 15(3): E2000797. https://doi.org/10.1371/journal.pbio.2000797 PMID: 28253258
13. Ramsey S, Scoggins J. Practicing on the Tip of an Information Iceberg? Evidence of Underpublication of Registered Clinical Trials in Oncology. Oncologist. NIH Public Access; 2008; 13: 925. https://doi.org/10.1634/theoncologist.2008-0133 PMID: 18794216
14. Rosenthal R. The file drawer problem and tolerance for null results. Psychol Bull. 1979; 86: 638–641. https://doi.org/10.1037/0033-2909.86.3.638
15. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single-versus double-blind peer review. Proc Natl Acad Sci U S A. National Academy of Sciences; 2017; 114: 12708–12713. https://doi.org/10.1073/pnas.1707323114 PMID: 29138317
16. Willingham S, Volkmer J, Gentles A, Sahoo D, Dalerba P, Mitra S, et al. The challenges of replication. Elife. eLife Sciences Publications Limited; 2017; 6: 6662–6667. https://doi.org/10.7554/eLife.23693 PMID: 28182866
17. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci. 2018; 2017: 201708274. https://doi.org/10.1073/pnas.1708274114
18. Nosek B, Spies J, Motyl M. Scientific Utopia. Perspect Psychol Sci. 2012; 7: 615–631. https://doi.org/10.1177/1745691612459058 PMID: 26168121
19. Freedman LP, Cockburn IM, Simcoe TS. The economics of reproducibility in preclinical research. PLoS Biol. 2015; 13(6): E1002165. https://doi.org/10.1371/journal.pbio.1002165 PMID: 26057340
20. Mazey L, Tzavella L. Barriers and solutions for early career researchers in tackling the reproducibility crisis in cognitive neuroscience. Cortex. 2018; 113: 357–359. https://doi.org/10.1016/j.cortex.2018.12.015 PMID: 30670310
21. Button KS, Ioannidis JPA, Mokrysz C, Nosek BA, Flint J, Robinson ESJ, et al. Power failure: Why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci. 2013; 14: 365–376. https://doi.org/10.1038/nrn3475 PMID: 23571845
22. Algermissen J, Mehler DMA. May the power be with you: Are there highly powered studies in neuroscience, and how can we get more of them? J Neurophysiol. 2018; 119: 2114–2117. https://doi.org/10.1152/jn.00765.2017 PMID: 29465324
23. Dienes Z. Understanding psychology as a science: An introduction to scientific and statistical inference. Palgrave Macmillan; 2008.
24. Introducing eLife’s first computationally reproducible article [Internet].2019. Available from: Https://elifesciences.org/labs/ad58f08d/introducing-elife-s-first-computationally-reproducible-article. [cited 2019 March 25].
25. Living Figures–an interview with Björn Brembs and Julien Colomb [Internet].2014. Available from: Https://blog.f1000.com/2014/09/09/living-figures-interview/. [cited 2019 March 25].
26. Tomkins A, Zhang M, Heavlin WD. Reviewer bias in single-versus double-blind peer review. Proc Natl Acad Sci U S A. National Academy of Sciences; 2017; 114: 12708–12713. https://doi.org/10.1073/pnas.1707323114 PMID: 29138317
27. McKiernan EC, Bourne PE, Brown CT, Buck S, Kenall A, Lin J, et al. How open science helps researchers succeed. Elife. 2016; 5: 1–19. https://doi.org/10.7554/eLife.16800 PMID: 27387362
28. Chambers CD. Registered Reports: A new publishing initiative at Cortex. Cortex. 2013; 49: 609–610. https://doi.org/10.1016/j.cortex.2012.12.016 PMID: 23347556
29. Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proc Natl Acad Sci. 2018; 2017: 201708274. https://doi.org/10.1073/pnas.1708274114
30. Toelch U, Ostwald D. Digital open science—Teaching digital tools for reproducible and transparent research. PLoS Biol. 2018; 16(7): E2006022. https://doi.org/10.1371/journal.pbio.2006022 PMID: 30048447
31. Kerr N. HARKing: -hypnothesizing after the results are known. Personal Soc Psychol Rev. 1998; 2: 196–217. https://doi.org/10.1207/s15327957pspr0203_4
32. McIntosh RD. Exploratory reports: A new article type for Cortex. Cortex. Elsevier Ltd; 2017; 96: A1–A4. https://doi.org/10.1016/j.cortex.2017.07.014 PMID: 29110814
33. Laakso M, Welling P, Bukvova H, Nyman L, Bjoerk B-C, Hedlund T. The Development of Open Access Journal Publishing from 1993 to2009. PLoS ONE. 2011; 6(6): E20961. https://doi.org/10.1371/journal.pone.0020961 PMID: 21695139
34. Hanel PHP., Mehler DMA. Beyond reporting statistical significance: Identifying informative effect sizes to improve scientific communication. Public Underst Sci. 2019; https://doi.org/10.1177/0963662519834193 PMID: 30848696
35. Kluyver Thomas, Benjamin Ragan-Kelley Fernando Pérez, Granger Brian, Bussonnier Matthias, Frederic Jonathan, Kelley Kyle, Hamrick Jessica, Grout Jason, Corlay Sylvain, Ivanov Paul, Avila Damián, Abdalla Safia, Carol Willing JDT. Jupyter Notebooks–a publishing format for reproducible computational workflows. Positioning and Power in Academic Publishing: Players, Agents and Agendas. 2016. pp. 87–90. https://doi.org/10.3233/978-1-61499-649-1-87
36. Markowetz F. Five selfish reasons to work reproducibly. Genome Biol. 2015; 16: 274. https://doi.org/10.1186/s13059-015-0850-7 PMID: 26646147
37. Wagenmakers EJ, Dutilh G. Seven Selfish Reasons for Preregistration. APS Obs. 2016;29.
38. Bishop DVM. Fallibility in science: Responding to errors in the work of oneself and others. Adv Methods Pract Psychol Sci. 2017; 1–9. https://doi.org/10.7287/peerj.preprints.3486v1
39. Lewandowsky S, Bishop D. Research integrity: Don’t let transparency damage science. Nature. 2016; 529: 459–461. https://doi.org/10.1038/529459a PMID: 26819029
40. National Academy of Sciences; National Academy of Engineering and Institute of Medicine. The Postdoctoral Experience Revisited. Washington, D.C.: The National Academies Press; 2014. https://doi.org/10.17226/18982 PMID: 25590106
41. Pickett CL, Corb BW, Matthews CR, Sundquist WI, Berg JM. Toward a sustainable biomedical research enterprise: Finding consensus and implementing recommendations. 2015; 112. https://doi.org/10.1073/pnas.1509901112 PMID: 26195768
42. Moher D, Naudet F, Cristea IA, Miedema F, Ioannidis JPA, Goodman SN. Assessing scientists for hiring, promotion, and tenure. PLoS Biol. 2018; 16(3): E2004089. https://doi.org/10.1371/journal.pbio.2004089 PMID: 29596415
43. Poldrack RA. NeuroView The Costs of Reproducibility NeuroView. Neuron. Elsevier Inc.; 2019; 101: 11–14. https://doi.org/10.1016/j.neuron.2018.11.030 PMID: 30605654
44. Baker M. 1,500 scientists lift the lid on reproducibility. Nature. 2016; 533: 452–454. https://doi.org/10.1038/533452a PMID: 27225100
45. Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ. Scientific standards. Promoting an open research culture. Science (80). 2015; 348. https://doi.org/10.1126/science.aab2374 PMID: 26113702
46. Kidwell MC, Lazarević LB, Baranski E, Hardwicke TE, Piechowski S, Falkenberg L-S, et al. Badges to Acknowledge Open Practices: A Simple, Low-Cost, Effective Method for Increasing Transparency. PLoS Biol. 2016; 14(5): E1002456. https://doi.org/10.1371/journal.pbio.1002456 PMID: 27171007
47. Hardwicke TE, Ioannidis JPA. Mapping the universe of registered reports. Nat Hum Behav. Springer US; 2018; 2: 793–796. https://doi.org/10.1038/s41562-018-0444-y
48. Smith PL, Little DR. Small is beautiful: In defense of the small-N design. Psychonomic Bulletin & Review; 2018; https://doi.org/10.3758/s13423-018-1451-8
49. Fontanarosa P, Bauchner H, Flanagin A. Authorship and Team Science. JAMA. American Medical Association; 2017; 318: 2433. https://doi.org/10.1001/jama.2017.19341 PMID: 29279909
50. Brecht K. “Bullied Into Bad Science”: An Interview with Corina Logan–JEPS Bulletin.2017. Available from: Http://blog.efpsa.org/2017/10/23/meet-corina-logan-from-the-bullied-into-bad-science-campaign/. [cited2017 Nov 14].
51. Flier J. Faculty promotion must assess reproducibility. Nature. 2017; 549: 133–133. https://doi.org/10.1038/549133a PMID: 28905925
52. Nuzzo R. How scientists fool themselves–and how they can stop. Nature. 2015; 526: 182–185. https://doi.org/10.1038/526182a PMID: 26450039
53. Fanelli D, Costas R, Ioannidis JPA. Meta-assessment of bias in science. Proc Natl Acad Sci. 2017; 114: 3714–3719. https://doi.org/10.1073/pnas.1618569114 PMID: 28320937
54. Team J. JASP (Version 0.8.5.1)[Computer software]. 2017.
55. Eglen SJ, Marwick B, Halchenko YO, Hanke M, Sufi S, Gleeson P, et al. Toward standard practices for sharing computer code and programs in neuroscience. Nat Neurosci. Nature Publishing Group; 2017; 20: 770–773. https://doi.org/10.1038/nn.4550 PMID: 28542156
56. Editorial. Referees’ rights. Peer reviewers should not feel pressured to produce a report if key data are missing. Nature. 2018; 560: 409. https://doi.org/10.1038/d41586-018-06006-y PMID: 30131542
57. Centre for Open Science Blog. A Preregistration Coaching Network [Internet]. 2017 Available from: Https://cos.io/blog/preregistration-coaching-network/. [cited 2 Nov 2017].
58. Moshontz H, Campbell L, Ebersole CR, IJzerman H, Urry HL, Forscher PS, et al. The Psychological Science Accelerator: Advancing Psychology through a Distributed Collaborative Network. Adv Methods Pract Psychol Sci. 2018; https://doi.org/10.31234/OSF.IO/785QU
59. Biecek P, Kosinski M. archivist: An R Package for Managing, Recording and Restoring Data Analysis Results. 2017;VV. https://doi.org/10.18637/jss.v082.i11
60. Baumer B, Cetinkaya-Rundel M, Bray A, Loi L, Horton NJ. R Markdown: Integrating A Reproducible Analysis Tool into Introductory Statistics. BioRxiV. 2014; https://doi.org/10.5811/westjem.2011.5.6700
61. Gorgolewski KJ, Alfaro-Almagro F, Auer T, Bellec P, Capotă M, Chakravarty MM, et al. BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods. PLoS Comput Biol. 2017; 13(3): E1005209. https://doi.org/10.1371/journal.pcbi.1005209 PMID: 28278228
62. Simonsohn U, Nelson LD, Simmons JP. P-Curve: A Key to the File Drawer [Internet].2013. Available from: Https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2256237. [cited 2019 March 25].
63. Bishop DVM., Thompson PA. Problems in using p -curve analysis and text-mining to detect rate of p -hacking and evidential value. PeerJ. PeerJ Inc.; 2016; 4: E1715. https://doi.org/10.7717/peerj.1715 PMID: 26925335
64. Nuijten MB, Hartgerink CHJ., van Assen MALM., Epskamp S, Wicherts JM. The prevalence of statistical reporting errors in psychology (1985–2013). Behav Res Methods. Behavior Research Methods; 2016; 48: 1205–1226. https://doi.org/10.3758/s13428-015-0664-2 PMID: 26497820
65. Kamens J. Addgene: Making Materials Sharing ‘“Science As Usual.”‘PLoS Biol. 2014; 12(11): E1001991. https://doi.org/10.1371/journal.pbio.1001991 PMID: 25387006
66. Jones AR, Overly CC, Sunkin SM. The Allen Brain Atlas: 5 years and beyond. Nat Rev Neurosci. Nature Publishing Group; 2009; 10: 821–828. https://doi.org/10.1038/nrn2722 PMID: 19826436
67. Van Essen DC, Ugurbil K, Auerbach E, Barch D, Behrens TEJ, Bucholz R, et al. The Human Connectome Project: A data acquisition perspective. Neuroimage. 2012; 62: 2222–2231. https://doi.org/10.1016/j.neuroimage.2012.02.018 PMID: 22366334
68. Sudlow C, Gallacher J, Allen N, Beral V, Burton P, Danesh J, et al. UK Biobank: An Open Access Resource for Identifying the Causes of a Wide Range of Complex Diseases of Middle and Old Age. PLoS Med. 2015; 12: 1–10. https://doi.org/10.1371/journal.pmed.1001779 PMID: 25826379
69. Thompson PM, Stein JL, Medland SE, Hibar DP, Vasquez AA, Renteria ME, et al. The ENIGMA Consortium: Large-scale collaborative analyses of neuroimaging and genetic data. Brain Imaging Behav. Springer; 2014; 8: 153–82. https://doi.org/10.1007/s11682-013-9269-5 PMID: 24399358
70. Modjarrad K, Moorthy VS, Millett P, Gsell P-S, Roth C, Kieny M-P. Developing Global Norms for Sharing Data and Results during Public Health Emergencies. PLoS Med. 2016; 13(1): E1001935. https://doi.org/10.1371/journal.pmed.1001935 PMID: 26731342
71. Steward O, Balice-Gordon R. Rigor or mortis: Best practices for preclinical research in neuroscience. Neuron. Elsevier Inc.; 2014; 84: 572–581. https://doi.org/10.1016/j.neuron.2014.10.042 PMID: 25442936
72. Yozwiak NL, Schaffner SF, Sabeti PC. Data sharing: Make outbreak research open access. Nature. 2015; 518: 477–479. https://doi.org/10.1038/518477a PMID: 25719649
73. Lowndes JSS, Best BD, Scarborough C, Afflerbach JC, Frazier MR, O’Hara CC, et al. Our path to better science in less time using open data science tools. Nat Ecol Evol. 2017; 1: 0160. https://doi.org/10.1038/s41559-017-0160 PMID: 28812630
74. Milham MP, Ai L, Koo B, Xu T, Amiez C, Balezeau F, et al. An Open Resource for Non-human Primate Imaging. Neuron. 2018; 100: 61–74.e2. https://doi.org/10.1016/j.neuron.2018.08.039 PMID: 30269990
75. Weiss A, Wilson ML, Collins DA, Mjungu D, Kamenya S, Foerster S, et al. Personality in the chimpanzees of Gombe National Park. Sci Data. Nature Publishing Group; 2017; 4: 170146. https://doi.org/10.1038/sdata.2017.146 PMID: 29064463
76. Piwowar HA, Vision TJ. Data reuse and the open data citation advantage. PeerJ. 2013; 1–25. https://doi.org/10.7717/peerj.175 PMID: 24109559
77. Sarabipour S, Debat HJ, Emmott E, Burgess SJ, Schwessinger B, Hensel Z. On the value of preprints: An early career researcher perspective. PLoS Biol. 2019; 17(2): E3000151. https://doi.org/10.1371/journal.pbio.3000151 PMID: 30789895
78. Allen CPG, Mehler DMA. Open Science challenges, benefits and tips in early career and beyond. Psy-ArXiv. 2018; https://doi.org/10.31234/OSF.IO/3CZYT
79. Warren M. First analysis of ‘pre-registered’ studies shows sharp rise in null findings. Nature. 2018;
80. Vines TH, Albert AYK, Andrew RL, Dé F, Bock DG, Franklin MT, et al. Report The Availability of Research Data Declines Rapidly with Article Age. 2014; https://doi.org/10.1016/j.cub.2013.11.014
81. Stodden V, McNutt M, Bailey DH, Deelman E, Gil Y, Hanson B, et al. Enhancing reproducibility for computational methods. Science (80-). 2016; 354: 1240–1241. https://doi.org/10.1126/science.aah6168 PMID: 27940837
82. Jarvis MF, Williams M. Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps. Trends Pharmacol Sci. Elsevier Ltd; 2016; 37: 290–302. https://doi.org/10.1016/j.tips.2015.12.001 PMID: 26776451
83. Many junior scientists need to take a hard look at their job prospects. Nature. 2017; 550: 429–429. https://doi.org/10.1038/550429a PMID: 29072277
84. Kiley R, Peatfield T, Hansen J, Reddington F. Data Sharing from Clinical Trials—A Research Funder’s Perspective. N Engl J Med. Massachusetts Medical Society; 2017; 377: 1990–1992. https://doi.org/10.1056/NEJMsb1708278 PMID: 29141170
85. Taichman DB, Sahni P, Pinborg A, Peiperl L, Laine C, James A, et al. Data Sharing Statements for Clinical Trials—A Requirement of the International Committee of Medical Journal Editors. N Engl J Med. Massachusetts Medical Society; 2017; 376: 2277–2279. https://doi.org/10.1056/NEJMe1705439 PMID: 28581902
86. Poupon V, Seyller A, Rouleau GA. The Tanenbaum Open Science Institute: Leading a Paradigm Shift at the Montreal Neurological Institute. Neuron. Cell Press; 2017; 95: 1002–1006. https://doi.org/10.1016/J.NEURON.2017.07.026 PMID: 28858611