How Modeling Standards, Software, and Initiatives Support Reproducibility in Systems Biology and Systems Medicine

IEEE Transactions on Biomedical Engineering

Volumn 63, Issue 10, 2016, Pages 1999-2006

  Waltemath, Dagmar ^a   Wolkenhauer, Olaf ^b  

^a UNIVERSITY OF ROSTOCK   (Germany)

^b STELLENBOSCH UNIVERSITY   (South Africa)

Author keywords

Reproducibility; standards; systems biology; Systems medicine

Indexed keywords

BIOLOGY; REUSABILITY; STANDARDS;

REPRODUCIBILITIES; REPRODUCIBLE RESEARCH; SIMULATION STUDIES; SYSTEMS BIOLOGY;

COMPUTER SOFTWARE;

INSTRUMENT VALIDATION; MEDICINE; MODEL; REPRODUCIBILITY; SOFTWARE; SYSTEMS BIOLOGY; BIOLOGICAL MODEL; HUMAN; MEDICAL RESEARCH; STANDARDS;

BIOMEDICAL RESEARCH; HUMANS; MODELS, BIOLOGICAL; REPRODUCIBILITY OF RESULTS; SOFTWARE; SYSTEMS BIOLOGY;

EID: 84991252525     PISSN: 00189294     EISSN: 15582531     Source Type: Journal    
DOI: 10.1109/TBME.2016.2555481     Document Type: Article

References (90)

1. B. Nosek et al., "Promoting an open research culture," Science, vol. 348, no. 6242, pp. 1422-1425, 2015.
2. J. Lehrer, "The truth wears off," New Yorker, vol. 13, 2010, Art. no. 52.
3. B. Richard, "Reproducibility undergoes scrutiny," BioScience, vol. 64, no. 4, 2014, Art. no. 368.
4. E. Garcá-Berthou and C. Alcaraz, "Incongruence between test statistics and p values in medical papers," BMC Med. Res. Methodology, vol. 4, no. 1, 2004, Art. no. 13.
5. J. P. Ioannidis et al., "Repeatability of published microarray gene expression analyses," Nature Genetics, vol. 41, no. 2, pp. 149-155, 2009.
6. A. W. Bell et al., "A HUPO test sample study reveals common problems in mass spectrometry-based proteomics," Nature Methods, vol. 6, no. 6, pp. 423-430, 2009.
7. J. Arrowsmith, "Trial watch: Phase II failures: 2008-2010," Nature Rev. Drug Discovery, vol. 10, no. 5, pp. 328-329, 2011.
8. F. Prinz, "Believe it or not: How much can we rely on published data on potential drug targets" Nature Rev. Drug Discovery, vol. 10, no. 9, pp. 712-712, 2011.
9. A. Mullard, "Reliability of 'new drug target' claims called into question," Nature Rev. Drug Discovery, vol. 10, no. 9, pp. 643-644, 2011.
10. C. G. Begley and L. M. Ellis, "Drug development: Raise standards for preclinical cancer research," Nature, vol. 483, no. 7391, pp. 531-533, 2012.
11. D. Garijo et al., "Quantifying reproducibility in computational biology: The case of the tuberculosis drugome," PLoS ONE, vol. 8, no. 11, 2013, Art. no. e80278.
12. T. M. Errington et al., "An open investigation of the reproducibility of cancer biology research," eLife, vol. 3, 2014, Art. no. e04333.
13. M. Topalidou et al., "A long journey into reproducible computational neuroscience," Frontiers Comput. Neurosci., vol. 9, 2015, Art. no. 30.
14. A. Mobley et al., "A survey on data reproducibility in cancer research provides insights into our limited ability to translate findings from the laboratory to the clinic," PLoS ONE, vol. 8, no. 5, 2013, Art. no. e63221.
15. J. P. Ioannidis et al., "Increasing value and reducing waste in research design, conduct, analysis," Lancet, vol. 383, no. 9912, pp. 166-175, 2014.
16. R. Nuzzo, "Howscientists fool themselves-and howthey can stop," Nature, vol. 526, no. 7572, pp. 182-185, 2015.
17. R. MacCoun and S. Perlmutter, "Blind analysis: Hide results to seek the truth." Nature, vol. 526, no. 7572, pp. 187-189, 2015.
18. H. A. Piwowar et al., "Data archiving is a good investment," Nature, vol. 473, no. 7347, pp. 285-285, 2011.
19. D. Waltemath et al., "Toward standards and tools for whole-cell modeling," vol. 63, no. 10, pp. 2007-2014, 2016.
20. B. Mons, "Which gene did you mean" BMC Bioinform., vol. 6, no. 1, 2005, Art. no. 142.
21. J. P. Ioannidis, "Why most published research findings are false," Chance, vol. 18, no. 4, pp. 40-47, 2005.
22. Anonymous, "Credit where credit is overdue," Nature Biotechnol., vol. 27, 2009, Art. no. 579.
23. V. Stodden et al., "Toward reproducible computational research: An empirical analysis of data and code policy adoption by journals," PLoS ONE, vol. 8, no. 6, 2013, Art. no. e67111.
24. F. T. Bergmann et al., "COMBINE archive and OMEX format: One file to share all information to reproduce a modeling project," BMC Bioinform., vol. 15, no. 1, 2014, Art. no. 369.
25. C. G. Begley and J. P. Ioannidis, "Reproducibility in science improving the standard for basic and preclinical research," Circulation Res., vol. 116, no. 1, pp. 116-126, 2015.
26. D. Pattinson, "PLOS ONE launches reproducibility initiative," PLOS ONE Community Blog, AUG. 2012, published online at http://blogs.plos. org/everyone/2012/08/14/plos-one-launches-reproducibi lity-initiative/.
27. D. Waltemath et al., "Reproducibility of model-based results in systems biology," in System Biology. Berlin, Germany: Springer, 2013, pp. 301-320.
28. D. Waltemath et al., "Meeting report from the fourth meeting of the computational modeling in biology network (COMBINE)," Standards Genomic Sci., vol. 9, no. 3, 2014.
29. M. Hucka et al., "Promoting coordinated development of communitybased information standards for modeling in biology: The COMBINE initiative," Frontiers Bioeng. Biotechnol., vol. 3, 2015, Art. no. 19.
30. A. A. Cuellar et al., "An overview of CellML 1.1, a biological model description language," Simulation, vol. 79, no. 12, pp. 740-747, 2003.
31. M. Hucka et al., "The systems biology markup language (SBML): A medium for representation and exchange of biochemical networkmodels," Bioinformatics, vol. 19, no. 4, pp. 524-531, 2003.
32. M. Galdzicki et al., "The synthetic biology open language (SBOL) provides a community standard for communicating designs in synthetic biology," Nature Biotechnol., vol. 32, no. 6, 2014, Art. no. 545.
33. E. Demir et al., "The BioPAX community standard for pathway data sharing," Nature Biotechnol., vol. 28, no. 9, pp. 935-942, 2010.
34. D. Waltemath et al., "Reproducible computational biology experiments with SED-ML-The simulation experiment description markup language," BMC Syst. Biol., vol. 5, no. 1, 2011, Art. no. 198.
35. N. Le Novère et al., "The systems biology graphical notation," Nature Biotechnol., vol. 27, pp. 735-741, 2009.
36. M. Hucka et al., "A profile of today's SBML-Compatible software," in Proc. IEEE e-Science Workshops, 2011, pp. 143-150.
37. M. Courtot et al., "Controlled vocabularies and semantics in systems biology," Molecular Syst. Biol., vol. 7, no. 1, 2011, Art. no. 543.
38. P. Gleeson et al., "NeuroML:Alanguage for describing data driven models of neurons and networks with a high degree of biological detail," PLoS Comput. Biol., vol. 6, no. 6, 2010, Art. no. e1000815.
39. M. J. Swat et al., "Pharmacometrics markup language (PharmML): Opening new perspectives for model exchange in drug development," CPT: Pharmacometrics Syst. Pharmacology, 2015, pp. 316-319.
40. R. D. Peng, "Reproducible research in computational science," Science, vol. 334, no. 6060, 2011, Art. no. 1226.
41. J. B. Buckheit and D. L. Donoho, "Wavelab and reproducible research," in Wavelets and Statistics, vol. 103, Berlin, Germnay: Springer, 1995, pp. 55-81.
42. S. R. Piccolo et al., "Tools and techniques for computational reproducibility," bioRxiv, 2015.
43. C. Boettiger, "An introduction to Docker for reproducible research," ACM SIGOPS Oper. Syst. Rev., vol. 49, no. 1, pp. 71-79, 2015.
44. ENCODE Project Consortium, "An integrated encyclopedia of DNA elements in the human genome," Nature, vol. 489, no. 7414, pp. 57-74, 2012.
45. J. Goecks et al., "Galaxy: A comprehensive approach for supporting accessible, reproducible, transparent computational research in the life sciences," Genome Biol, vol. 11, no. 8, 2010, Art. no. R86.
46. S. O'hagan and D. B. Kell, "Software review: The KNIME workflow environment and its applications in genetic programming and machine learning," Genetic Program. Evolvable Mach., vol. 16, no. 3, pp. 387-391, Sep. 2015. [Online]. Available: http://dx.doi.org/10.1007/s 10710-015-9247-3
47. A. Weidemann et al., "SYCAMORE-A systems biology computational analysis and modeling research environment," Bioinformatics, vol. 24, no. 12, pp. 1463-1464, 2008.
48. B. G. Olivier and J. L. Snoep, "Web-based kinetic modelling using JWS Online," Bioinformatics, vol. 20, no. 13, pp. 2143-2144, 2004.
49. F. Krause et al., "Annotation and merging of SBML models with semanticSBML," Bioinformatics, vol. 26, no. 3, pp. 421-422, 2010.
50. N. Rodriguez et al., "The systems biology format converter," BMC Bioinform., vol. 17, no. 1, 2016, Art. no. 1.
51. M. Scharm and D. Waltemath, "Extracting reproducible simulation studies from model repositories using the combinearchive toolkit," in Datenbanksysteme für Business, Technologie und Web, N. Ritter, A. Henrich, W. Lehner et al., Eds., 2015.
52. S. Hoops et al., "-A complex pathway simulator," Bioinformatics, vol. 22, no. 24, pp. 3067-3074, 2006.
53. F. T. Bergmann and H. M. Sauro, "SBW-A modular framework for systems biology," in Proc. 38th Conf. Winter Simul., 2006, pp. 1637-1645.
54. S. N. Murphy et al., "Serving the enterprise and beyond with informatics for integrating biology and the bedside (i2b2)," J. Amer. Med. Inform. Assoc., vol. 17, no. 2, pp. 124-130, 2010.
55. N. Le Novère, "Model storage, exchange and integration,"BMCNeurosci., vol. 7, 2006, Art. no. S11.
56. R. Henkel et al., "Combining computationalmodels, semantic annotations and simulation experiments in a graph database," Database, vol. 2015, 2015, Art. no. bau130.
57. R. Henkel et al., "Ranked retrieval of computational biology models," BMC Bioinform., vol. 11, no. 1, 2010, Art. no. 423.
58. N. Juty et al., "BioModels: Content, features, functionality, use," CPT Pharmacometrics Syst. Pharmacology, vol. 4, no. 2, pp. 1-14, 2015.
59. C. Li et al., "Biomodels database: An enhanced, curated and annotated resource for published quantitative kinetic models," BMC Syst. Biol., vol. 4, no. 1, 2010, Art. no. 92.
60. R. Machné et al., "The SBML ODE solver library: A native API for symbolic and fast numerical analysis of reaction networks," Bioinformatics, vol. 22, no. 11, pp. 1406-1407, 2006.
61. N. Juty et al., "Identifiers.org and MIRIAM registry: Community resources to provide persistent identification," Nucleic Acids Res., vol. 40, pp. D580-D586, 2012.
62. M. L. Hines et al., "ModelDB: A database to support computational neuroscience," J. Comput. Neuroscie., vol. 17, no. 1, pp. 7-11, 2004.
63. P. Gleeson et al., "The open source brain initiative: Enabling collaborative modelling in computational neuroscience," BMC Neurosci., vol. 13, 2012, Art. no. O7.
64. T. Yu et al., "The physiome model repository 2," Bioinformatics, vol. 27, no. 5, pp. 743-744, Mar. 2011.
65. J. Hetherington et al., "Addressing the challenges of multiscale model management in systems biology," Comput. Chemi. Eng., vol. 31, no. 8, pp. 962-979, 2007.
66. M. Scharm et al., "An algorithm to detect and communicate the differences in computational models describing biological systems," Bioinformatics, 2015, Art. no. btv484.
67. K. Wolstencroft et al., "SEEK: A systems biology data and model management platform," BMC Syst. Biol., vol. 9, no. 1, 2015, Art. no. 33.
68. J. T. Leek and R. D. Peng, "Opinion: Reproducible research can still be wrong: Adopting a prevention approach," Proc. Nat. Acad. Sci., vol. 112, no. 6, pp. 1645-1646, 2015.
69. S. Schnell, "Ten simple rules for a computational biologists laboratory notebook," PLoS Comput. Biol., vol. 11, no. 9, 2015, Art. no. e1004385.
70. The Reproducibility Initiative, "Replication best practices," 2015, last accessed 2015-08-03.
71. G. K. Sandve et al., "Ten simple rules for reproducible computational research," PLoS Comput. Biol., vol. 9, no. 10, 2013, Art. no. e1003285.
72. N. Le Novère et al., "Minimum information requested in the annotation of biochemical models (MIRIAM)," Nature Biotechnol., vol. 23, no. 12, pp. 1509-1515, 2005.
73. D.Waltemath et al., "Minimum information about a simulation experiment (MIASE)," PLoS Comput. Biol., vol. 7, no. 4, 2011, Art. no. e10011221.
74. T. P. Sneddon et al., "GigaDB: Promoting data dissemination and reproducibility," Database, vol. 2014, 2014, Art. no. bau018.
75. S. Bechhofer et al., "Research objects: Towards exchange and reuse of digital knowledge," Future Web Collaborative Sci., http://dx.doi.org/10.1038/npre.2010.4626.1
76. F. T. Bergmann et al., "Simulation experiment description markup language (sed-ml) level 1 version 2," J. Integrative Bioinform., vol. 12, no. 2, 2015, Art. no. 262.
77. V. Stodden et al., "Setting the default to reproducible," Comput. Sci. Res. SIAM News, vol. 46, pp. 4-6, 2013.
78. S. Bechhofer et al., "Why linked data is not enough for scientists," Future Generation Comput. Syst., vol. 29, no. 2, pp. 599-611, 2013.
79. S.-A. Sansone et al., "Toward interoperable bioscience data," Nature Genetics, vol. 44, no. 2, pp. 121-126, 2012.
80. D. G. Altman et al., "Equator: Reporting guidelines for health research," Open Med., vol. 2, no. 2, 2008, Art. no. e49.
81. P. Lord et al., Large-scale data sharing in the life sciences: Data standards, incentives, barriers and funding models, the joint data standards study (2006) [Online]. Available: http://www. mrc. ac. uk/pdfjdss-final-report. pdf
82. W. Wruck et al., "Data management strategies for multinational largescale systems biology projects," Briefings Bioinform., vol. 15, no. 1, pp. 65-78, 2014.
83. F. S. Collins and L. A. Tabak, "NIH plans to enhance reproducibility," Nature, vol. 505, no. 7485, 2014, Art. no. 612.
84. G. Penkler et al., "Construction and validation of a detailed kinetic model of glycolysis in plasmodium falciparum," FEBS J., vol. 282, no. 8, pp. 1481-1511, 2015.
85. M. Scharm, "Combine archive show case," University of Rostock, Rostock, Germany, Tech. Rep., 2015. [Online]. Available github, https://github.com/SemsProject/CombineArchiveShowCase
86. J. P. Ioannidis, "How to make more published research true," PLoS MEDICINE, vol. 11, no. 10, 2014, Art. no. e1001747.
87. P. Ronald, "Lab life: The anatomy of a retraction," Scientific American, OCT. 2013, last accessed online: 2015-08-07 Available: http://blogs.scientificamerican. com/food-matters/lab-life-the-anatomy-of-a-retraction/.
88. H. A. Piwowar et al., "Sharing detailed research data is associated with increased citation rate," PLoS ONE, vol. 2, no. 3, 2007, Art. no. e308.
89. J. Cooper et al., "The cardiac electrophysiology web lab," Biophys. J., vol. 110, no. 2, pp. 292-300, 2016.
90. K. Popper, The Logic of Scientific Discovery. Evanston, IL, USA: Routledge, 2005.