Tools and techniques for computational reproducibility

GigaScience

Volumn 5, Issue 1, 2016, Pages

  Piccolo, Stephen R ^a   Frampton, Michael B ^b  

^a BRIGHAM YOUNG UNIVERSITY   (United States)

^b Brigham Young University   (United States)

Author keywords

Computational reproducibility; Literate programming; Practice of science; Software containers; Software frameworks; Virtualization

Indexed keywords

CALCULATION; COMPUTER INTERFACE; COMPUTER PROGRAM; INFORMATION PROCESSING; INFORMATION RETRIEVAL; MATHEMATICAL ANALYSIS; PRIORITY JOURNAL; REPRODUCIBILITY; RESEARCH; REVIEW; VALIDITY; BIOLOGY; METHODOLOGY; PROCEDURES; SOFTWARE;

COMPUTATIONAL BIOLOGY; REPRODUCIBILITY OF RESULTS; RESEARCH DESIGN; SOFTWARE;

EID: 84991529234     PISSN: None     EISSN: 2047217X     Source Type: Journal    
DOI: 10.1186/s13742-016-0135-4     Document Type: Review

References (127)

1. Fisher RA. The Design of Experiments. New York: Hafner Press; 1935.
2. Popper KR. The logic of scientific discovery. London: Routledge; 1959.
3. Peng RD. Reproducible research in computational science. Science. 2011;334:1226-7.
4. Russell JF. If a job is worth doing, it is worth doing twice. Nature. 2013;496:7.
5. Feynman RP. Six Easy Pieces: Essentials of Physics Explained by Its Most Brilliant Teacher. Boston, MA: Addison-Wesley; 1995. p. 34-5.
6. Murray-Rust P, Murray-Rust D. Reproducible Physical Science and the Declaratron. In: Stodden VC, Leisch F, Peng RD, editors. Implementing Reproducible Research. Boca Raton, FL: CRC Press; 2014. p. 113.
7. Hey AJG, Tansley S, Tolle KM, Others. The fourth paradigm: data-intensive scientific discovery. Redmond, WA: Microsoft Research Redmond, WA; 2009.
8. Millman KJ, Pérez F. Developing Open-Source Scientific Practice. Implementing Reproducible Research. Boca Raton, FL: CRC Press; 2014;149.
9. Wilson G, Aruliah DA, Brown CT, Chue Hong NP, Davis M, Guy RT, et al. Best practices for scientific computing. PLoS Biol. 2014;12:e1001745.
10. Software with impact. Nat Methods. 2014;11:211.
11. Hong NC. We are the 92% [Internet]. Figshare; 2014. Available from: http://dx.doi.org/10.6084/M9.FIGSHARE.1243288. Accessed 1 March 2016.
12. Sacks J, Welch WJ, Mitchell TJ, Wynn HP. Design and analysis of computer experiments. Stat Sci. 1989;4:409-23.
13. Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, Bourne PE, et al. Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PLoS One. 2013;8:e80278.
14. Error prone. Nature. 2012;487:406.
15. Vandewalle P, Barrenetxea G, Jovanovic I, Ridolfi A, Vetterli M. Experiences with reproducible research in various facets of signal processing research. IEEE International Conference on Acoustics, Speech and Signal Processing-ICASSP'07. IEEE. 2007;2007:IV-1253-6.
16. Cassey P, Cassey P, Blackburn T, Blackburn T. Reproducibility and repeatability in ecology. Bioscience. 2006;56:958-9.
17. Murphy JM, Sexton DMH, Barnett DN, Jones GS, Webb MJ, Collins M, et al. Quantification of modelling uncertainties in a large ensemble of climate change simulations. Nature. 2004;430:768-72.
18. McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Cazier J-B, et al. Choice of transcripts and software has a large effect on variant annotation. Genome Med. 2014;6:26.
19. Neuman JA, Isakov O, Shomron N. Analysis of insertion-deletion from deep-sequencing data: Software evaluation for optimal detection. Brief Bioinform. 2013;14:46-55.
20. Bradnam KR, Fass JN, Alexandrov A, Baranay P, Bechner M, Birol I, et al. Assemblathon 2: evaluating de novo methods of genome assembly in three vertebrate species. Gigascience. 2013;2:10.
21. Bilal E, Dutkowski J, Guinney J, Jang IS, Logsdon BA, Pandey G, et al. Improving breast cancer survival analysis through competition-based multidimensional modeling. PLoS Comput Biol. 2013;9:e1003047.
22. Gronenschild EHBM, Habets P, Jacobs HIL, Mengelers R, Rozendaal N, van Os J, et al. The effects of FreeSurfer version, workstation type, and Macintosh operating system version on anatomical volume and cortical thickness measurements. PLoS One. 2012;7:e38234.
23. Moskvin OV, McIlwain S, Ong IM. CAMDA 2014: Making sense of RNA-Seq data: From low-level processing to functional analysis. Systems Biomedicine. 2014;2:31-40.
24. Reducing our irreproducibility. Nature. 2013;496:398-398.
25. Michael CM, Nass SJ, Omenn GS, editors. Evolution of Translational Omics: Lessons Learned and the Path Forward. Washington, D.C: The National Academies Press; 2012.
26. Collins FS, Tabak LA. Policy: NIH plans to enhance reproducibility. Nature. 2014;505:612-3.
27. Chambers JM. S as a Programming Environment for Data Analysis and Graphics. Problem Solving Environments for Scientific Computing, Proceedings 17th Symposium on the Interface of Statistics and Computing North Holland; 1985. p. 211-4.
28. LeVeque RJ, Mitchell IM, Stodden V. Reproducible research for scientific computing: Tools and strategies for changing the culture. Comput Sci Eng. 2012;14:13.
29. Stodden V, Guo P, Ma Z. Toward reproducible computational research: an empirical analysis of data and code policy adoption by journals. PLoS One. 2013;8:2-9.
30. Morin A, Urban J, Adams PD, Foster I, Sali A, Baker D, et al. Research priorities. Shining light into black boxes. Science. 2012;336:159-60.
31. Rebooting review. Nat Biotechnol. 2015;33:319.
32. Ioannidis JP a, Allison DB, Ball C a, Coulibaly I, Cui X, Culhane AC, et al. Repeatability of published microarray gene expression analyses. Nat Genet. 2009;41:149-55.
33. Nekrutenko A, Taylor J. Next-generation sequencing data interpretation: enhancing reproducibility and accessibility. Nat Rev Genet. 2012;13:667-72.
34. Baggerly K a, Coombes KR. Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. Ann Appl Stat. 2009;3:1309-34.
35. Decullier E, Huot L, Samson G, Maisonneuve H. Visibility of retractions: a cross-sectional one-year study. BMC Res Notes. 2013;6:238.
36. Claerbout JF, Karrenbach M. Electronic Documents Give Reproducible Research a New Meaning. Meeting of the Society of Exploration Geophysics. New Orleans, LA; 1992.
37. Stodden V, Miguez S. Best practices for computational science: software infrastructure and environments for reproducible and extensible research. J Open Res Softw. 2014;2:21.
38. Ravel J, Wommack KE. All hail reproducibility in microbiome research. Microbiome. 2014;2:8.
39. Stodden V. 2014: What scientific idea is ready for retirement? [Internet]. http://edge.org/response-detail/25340. 2014. Available from: http://edge.org/response-detail/25340. Accessed 1 March 2016.
40. Birney E, Hudson TJ, Green ED, Gunter C, Eddy S, Rogers J, et al. Prepublication data sharing. Nature. 2009;461:168-70.
41. Hothorn T, Leisch F. Case studies in reproducibility. Brief Bioinform. 2011;12:288-300.
42. Schofield PN, Bubela T, Weaver T, Portilla L, Brown SD, Hancock JM, et al. Post-publication sharing of data and tools. Nature. 2009;461:171-3.
43. Piwowar HA., Day RS, Fridsma DB. Sharing detailed research data is associated with increased citation rate. PLoS One. 2007;2.
44. Johnson VE. Revised standards for statistical evidence. Proc Natl Acad Sci U S A. 2013;110:19313-7.
45. Halsey LG, Curran-everett D, Vowler SL, Drummond GB. The fickle P value generates irreproducible results. Nat Methods. 2015;12:179-85.
46. Wilson G. Software Carpentry: lessons learned. F1000Res. 2016;3:62.
47. Sandve GK, Nekrutenko A, Taylor J, Hovig E. Ten simple rules for reproducible computational research. PLoS Comput Biol. 2013;9:1-4.
48. GNU Make [Internet]. 2016. Available from https://www.gnu.org/software/make. Accessed 1 March 2016.
49. Make for Windows [Internet]. 2016. Available from http://gnuwin32.sourceforge.net/packages/make.htm. Accessed 1 March 2016.
50. Puppet [Internet]. 2016. Available from https://puppetlabs.com. Accessed 1 March 2016.
51. Code share. Nature. 2014;514:536.
52. Blischak JD, Davenport ER, Wilson G. A quick introduction to version control with Git and GitHub. PLoS Comput Biol. 2016;12:e1004668.
53. Loeliger J, McCullough M. Version Control with Git: Powerful Tools and Techniques for Collaborative Software Development. Sebastopol, California: "O'Reilly Media, Inc."; 2012. p. 456.
54. Huber W, Carey VJ, Gentleman R, Anders S, Carlson M, Carvalho BS, et al. Orchestrating high-throughput genomic analysis with Bioconductor. Nat Methods. 2015;12:115-21.
55. R Core Team. R: A Language and Environment for Statistical Computing [Internet]. Vienna, Austria: R Foundation for Statistical Computing; 2014. Available from: http://www.r-project.org. Accessed 1 March 2016.
56. Tóth G, Sokolov IV, Gombosi TI, Chesney DR, Clauer CR, De Zeeuw DL, et al. Space weather modeling framework: a new tool for the space science community. J Geophys Res. 2005;110:A12226.
57. Tan E, Choi E, Thoutireddy P, Gurnis M, Aivazis M. GeoFramework: Coupling multiple models of mantle convection within a computational framework. Geochem Geophys Geosyst. [Internet]. 2006;7. Available from: http://doi.wiley.com/10.1029/2005GC001155.
58. Heisen B, Boukhelef D, Esenov S, Hauf S, Kozlova I, Maia L, et al. Karabo: An Integrated Software Framework Combining Control, Data Management, and Scientific Computing Tasks. 14th International Conference on Accelerator & Large Experimental Physics Control Systems, ICALEPCS2013. San Francisco, CA; 2013.
59. Schneider CA, Rasband WS, Eliceiri KW. NIH Image to ImageJ: 25 years of image analysis. Nat Methods. 2012;9:671-5.
60. Schindelin J, Arganda-Carreras I, Frise E, Kaynig V, Longair M, Pietzsch T, et al. Fiji: an open-source platform for biological-image analysis. Nat Methods. 2012;9:676-82.
61. Biasini M, Schmidt T, Bienert S, Mariani V, Studer G, Haas J, et al. OpenStructure: an integrated software framework for computational structural biology. Acta Crystallogr D Biol Crystallogr. 2013;69:701-9.
62. Ivy, the agile dependency manager [Internet]. 2016. Available from http://ant.apache.org/ivy. Accessed 1 March 2016.
63. aRchive: Enabling reproducibility of Bioconductor package versions (for Galaxy) [Internet]. 2016. Available from http://bioarchive.github.io. Accessed 1 March 2016.
64. Martin RC. Clean code: a handbook of agile software craftsmanship. Pearson Education. 2009.
65. Knuth DE. Literate programming. Comput J. 1984;27:97-111.
66. Pérez F, Granger BE. IPython: a system for interactive scientific computing. Comput Sci Eng. 2007;9:21-9.
67. Shen H. Interactive notebooks: Sharing the code. Nature. 2014;515:151-2.
68. Xie Y. Dynamic Documents with R and knitr. Boca Raton, FL: CRC Press; 2013. p. 216.
69. RStudio Team. RStudio: Integrated Development for R [Internet]. [cited 2015 Nov 20]. Available from: http://www.rstudio.com. Accessed 1 March 2016.
70. Gross AM, Orosco RK, Shen JP, Egloff AM, Carter H, Hofree M, et al. Multitiered genomic analysis of head and neck cancer ties TP53 mutation to 3p loss. Nat Genet. 2014;46:1-7.
71. Ding T, Schloss PD. Dynamics and associations of microbial community types across the human body. Nature. 2014;509:357-60.
72. Ram Y, Hadany L. The probability of improvement in Fisher's geometric model: A probabilistic approach. Theor Popul Biol. 2015;99:1-6.
73. Meadow JF, Altrichter AE, Kembel SW, Moriyama M, O'Connor TK, Womack AM, et al. Bacterial communities on classroom surfaces vary with human contact. Microbiome. 2014;2:7.
74. White E. Programming for Biologists [Internet]. Available from: http://www.programmingforbiologists.org. Accessed 1 March 2016.
75. Peng RD, Leek J, Caffo B. Coursera course: Exploratory Data Analysis [Internet]. Available from: https://www.coursera.org/learn/exploratory-data-analysis.
76. Bioconductor-Courses and Conferences [Internet]. [cited 2015 Nov 20]. Available from: http://master.bioconductor.org/help/course-materials. Accessed 1 March 2016.
77. Gil Y, Deelman E, Ellisman M, Fahringer T, Fox G, Gannon D, et al. Examining the challenges of scientific workflows. Computer. 2007;40:24-32.
78. Giardine B, Riemer C, Hardison RC, Burhans R, Elnitski L, Shah P, et al. Galaxy: a platform for interactive large-scale genome analysis. Genome Res. 2005;15:1451-5.
79. Goecks J, Nekrutenko A, Taylor J. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 2010;11:R86.
80. Afgan E, Baker D, Coraor N, Goto H, Paul IM, Makova KD, et al. Harnessing cloud computing with Galaxy Cloud. Nat Biotechnol. 2011;29:972-4.
81. Callahan SP, Freire J, Santos E, Scheidegger CE, Silva CT, Vo HT. VisTrails: Visualization Meets Data Management. Proceedings of the 2006 ACM SIGMOD International Conference on Management of Data. New York, NY, USA: ACM; 2006. p. 745-7.
82. Davidson SB, Freire J. Provenance and scientific workflows. Proceedings of the 2008 ACM SIGMOD international conference on Management of data-SIGMOD'08. 2008. p. 1345.
83. Lazarus R, Kaspi A, Ziemann M. Creating re-usable tools from scripts: The Galaxy Tool Factory. Bioinformatics. 2012;28:3139-40.
84. Dudley JT, Butte AJ. In silico research in the era of cloud computing. Nat Biotechnol. 2010;28:1181-5.
85. Hurley DG, Budden DM, Crampin EJ. Virtual Reference Environments: a simple way to make research reproducible. Brief Bioinform. 2015;16(5)901-903.
86. Gent IP. The Recomputation Manifesto. arXiv [Internet]. 2013; Available from: http://arxiv.org/abs/1304.3674. Accessed 1 March 2016.
87. Howe B. Virtual appliances, cloud computing, and reproducible research. Comput Sci Eng. 2012;14:36-41.
88. Brown CT. Virtual machines considered harmful for reproducibility [Internet]. 2012. Available from: http://ivory.idyll.org/blog/vms-considered-harmful.html. Accessed 1 March 2016.
89. Piccolo SR. Building portable analytical environments to improve sustainability of computational-analysis pipelines in the sciences [Internet]. 2014. Available from: http://dx.doi.org/10.6084/m9.figshare.1112571. Accessed 1 March 2016.
90. Krampis K, Booth T, Chapman B, Tiwari B, Bicak M, Field D, et al. Cloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community. BMC Bioinformatics. 2012;13:42.
91. CloudBioLinux: configure virtual (or real) machines with tools for biological analyses [Internet]. 2016. Available from https://github.com/chapmanb/cloudbiolinux. Accessed 1 March 2016.
92. Felter W, Ferreira A, Rajamony R, Rubio J. An Updated Performance Comparison of Virtual Machines and Linux Containers [Internet]. IBM Research Division; 2014. Available from: http://domino.research.ibm.com/library/CyberDig.nsf/papers/0929052195DD819C85257D2300681E7B/$File/rc25482.pdf. Accessed 1 March 2016.
93. Eglen SJ, Weeks M, Jessop M, Simonotto J, Jackson T, Sernagor E. A data repository and analysis framework for spontaneous neural activity recordings in developing retina. Gigascience. 2014;3:3.
94. Eglen SJ. Bivariate spatial point patterns in the retina: a reproducible review. Journal de la Société Française de Statistique. 2016;157:33-48.
95. Bremges A, Maus I, Belmann P, Eikmeyer F, Winkler A, Albersmeier A, et al. Deeply sequenced metagenome and metatranscriptome of a biogasproducing microbial community from an agricultural production-scale biogas plant. Gigascience. 2015;4:33.
96. Belmann P, Dröge J, Bremges A, McHardy AC, Sczyrba A, Barton MD. Bioboxes: standardised containers for interchangeable bioinformatics software. Gigascience. 2015;4:47.
97. Barton M. nucleotides · genome assembler benchmarking [Internet]. [cited 2015 Nov 20]. Available from: http://nucleotid.es. Accessed 1 March 2016.
98. Hones MJ. Reproducibility as a Methodological Imperative in Experimental Research. PSA: Proceedings of the Biennial Meeting of the Philosophy of Science Association. Philosophy of Science Association. 1990. p. 585-99.
99. Rosenberg DM, Horn CC. Neurophysiological analytics for all! Free opensource software tools for documenting, analyzing, visualizing, and sharing using electronic notebooks. J Neurophysiol American Physiological Society; Apr2016;jn.00137.2016.
100. everware [Internet]. 2016. Available from https://github.com/everware/everware. Accessed 1 March 2016.
101. Crick T. "Share and Enjoy": Publishing Useful and Usable Scientific Models. Available from: http://arxiv.org/abs/1409.0367v2. Accessed 1 March 2016.
102. Donoho DL. An invitation to reproducible computational research. Biostatistics. 2010;11:385-8.
103. Goldberg D. What every computer scientist should know about floating-point arithmetic. ACM Comput Surv. 1991;23:5-48.
104. Shirts M, Pande VS. COMPUTING: screen savers of the world unite! Science. 2000;290:1903-4.
105. Bird I. Computing for the large hadron Collider. Annu Rev Nucl Part Sci. 2011;61:99-118.
106. Anderson DP. BOINC: A System for Public Resource Computing and Storage. Proceedings of the Fifth IEEE/ACM International Workshop on Grid Computing (GRID'04). 2004.
107. Ransohoff DF. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer. 2005;5:142-9.
108. Bild AH, Chang JT, Johnson WE, Piccolo SR. A field guide to genomics research. PLoS Biol. 2014;12:e1001744.
109. Köster J, Rahmann S. Snakemake-a scalable bioinformatics workflow engine. Bioinformatics. 2012;28:2520-2.
110. Sadedin SP, Pope B, Oshlack A. Bpipe: a tool for running and managing bioinformatics pipelines. Bioinformatics. 2012;28:1525-6.
111. Tange O. GNU Parallel-The Command-Line Power Tool.;login: The USENIX Magazine. Frederiksberg, Denmark; 2011;36:42-7.
112. Albrecht M, Donnelly P, Bui P, Thain D. Makeflow: A portable abstraction for data intensive computing on clusters, clouds, and grids. Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies. 2012.
113. Knight S, Austin C, Crain C, Leblanc S, Roach A. Scons software construction tool [Internet]. 2011. Available from: http://www.scons.org. Accessed 1 March 2016.
114. Altintas I, Berkley C, Jaeger E, Jones M, Ludascher B, Mock S. Kepler: an extensible system for design and execution of scientific workflows. Proceedings. 16th International Conference on Scientific and Statistical Database Management, 2004. IEEE; 2004. p. 423-4.
115. Goff SA, Vaughn M, McKay S, Lyons E, Stapleton AE, Gessler D, et al. The iPlant collaborative: cyberinfrastructure for plant biology. Front Plant Sci Frontiers. 2011;2:34.
116. Reich M, Liefeld T, Gould J, Lerner J, Tamayo P, Mesirov JP. GenePattern 2.0. Nat Genet. 2006;38:500-1.
117. Reich M, Liefeld J, Thorvaldsdottir H, Ocana M, Polk E, Jang D, et al. GenomeSpace: An environment for frictionless bioinformatics. Cancer Res. 2012;72:3966-3966.
118. GenePattern: A platform for reproducible bioinformatics [Internet]. 2016. Available from http://www.broadinstitute.org/cancer/software/genepattern]. Accessed 1 March 2016.
119. Wolstencroft K, Haines R, Fellows D, Williams A, Withers D, Owen S, et al. The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud. Nucleic Acids Res. 2013;41:557-61.
120. Rex DE, Ma JQ, Toga AW. The LONI pipeline processing environment. Neuroimage. 2003;19:1033-48.
121. LONI Pipeline Processing Environment [Internet]. 2016. Available from http://www.loni.usc.edu/Software/Pipeline. Accessed 1 March 2016.
122. Vortex [Internet]. 2016. Available from https://github.com/websecurify/node-vortex. Accessed 1 March 2016.
123. Amazon Web Services [Internet]. 2016. Available from http://aws.amazon.com. Accessed 1 March 2016.
124. Google Cloud Platform [Internet]. 2016. Available from https://cloud.google.com/compute. Accessed 1 March 2016.
125. Microsoft Azure [Internet]. 2016. Available from https://azure.microsoft.com. Accessed 1 March 2016.
126. lmctfy-Let Me Contain That For You [Internet]. 2016. Available from https://github.com/google/lmctfy. Accessed 1 March 2016.
127. Warden [Internet]. 2016. Available from http://docs.cloudfoundry.org/concepts/architecture/warden.html. Accessed 1 March 2016.