Experiences with workflows for automating data-intensive bioinformatics

Biology Direct

Volumn 10, Issue 1, 2015, Pages

  Spjuth, Ola ^a   Bongcam Rudloff, Erik ^b   Hernández, Guillermo Carrasco ^a   Forer, Lukas ^c   Giovacchini, Mario ^a   Guimera, Roman Valls ^a   Kallio, Aleksi ^d   Korpelainen, Eija ^d   Kańdula, Maciej M ^e   Krachunov, Milko ^f   Kreil, David P ^e   Kulev, Ognyan ^f   Łabaj, Pawel P ^e   Lampa, Samuel ^a   Pireddu, Luca ^g   Schönherr, Sebastian ^c   Siretskiy, Alexey ^h   Vassilev, Dimitar ⁱ  

^a SCIENCE FOR LIFE LABORATORY   (Sweden)

^b SWEDISH UNIVERSITY OF AGRICULTURAL SCIENCES   (Sweden)

^c INNSBRUCK MEDICAL UNIVERSITY   (Austria)

^d CSC Scientific Computing Ltd   (Finland)

^e UNIVERSITY OF NATURAL RESOURCES AND LIFE SCIENCES   (Austria)

^f UNIVERSITY OF SOFIA   (Bulgaria)

^g Sofia University Faculty of Mathematics and Informatics ^*  (Italy)

^h UPPSALA UNIVERSITY   (Sweden)

ⁱ AGROBIOINSTITUTE   (Bulgaria)

Author keywords

Automation; Big data; Data intensive; High performance computing; Reproducibility; Workflow

Indexed keywords

BIOLOGY; HIGH THROUGHPUT SEQUENCING; INFORMATION PROCESSING; PROCEDURES; REPRODUCIBILITY; WORKFLOW;

AUTOMATIC DATA PROCESSING; COMPUTATIONAL BIOLOGY; HIGH-THROUGHPUT NUCLEOTIDE SEQUENCING; REPRODUCIBILITY OF RESULTS; WORKFLOW;

EID: 84939529358     PISSN: None     EISSN: 17456150     Source Type: Journal    
DOI: 10.1186/s13062-015-0071-8     Document Type: Review

References (51)

1. Marx V. Biology: The big challenges of big data. Nature. 2013; 498(7453):255-60. doi: 10.1038/498255a .
2. Bux M, Leser U. Parallelization in Scientific Workflow Management Systems. ArXiv e-prints. 2013. 1303.7195.
3. Oinn T, Addis M, Ferris J, Marvin D, Senger M, Greenwood M, et al.Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics. 2004; 20(17):3045-54. doi: 10.1093/bioinformatics/bth361 .
4. Blankenberg D, Von Kuster G, Coraor N, Ananda G, Lazarus R, Mangan M, et al.Galaxy: a web-based genome analysis tool for experimentalists. Curr Protoc Mol Biol. 2010; Chapter 19:19.10.1-21. doi: 10.1002/0471142727.mb1910s89 .
5. Giardine B, Riemer C, Hardison RC, Burhans R, Elnitski L, Shah P, et al.Galaxy: a platform for interactive large-scale genome analysis. Genome Res. 2005; 15(10):1451-5. doi: 10.1101/gr.4086505 .
6. Goecks J, Nekrutenko A, Taylor J, Galaxy Team. Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol. 2010; 11(8):86. doi: 10.1186/gb-2010-11-8-r86 .
7. Altintas I, Berkley C, Jaeger E, Jones M, Ludascher B, Mock S. Kepler: an extensible system for design and execution of scientific workflows. In: Scientific and Statistical Database Management, 2004. Proceedings. 16th International Conference On: 2004. p. 423-4. doi: 10.1109/SSDM.2004.1311241 .
8. Kallio MA, Tuimala JT, Hupponen T, Klemelä P, Gentile M, Scheinin I, et al. Chipster: user-friendly analysis software for microarray and other high-throughput data. BMC Genomics. 2011; 12:507. doi: 10.1186/1471-2164-12-507 .
9. Sadedin SP, Pope B, Oshlack A. Bpipe: a tool for running and managing bioinformatics pipelines. Bioinformatics. 2012; 28(11):1525-6. doi: 10.1093/bioinformatics/bts167 .
10. Köster J, Rahmann S. Snakemake-a scalable bioinformatics workflow engine. Bioinformatics. 2012; 28(19):2520-2. doi: 10.1093/bioinformatics/bts480 .
11. Feldman SI. Make - a program for maintaining computer programs a program for maintaining computer programs. Softw Pract Experience. 1979; 9(4):255-65. doi: 10.1002/spe.4380090402 .
12. Schwab M, Schroeder J. Reproducible research documents using gnumake. In: Stanford Exploration Project: 1995. p. 217-26.
13. Schatz M, Langmead B, Salzberg S. Cloud computing and the DNA data race. Nat Biotechnol. 2010; 28:691-3. doi: 10.1038/nbt0710-691 .
14. Stein L. The case for cloud computing in genome informatics. Genome Biol. 2010; 11:207. doi: 10.1186/gb-2010-11-5-207 .
15. Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters. In: Sixth Symposium on Operating System Design and Implementation: 2004; San Francisco, CA. 2004.
16. White T. Hadoop: The Definitive Guide, 1st edn. Sebastopol, CA: O'Reilly; 2009. http://oreilly.com/catalog/9780596521981 .
17. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I. Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing: 2010. p. 10-10.
18. Lampa S, Dahlö M, Olason PI, Hagberg J, Spjuth O. Lessons learned from implementing a national infrastructure in sweden for storage and analysis of next-generation sequencing data. Gigascience. 2013; 2(1):9. doi: 10.1186/2047-217X-2-9 .
19. Rodríguez D, Bello X, Gutiérrez-de-Terán H. Molecular modelling of g protein-coupled receptors through the web. Mol Inf. 2012; 31(5):334-41. doi: 10.1002/minf.201100162 .
20. Schönherr S, Forer L, Weißensteiner H, Kronenberg F, Specht G, Kloss-Brandstätter A. Cloudgene: a graphical execution platform for mapreduce programs on private and public clouds. BMC Bioinformatics. 2012; 13:200. doi: 10.1186/1471-2105-13-200 .
21. Siretskiy A, Sundqvist T, Voznesenskiy M, Spjuth O. A quantitative assessment of the hadoop framework for analyzing massively parallel dna sequencing data. Gigascience. 2015; 4:26. doi: 10.1186/s13742-015-0058-5 .
22. Siretskiy A, Spjuth O. Htseq-hadoop: Extending htseq for massively parallel sequencing data analysis using hadoop. In: eScience (eScience), 2014 IEEE 10th International Conference On: 2014.
23. Anders S, Pyl PT, Huber W. Htseq-a python framework to work with high-throughput sequencing data. Bioinformatics. 2015; 31(2):166-169. doi: 10.1093/bioinformatics/btu638 .
24. SEQC/MAQC-III Consortium. A comprehensive assessment of rna-seq accuracy, reproducibility and information content by the sequencing quality control consortium. Nat Biotechnol. 2014; 32(9):903-14. doi: 10.1038/nbt.2957 .
25. Li S, Łabaj PP, Zumbo P, Sykacek P, Shi W, Shi L, et al.Detecting and correcting systematic variation in large-scale rna sequencing data. Nat Biotechnol. 2014; 32(9):888-95. doi: 10.1038/nbt.3000 .
26. Mueckstein U, Leparc GG, Posekany A, Hofacker I, Kreil DP. Hybridization thermodynamics of nimblegen microarrays. BMC Bioinformatics. 2010; 11:35. doi: 10.1186/1471-2105-11-35 .
27. Leparc GG, Tüchler T, Striedner G, Bayer K, Sykacek P, Hofacker IL, et al.Model-based probe set optimization for high-performance microarrays. Nucleic Acids Res. 2009; 37(3):18. doi: 10.1093/nar/gkn1001 .
28. Goodstadt L. Ruffus: a lightweight python library for computational pipelines. Bioinformatics. 2010; 26(21):2778-9. doi: 10.1093/bioinformatics/btq524 .
29. Cancer Genome Atlas Research Network, Weinstein JN, Collisson EA, Mills GB, Shaw KRM, Ozenberger BA, et al. The cancer genome atlas pan-cancer analysis project. Nat Genet. 2013; 45(10):1113-20. doi: 10.1038/ng.2764 .
30. Niemenmaa M, Kallio A, Schumacher A, Klemelä P, Korpelainen E, Heljanko K. Hadoop-bam: directly manipulating next generation sequencing data in the cloud. Bioinformatics. 2012; 28(6):876-7. doi: 10.1093/bioinformatics/bts054 . http://bioinformatics.oxfordjournals.org/content/28/6/876.full.pdf+html .
31. Schumacher A, Pireddu L, Niemenmaa M, Kallio A, Korpelainen E, Zanetti G, et al.Seqpig: simple and scalable scripting for large sequencing data sets in hadoop. Bioinformatics. 2014; 30(1):119-20. doi: 10.1093/bioinformatics/btt601 . http://bioinformatics.oxfordjournals.org/content/30/1/119.full.pdf+html .
32. Merali Z. Computational science:..error. Nature. 2010; 467(7317):775-7. doi: 10.1038/467775a .
33. Orrù V, Steri M, Sole G, Sidore C, Virdis F, Dei M, et al. Genetic variants regulating immune cell levels in health and disease.Cell. 2013; 155(1):242-56. doi: 10.1016/j.cell.2013.08.041 .
34. Francalacci P, Morelli L, Angius A, Berutti R, Reinier F, Atzeni R, et al. Low-pass DNA sequencing of 1200 Sardinians reconstructs European Y-chromosome phylogeny.Science (New York, N.Y.) 2013; 341(6145):565-9. doi: 10.1126/science.1237947 .
35. Cuccuru G, Leo S, Lianas L, Muggiri M, Pinna A, Pireddu L, et al.An automated infrastructure to support high-throughput bioinformatics. In: High Performance Computing Simulation (HPCS), 2014 International Conference On: 2014. p. 600-7. doi: 10.1109/HPCSim.2014.6903742 .
36. Pireddu L, Leo S, Zanetti G. Seal: a distributed short read mapping and duplicate removal tool. Bioinformatics. 2011. doi: 10.1093/bioinformatics/btr325 . http://bioinformatics.oxfordjournals.org/content/early/2011/06/22/bioinformatics.btr325.full.pdfhtml .
37. Pireddu L, Leo S, Soranzo N, Zanetti G. A hadoop-galaxy adapter for user-friendly and scalable data-intensive bioinformatics in galaxy. In: Proceedings of the 5 th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics. BCB '14. New York, NY, USA: ACM: 2014. p. 184-91. doi: 10.1145/2649387.2649429 .
38. Kloss-Brandstätter A, Pacher D, Schönherr S, Weissensteiner H, Binna R, Specht G, et al.Haplogrep: a fast and reliable algorithm for automatic classification of mitochondrial dna haplogroups. Hum Mutat. 2011; 32(1):25-32. doi: 10.1002/humu.21382 .
39. Afgan E, Baker D, Coraor N, Chapman B, Nekrutenko A, Taylor J. Galaxy cloudman: delivering cloud compute clusters. BMC Bioinformatics. 2010; 11(Suppl 12):4. doi: 10.1186/1471-2105-11-S12-S4 .
40. Afgan E, Chapman B, Jadan M, Franke V, Taylor J. Using cloud computing infrastructure with cloudbiolinux, cloudman, and galaxy. Curr Protoc Bioinformatics. 2012; Chapter 11:11-9. doi: 10.1002/0471250953.bi1109s38 .
41. Forer L, Lipic T, Schonherr S, Weisensteiner H, Davidovic D, Kronenberg F, et al.Delivering bioinformatics mapreduce applications in the cloud. In: Information and Communication Technology, Electronics and Microelectronics (MIPRO), 2014 37th International Convention On: 2014. p. 373-7. doi: 10.1109/MIPRO.2014.6859593 .
42. Krachunov M. Hierarchy and expressions for automated workflows for ngs data processing. In: Proceedings of the 8th International Conference on Information Systems & Grid Technologies (ISGT). Sofia, Bulgaria: 2015. p. 38-48.
43. Schaaff A, Verdes-Montenegro L, Ruiz J, Vela JS. Scientific workflows in astronomy. In: Proceeding of Astronomical Data Analysis Software and Systems: 2012.
44. Lih A, Zadok E. Pgmake: A portable distributed make system. 1994. Technical report.
45. Taura K, Matsuzaki T, Miwa M, Kamoshida Y, Yokoyama D, Dun N, et al.Design and implementation of gxp make - a workflow system based on make. Future Gener Comput Syst. 2013; 29(2):662-72. doi: 10.1016/j.future.2011.05.026 .
46. Albrecht M, Donnelly P, Bui P, Thain D. Makeflow: A portable abstraction for data intensive computing on clusters, clouds, and grids. In: Proceedings of the 1st ACM SIGMOD Workshop on Scalable Workflow Execution Engines and Technologies. SWEET '12. New York, NY, USA: ACM: 2012. p. 1-1113. doi: 10.1145/2443416.2443417 .
47. Seibel P, Kruger J, Hartmeier S, Schwarzer K, Lowenthal K, Mersch H, et al.XML schemas for common bioinformatic data types and their application in workflow systems. BMC Bioinformatics. 2006; 7(1):490.
48. Kalas M, Puntervoll P, Joseph A, Bartaseviciute E, Töpfer A, Venkataraman P, et al.Bioxsd: the common data-exchange format for everyday bioinformatics web services. Bioinformatics. 2010; 26(18):540-6.
49. Wilkinson M. Interoperability With Moby 1.0 - It's Better Than Sharing Your Toothbrush!. 2008. Available from Nature Precedings.
50. Linke B, Giegerich R, Goesmann A. Conveyor: a workflow engine for bioinformatic analyses. Bioinformatics. 2011; 27(7):903-11.
51. Wassink I, van der Vet PE, Wolstencroft K, Neerincx PBT, Roos M, Rauwerda H, et al.Analysing scientific workflows: Why workflows not only connect web services. In: Services - I, 2009 World Conference On: 2009. p. 314-21.