A quantitative assessment of the Hadoop framework for analyzing massively parallel DNA sequencing data

GigaScience

Volumn 4, Issue 1, 2015, Pages

  Siretskiy, Alexey ^a   Sundqvist, Tore ^a   Voznesenskiy, Mikhail ^b   Spjuth, Ola ^a  

^a UPPSALA UNIVERSITY   (Sweden)

^b ST PETERSBURG STATE UNIVERSITY   (Russian Federation)

Author keywords

Bioinformatics; DNA seq; Hadoop; High performance computing; Massively parallel sequencing; Next generation sequencing

Indexed keywords

ARTICLE; COMPUTER INTERFACE; CONTROLLED STUDY; DATA ANALYSIS; DATA ANALYSIS SOFTWARE; DNA SEQUENCE; INFORMATION; INFORMATION SYSTEM; MEASUREMENT; PRIORITY JOURNAL; QUANTITATIVE ANALYSIS; BIOLOGY; COMPUTER PROGRAM; INTERNET; PROCEDURES;

COMPUTATIONAL BIOLOGY; INTERNET; SEQUENCE ANALYSIS, DNA; SOFTWARE;

EID: 84979619031     PISSN: None     EISSN: 2047217X     Source Type: Journal    
DOI: 10.1186/s13742-015-0058-5     Document Type: Article

References (51)

1. Metzker ML. Sequencing technologies-the next generation. Nat Rev Genet. 2010;11(1):31-46.
2. Marx V. Biology: The big challenges of big data. Nature. 2013;498(7453): 255-60.
3. Hiseq Comparison. Available from: http://www.illumina.com/systems/ sequencing.ilmn.
4. Li H, Durbin R. Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics. 2009;25(14):1754-60.
5. Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol. 2009;10(3):R25.
6. Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, et al. The sequence alignment/map format and SAMtools. Bioinformatics. 2009;25(16):2078-79.
7. The OpenMP® API specification for parallel programming. Available from:http://openmp.org/.
8. Top 500 Supercomputer Sites. Available from: http://www.top500.org/ statistics/list/.
9. Tange O. GNU Parallel-The Command-Line Power Tool. The USENIX Magazine. 2011;36(1):42-7. Available from: http://www.gnu.org/s/parallel.
10. The Message Passing Interface (MPI) standard. Available from: http://www.mcs.anl.gov/research/projects/mpi/.
11. The Extended Randomized Numerical alignEr. Available from: http://erne.sourceforge.net.
12. pMap: Parallel Sequence Mapping Tool. Available from: http://bmi.osu.edu/hpc/software/pmap/pmap.html.
13. Giardine B, Riemer C, Hardison RC, Burhans R, Elnitski L, Shah P, et al. Galaxy: a platform for interactive large-scale genome analysis. Genome Res. 2005;15(10):1451-5.
14. McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, et al. The genome analysis toolkit: a MapReduce framework for analyzing next-generation DNA sequencing data. Genome Res. 2010;20(9): 1297-303.
15. Dean J, Ghemawat S. MapReduce: simplified data processing on large clusters. In: OSDI '04: 6th Symposium on Operating Systems Design and Implementation. 2004. https://www.usenix.org/legacy/event/osdi04/ tech/full_papers/dean/dean.pdf.
16. Hadoop Wiki-Powered By. Available from: https://wiki.apache.org/ hadoop/PoweredBy.
17. Lin J, Dyer C. Data-Intensive Text Processing with MapReduce. Morgan and Claypool Publishers: College Park; 2010.
18. How Facebook keeps 100 petabytes of Hadoop data online. Available from: https://gigaom.com/2012/06/13/how-facebook-keeps-100-petabytes-of-hadoop-data-online/.
19. White T. Hadoop: The Definitive Guide. 1st ed. Sebastopol: O'Reilly; 2009. Available from: http://oreilly.com/catalog/9780596521981.
20. Sammer E. Hadoop Operations. 1st ed. Sebastopol. O'Reilly Media: Inc.; 2012.
21. Schatz MC. CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics. 2009;25(11):1363-9.
22. Langmead B, Schatz MC, Lin J, Pop M, Salzberg SL. Searching for SNPs with cloud computing. Genome Biol. 2009;10(11):R134.
23. Langmead B, Hansen KD, Leek JT, et al. Cloud-scale RNA-sequencing differential expression analysis with Myrna. Genome Biol. 2010;11(8):R83.
24. Schatz M, Sommer D, Kelley D, Pop M. Contrail: Assembly of large genomes using cloud computing. In: CSHL Biology of Genomes Conference; 2010.
25. Taylor R. An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics. BMC Bioinformatics. 2010;11(Suppl 12):S1. Available from: http://www.biomedcentral.com/1471-2105/11/S12/S1.
26. The Arabidopsis Information Resource (TAIR). Available from: www.arabidopsis.org.
27. Gilchrist J, Nikolov Y. Parallel BZIP2 (pbzip2). http://compression.ca/pbzip2/.
28. UPPMAX. Available from: http://uppmax.uu.se.
29. Open Nebula. Available from: http://opennebula.org.
30. Cloudera. http://www.cloudera.com/content/cloudera/en/whycloudera/hadoop-and-big-data.html.
31. Habib I. Virtualization with KVM. Linux J. 2008 Feb;2008(166). Available from: http://dl.acm.org/citation.cfm?id=1344209.1344217.
32. Li Y, Chen W, Liu EY, Zhou YH. Single Nucleotide Polymorphism (SNP) Detection and Genotype Calling from Massively Parallel Sequencing (MPS) Data. Stat Biosci. 2013;5(1):3-25.
33. Short Oligonucleotide Analysis Package. Available from: http://soap.genomics.org.cn/soapsnp.html.
34. Siretskiy A. HPC_bash_align. Available from: https://github.com/raalesir/HPC_bash_align.
35. Siretskiy A. mr_python. Available from: https://github.com/raalesir/mr_python.
36. The NCBI Sequence Read Archive. Available from: http://www.ncbi.nlm.nih.gov/Traces/sra.
37. Mark A. A parallel implementation of gzip for modern multi-processor, multi-core machines. Available from: http://zlib.net/pigz/.
38. 1001 Genomes Project database. Available from: http://1001genomes. org/data/software/shoremap/shoremap%5C_2.0%5C%5C/data/reads/Schneeberger.2009/Schneeberger.2009.single%5C_end.gz.
39. Pireddu L, Leo S, Zanetti G. SEAL: a distributed short read mapping and duplicate removal tool. Bioinformatics. 2011;27(15):2159-60.
40. Dean J, Ghemawat S. MapReduce: Simplified data processing on large clusters. In: Sixth Symposium on Operating System Design and Implementation: 2004; San Francisco, CA. 2004. https://www.usenix.org/ legacy/event/osdi04/tech/full_papers/dean/dean.pdf.
41. Yoo AB, Jette MA, Grondona M. SLURM: Simple linux utility for resource management. In: Job Scheduling Strategies for Parallel Processing. Berlin Heidelberg: Springer; 2003. p. 44-60.
42. Afgan E, Baker D, Coraor N, Chapman B, Nekrutenko A, Taylor J. Galaxy CloudMan: delivering cloud compute clusters. BMC Bioinformatics. 2010;11 Suppl 12:S4.
43. Schönherr S, Forer L, Weissensteiner H, Kronenberg F, Specht G, Kloss-Brandstatter A. Cloudgene: a graphical execution platform for MapReduce programs on private and public clouds. BMC Bioinformatics. 2012;13:200.
44. Gurtowski J, Schatz MC, Langmead B. Genotyping in the cloud with Crossbow. Curr Protoc Bioinformatics. 2012. Sep;Chapter 15:Unit15.3.
45. Fadika Z, Dede E, Govindaraju M, Ramakrishnan L. MARIANE: Using MapReduce in HPC environments. Future Generation Comput Syst. 2014;36(0):379-88. Special Section: Intelligent Big Data Processing Special Section: Behavior Data Security Issues in Network Information Propagation Special Section: Energy-efficiency in Large Distributed Computing Architectures Special Section: eScience Infrastructure and Applications. Available from: http://www.sciencedirect.com/science/ article/pii/S0167739X13002719.
46. Nordberg H, Bhatia K, Wang K, Wang Z. BioPig: a Hadoop-based analytic toolkit for large-scale sequence data. Bioinformatics. 2013;29(23):3014-9.
47. Schumacher A, Pireddu L, Niemenmaa M, Kallio A, Korpelainen E, Zanetti G, et al. SeqPig: simple and scalable scripting for large sequencing data sets in Hadoop. Bioinformatics. 2014;30(1):119-20.
48. Krishnan S, Tatineni M, Baru C. myHadoop-Hadoop-on-Demand on Traditional HPC Resources. San Diego Supercomputer Center Technical Report TR-2011-2, University of California, San Diego; 2011.
49. Zaharia M, Chowdhury M, Franklin MJ, Shenker S, Stoica I. Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX conference on Hot topics in cloud computing; 2010. p. 10. http://dl.acm. org/citation.cfm?id=1863103.1863113.
50. Wiewiórka MS, Messina A, Pacholewska A, Maffioletti S, Gawrysiak P, Okoniewski MJ. SparkSeq: fast, scalable, cloud-ready tool for the interactive genomic data analysis with nucleotide precision. Bioinformatics. 2014. p. btu343 http://dx.doi.org/10.1093/bioinformatics/btu343.
51. Massie M, Nothaft F, Hartl C, Kozanitis C, Schumacher A, Joseph AD, et al. ADAM: Genomics Formats and Processing Patterns for Cloud Scale Computing. Berkeley: EECS Department, University of California; 2013. UCB/EECS-2013-207. Available from: http://www.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-207.html.