'Big data', Hadoop and cloud computing in genomics

Journal of Biomedical Informatics

Volumn 46, Issue 5, 2013, Pages 774-781

  O'Driscoll, Aisling ^a   Daugelaite, Jurate ^a   Sleator, Roy D ^a  

^a Munster Technological University   (Ireland)

Author keywords

Big data; Bioinformatics; Cloud computing; Genomics; Hadoop

Indexed keywords

BIG DATUM; DATA PROCESSING AND ANALYSIS; GENOMIC DATA SETS; GENOMIC-SEQUENCE DATA; GENOMICS; HADOOP; HUMAN GENOME PROJECT;

CLOUD COMPUTING; COMPUTER SOFTWARE; GENES;

BIOINFORMATICS;

BIOINFORMATICS; COMPUTER; COMPUTER PROGRAM; DATA PROCESSING; GENE TECHNOLOGY; GENOMICS; HUMAN GENOME PROJECT; INFORMATION PROCESSING; PRIORITY JOURNAL; REVIEW; RNA SEQUENCE;

BIG DATA; BIOINFORMATICS; CLOUD COMPUTING; GENOMICS; HADOOP;

HUMAN GENOME PROJECT; HUMANS; INTERNET; SOFTWARE;

EID: 84883788980     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2013.07.001     Document Type: Review

References (93)

1. Quail M.A., Smith M., Coupland P., Otto T.D., Harris S.R., Connor T.R., et al. A tale of three next generation sequencing platforms: comparison of Ion Torrent, Pacific Biosciences and Illumina MiSeq sequencers. BMC Genomics 2012, 13:341.
2. Pollack A. DNA sequencing: caught in the deluge of data 2011, New York Times, NY.
3. Moore G. Cramming more components into integrated circuits. Electronics 1965, 38:4-7.
4. Walter C. Kryder's Law Sci Am 2005, 293(August):32-33.
5. As We May Communicate. <>. http://www.tmcnet.com/articles/comsol/0100/0100pubout.htm.
6. Loman N.J., Constantinidou C., Chan J.Z., Halachev M., Sergeant M., Penn C.W., et al. High-throughput bacterial genome sequencing: an embarrassment of choice, a world of opportunity. Nat Rev Microbiol 2012, 10:599-606.
7. Davies K. The $1,000 genome: the revolution in DNA sequencing and the new era of personalized medicine [hardcover] 2010, Free Press, New York.
8. Mathe C., Sagot M.F., Schiex T., Rouze P. Current methods of gene prediction, their strengths and weaknesses. Nucl Acids Res 2002, 30:4103-4117.
9. Stein L. The case for cloud computing in genome informatics. Rev J: Genome Biol 2010, 11(May):207.
10. Mason C.E., Elemento O. Faster sequencers, larger datasets, new challenges. Genome Biol 2012, 13:314.
11. Managing and Analysing 1,000,000 Genomes. <>. http://rgrossman.com/2012/09/18/million-genomes-challeng/.
12. Genomics Takes Flight. To the Cloud. <>. https://idc-insights-community.com/health/life-sciences/genomics-takes-flight-to-the-cloud.
13. Gantz J, Reinsel, D. The Digital Universe in 2020: big data, bigger digital shadows, and biggest growth in the far east. In: IDC iView: IDC Analyze the, Future; 2012.
14. Social Media And The Big Data Explosion. <>. http://www.forbes.com/sites/onmarketing/2012/06/28/social-media-and-the-big-data-explosion/.
15. Big Data Offers Big Opportunities for Retail, Financial, Web Companies. <>. http://www.eweek.com/enterprise-apps/big-data-offers-big-opportunities-for-retail-financial-web-companies/.
16. Data Deluge and the Human Microbiome Project. <>. http://www.issues.org/28.4/sagoff.html.
17. Chae H., Jung I., Lee H., Marru S., Lee S.-W., Kim S. Bio and health informatics meets cloud: BioVLab as an example. Health Inform Sci Syst 2013, 1:6.
18. Davenport T.H.P. D. J. Data scientist: the sexiest job of the 21st century. Harward Business 2012, 90(70-6):128.
19. EMC Sitting In Sweet Spot Of $70 Billion Big Data Industry. <>. http://www.forbes.com/sites/greatspeculations/2011/11/18/emc-sitting-in-sweet-spot-of-70-billion-big-data-industry/.
20. Yeh R.F., Lim L.P., Burge C.B. Computational inference of homologous gene structures in the human genome. Genome Res 2001, 11:803-816.
21. Obama Administration Unveils "Big Data" Initiative: Announces $200 Million In New R&D Investments. <>. http://www.whitehouse.gov/sites/default/files/microsites/ostp/big_data_press_release_final_2.pdf.
22. Manyika J., Chui M., Brown B., Bughin J., Dobbs R., Roxburgh C., et al. Big data: the next frontier for innovation, competition, and productivity 2011, McKinsey Global Institute.
23. The Benefits Of Data Center Virtualization For Businesses. <>. http://www.cloudtweaks.com/2012/03/the-benefits-of-data-center-virtualization-for-businesses/.
24. Big Data, Meet the Huge Data That Will Shape Your Future. <>. http://www.information-management.com/news/big-data-meet-the-huge-data-that-will-shape-your-future-10023324-1.html.
25. Bridging the gap between HPC and IaaS clouds. <>. http://datasys.cs.iit.edu/seminar/2012-03-06_bogdan-nicolae.html.
26. Dai L., Gao X., Guo Y., Xiao J., Zhang Z. Bioinformatics clouds for big data manipulation. Biology Direct 2012, 7:43. [discussion 43].
27. What will happen to Amazon's massive cloud business? <>. http://tech.fortune.cnn.com/2012/05/22/aws/.
28. Fusaro V., Patil P., Gafni E., Wall D., Tonellato P. Biomedical cloud computing with amazon web services. PLOS J 2011, [August].
29. Shachak A., Shuval K., Fine S. Barriers and enablers to the acceptance of bioinformatics tools: a qualitative study. J Med Libr Assoc 2007, 95:454-458.
30. Krampis K, Booth T, Chapman B, Tiwari B, Bicak M, Field D, et al. Cloud BioLinux: pre-configured and on-demand bioinformatics computing for the genomics community. BMC Bioinform; 2012;13:42. .
31. Giardine B., Riemer C., Hardison R.C., Burhans R., Elnitski L., Shah P., et al. Galaxy: a platform for interactive large-scale genome analysis. Genome Res 2005, 15:1451-1455.
32. Angiuoli S.V., Matalka M., Gussman A., Galens K., Vangala M., Riley D.R., et al. CloVR: a virtual machine for automated and portable sequence analysis from the desktop using cloud computing. BMC Bioinform 2011, 12.
33. Oinn T., Addis M., Ferris J., Marvin D., Senger M., Greenwood M., et al. Taverna: a tool for the composition and enactment of bioinformatics workflows. Bioinformatics 2004, 20:3045-3054.
34. Hong D., Rhie A., Park S.S., Lee J., Ju Y.S., Kim S., et al. FX: an RNA-Seq analysis tool on the cloud. Bioinformatics 2012, 28:721-723.
35. O'Connor B.D., Merriman B., Nelson S.F. SeqWare Query Engine: storing and searching sequence data in the cloud. BMC Bioinform 2010, 11.
36. Available at https://dnanexus.com/.
37. How Hadoop Makes Short Work of Big Data. <>. http://www.forbes.com/sites/netapp/2012/09/24/hadoop-big-data/.
38. Taylor R.C. An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics. BMC Bioinform 2010, 11(Suppl. 12):S1.
39. Hadoop Sorts a Petabyte in 16.25 Hours and a Terabyte in 62 Seconds. <>. http://developer.yahoo.com/blogs/hadoop/posts/2009/05/hadoop_sorts_a_petabyte_in_162/.
40. Dai L., Gao X., Guo Y., Xiao J.F., Zhang Z. Bioinformatics clouds for big data manipulation. Biol Direct 2012, 7.
41. Cloudera and Mount Sinai: The structure of a Big Data Revolution? <>. http://www.zdnet.com/cloudera-and-mount-sinai-the-structure-of-a-big-data-revolution-7000000354/.
42. Zou Q., Li X.B., Jiang W.R., Lin Z.Y., Li G.L., Chen K. Survey of MapReduce frame operation in bioinformatics. Brief Bioinform 2013.
43. McKenna A., et al. The genome analysis toolkit: a MapReduce framework for analysing next-generation DNA sequencing data. Genome Res 2010, 20(July):1297-1303.
44. Gurtowski J., Schatz M.C., Langmead B. Genotyping in the cloud with Crossbow. Current Protocol Bioinform 2012, [chapter 15, Unit 153].
45. Schatz M.C., Langmead B., Salzberg S.L. Cloud computing and the DNA data race. Nat Biotechnol 2010, 28:691-693.
46. Langmead B., Schatz M.C., Lin J., Pop M., Salzberg S.L. Searching for SNPs with cloud computing. Genome Biol 2009, 10:R134.
47. Nguyen T., Shi W., Ruden D. CloudAligner: a fast and full-featured MapReduce based tool for sequence mapping. BMC Res Notes 2011, 4:171.
48. Langmead B., Hansen K.D., Leek J.T. Cloud-scale RNA-sequencing differential expression analysis with Myrna. Genome Biol 2010, 11.
49. Helping accelerate treatment for pediatric cancer with Dell cloud technology. <>. http://content.dell.com/us/en/corp/d/corp-comm/pediatric-cancer.
50. NextBio, Intel to collaborate on improving Hadoop Stack for Genomic Data Analysis. <>. http://www.genomeweb.com/informatics/nextbio-intel-collaborate-improving-hadoop-stack-genomic-data-analysis.
51. Cloudera Chief Scientist Jeff Hammerbacher Teams with Mount Sinai School of Medicine to Solve Medical Challenges Using Big Data. <>. http://www.marketwire.com/press-release/Cloudera-Chief-Scientist-Jeff-Hammerbacher-Teams-With-Mount-Sinai-School-Medicine-1676135.htm.
52. Schadt E.E., Linderman M.D., Sorenson J., Lee L., Nolan G.P. Computational solutions to large-scale data management and analysis. Nat Rev Genet 2010, 11:647-657.
53. Healthcare Cloud Computing (Clinical, EMR, SaaS, Private, Public, Hybrid) Market - Global Trends, Challenges, Opportunities & Forecasts (2012-2017). <. http://www.reportlinker.com/p0924631-summary/Healthcare-Cloud-Computing-Clinical-EMR-SaaS-Private-Public-Hybrid-Market-Global-Trends-Challenges-Opportunities-Forecasts-.html.
54. Schoenherr S., Forer L., Weissensteiner H., Specht G., Kronenberg F., Kloss-Brandstaetter A. Cloudgene: a graphical execution platform for MapReduce programs on private and public clouds. BMC Bioinform 2012, 13(1):200.
55. Pennisi E. Will computers crash genomics?. Science 2011, 331:666-668.
56. 1,000 Genomes in the Cloud and NCBI Experiences. <>. https://respond.niaid.nih.gov/conferences/bioinformatics2012/Festival%20Proceedings/Preuss_1000_Genomes.pdf.
57. Available at http://asperasoft.com/.
58. How "Cloud" Services Democratize DNA Sequencing. <>. http://techonomy.com/2012/08/how-cloud-services-democratize-dna-sequencing/.
59. Schadt E.E. The changing privacy landscape in the era of big data. Mol Syst Biol 2012, 8:612.
60. Creating HIPAA-Compliant Medical Data Applications With AWS. <>. http://aws.amazon.com/about-aws/whats-new/2009/04/06/whitepaper-hipaa/.
61. Managing data in the Cloud Age. <>. http://www.dddmag.com/articles/2012/10/managing-data-cloud-age.
62. Robertson J.A. The $1000 genome: ethical and legal issues in whole genome sequencing of individuals. Am J Bioeth 2003, 3. W-IF1.
63. Klein C.A. Cloudy confidentiality: clinical and legal implications of cloud computing in health care. J Am Acad Psychiatry Law 2011, 39:571-578.
64. Sleator R.D., Shortall C., Hill C. Metagenomics. Lett Appl Microbiol 2008, 47:361-366.
65. Sleator R.D. An overview of the processes shaping protein evolution. Sci Prog 2010, 93:1-6.
66. Sleator R.D. Prediction of protein functions. Methods Mol Biol 2012, 815:15-24.
67. Sleator R.D. Proteins: form and function. Bioeng Bugs 2012, 3:80-85.
68. Marianayagam N.J., Fawzi N.L., Head-Gordon T. Protein folding by distributed computing and the denatured state ensemble. Proc Natl Acad Sci USA 2005, 102:16684-16689.
69. Cooper S., Khatib F., Treuille A., Barbero J., Lee J., Beenen M., et al. Predicting protein structures with a multiplayer online game. Nature 2010, 466:756-760.
70. Murray J.F. Personalized medicine: been there, done that, always needs work!. Am J Respir Crit Care Med 2012, 185:1251-1252.
71. Furusawa C., Kaneko K. A dynamical-systems view of stem cell biology. Science 2012, 338:215-217.
72. Sleator R.D. The human superorganism - of microbes and men. Med Hypotheses 2010, 74:214-215.
73. O'Driscoll A., Sleator R.D. Synthetic DNA: the next generation of big data storage. Bioengineered 2013, 4.
74. Karr J.R., Sanghvi J.C., Macklin D.N., Gutschow M.V., Jacobs J.M., Bolival B., et al. A whole-cell computational model predicts phenotype from genotype. Cell 2012, 150:389-401.
75. Sleator R.D. Digital biology: a new era has begun. Bioengineered 2012, 3:311-312.
76. Schatz M.C. CloudBurst: highly sensitive read mapping with MapReduce. Bioinformatics 2009, 25:1363-1369.
77. Pireddu L., Leo S., Zanetti G. SEAL: a distributed short read mapping and duplicate removal tool. Bioinformatics 2011, 27:2159-2160.
78. Blastreduce: high performance short read mapping with mapreduce. <>. http://www.cbcb.umd.edu/software/blastreduce/.
79. Schatz M., Sommer D., Kelley D., Pop M. De Novo assembly of large genomes with cloud computing. Biology of genomes 2010, vol. 10. Cold Spring Harbor, NY.
80. Chang Y.J., Chen C.C., Chen C.L., Ho J.M. A de novo next generation genomic sequence assembler based on string graph and MapReduce cloud computing framework. BMC Genomics 2012, 13(Suppl. 7):S28.
81. Jourdren L., Bernard M., Dillies M.A., Le Crom S. Eoulsan: a cloud computing-based framework facilitating high throughput sequencing analyses. Bioinformatics 2012, 28:1542-1543.
82. Niemenmaa M., Kallio A., Schumacher A., Klemela P., Korpelainen E., Heljanko K. Hadoop-BAM: directly manipulating next generation sequencing data in the cloud. Bioinformatics 2012, 28:876-877.
83. Matthews S.J., Williams T.L. MrsRF: an efficient MapReduce algorithm for analyzing large collections of evolutionary trees. BMC Bioinform 2010, 11(Suppl. 1):S15.
84. Colosimo M.E., Peterson M.W., Mardis S., Hirschman L. Nephele: genotyping via complete composition vectors and MapReduce. Source Code Biol Med 2011, 6:13.
85. Vouzis P.D., Sahinidis N.V. GPU-BLAST: using graphics processors to accelerate protein sequence alignment. Bioinformatics 2011, 27:182-188.
86. Liu C.M., Wong T., Wu E., Luo R.B., Yiu S.M., Li Y.R., et al. SOAP3: ultra-fast GPU-based parallel alignment tool for short reads. Bioinformatics 2012, 28:878-879.
87. Lewis S., Csordas A., Killcoyne S., Hermjakob H., Hoopmann M.R., Moritz R.L., et al. Hydra: a scalable proteomic search engine which utilizes the Hadoop distributed computing framework. BMC Bioinform 2012, 13.
88. Matsunaga A, Tsugawa M, and Fortes J. CloudBLAST: Combining MapReduce and Virtualization on Distributed Resources for Bioinformatics Applications. IEEE Fourth International Conference on eScience, Indiana, USA, 2008 222-229.
89. Leo S, Santoni F, Zanetti G. Biodoop: bioinformatics on hadoop. In: Parallel processing workshops, 2009. ICPPW '09. International Conference on; 2009. p. 415-22.
90. Huang H.L., Tata S., Prill R.J. BlueSNP: R package for highly scalable genome-wide association studies using Hadoop clusters. Bioinformatics 2013, 29:135-136.
91. Kelley D.R., Schatz M.C., Salzberg S.L. Quake: quality-aware detection and correction of sequencing errors. Genome Biol 2010, 11:R116.
92. Zhang L., Gu S., Liu Y., Wang B., Azuaje F. Gene set analysis in the cloud. Bioinformatics 2012, 28:294-295.
93. Feng X., Grossman R., Stein L. PeakRanger: a cloud-enabled peak caller for ChIP-seq data. BMC Bioinform 2011, 12:139.