Scalable and efficient whole-exome data processing using workflows on the cloud

Future Generation Computer Systems

Volumn 65, Issue , 2016, Pages 153-168

  Cala J ^a   Marei, E ^a   Xu, Y ^b   Takeda, K ^c   Missier, P ^a  

^a NEWCASTLE UNIVERSITY   (United Kingdom)

^b NEWCASTLE UNIVERSITY   (United Kingdom)

^c MICROSOFT RESEARCH   (United Kingdom)

Author keywords

Cloud computing; HPC; Performance analysis; Whole exome sequencing; Workflow based application

Indexed keywords

BIOINFORMATICS; CLOUD COMPUTING; COMPUTATION THEORY; COMPUTER ARCHITECTURE; DATA FLOW ANALYSIS; DATA HANDLING; INFORMATION MANAGEMENT; PIPELINE PROCESSING SYSTEMS; PIPELINES; SOFTWARE PROTOTYPING; WINDOWS OPERATING SYSTEM; WORK SIMPLIFICATION;

CLOUD COMPUTING ARCHITECTURES; CLOUD COMPUTING INFRASTRUCTURES; DATA PROCESSING PIPELINES; HIGH-LEVEL PROGRAMMING MODELS; PERFORMANCE ANALYSIS; WHOLE-EXOME SEQUENCING; WORKFLOW MANAGEMENT SYSTEMS; WORKFLOW-BASED APPLICATIONS;

BIG DATA;

EID: 84959191705     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2016.01.001     Document Type: Article

References (38)

1. [1] Stein, L.D., Knoppers, B.M., Campbell, P., Getz, G., Korbel, J.O., Data analysis: Create a cloud commons. Nature 523:7559 (2015), 149–151, 10.1038/523149a.
2. [2] K. Wetterstrand, DNA sequencing costs: data from the NHGRI Genome Sequencing Program (GSP), 2015.
3. [3] Stein, L.D., The case for cloud computing in genome informatics. Genome Biol., 11(5), 2010, 207, 10.1186/gb-2010-11-5-207.
4. [4] Xuan, J., Yu, Y., Qing, T., Guo, L., Shi, L., Next-generation sequencing in the clinic: promises and challenges. Cancer lett. 340:2 (2013), 284–295, 10.1016/j.canlet.2012.11.025.
5. [5] T. Camerlengo, H.G. Ozer, R. Onti-Srinivasan, P. Yan, T. Huang, J. Parvin, K. Huang, From sequencer to supercomputer: an automatic pipeline for managing and processing next generation sequencing data, in: AMIA Summits on Translational Science proceedings AMIA Summit on Translational Science, 2012, pp. 1–10.
6. [6] Fischer, M., Snajder, R., Pabinger, S., Dander, A., Schossig, A., Zschocke, J., Trajanoski, Z., Stocker, G., SIMPLEX: cloud-enabled pipeline for the comprehensive analysis of exome sequencing data. PLoS ONE, 7(8), 2012, e41948, 10.1371/journal.pone.0041948.
7. [7] Reid, J.G., Carroll, A., Veeraraghavan, N., Dahdouli, M., Sundquist, A., English, A., Bainbridge, M., White, S., Salerno, W., Buhay, C., Yu, F., Muzny, D., Daly, R., Duyk, G., Gibbs, R.A., Boerwinkle, E., Launching genomics into the cloud: deployment of mercury, a next generation sequence analysis pipeline. BMC Bioinformatics, 15(1), 2014, 30, 10.1186/1471-2105-15-30.
8. [8] Dander, A., Pabinger, S., Sperk, M., Fischer, M., Stocker, G., Trajanoski, Z., SeqBench: integrated solution for the management and analysis of exome sequencing data. BMC Res. Notes, 7(1), 2014, 43, 10.1186/1756-0500-7-43.
9. [9] Pabinger, S., Dander, A., Fischer, M., Snajder, R., Sperk, M., Efremova, M., Krabichler, B., Speicher, M.R., Zschocke, J., Trajanoski, Z., A survey of tools for variant analysis of next-generation genome sequencing data. Briefings Bioinform. 15:2 (2014), 256–278, 10.1093/bib/bbs086.
10. [10] Koboldt, D.C., Ding, L., Mardis, E.R., Wilson, R.K., Challenges of sequencing human genomes. Briefings Bioinform. 11:5 (2010), 484–498, 10.1093/bib/bbq016.
11. [11] Hiden, H., Woodman, S., Watson, P., Cala, J., Developing cloud applications using the e-Science central platform. Philosophical transactions. Series A, Math. Phys. Eng. Sci., 371, 1983, 10.1098/rsta.2012.0085.
12. [12] Bhuvaneshwar, K., Sulakhe, D., Gauba, R., Rodriguez, A., Madduri, R., Dave, U., Lacinski, L., Foster, I., Gusev, Y., Madhavan, S., A case study for cloud based high throughput analysis of NGS data using the globus genomics system. Comput. Struct. Biotechnol. J., 2014, 1–11, 10.1016/j.csbj.2014.11.001.
13. [13] Cała, J., Xu, Y., Wijaya, E.A., Missier, P., From scripted HPC-based NGS pipelines to workflows on the cloud. 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, 2014, IEEE, 694–700, 10.1109/CCGrid.2014.128.
14. [14] Blanquer, I.B., Brasche, G., Cala, J., Gagliardi, F., Gannon, D., Hiden, H., Soncu, H., Takeda, K., Tomás, A., Woodman, S., Supporting NGS pipelines in the cloud. EMBnet. J., 19(A), 2013, 14, 10.14806/ej.19.A.625.
15. [15] Deelman, E., Gannon, D., Shields, M., Taylor, I., Workflows and e-Science: An overview of workflow system features and capabilities. Future Gener. Comput. Syst. 25:5 (2009), 528–540, 10.1016/j.future.2008.06.012.
16. [16] Deelman, E., Vahi, K., Juve, G., Rynge, M., Callaghan, S., Maechling, P.J., Mayani, R., Chen, W., Ferreira da Silva, R., Livny, M., Wenger, K., Pegasus, a workflow management system for science automation. Future Gener. Comput. Syst. 46 (2014), 17–35, 10.1016/j.future.2014.10.008.
17. [17] K. Wolstencroft, R. Haines, D. Fellows, A. Williams, D. Withers, S. Owen, S. Soiland-Reyes, I. Dunlop, A. Nenadic, P. Fisher, J. Bhagat, K. Belhajjame, F. Bacall, A. Hardisty, A. Nieva de la Hidalga, M.P. Balcazar Vargas, S. Sufi, C. Goble, The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, web or in the cloud, Nucleic acids research 41 (Web Server issue). http://dx.doi.org/10.1093/nar/gkt328.
18. [18] Cała, J., Hiden, H., Woodman, S., Watson, P., Cloud computing for fast prediction of chemical activity. Future Gener. Comput. Syst. 29:7 (2013), 1860–1869, 10.1016/j.future.2013.01.011.
19. [19] Li, H., Durbin, R., Fast and accurate long-read alignment with Burrows-Wheeler transform. Bioinformatics (Oxford, England) 26:5 (2010), 589–595, 10.1093/bioinformatics/btp698.
20. [20] McKenna, A., Hanna, M., Banks, E., Sivachenko, A., Cibulskis, K., Kernytsky, A., Garimella, K., Altshuler, D., Gabriel, S., Daly, M., DePristo, M.A., The genome analysis toolkit: A MapReduce framework for analyzing next-generation DNA sequencing data. Genome Res. 20:9 (2010), 1297–1303, 10.1101/gr.107524.110.
21. [21] Wang, K., Li, M., Hakonarson, H., ANNOVAR: functional annotation of genetic variants from high-throughput sequencing data. Nucleic Acids Res., 38(16), 2010 e164–e164. http://dx.doi.org/10.1093/nar/gkq603.
22. [22] Hull, D., Stevens, R., Lord, P., Wroe, C., Goble, C., Treating shimantic web syndrome with ontologies. Domingue, J., Cabral, L., Motta, E., (eds.) AKT Workshop on Semantic Web Services, 2004, Milton Keynes, UK, 1–8.
23. [23] Sadooghi, I., Hernandez Martin, J., Li, T., Brandstatter, K., Zhao, Y., Maheshwari, K., Ruivo, T.~P.P.d.~L., Timm, S., Garzoglio, G., Raicu, I., Understanding the performance and potential of cloud computing for scientific applications. IEEE Trans. Cloud Comput., PP(99), 2015 1–1. http://dx.doi.org/10.1109/TCC.2015.2404821.
24. [24] Wozniak, J.M., Armstrong, T.G., Maheshwari, K., Lusk, E.L., Katz, D.S., Wilde, M., Foster, I.T., Turbine: A distributed-memory dataflow engine for high performance many-task applications. Fund. Inform., 2012, 1001–1032, 10.3233/FI-2012-0000.
25. [25] T.G. Armstrong, J.M. Wozniak, M. Wilde, K. Maheshwari, D.S. Katz, M. Ripeanu, E.L. Lusk, I.T. Foster, ExM: High level dataflow programming for extreme-scale systems, in: 4th USENIX Workshop on Hot Topics in Parallelism (HotPar), poster, Berkeley, CA, 2012.
26. [26] de~Oliveira, D., Ocaña, K.A., Ogasawara, E., Dias, J., Gonçalves, J., Baião, F., Mattoso, M., Performance evaluation of parallel strategies in public clouds: A study with phylogenomic workflows. Future Gener. Comput. Syst. 29:7 (2013), 1816–1825, 10.1016/j.future.2012.12.019.
27. [27] Kelly, B.J., Fitch, J.R., Hu, Y., Corsmeier, D.J., Zhong, H., Wetzel, A.N., Nordquist, R.D., Newsom, D.L., White, P., Churchill: an ultra-fast, deterministic, highly scalable and balanced parallelization strategy for the discovery of human genetic variation in clinical and population-scale genomics. Genome Biol., 16(1), 2015, 6, 10.1186/s13059-014-0577-x.
28. [28] Liu, B., Madduri, R.K., Sotomayor, B., Chard, K., Lacinski, L., Dave, U.J., Li, J., Liu, C., Foster, I.T., Cloud-based bioinformatics workflow platform for large-scale next-generation sequencing analyses. J. Biomed. Inf. 49 (2014), 119–133, 10.1016/j.jbi.2014.01.005.
29. [29] Goecks, J., Nekrutenko, A., Taylor, J., Team, T.G., Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biol., 11(8), 2010, R86, 10.1186/gb-2010-11-8-r86.
30. [30] Missier, P., Wijaya, E., Kirby, R., SVI: a simple single-nucleotide human variant interpretation tool for clinical use - springer. Ashish, N., Ambite, J.-L., (eds.) Data Integration in the Life Sciences, vol. 9162, 2015, Springer International Publishing, 180–194, 10.1007/978-3-319-21843-4_14.
31. [31] N.M. Mohamed, H. Lin, W. Feng, Accelerating data-intensive genome analysis in the cloud, in: Proceedings of the 5th International Conference on Bioinformatics and Computational Biology, Honolulu, 2013.
32. [32] Hsi-Yang Fritz, M., Leinonen, R., Cochrane, G., Birney, E., Efficient storage of high throughput DNA sequencing data using reference-based compression. Genome Res. 21:5 (2011), 734–740, 10.1101/gr.114819.110.
33. [33] Gao, X., Xu, J., Starmer, J., Fastq2vcf: a concise and transparent pipeline for whole-exome sequencing data analyses. BMC Res. Notes, 8(1), 2015, 72, 10.1186/s13104-015-1027-x.
34. [34] P. Carrier, B. Long, R. Walsh, J. Dawson, C.P. Sosa, B. Haas, T. Tickle, T. William, The Impact of High-Performance Computing Best Practice Applied to Next-Generation Sequencing Workflows, bioRxiv, 2015, 017665. http://dx.doi.org/10.1101/017665.
35. [35] Zhao, D., Yang, X., Sadooghi, I., Garzoglio, G., Timm, S., Raicu, I., High-Performance storage support for scientific applications on the cloud. Proceedings of the 6th Workshop on Scientific Cloud Computing - ScienceCloud’15, 2015, ACM Press, New York, New York, USA, 33–36, 10.1145/2755644.2755648.
36. [36] Afgan, E., Chapman, B., Taylor, J., CloudMan as a platform for tool, data, and analysis distribution. BMC Bioinformatics, 13(1), 2012, 315, 10.1186/1471-2105-13-315.
37. [37] Walker, E., Benchmarking amazon EC2 for high-performance scientific computing. login 33:5 (2008), 18–23.
38. [38] Ostermann, S., Iosup, A., Yigitbasi, N., Prodan, R., Fahringer, T., Epema, D., A performance analysis of EC2 cloud computing services for scientific computing. Avresky, Dimiter R., Diaz, M., Bode, A., Ciciani, B., Dekel, E., (eds.) Cloud Computing, 2010, Springer, Berlin, Heidelberg, 115–131 (Chapter 4) http://dx.doi.org/10.1007/978-3-642-12636-9_9.