A hybrid HPC/cloud distributed infrastructure: Coupling EC2 cloud resources with HPC clusters to run large tightly coupled multiscale applications

Future Generation Computer Systems

Volumn 42, Issue , 2015, Pages 11-21

  Ben Belgacem, Mohamed ^a   Chopard, Bastien ^a  

^a UNIVERSITY OF GENEVA   (Switzerland)

Author keywords

Cloud resources; Distributed computation; HPC infrastructure; MAPPER project; Multiscale applications; Performance measures

Indexed keywords

SOFTWARE ENGINEERING;

DISTRIBUTED COMPUTATIONS; HPC INFRASTRUCTURE; MAPPER PROJECT; MULTISCALE APPLICATIONS; PERFORMANCE MEASURE;

PROGRAM PROCESSORS;

EID: 84908431309     PISSN: 0167739X     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.future.2014.08.003     Document Type: Article

References (51)

1. N. Sultan, Discovering the potential of cloud computing in accelerating the search for curing serious illnesses, Int. J. Inf. Manage. 34 (2) (2014) 221-225. http://dx.doi.org/10.1016/j.ijinfomgt.2013.12.012. URL: http://www.sciencedirect.com/science/article/pii/S0268401213001692.
2. R.S. Barga, D. Gannon, D.A. Reed, The client and the cloud: Democratizing research computing, IEEE Internet Comput. 15 (1) (2011) 72-75. http://doi.ieeecomputersociety.org/10.1109/MIC.2011.20.
3. VENUS-C European project, http://www.venus-c.eu/.
4. IBM Cloud, http://www.ibm.com/cloud-computing/us/en/.
5. Google Apps Engine, https://developers.google.com/appengine/docs/whatisgoogleappengine.
6. O. Evangelinos, C.N. Hill, Cloud computing for parallel scientific HPC applications: Feasibility of running coupled atmosphere-ocean climate models on Amazon's EC2, in: in Cloud Computing and its Applications, CCA 2008, 2008. URL: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.296.3779&rep=rep1&type=pdf.
7. K. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H.J. Wasserman, N. Wright, Performance analysis of high performance computing applications on the Amazon web services cloud, in: Cloud Computing Technology and Science, CloudCom, 2010 IEEE Second International Conference on, 2010, pp. 159-168. URL: http://dx.doi.org/10.1109/CloudCom.2010.69.
8. A. Iosup, S. Ostermann, N. Yigitbasi, R. Prodan, T. Fahringer, D. Epema, Performance analysis of cloud computing services for many-tasks scientific computing, IEEE Trans. Parallel Distrib. Syst. 22 (6) (2011) 931-945. URL: http://dx.doi.org/10.1109/TPDS.2011.66.
9. P. Zaspel, M. Griebel, Massively parallel fluid simulations on Amazon's HPC cloud, in: Network Cloud Computing and Applications, NCCA, 2011 First International Symposium on, 2011, pp. 73-78. URL: http://dx.doi.org/10.1109/NCCA.2011.19.
10. P. Mehrotra, J. Djomehri, S. Heistand, R. Hood, H. Jin, A. Lazanoff, S. Saini, R. Biswas, Performance evaluation of Amazon EC2 for NASA HPC applications, in: Proceedings of the 3rd workshop on Scientific Cloud Computing Date, ScienceCloud'12, ACM, New York, NY, USA, 2012, pp. 41-50. URL: http://doi.acm.org/10.1145/2287036.2287045.
11. R.R. Expósito, G.L. Taboada, S. Ramos, J. Touriño, R. Doallo, Performance analysis of HPC applications in the cloud, Future Gener. Comput. Syst. 29 (1) (2013) 218-229. including Special section: AIRCC-NetCoM 2009 and Special section: Clouds and Service-Oriented Architectures. URL: http://dx.doi.org/10.1016/j.future.2012.06.009.
12. M. Ben Belgacem, B. Chopard, J. Borgdorff, M. Mamonski, K. Rycerz, D. Harezlak, Distributed multiscale computations using the MAPPER framework, in: V.N. Alexandrov, M. Lees, V.V. Krzhizhanovskaya, J. Dongarra, P.M.A. Sloot (Eds.), ICCS, Procedia Comput. Sci. 18 (2013), 1106-1115. http://dx.doi.org/10.1016/j.procs.2013.05.276.
13. M. Ben Belgacem, B. Chopard, A. Parmigiani, Coupling Method for building a network of irrigation canals on a distributed computing environment, in: G. Sirakoulis, S. Bandini (Eds.), Cellular Automata, in: Lecture Notes in Computer Science, vol. 7495, Springer, Berlin, Heidelberg, 2012, pp. 309-318. URL: http://dx.doi.org/10.1007/978-3-642-33350-7-32.
14. A. Hoekstra, Multiscale applications on European e-infrastructures, VPH NoE Newsletter 5, January 2011. URL: http://vph-portal.eu/documents/10180/16729/VPH%20NoE%20newsletter%20n%C2%B05-Jan2011.pdf.
15. Mapper project, http://www.mapper-project.eu/.
16. J. Borgdorff, E. Lorenz, A. Hoekstra, J. Falcone, B. Chopard, A principled approach to distributed multiscale computing, from formalization to execution, in: e-Science Workshops (eScienceW), 2011 IEEE Seventh International Conference on, 2011, pp. 97-104. URL: http://dx.doi.org/10.1109/eScienceW.2011.9.
17. J. Borgdorff, J.-L. Falcone, E. Lorenz, C. Bona-Casas, B. Chopard, A.G. Hoekstra, Foundations of distributed multiscale computing: Formalization, specification, and analysis, J. Parallel Distrib. Comput. 73 (4) (2013) 465-483. URL: http://dx.doi.org/10.1016/j.jpdc.2012.12.011.
18. N. Kennedy, A. Mizeranschi, P. Thompson, H. Zheng, W. Dubitzky, Reverse engineering of gene regulation models from multi-condition experiments, in: Computational Intelligence in Bioinformatics and Computational Biology, CIBCB, 2013 IEEE Symposium on, 2013, pp. 112-119. http://dx.doi.org/10.1109/CIBCB.2013.6595396.
19. N. Kennedy, P. Thompson, O. Schmidt, W. Dubitzky, H. Zheng, Bile acid and xenobiotic system, in: W. Dubitzky, O. Wolkenhauer, K.-H. Cho, H. Yokota (Eds.), Encyclopedia of Systems Biology, Springer, New York, 2013, pp. 87-91. http://dx.doi.org/10.1007/978-1-4419-9863-7-1608.
20. K. Rycerz, E. Ciepiela, G. Dyk, D. Groen, T. Gubala, D. Harezlak, M. Pawlik, J.J. Suter, S.J. Zasada, P. Coveney, M. Bubak, Support for multiscale simulations with molecular dynamics, in: V.N. Alexandrov, M. Lees, V.V. Krzhizhanovskaya, J. Dongarra, P.M.A. Sloot (Eds.), ICCS, Procedia Comput. Sci. 18 (2013), 1116-1125. http://dx.doi.org/10.1016/j.procs.2013.05.277.
21. R.W. Nash, H.B. Carver, M.O. Bernabeu, J. Hetherington, D. Groen, T. Krüger, P.V. Coveney, Choice of boundary condition for lattice-Boltzmann simulation of moderate-Reynolds-number flow in complex domains, Phys. Rev. E 89 (2014) 023303. http://link.aps.org/doi/10.1103/PhysRevE.89.023303.
22. D. Groen, J.J Suter, P.V. Coveney, Modelling distributed multiscale simulation performance: An application to nanocomposites, in: e-Science Workshops, IEEE Computer Society, 2011, pp. 105-111. http://doi.ieeecomputersociety.org/10.1109/eScienceW.2011.37.
23. J. Borgdorff, M. Mamonski, B. Bosak, K. Kurowski, M. Ben Belgacem, B. Chopard, D. Groen, P. Coveney, A. Hoekstra, Distributed multiscale computing with MUSCLE 2, the multiscale coupling library and environment, J. Comput. Sci. 5 (5) (2014) 719-731. http://dx.doi.org/10.1016/j.jocs.2014.04.004.
24. MUSCLE2, http://apps.man.poznan.pl/trac/muscle.
25. T. Gubala, M. Bubak, P.M. Sloot, Semantic integration of collaborative research environments, Information Science Reference IGI Global, 2009, Chapter XXVI, pp. 514-530. http://dx.doi.org/10.4018/978-1-60566-374-6.ch026.
26. MAD software, http://www.mapper-project.eu/web/guest/mad-mame-ew.
27. PALABOS toolkit, http://www.palabos.org/.
28. M.C. Sukop, D.T. Thorne, Lattice Boltzmann Modeling: An Introduction for Geoscientists and Engineers, first ed., Springer, 2007, URL: http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/3540279814.
29. B. Chopard, Cellular automata modeling of physical systems, in: R.A. Meyers (Ed.), Encyclopedia of Complexity and Systems Science, Springer, 2009, pp. 865-892. http://dblp.uni-trier.de/db/reference/complexity/ecss2009.html#Chopard09.
30. A. Parmigiani, J. Latt, M. Ben Belgacem, B. Chopard, A lattice Boltzmann simulation of the Rhone river, Internat. J. Modern Phys. C 11 (24) (2013) 1340008. arXiv:http://www.worldscientific.com/doi/pdf/10.1142/S0129183113400081.
31. PRACE, http://www.prace-ri.eu/.
32. V. Mauch, M. Kunze, M. Hillenbrand, High performance cloud computing, Future Gener. Comput. Syst. 29 (6) (2013) 1408-1416. Including Special sections: High Performance Computing in the Cloud; Resource Discovery Mechanisms for P2P Systems. URL: http://dx.doi.org/10.1016/j.future.2012.03.011.
33. E. Weinan, B. Engquist, X. Li, W. Ren, E. Vanden-Eijnden, Heterogeneous multiscale methods: A review, Commun. Comput. Phys. 2 (3) (2007) 367-450. URL: http://www.global-sci.com/openaccess/v2-367.pdf.
34. G. Ingram, I. Cameron, K. Hangos, Classification and analysis of integrating frameworks in multiscale modelling, Chem. Eng. Sci. 59 (11) (2004) 2171-2187. http://dx.doi.org/10.1016/j.ces.2004.02.010. URL: http://www.sciencedirect.com/science/article/pii/S0009250904001307.
35. J.O. Dada, P. Mendes, Multi-scale modelling and simulation in systems biology, Integr. Biol. 2 (3) (2011) 86-96. URL: http://dx.doi.org/10.1039/c0ib00075b.
36. D. Groen, S. Zasada, P. Coveney, Taxonomy of multiscale computing communities, in: e-Science Workshops (eScienceW), 2011 IEEE Seventh International Conference on, 2011, pp. 120-127. URL: http://dx.doi.org/10.1109/eScienceW.2011.11.
37. A. Hoekstra, E. Lorenz, J.-L. Falcone, B. Chopard, Towards a complex automata framework for multi-scale modeling: Formalism and the scale separation map, in: Y. Shi, G. Albada, J. Dongarra, P. Sloot (Eds.), Computational Science-ICCS 2007, in: Lecture Notes in Computer Science, vol. 4487, Springer, Berlin, Heidelberg, 2007, pp. 922-930. URL: http://dx.doi.org/10.1007/978-3-540-72584-8-121.
38. A. Hoekstra, J.-L. Falcone, A. Caiazzo, B. Chopard, Multi-scale modeling with cellular automata: The complex automata approach, in: H. Umeo, S. Morishita, K. Nishinari, T. Komatsuzaki, S. Bandini (Eds.), Cellular Automata, in: Lecture Notes in Computer Science, vol. 5191, Springer, Berlin, Heidelberg, 2008, pp. 192-199. URL: http://dx.doi.org/10.1007/978-3-540-79992-4-25.
39. A. Hoekstra, A. Caiazzo, E. Lorenz, J.-L. Falcone, B. Chopard, Complex automata: Multi-scale modeling with coupled cellular automata, in: J. Kroc, P.M. Sloot, A.G. Hoekstra (Eds.), Simulating Complex Systems by Cellular Automata, Understanding Complex Systems, Springer, Berlin, Heidelberg, 2010, pp. 29-57. http://dx.doi.org/10.1007/978-3-642-12203-3-3.
40. D. Groen, S. Rieder, P. Grosso, C. de Laat, S.P. Zwart, A lightweight communication library for distributed computing, Comput. Sci. Discovery 3 (1) (2010) 015002. URL: http://stacks.iop.org/1749-4699/3/i=1/a=015002.
41. B. Bosak, J. Komasa, P. Kopta, K. Kurowski, M. Mamonski, T. Piontek, New capabilities in qoscosgrid middleware for advanced job management, advance reservation and co-allocation of computing resources-quantum chemistry application use case, in: M. Bubak, T. Szepieniec, K. Wiatr (Eds.), Building a National Distributed e-Infrastructure-PL-Grid, in: Lecture Notes in Computer Science, vol. 7136, Springer, Berlin, Heidelberg, 2012, pp. 40-55. URL: http://dx.doi.org/10.1007/978-3-642-28267-6-4.
42. E. Ciepiela, D. Harezlak, J. Kocot, T. Bartyński, M. Kasztelnik, P. Nowakowski, T. Gubala, M. Malawski, M. Bubak, Exploratory programming in the virtual laboratory, in: Computer Science and Information Technology, IMCSIT, Proceedings of the 2010 International Multiconference on, 2010, pp. 621-628.
43. E. Ciepiela, P. Nowakowski, J. Kocot, D. Hare¸zlak, T. Gubała, J. Meizner, M. Kasztelnik, T. Bartyński, M. Malawski, M. Bubak, Managing entire lifecycles of e-science applications in the gridspace2 virtual laboratory-from motivation through idea to operable web-accessible environment built on top of PL-grid e-infrastructure, in: M. Bubak, T. Szepieniec, K. Wiatr (Eds.), Building a National Distributed e-Infrastructure-PL-Grid, in: Lecture Notes in Computer Science, vol. 7136, Springer, Berlin, Heidelberg, 2012, pp. 228-239. http://dx.doi.org/10.1007/978-3-642-28267-6-18.
44. EGI, http://www.egi.eu/.
45. Amazon EC2 cloud computing. http://aws.amazon.com/ec2/.
46. MUSCLE package. http://apps.man.poznan.pl/trac/muscle.
47. StartCluster package. http://star.mit.edu/cluster/.
48. J. Borgdorff, M. Mamonski, B. Bosak, D. Groen, M. Ben Belgacem, K. Kurowski, A.G. Hoekstra, Multiscale computing with the multiscale modeling library and runtime environment, in: V.N. Alexandrov, M. Lees, V.V. Krzhizhanovskaya, J. Dongarra, P.M.A. Sloot (Eds.), ICCS, Procedia Comput. Sci. 18 (2013), 1097-1105. http://dx.doi.org/10.1016/j.procs.2013.05.275.
49. P. Kopta, M. Kulczewski, K. Kurowski, T. Piontek, P. Gepner, M. Puchalski, J. Komasa, Parallel application benchmarks and performance evaluation of the intel xeon 7500 family processors, in: M. Sato, S. Matsuoka, P.M.A. Sloot, G.D. van Albada, J. Dongarra (Eds.), ICCS, Procedia Comput. Sci. 4 (2011), 372-381. http://dx.doi.org/10.1016/j.procs.2011.04.039.
50. G. Wang, T.S.E. Ng, The impact of virtualization on network performance of Amazon ec2 data center, in: Proceedings of the 29th Conference on Information Communications, INFOCOM'10, IEEE Press, Piscataway, NJ, USA, 2010, pp. 1163-1171. URL: http://dl.acm.org/citation.cfm?id=1833515.1833691.
51. T. Gunarathne, T.-L. Wu, J.Y. Choi, S.-H. Bae, J. Qiu, Cloud computing paradigms for pleasingly parallel biomedical applications, Concurr. Comput.: Pract. Exp. 23 (17) (2011) 2338-2354. http://dx.doi.org/10.1002/cpe.1780.