Evaluating and Improving the Performance and Scheduling of HPC Applications in Cloud

IEEE Transactions on Cloud Computing

Volumn 4, Issue 3, 2016, Pages 307-321

  Gupta, Abhishek ^a   Faraboschi, Paolo ^b   Gioachin, Filippo ^b   Kale, Laxmikant V ^a   Kaufmann, Richard ^b   Lee, Bu Sung ^b   March, Verdi ^b   Milojicic, Dejan ^b   Suen, Chun Hui ^b  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

application awareness; characterization; cloud; economics; HPC; job scheduling; performance analysis

Indexed keywords

CHARACTERIZATION; CLOUDS; ECONOMICS; SUPERCOMPUTERS; TURNAROUND TIME; VIRTUAL REALITY;

APPLICATION AWARENESS; COMPREHENSIVE PERFORMANCE; HETEROGENEOUS RESOURCES; HIGH PERFORMANCE COMPUTING (HPC); INTELLIGENT APPLICATIONS; JOB SCHEDULING; MULTI-PLATFORM ENVIRONMENT; PERFORMANCE ANALYSIS;

SCHEDULING;

EID: 84986598841     PISSN: None     EISSN: 21687161     Source Type: Journal    
DOI: 10.1109/TCC.2014.2339858     Document Type: Article

References (41)

1. K. Yelick, S. Coghlan, B. Draney, R. S. Canon, L. Ramakrishnan, A. Scovel, I. Sakrejda, A. Liu, S. Campbell, P. T. Zbiegiel, T. Declerck, P. Rich, "The magellan report on cloud computing for science", U. S. Department of Energy Office of Science, Office of Advanced Scientific Computing Research (ASCR), Dec. 2011.
2. P. Mehrotra, J. Djomehri, S. Heistand, R. Hood, H. Jin, A. Lazanoff, S. Saini, and R. Biswas, "Performance evaluation of Amazon EC2 for NASA HPC applications, " in Proc. 3rd Workshop Scientific Cloud Comput. 2012, pp. 41-50.
3. C. Evangelinos and C. N. Hill, "Cloud computing for parallel scientific HPC applications: Feasibility of running coupled atmosphere- ocean climate models on Amazon's EC2, " in Proc. IEEE Cloud Comput. Appl., Oct. 2008, pp. 2-34.
4. A. Gupta and D. Milojicic, "Evaluation of HPC Applications on Cloud, " in Proc. Open Cirrus Summit (Best Student Paper), Atlanta, GA, USA, Oct. 2011, pp. 22-26.
5. E. Walker, "Benchmarking Amazon EC2 for high-performance scientific computing, " LOGIN, vol. 33, pp. 18-23, 2008.
6. A. Gupta, L. V. Kalé, D. S. Milojicic, P. Faraboschi, R. Kaufmann, V. March, F. Gioachin, C. H. Suen, and B.-S. Lee, "The who, what, why, and how of HPC applications in the cloud, " in Proc. 5th IEEE Intl. Conf. Cloud Comp. Techno. and Sc. Best Paper, 2013, pp. 306-314.
7. High Performance Computing (HPC) on AWS [Online]. Available: http://aws.amazon.com/hpc-applications
8. A. Iosup, S. Ostermann, M. N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. J. Epema, "Performance Analysis of Cloud Computing Services for Many-Tasks Scientific Computing, " IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 6, pp. 931-945, Jun. 2011.
9. J. Napper and P. Bientinesi, "Can Cloud Computing reach the Top500?' in Proceedings of the Combined Workshops on Unconventional High Performance Computing Workshop Plus Memory Access Workshop. New York, NY, USA: ACM, 2009.
10. Ranger User Guide. [Online]. Available: http://services.tacc. utexas.edu/index.php/ranger-user-guide
11. A. I. Avetisyan, R. Campbell, I. Gupta, M. T. Heath, S. Y. Ko, G. R. Ganger, M. A. Kozuch, D. O'Hallaron, M. Kunze, T. T. Kwan, and others, "Open Cirrus: A Global Cloud Computing Testbed, " Computer, vol. 43, no. 4, pp. 35-43, Apr. 2010.
12. D. Nurmi R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, "Cluster Computing and the Grid, 2009. CCGRID'09. 9th IEEE/ACM International Symposium on, " in Proc. Cloud Comput. Appl., Oct. 2009, pp. 124-131.
13. Kivity, A. Y. Kamay, D. Laor, U. Lublin, and A. Liguori, "kvm: the Linux virtual machine monitor, " in Proc. of the Linux Symposium, vol. 1, pp. 225-230, 2007.
14. A. J. Younge, R. Henschel, J. T. Brown, G. Von Laszewski, J. Qiu, and G. C. Fox, "Analysis of virtualization technologies for high performance computing environments, " in Proc. IEEE Int. Conf. Cloud Comput., 2011, pp. 9-16.
15. D. Schauer et al, (June). Linux containers version 0.7.0. (2010) [Online]. Available: http://lxc.sourceforge.net/
16. Intel Corporation. (Feb.) Intel(r) Virtualization Technology for Directed I/O. Tech. Rep. D51397-006, (2011) [Online]. Available: http://download.intel.com/technology/computing/vptech/Intel (r)-VT-for-D irect-IO.pdf
17. NPB [Online]. Available: http://nas.nasa.gov/publications/npb. html
18. "MPI: A Message Passing Interface Standard, " in MPI Forum, 1994.
19. L. Kalé and S. Krishnan, "CHARM++: A Portable Concurrent Object Oriented System Based on C++, " in Proc. 8th Annu. Conf. Object-Oriented Program. Syst., Languages, Appl., 1993, pp. 91-108.
20. A. Bhatele, S. Kumar, C. Mei, J. C. Phillips, G. Zheng, and L. V. Kale, "Overcoming scaling challenges in biomolecular simulations across multiple platforms, " in Proc. IEEE Int. Symp. Parallel Distrib. Process., 2008, pp. 1-12.
21. P. Jetley, F. Gioachin, C. Mendes, L. V. Kale, and T. R. Quinn, "Massively Parallel Cosmological Simulations with ChaNGa, " in Proc. IEEE Int. Symp. Parallel Distrib. Processing, 2008, pp. 1-12.
22. The ASCII Sweep3D code [Online]. Available: http://wwwc3. lanl.gov/pal/software/sweep3d
23. O. Zaki, E. Lusk, W. Gropp, and D. Swider, "Toward scalable performance visualization with Jumpshot, " Int. J. High Perform. Comput. Appl., vol. 13, no. 3, pp. 277-288, Fall 1999.
24. T. Watanabe, M. Nakao, T. Hiroyasu, T. Otsuka, and M. Koibuchi, "Impact of topology and link aggregation on a PC cluster with ethernet, " in Proc. IEEE CLUSTER, Sep./Oct. 2008, pp. 280-285.
25. C. Bischof, D. anMey, and C. Iwainsky. (2011). Brainware for Green HPC. CS - Research and Development, pp. 1-7 [Online]. Available: http://dx.doi.org/10.1007/s00450-011-0198-5
26. NVIDIA GRID [Online]. Available: http://nvidia.com/object/ virtual-gpus.html
27. A. Gupta, Abhishek, Kalé, V. Laxmikant, D. S. Milojicic, P. Faraboschi, R. Kaufmann, V. March, F. Gioachin, C. H. Suen, and B. -S. Lee, "Exploring the Performance and Mapping of HPC Applications to Platforms in the cloud, " in Proc. 21st Int. Symp. High-Perform. Parallel Distrib. Comput., 2012, pp. 121-122.
28. R. N. Calheiros, R. Ranjan, A. Beloglazov, C. A. F. De Rose, and R. Buyya, "CloudSim: A toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, " Softw. Pract. Exper., vol. 41, no. 1, pp. 23-50, Jan. 2011.
29. Parallel Workloads Archive. [Online]. Available: http://www.cs. huji.ac.il/labs/parallel/workload/
30. J. Ekanayake, X. Qiu, T. Gunarathne, S. Beason, and G. C. Fox, "High performance parallel computing with clouds and cloud technologies, " in Cloud Computing. New York, NY, USA: Springer, 2010.
31. E. Roloff, M. Diener, A. Carissimi, and P. Navaux, "High performance computing in the cloud: Deployment, performance and cost efficiency, " in Proc. CloudCom, 2012, pp. 371-378.
32. A. Marathe, R. Harris, D. K. Lowenthal, B. R. de Supinski, B. Rountree, M. Schulz, and X. Yuan, "a comparative study of high-performance computing on the cloud, " in Proc. 21st Int. Symp. High-Performance Parallel Distrib. Comput., 2013, pp. 239-250.
33. J. Lange, K. Pedretti, T. Hudson, P. Dinda, Z. Cui, L. Xia, P. Bridges, A. Gocke, S. Jaconette, M. Levenhagen, and R. Brightwell, "Palacios and Kitten: New high performance operating systems for scalable virtualized and native supercomputing, " in Proc. IEEE Int. Symp. Parallel Distrib. Process., 2010, pp. 1-12.
34. B. Kocoloski, J. Ouyang, and J. Lange, "A case for dual stack virtualization: consolidating HPC and commodity applications in the cloud, " in Proc. 3rd ACM Symp. Cloud Comput., New York, NY, USA, 2012, pp. 23:1-23:7.
35. Heterogeneous Architecture Scheduler. [Online]. Available: http:// wiki.openstack.org/HeterogeneousArchitectureScheduler
36. A. Gupta, L. Kale, D. Milojicic, P. Faraboschi, and S. Balle, "HPCAware VM Placement in Infrastructure Clouds, " in Proc. IEEE Int. Conf. Cloud Eng., Mar. 2013, pp. 11-20.
37. A. Gupta, O. Sarood, L. Kale, and D. Milojicic, "Improving HPC application performance in cloud through dynamic load balancing, " in Proc. 13th IEEE/ACM Int. Symp. Cluster, Cloud, Grid Comput., 2013, pp. 402-409.
38. P. Fan, Z. Chen, J. Wang, Z. Zheng, and M. R. Lyu, "Topologyaware deployment of scientific applications in cloud computing, " in Proc. IEEE 5th Int. Conf. Cloud Comput., 2012, pp. 319-326.
39. F. Berman, A. Chien, K. Cooper, J. Dongarra, I. Foster, D. Gannon, L. Johnsson, K. Kennedy, C. Kesselman, J. Mellor-Crumme, and others, "The grads project: Software support for high-level grid application development, " Int. J. High Perform. Comput. Appl., vol. 15, pp. 327-344, 2001.
40. H. Kim, Y. el Khamra, I. Rodero, S. Jha, and M. Parashar, "Autonomic management of application workflows on hybrid computing infrastructure, " Sci. Program., vol. 19, no. 2, pp. 75-89, Jan. 2011.
41. M. Kulkarni, M. Burtscher, R. Inkulu, K. Pingali, and C. Casçaval, "How much parallelism is there in irregular applications?" SIGPLAN Not., vol. 44, no. 4, pp. 3-14, Feb. 2009.