Performance analysis of cloud computing services for many-tasks scientific computing

IEEE Transactions on Parallel and Distributed Systems

Volumn 22, Issue 6, 2011, Pages 931-945

  Iosup, Alexandru ^a   Ostermann, Simon ^b   Yigitbasi, Nezih ^a   Prodan, Radu ^b   Fahringer, Thomas ^b   Epema, Dick ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

^b UNIVERSITY OF INNSBRUCK   (Austria)

Author keywords

distributed applications; Distributed systems; metrics measurement; performance evaluation; performance measures.

Indexed keywords

NETWORK FUNCTION VIRTUALIZATION; VIRTUAL REALITY; VIRTUALIZATION;

DISTRIBUTED APPLICATIONS; DISTRIBUTED SYSTEMS; METRICS/MEASUREMENT; PERFORMANCE EVALUATION; PERFORMANCE MEASURE;

CLOUD COMPUTING;

EID: 85027938495     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2011.66     Document Type: Article

References (74)

1. Amazon, Inc., "Amazon Elastic Compute Cloud (Amazon EC2)," http://aws.amazon.com/ec2/, Dec. 2008.
2. GoGrid, "GoGrid Cloud-Server Hosting," http://www.gogrid. com, Dec. 2008.
3. A. Iosup, O.O. Sonmez, S. Anoep, and D.H.J. Epema, "The Performance of Bags-of-Tasks in Large-Scale Distributed Systems," Proc. ACM Int'l Symp. High Performance Distributed Computing (HPDC), pp. 97-108, 2008.
4. I. Raicu, Z. Zhang, M. Wilde, I.T. Foster, P.H. Beckman, K. Iskra, and B. Clifford, "Toward Loosely Coupled Programming on Petascale Systems," Proc. ACM Conf. Supercomputing (SC), p. 22, 2008.
5. A. Iosup, C. Dumitrescu, D.H.J. Epema, H. Li, and L. Wolters, "How Are Real Grids Used? The Analysis of Four Grid Traces and Its Implications," Proc. IEEE Seventh Int'l Conf. Grid Computing, pp. 262-269, 2006.
6. U. Lublin, D.G. Feitelson, "Workload on Parallel Supercomputers: Modeling Characteristics of Rigid Jobs," J. Parallel and Distributed Computing, vol. 63, no. 11, pp. 1105-1122, 2003. (Pubitemid 37457240)
7. D.G. Feitelson, L. Rudolph, U. Schwiegelshohn, K.C. Sevcik, and P. Wong, "Theory and Practice in Parallel Job Scheduling," Proc. Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1-34, 1997. (Pubitemid 127119055)
8. L. Youseff, R. Wolski, B.C. Gorda, and C. Krintz, " Paravirtualization for HPC Systems," Proc. ISPA Workshops, pp. 474-486, 2006.
9. E. Deelman, G. Singh, M. Livny, J.B. Berriman, and J. Good, "The Cost of Doing Science on the Cloud: The Montage Example," Proc. IEEE/ACM Supercomputing (SC), p. 50, 2008.
10. M.R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, "Amazon S3 for Science Grids: A Viable Solution?" Proc. DADC '08: ACM Int'l Workshop Data-Aware Distributed Computing, pp. 55- 64, 2008.
11. E. Walker, "Benchmarking Amazon EC2 for HP Scientific Computing," Login, vol. 33, no. 5, pp. 18-23, Nov. 2008.
12. L. Wang, J. Zhan, W. Shi, Y. Liang, L. Yuan, "In Cloud, Do Mtc or Htc Service Providers Benefit from the Economies of Scale?" Proc. Second Workshop Many-Task Computing on Grids and Supercomputers (SC-MTAGS), 2009.
13. A. Iosup, H. Li, M. Jan, S. Anoep, C. Dumitrescu, L. Wolters, and D. Epema, "The Grid Workloads Archive," Future Generation Computer Systems, vol. 24, no. 7, pp. 672-686, 2008.
14. D. Thain, J. Bent, A.C. Arpaci-Dusseau, R.H. Arpaci-Dusseau, and M. Livny, "Pipeline and Batch Sharing in Grid Workloads," Proc. IEEE 12th Int'l Symp. High Performance Distributed Computing (HPDC), pp. 152-161, 2003.
15. S. Ostermann, A. Iosup, R. Prodan, T. Fahringer, and D.H.J. Epema, "On the Characteristics of Grid Workflows," Proc. Workshop Integrated Research in Grid Computing (CGIW), pp. 431-442, 2008.
16. The Parallel Workloads Archive Team, "The Parallel Workloads Archive Logs," http://www.cs.huji.ac.il/labs/parallel/ workload/logs.html, Jan. 2009.
17. Y.-S. Kee, H. Casanova, and A.A. Chien, "Realistic Modeling and Svnthesis of Resources for Computational Grids," Proc. ACM/IEEE Conf. Supercomputing (SC), p. 54, 2004.
18. A. Iosup, O.O. Sonmez, and D.H.J. Epema, "DGSim: Comparing Grid Resource Management Architectures through Trace-Based Simulation," Proc. 14th Int'l Euro-Par Conf. Parallel Processing, pp. 13-25, 2008.
19. L. Youseff, M. Butrico, and D. DaSilva, "Towards a Unified Ontology of Cloud Computing," Proc. Grid Computing Environments Workshop (GCE '08), Nov. 2008.
20. M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R.H. Katz, A. Konwinski, G. Lee, D.A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "Above the Clouds: A Berkeley View of Cloud Computing," Technical Report UCB/EECS-2009-28, EECS Dept, Univ. of California, Berkeley, Feb. 2009.
21. R. Prodan and S. Ostermann, "A Survey and Taxonomy of Infrastructure as a Service and Web Hosting Cloud Providers," Proc. Int'l Conf. Grid Computing, pp. 1-10, 2009.
22. A. Iosup, M. Jan, O.O. Sonmez, and D.H.J. Epema, "The Characteristics and Performance of Groups of Jobs in Grids," Proc. Euro-Par, pp. 382-393, 2007.
23. R.H. Saavedra, A.J. Smith, "Analysis of Benchmark Characteristics and Benchmark Performance Prediction," ACM Trans. Computer Systems, vol. 14, no. 4, pp. 344-384, 1996. (Pubitemid 126418627)
24. P. Barham, B. Dragovic, K. Fraser, S. Hand, T.L. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, "Xen and the Art of Virtualization," Proc. 19th ACM Symp. Operating Systems Principles (SOSP), pp. 164-177, 2003. (Pubitemid 40929695)
25. A. Kowalski, "Bonnie-File System Benchmarks," technical report, Jefferson Lab, http://cc.jlab.org/docs/scicomp/ benchmark/bonnie.html, Oct. 2002.
26. M. Babcock, "XEN Benchmarks, Tech. Rep.," mikebabcock.ca/ linux/xen/, Aug. 2007.
27. J.S. Vetter, S.R. Alam, T.H.D. Jr, M.R. Fahey, P.C. Roth, P.H. Worley, "Early Evaluation of the Cray XT3," Proc. 20th Int'l Conf. Parallel and Distributed Processing Symp. (IPDPS), 2006.
28. P. Luszczek, D.H. Bailey, J. Dongarra, J. Kepner, R.F. Lucas, R. Rabenseifner, and D. Takahashi, "S12-The HPC Challenge (HPCC) Benchmark Suite," Proc. ACM Supercomputing Conf. High Performance Networking and Computing (SC), p. 213, 2006.
29. S. Saini, D. Talcott, D.C. Jespersen, M.J. Djomehri, H. Jin, and R. Biswas, "Scientific Application-Based Performance Comparison of SGI Altix 4700, IBM POWER5+, and SGI ICE 8200 Supercomputers," Proc. IEEE/ACM Conf. Supercomputing (SC), p. 7, 2008.
30. T.H. Dunigan, M.R. Fahey, J.B.W. III, and P.H. Worley, "Early Evaluation of the Cray X1," Proc. IEEE/ACM Conf. Supercomputing (SC), p. 18, 2003.
31. S.R. Alam, R.F. Barrett, M. Bast, M.R. Fahey, J.A. Kuehn, C. McCurdy, J. Rogers, P.C. Roth, R. Sankaran, J.S. Vetter, P.H. Worley, and W. Yu, "Early Evaluation of IBM BlueGene/P," Proc. ACM Conf. Supercomputing (SC), p. 23, 2008.
32. R. Biswas, M.J. Djomehri, R. Hood, H. Jin, C.C. Kiris, and S. Saini, "An Application-Based Performance Characterization of the Columbia Supercluster," Proc. IEEE Conf. Supercomputing (SC), p. 26, 2005.
33. L. McVoy and C. Staelin, "LMbench-Tools for Performance Analysis," http://www.bitmover.com/lmbench/, Dec. 2008.
34. T. Bray, "Bonnie," 1996, http://www.textuality.com/bonnie/, Dec. 2008.
35. P.J. Mucci and K.S. London, "Low Level Architectural Characterization Benchmarks for Parallel Computers," Technical Report UT-CS-98-394, Univ. of Tennessee, 1998.
36. N. Yigitbasi, A. Iosup, S. Ostermann, and D. Epema, "C-Meter: A Framework for Performance Analysis of Computing Clouds," Proc. IEEE Ninth Int'l Symp. Cluster Computing and the GRID (CCGRID '09), pp. 472-477, 2009.
37. A. Iosup and D.H.J. Epema, "GrenchMark: A Framework for Analyzing, Testing, and Comparing Grids," Proc. IEEE Sixth Int'l Symp. Cluster Computing and the GRID (CCGrid), pp. 313-320, 2006. (Pubitemid 44761720)
38. The HPCC Team, "HPCChallenge Results," http://icl.cs.utk.edu/ hpcc/hpcc-results.cgi, Jan. 2009.
39. J. Worringen and K. Scholtyssik, "MP-MPICH: User Documentation & Technical Notes," June 2002.
40. J. Dongarra et al., "Basic Linear Algebra Subprograms Technical Forum Standard," Int'l. J. High Performance Applications and Supercomputing, vol. 16, no. 1, pp. 1-111, 2002.
41. K. Goto, R.A.v.d. Geijn, "Anatomy of High-Performance Matrix Multiplication," ACM Trans. Math. Software, vol. 34, no. 3, pp. 1-25, 2008.
42. Advanced Clustering Tech., "Linpack Problem Size Analyzer," http://www.advancedclustering.com/, Dec. 2008.
43. J. Dongarra, P. Luszczek, A. Petitet, "The Linpack Benchmark: Past, Present and Future," Concurrency and Computation: Practice and Experience, vol. 15, no. 9, pp. 803-820, 2003.
44. A. Verma, P. Ahuja, and A. Neogi, "Power-Aware Dynamic Placement of Hpc Applications," Proc. ACM 22nd Ann. Int'l Conf. Supercomputing (ICS), pp. 175-184, 2008.
45. M. Zaharia, A. Konwinski, A.D. Joseph, R.H. Katz, and I. Stoica, "Improving Mapreduce Performance in Heterogeneous Environments," Proc. Eighth USENIX Conf. Operating Systems Design and Implementation (OSDI), pp. 29-42, 2008.
46. S. Ostermann, A. Iosup, N.M. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema, "An Early Performance Analysis of Cloud Computing Services for Scientific Computing," Technical Report PDS-2008-006, TU Delft, http://www.eecs.berkeley.edu/Pubs/ TechRpts/2009/EECS-2009-28.html, Dec. 2008.
47. F. Petrini, D.J. Kerbyson, and S. Pakin, "The Case of the Missing Supercomputer Performance: Achieving Optimal Performance on the 8,192 Processors of ASCI Q," Proc. IEEE/ACM Conf. Supercomputing (SC), p. 55, 2003.
48. D.J. Kerbyson, A. Hoisie, H.J. Wasserman, "A Performance Comparison between the Earth Simulator and Other Terascale Systems on a Characteristic ASCI Workload," Concurrency- Practice and Experience, vol. 17, no. 10, pp. 1219-1238, 2005. (Pubitemid 41092969)
49. F. Petrini, G. Fossum, J. Fernández, A.L. Varbanescu, M. Kistler, and M. Perrone, "Multicore Surprises: Lessons Learned from Optimizing Sweep3D on the Cell Broadband Engine," Proc. IEEE Int'l Parallel and Distributed Processing Symp. (IPDPS), pp. 1-10, 2007.
50. R.H. Arpaci-Dusseau, A.C. Arpaci-Dusseau, A. Vahdat, L.T. Liu, T.E. Anderson, and D.A. Patterson, "The Interaction of Parallel and Sequential Workloads on a Network of Workstations," Proc. ACM SIGMETRICS Joint Int'l Conf. Measurement and Modeling of Computer Systems, pp. 267-278, 1995.
51. B. Clark, T. Deshane, E. Dow, S. Evanchik, M. Finlayson, J. Herne, and J.N. Matthews, "Xen and the Art of Repeated Research," Proc. USENIX Ann. Technical Conf. (ATC), pp. 135-144, 2004.
52. A. Iosup, N.M. Yigitbasi, and D. Epema, "On the Performance Variability of Production Cloud Services," Technical Report PDS- 2010-002, TU Delft, http://pds.twi.tudelft.nl/reports/2010/PDS- 2010-002.pdf. Jan. 2010.
53. T. Killalea, "Meet the Virts," Queue, vol. 6, no. 1, pp. 14-18, 2008.
54. D. Nurmi, R. Wolski, and J. Brevik, "Varq: Virtual Advance Reservations for Queues," Proc. ACM 17th Int'l Symp. High Performance Distributed Computing (HPDC), pp. 75-86, 2008.
55. A. Iosup, D.H.J. Epema, T. Tannenbaum, M. Farrellee, and M. Livny, "Inter-Operating Grids through Delegated Matchmaking," Proc. ACM/IEEE Conf. Supercomputing (SC), p. 13, 2007.
56. D. Nurmi, R. Wolski, C. Grzegorczyk, G. Obertelli, S. Soman, L. Youseff, and D. Zagorodnov, "The Eucalyptus Open-Source Cloud-Computing System," Technical Report 2008-10, uCSD, http://eucalyptus.cs.ucsb.edu/, 2008.
57. B. Quétier, V. Néri, F. Cappello, "Scalability Comparison of Four Host Virtualization Tools," J. Grid Computing, vol. 5, pp. 83-98, 2007. (Pubitemid 46909014)
58. A. Menon, J.R. Santos, Y. Turner, G.J. Janakiraman, and W. Zwaenepoel, "Diagnosing Performance Overheads in the Xen Virtual Machine Environment," Proc. ACM First Int'l Conf. Virtual Execution Environments (VEE), pp. 13-23, 2005. (Pubitemid 43195497)
59. N. Sotomayor, K. Keahey, and I. Foster, "Overhead Matters: A Model for Virtual Resource Management," Proc. IEEE First Int'l Workshop Virtualization Technology in Distributed Technology (VTDC), pp. 4-11, 2006.
60. A.B. Nagarajan, F. Mueller, C. Engelmann, and S.L. Scott, "Proactive Fault Tolerance for HPC with Xen Virtualization," Proc. ACM 21st Ann. Int'l Conf. Supercomputing (ICS), pp. 23-32, 2007. (Pubitemid 47281603)
61. L. Youseff, K. Seymour, H. You, J. Dongarra, and R. Wolski, "The Impact of Paravirtualized Memory Hierarchy on Linear Algebra Computational Kernels and Software," Proc. ACM 17th Int'l Symp. High Performance Distributed Computing (HPDC), pp. 141-152, 2008.
62. W. Yu and J.S. Vetter, "Xen-Based HPC: A Parallel I/O Perspective," Proc. IEEE Eighth Int'l Symp. Cluster Computing and the Grid (CCGrid), pp. 154-161, 2008.
63. L. Cherkasova and R. Gardner, "Measuring CPU Overhead for I/ O Processing in the Xen Virtual Machine Monitor," Proc. USENIX Ann. Technical Conf. (ATC), pp. 387-390, 2005.
64. U.F. Minhas, J. Yadav, A. Aboulnaga, and K. Salem, "Database Systems on Virtual Machines: How Much Do You Lose?" Proc. IEEE 24th Int'l Conf. Data Eng. (ICDE) Workshops, pp. 35-41, 2008.
65. W. Huang, J. Liu, B. Abali, and D.K. Panda, "A Case for High Performance Computing with Virtual Machines," Proc. ACM 20th Ann. Int'l Conf. Supercomputing (ICS), pp. 125-134, 2006. (Pubitemid 47168498)
66. L. Gilbert, J. Tseng, R. Newman, S. Iqbal, R. Pepper, O. Celebioglu, J. Hsieh, and M. Cobban, "Performance Implications of Virtualization and Hyper-Threading on High Energy Physics Applications in a Grid Environment," Proc. IEEE 19th Int'l Parallel and Distributed Processing Symp. (IPDPS), 2005.
67. J. Zhan, L. Wang, B. Tu, Y. Li, P. Wang, W. Zhou, and D. Meng, "Phoenix Cloud: Consolidating Different Computing Loads on Shared Cluster System for Large Organization," Proc. First Workshop Cloud Computing and Its Application (CCA '08) Posters, pp. 7-11, 2008.
68. C. Evangelinos and C.N. Hill, "Cloud Computing for Parallel Scientific Hpc Applications: Feasibility of Running Coupled Atmosphere-Ocean Climate Models on Amazons ec2," Proc. First Workshop Cloud Computing and Its Application (CCA '08), pp. 1-6, 2008.
69. M.-E. Bgin, B. Jones, J. Casey, E. Laure, F. Grey, C. Loomis, and R. Kubli, "Comparative Study: Grids and Clouds, Evolution or Revolution?," CERN'' EGEE-II Report, https://edms.cern.ch/ file/925013/3/EGEE-Grid-Cloud.pdf, June 2008.
70. S. Williams, J. Shalf, L. Oliker, S. Kamil, P. Husbands, and K.A. Yelick, "The Potential of the Cell Processor for Scientific Computing," Proc. ACM Conf. Computing Frontiers, pp. 9-20, 2006. (Pubitemid 46644667)
71. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A.D. Joseph, R.H. Katz, S. Shenker, and I. Stoica, "Mesos: A Platform for Fine- Grained Resource Sharing in the Data Center," Technical Report UCB/EECS-2010-87, UCBerkeley, http://www.eecs.berkeley. edu/Pubs/TechRpts/2010/EECS-2010-87.html, May 2010.
72. L. Vu, "Berkeley Lab Contributes Expertise to New Amazon Web Services Offering," http://www.lbl.gov/cs/Archive/ news071310.html, July 2010.
73. J.P. Jones and B. Nitzberg, "Scheduling for Parallel Supercomputing: A Historical Perspective of Achievable Utilization," Proc. Job Scheduling Strategies for Parallel Processing (JSSPP), pp. 1-16, 1999.
74. lxc Linux Containers Team, "Linux Containers Overview," http://lxc.sourceforge.net/, Aug. 2010.