Cloud Benchmarking for Maximising Performance of Scientific Applications

IEEE Transactions on Cloud Computing

Volumn 7, Issue 1, 2019, Pages 170-182

  Varghese, Blesson ^a   Akgun, Ozgur ^b   Miguel, Ian ^b   Thai, Long ^b   Barker, Adam ^b  

^a QUEEN'S UNIVERSITY BELFAST   (United Kingdom)

^b UNIVERSITY OF ST ANDREWS   (United Kingdom)

Author keywords

benchmarking methodology; Cloud benchmark; cloud performance; cloud ranking

Indexed keywords

COST EFFECTIVENESS; VIRTUAL ADDRESSES;

BENCHMARKING DATA; BENCHMARKING METHODOLOGY; CASE-STUDIES; CLOUD PERFORMANCE; COST EFFECTIVE; LOCAL COMMUNICATIONS; PERFORMANCE CAPABILITY; SCIENTIFIC APPLICATIONS;

BENCHMARKING;

EID: 85062706241     PISSN: None     EISSN: 21687161     Source Type: Journal    
DOI: 10.1109/TCC.2016.2603476     Document Type: Article

References (43)

1. Z. Li, L. O'Brien, H. Zhang, and R. Cai, "On a catalogue of metrics for evaluating commercial cloud services, " in Proc. 13th Int. Conf. Grid Comput., 2012, pp. 164-173.
2. A. Iosup, R. Prodan, and D. Epema, "IaaS cloud benchmarking: Approaches, challenges, and experience, " in Proc. Int. Workshop Hot Topics Cloud Services, 2013, pp. 1-2.
3. B. Varghese, O. Akgun, I. Miguel, L. Thai, and A. Barker, "Cloud benchmarking for performance, " in Proc. 6th IEEE Int. Conf. Cloud Comput. Technol. Sci., 2014, pp. 535-540.
4. B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking cloud serving systems with YCSB, " in Proc. 1st ACM Symp. Cloud Comput., 2010, pp. 143-154.
5. P. Mehrotra, et al., "Performance evaluation of Amazon EC2 for NASA HPC applications, " in Proc. 3rd Workshop Sci. Cloud Comput., 2012, pp. 41-50.
6. J. J. Rehr, F. D. Vila, J. P. Gardner, L. Svec, and M. Prange, "Scientific computing in the cloud, " IEEE Comput. Sci. Eng., vol. 12, no. 3, pp. 34-43, May/Jun. 2010.
7. O. Niehorster, A. Brinkmann, G. Fels, J. Kruger, and J. Simon, "Enforcing SLAs in scientific clouds, " in Proc. IEEE Int. Conf. Cluster Comput., 2010, pp. 178-187.
8. L. Gillam, B. Li, and J. O'Loughlin, "Benchmarking cloud performance for service level agreement parameters, " Int. J. Cloud Comput., vol. 3, no. 1, pp. 3-23, 2014.
9. L. Gillam, B. Li, J. O'Loughlin, and A. P. Tomar, "Fair benchmarking for cloud computing systems, " J. Cloud Comput. Advances Syst. Appl., vol. 2, no. 6, 2013, Art. no. 44.
10. E. Kreyszig, Advanced Engineering Mathematics, 8th ed. Hoboken, NJ, USA: Wiley, 2000.
11. E. Walker, "Benchmarking Amazon EC2 for high-performance scientific computing, " USENIX Login, vol. 33, no. 5, pp. 18-23, 2008.
12. A. Gupta and D. Milojicic, "Evaluation of HPC applications on cloud, " HP Lab., Palo Alto, CA, USA, Tech. Rep. HPL-2011-132, 2011.
13. T. Rabl, M. Frank, H. M. Sergieh, and H. Kosch, "A data generator for cloud-scale benchmarking, " in Proc. 2nd TPC Technol. Conf. Perform. Eval. Meas. Characterization Complex Syst., 2010, pp. 41-56.
14. K. R. Jackson, et al., "Performance analysis of high performance computing applications on the Amazon web services cloud, " in Proc. IEEE 2nd Int. Conf. Cloud Comput. Technol. Sci., 2010, pp. 159-168.
15. A. Gupta, et al., "Exploring the performance and mapping of HPC applications to platforms in the cloud, " in Proc. 21st Int. Symp. High-Perform. Parallel Distrib. Comput., 2012, pp. 121-122.
16. Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, "Case study for running HPC applications in public clouds, " in Proc. 19th ACM Int. Symp. High Perform. Distrib. Comput., 2010, pp. 395-401.
17. C. Luo, et al., "CloudRank-D: Benchmarking and ranking cloud computing systems for data processing applications, " Frontiers Comput. Sci., vol. 6, no. 4, pp 347-362, 2012.
18. Amazon Elastic Compute Cloud (EC2) instance pricing webpage. (2016). [Online]. Available: http://aws. amazon. com/ec2/pricing/, Accessed on: Feb. 16, 2014.
19. C. Staelin, "Imbench-An extensible micro-benchmark suite, " Softw.-Practice Exp., vol. 35, pp. 1079-1105, 2005.
20. D. H. Bailey, et al., "The NAS parallel benchmarks-Summary and preliminary results, " in Proc. ACM/IEEE Conf. Supercomputing, 1991, pp. 158-165.
21. A. K. Bahl, O. Baltzer, A. Rau-Chaplin, and B. Varghese, "Parallel simulations for analysing portfolios of catastrophic event risk, " in Proc. Int. Conf. SC Companion High Perform. Comput. Netw. Storage Anal., 2012, pp. 1176-1184.
22. B. Varghese and A. Barker, "Are clouds ready to accelerate ad hoc financial simulations?" in Proc. Int. Symp. Big Data Comput., 2014, pp. 54-63.
23. G. Woo, "Natural catastrophe probable maximum loss, " Brit. Actuarial J., vol. 8, no. 5, pp. 943-959, 2002.
24. K. J. Bowers, et al., "Scalable algorithms for molecular dynamics simulations on commodity clusters, " in Proc. ACM/IEEE Conf. Supercomputing, 2006, Art. no. 84.
25. J. C. Phillips, et al., "Scalable molecular dynamics with NAMD, " J. Comput. Chemistry, vol. 26, no. 16, pp. 1781-1802, 2005.
26. D. H. Bailey, et al., "The NAS parallel benchmarks-Summary and preliminary results, " in Proc. ACM/IEEE Conf. Supercomputing, 1991, pp. 158-165.
27. K. Huppler, "The art of building a good benchmark, " in Performance Evaluation and Benchmarking, R. Nambiar and M. Poess, Eds. Berlin, Germany: Springer, 2009, pp. 18-30.
28. D. Kaeli and K. Sachs, Computer Performance Evaluation and Benchmarking. Berlin Germany: Springer, 2009.
29. J. J. Dongarra, P. Luszczek, and A. Petitet, "The LINPACK benchmark: Past, present and future, " Concurrency Comput. Practice Experience, vol. 15, pp. 803-820, 2013.
30. B. Gregg, Systems Performance: Enterprise and the Cloud, 1st ed. Englewood Cliffs, NJ, USA: Prentice Hall, 2013.
31. E. Folkerts, A. Alexandrov, K. Sachs, A. Iosup, V. Markl, and C. Tosun, "Benchmarking in the cloud: What it should, can, and cannot be, " in Selected Topics in Performance Evaluation and Benchmarking, R. Nambiar and M. Poess, Eds. Berlin, Germany: Springer, 2013, pp. 173-188.
32. V. Vedam and J. Vemulapati, "Demystifying cloud benchmarking paradigm: An in depth view, " in Proc. 36th IEEE Int. Conf. Comput. Softw. Appl., 2012, pp. 416-421.
33. M. Ferdman, et al., "Clearing the clouds: A study of emerging scale-out workloads on modern hardware" in Proc. 17th Int. Conf. Archit. Support Program. Languages Operating Syst., 2012, pp. 37-48.
34. A. Lenk, M. Menzel, J. Lipsky, S. Tai, and P. Offermann, "What are you paying for? Performance benchmarking for infrastructure-as-a-service offerings, " in Proc. 4th Int. Conf. Cloud Comput., 2011, pp. 484-491.
35. A. Iosup, N. Yigitbasi, and D. Epema, "On the performance variability of production cloud services, " in Proc. 11th IEEE/ACM Int. Symp. Cluster Cloud Grid Comput., 2011, pp. 104-113.
36. Y. Mei, L. Liu, X. Pu, and S. Sivathanu, "Performance measurements and analysis of network I/O applications in virtualised cloud, " in Proc. 3rd Int. Conf. Cloud Comput., 2010, pp. 59-66.
37. A. Li, X. Yang, S. Kandula, and M. Zhang, "CloudCmp: Comparing public cloud providers, " in Proc. 10th ACM SIGCOMM Conf. Internet Meas., 2010, pp. 1-14.
38. W. Sobel, et al., "Cloudstone: Multi-platform multi-language benchmark and measurement tools for web 2. 0, " in Proc. Cloud Comput. Appl., 2008.
39. A. Li, X. Zong, S. Kandula, X. Yang, and M. Zhang, "CloudProphet: Towards application performance prediction in cloud, " in Proc. ACM SIGCOMM Conf., 2011, pp. 426-427.
40. M. Ferdman, et al., "Clearing the clouds: A study of emerging scale-out workloads on modern hardware, " in Proc. 17th Int. Conf. Archit. Support Program. Languages Operating Syst., 2012, pp. 37-48.
41. J. Schad, J. Dittrich, and J.-A. Quiane-Ruiz, "Runtime measurements in the cloud: Observing, analyzing, and reducing variance, " Proc. VLDB Endowment, vol. 3, no. 1/2, pp. 460-471, 2010.
42. A. Iosup, S. Ostermann, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema, "Performance analysis of cloud computing services for many-tasks scientific computing, " IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 6, pp. 931-945, Jun. 2011.
43. S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. Epema, "A performance analysis of EC2 cloud computing services for scientific computing, " in Cloud Computing, Berlin, Germany: Springer, 2010, pp 115-131.