Patterns in the chaos-A study of performance variation and predictability in public IaaS clouds

ACM Transactions on Internet Technology

Volumn 16, Issue 3, 2016, Pages

  Leitner, Philipp ^a   Cito, Jürgen ^a  

^a UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

Benchmarking; Infrastructure as a service; Public cloud

Indexed keywords

BENCHMARKING; ELECTROSTATIC ACTUATORS;

CLOUD PERFORMANCE; CLOUD PLATFORMS; EXPERIMENT SET-UP; LITERATURE REVIEWS; MULTI TENANCIES; PERFORMANCE VARIATIONS; PUBLIC CLOUDS; RESEARCH TOPICS;

INFRASTRUCTURE AS A SERVICE (IAAS);

EID: 84971432685     PISSN: 15335399     EISSN: 15576051     Source Type: Journal    
DOI: 10.1145/2885497     Document Type: Article

References (54)

1. Sayaka Akioka and Yoichi Muraoka. 2010. HPC benchmarks on Amazon EC2. In Proceedings of the 2010 IEEE 24th International Conference on Advanced Information Networking and Applications Workshops (WAINA'10). 1029-1034. DOI:http://dx.doi.org/10.1109/WAINA.2010.166.
2. Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia. 2010. A view of cloud computing. Communications of the ACM 53, 4, 50-58.
3. Gültekin Atas and Vehbi Cagri Gungor. 2014. Performance evaluation of cloud computing platforms using statistical methods. Computers and Electrical Engineering 40, 5, 1636-1649. DOI:http://dx.doi.org/10.1016/j.compeleceng.2014.03.017.
4. Sean Kenneth Barker and Prashant Shenoy. 2010. Empirical evaluation of latency-sensitive application performance in the cloud. In Proceedings of the 1st Annual ACM SIGMM Conference on Multimedia Systems (MMSys'10). ACM, New York, NY, 35-46. DOI:http://dx.doi.org/10.1145/1730836.1730842.
5. Amir Hossein Borhani, Philipp Leitner, Bu-Sung Lee, Xiaorong Li, and Terence Hung. 2014. WPress: Benchmarking infrastructure-as-a-service cloud computing systems for on-line transaction processing applications. In Proceedings of the 18th IEEE International Enterprise Distributed Object Computing Conference (EDOC'14).
6. Rajkumar Buyya, Chee Shin Yeo, Srikumar Venugopal, James Broberg, and Ivona Brandic. 2009. Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility. Future Generation Computing Systems 25, 6, 599-616.
7. Davide Cerotti, Marco Gribaudo, Pietro Piazzolla, and Giuseppe Serazzi. 2012. Flexible CPU provisioning in clouds: A new source of performance unpredictability. International Conference on Quantitative Evaluation of Systems 0, 230-237. DOI:http://dx.doi.org/10.1109/QEST.2012.23.
8. Nuttapong Chakthranont, Phonlawat Khunphet, Ryousei Takano, and Tsutomu Ikegami. 2014. Exploring the performance impact of virtualization on an HPC cloud. In Proceedings of the 2014 IEEE 6th International Conference on Cloud Computing Technology and Science (CloudCom). 426-432. DOI:http://dx.doi.org/10.1109/CloudCom.2014.71.
9. Matheus Cunha, Nabor Mendonca, and Americo Sampaio. 2013. A declarative environment for automatic performance evaluation in IaaS clouds. In Proceedings of the 2013 IEEE 6th International Conference on Cloud Computing (CLOUD'13). 285-292. DOI:http://dx.doi.org/10.1109/CLOUD.2013.12.
10. Jiang Dejun, Guillaume Pierre, and Chi-Hung Chi. 2009. EC2 performance analysis for resource provisioning of service-oriented applications. In Proceedings of the 2009 International Conference on Service- Oriented Computing (ICSOC/ServiceWave'09). Springer, Berlin, 197-207. http://dl.acm.org/citation.cfmid=1926618.1926641.
11. Roberto R. Exposito, Guillermo L. Taboada, Xoan C. Pardo, Juan Tourino, and Ramon Doallo. 2013. Running scientific codes on Amazon EC2: A performance analysis of five high-end instances. Computer Science and Technology 13, 3.
12. Benjamin Farley, Ari Juels, Venkatanathan Varadarajan, Thomas Ristenpart, Kevin D. Bowers, and Michael M. Swift. 2012. More for your money: Exploiting performance heterogeneity in public clouds. In Proceedings of the 3rd ACM Symposium on Cloud Computing (SoCC'12). ACM, New York, NY, Article 20, 14 pages. DOI:http://dx.doi.org/10.1145/2391229.2391249.
13. Florian Fittkau, Sören Frey, and Wilhelm Hasselbring. 2012. CDOSim: Simulating cloud deployment options for software migration support. In Proceedings of the 6th IEEE International Workshop on the Maintenance and Evolution of Service-Oriented and Cloud-Based Systems (MESOCA'12). 37-46. DOI:http://dx.doi.org/10.1109/MESOCA.2012.6392599.
14. Marc Frincu, Stephane Genaud, and Julien Gossa. 2014. On the efficiency of several VMprovisioning strategies for workflows with multi-threaded tasks on clouds. Computing 1-28. DOI:http://dx.doi.org/10.1007/s00607-014-0410-0.
15. Devarshi Ghoshal, Richard Shane Canon, and Lavanya Ramakrishnan. 2011. I/O performance of virtualized cloud environments. In Proceedings of the 2nd International Workshop on Data Intensive Computing in the Clouds (DataCloud-SC'11). ACM, New York, NY, 71-80. DOI:http://dx.doi.org/10.1145/2087522.2087535.
16. Lee Gillam, Bin Li, John O'Loughlin, and Anuz Pratap Singh Tomar. 2013. Fair benchmarking for cloud computing systems. Journal of Cloud Computing: Advances, Systems and Applications 2, 6.
17. Lazaros Gkatzikis and Iordanis Koutsopoulos. 2013. Migrate or not Exploiting dynamic task migration in mobile cloud computing systems. IEEE Wireless Communication 20, 3, 24-32.
18. Scott Hazelhurst. 2008. Scientific computing using virtual high-performance computing: A case study using the Amazon elastic computing cloud. In Proceedings of the 2008 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists on IT Research in Developing Countries (SAICSIT'08), Vol. 338. ACM, 94-103.
19. Zach Hill andMarty Humphrey. 2009. A quantitative analysis of high performance computing with Amazon's EC2 infrastructure: The death of the local cluster In GRID. IEEE, 26-33.
20. Zach Hill, Jie Li, Ming Mao, Arkaitz Ruiz-Alvarez, and MartyHumphrey. 2010. Early observations on the performance of Windows Azure. In Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing (HPDC'10). ACM, New York, NY, 367-376. DOI:http://dx.doi.org/10.1145/1851476.1851532.
21. Shigeru Imai, Thomas Chestna, and Carlos A. Varela. 2013. Accurate resource prediction for hybrid IaaS clouds using workload-tailored elastic compute units. In Proceedings of the 6th IEEE/ACMInternational Conference on Utility and Cloud Computing (UCC'13). Dresden, Germany.
22. Alexandru Iosup, Simon Ostermann, Nezih Yigitbasi, Radu Prodan, Thomas Fahringer, and Dick Epema. 2011a. Performance analysis of cloud computing services for many-tasks scientific computing. IEEE Transactions on Parallel and Distributed Systems 22, 6, 931-945. DOI:http://dx.doi.org/10.1109/TPDS.2011.66.
23. Alexandru Iosup, Nezih Yigitbasi, and Dick Epema. 2011b. On the performance variability of production cloud services. In Proceedings of the 2011 11th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid'11). 104-113. DOI:http://dx.doi.org/10.1109/CCGrid.2011.22.
24. Keith R. Jackson, Lavanya Ramakrishnan, Krishna Muriki, Shane Canon, Shreyas Cholia, John Shalf, Harvey J.Wasserman, and Nicholas J. Wright. 2010. Performance analysis of high performance computing applications on the Amazon web services cloud. In Proceedings of the 2010 IEEE 2nd International Conference on Cloud Computing Technology and Science (CLOUDCOM'10). IEEE Computer Society, Washington, DC, 159-168. DOI:http://dx.doi.org/10.1109/CloudCom.2010.69.
25. Deepal Jayasinghe, Galen Swint, Simon Malkowski, Jack Li, Qingyang Wang, Junhee Park, and Calton Pu. 2012. Expertus: A generator approach to automate performance testing in IaaS clouds. In Proceedings of the'12 IEEE 5th International Conference on Cloud Computing (CLOUD'12). IEEE Computer Society, Washington, DC, 115-122. DOI:http://dx.doi.org/10.1109/CLOUD.2012.98.
26. Barbara A. Kitchenham, Tore Dyba, and Magne Jorgensen. 2004. Evidence-based software engineering. In Proceedings of the 26th International Conference on Software Engineering (ICSE'04). IEEE Computer Society, Washington, DC, 273-281. http://dl.acm.org/citation.cfmid=998675.999432.
27. Donald Kossmann, Tim Kraska, and Simon Loesing. 2010. An evaluation of alternative architectures for transaction processing in the cloud. In Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data (SIGMOD'10). ACM, New York, NY, 579-590. DOI:http://dx.doi.org/10.1145/1807167.1807231.
28. LarsKotthoff. 2014. Reliability of computational experiments on virtualised hardware. Journal ofExperimental& Theoretical Artificial Intelligence 26, 1 33-49. DOI:http://dx.doi.org/10.1080/0952813X.2013.784812.
29. Philipp Leitner, Benjamin Satzger, Waldemar Hummer, Christian Inzinger, and Schahram Dustdar. 2012. CloudScale: A novel middleware for building transparently scaling cloud applications. In Proceedings of the 27th Annual ACM Symposium on Applied Computing (SAC'12). 434-440. http://doi.acm.org/10.1145/2245276.2245360.
30. Philipp Leitner and Joel Scheuner. 2015. Bursting with possibilities-an empirical study of credit-based bursting cloud instance types. In Proceedings of the 8th IEEE/ACM International Conference on Utility and Cloud Computing (UCC'15).
31. Alexander Lenk, Michael Menzel, Johannes Lipsky, Stefan Tai, and Philipp Offermann. 2011. What are you paying for Performance benchmarking for infrastructure-as-a-service offerings. In IEEE CLOUD, Ling Liu and Manish Parashar (Eds.). 484-491.
32. Ang Li, Xiaowei Yang, Srikanth Kandula, and Ming Zhang. 2010. CloudCmp: Comparing public cloud providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement (IMC'10). ACM, New York, NY, 1-14. DOI:http://dx.doi.org/10.1145/1879141.1879143.
33. Zheng Li, Liam O'Brien, Rajiv Ranjan, and Miranda Zhang. 2013b. Early observations on performance of Google compute engine for scientific computing. In Proceedings of the 2013 IEEE International Conference on Cloud Computing Technology and Science (CLOUDCOM'13). IEEE Computer Society, Washington, DC, 1-8. DOI:http://dx.doi.org/10.1109/CloudCom.2013.7.
34. Zheng Li, L. O'Brien, and He Zhang. 2013a. CEEM: A practical methodology for cloud services evaluation. In Proceedings of the 2013 IEEE 9th World Congress on Services (SERVICES'13). 44-51. DOI:http://dx.doi.org/10.1109/SERVICES.2013.73.
35. Zheng Li, Liam O'Brien, He Zhang, and Rainbow Cai. 2012. On a catalogue of metrics for evaluating commercial cloud services. In Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing (GRID'12). IEEE Computer Society, Washington, DC, 164-173. DOI:http://dx.doi.org/10.1109/Grid.2012.15.
36. Zheng Li, Liam O'Brien, He Zhang, and Rainbow Cai. 2013. On the conceptualization of performance evaluation of IaaS services. IEEE Transactions on Services Computing.
37. Ming Mao and Marty Humphrey. 2012. A performance study on the VM startup time in the cloud. In Proceedings of the 2012 IEEE 5th International Conference on Cloud Computing (CLOUD'12). IEEE Computer Society, Washington, DC, 423-430. DOI:http://dx.doi.org/10.1109/CLOUD.2012.103.
38. Piyush Mehrotra, Jahed Djomehri, Steve Heistand, Robert Hood, Haoqiang Jin, Arthur Lazanoff, Subhash Saini, and Rupak Biswas. 2012. Performance evaluation of Amazon EC2 for NASA HPC applications. In Proceedings of the 3rd Workshop on Scientific Cloud Computing (ScienceCloud'12). ACM, New York, NY, 41-50. DOI:http://dx.doi.org/10.1145/2287036.2287045.
39. John OLoughlin and Lee Gillam. 2013. Towards performance prediction for public infrastructure clouds: An EC2 case study. Proceedings of the 2013 IEEE 5th International Conference on Cloud Computing Technology and Science (CloudCom'13) 1, 475-480. DOI:http://dx.doi.org/10.1109/CloudCom.2013.69.
40. Zhonghong Ou, Hao Zhuang, Andrey Lukyanenko, Jukka K. Nurminen, Pan Hui, Vladimir Mazalov, and Antti Yla-Jaaski. 2013. Is the same instance type created equal Exploiting heterogeneity of public clouds. IEEE Transactions on Cloud Computing 1, 2, 201-214. DOI:http://dx.doi.org/10.1109/TCC.2013.12.
41. Stephen C. Phillips, Vegard Engen, and Juri Papay. 2011. Snow white clouds and the seven dwarfs. In Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science (CLOUDCOM'11). IEEE Computer Society, Washington, DC, 738-745. DOI:http://dx.doi.org/10.1109/CloudCom.2011.114.
42. Iman Sadooghi, Jesus Hernandez Martin, Tonglin Li, Kevin Brandstatter, Ketan Maheshwari, Tiago Pais Pitta de Lacerda Ruivo, Gabriele Garzoglio, Steven Timm, Yong Zhao, and Ioan Raicu. 2015. Understanding the performance and potential of cloud computing for scientific applications. IEEE Transactions on Cloud Computing PP, 99. DOI:http://dx.doi.org/10.1109/TCC.2015.2404821.
43. Jorg Schad, Jens Dittrich, and Jorge-Arnulfo Quiańe-Ruiz. 2010. Runtime measurements in the cloud: Observing, analyzing, and reducing variance. Proceedings of the VLDB Endowment 3, 1-2, 460-471. http://dl.acm.org/citation.cfmid=1920841.1920902.
44. Joel Scheuner, Jü rgen Cito, Philipp Leitner, and Harald C. Gall. 2015. Cloud workbench: Benchmarking IaaS providers based on infrastructure-as-code. In Proceedings of the 24th InternationalWorldWideWeb Conference (WWW 2015)-Companion Volume. 239-242. DOI:http://dx.doi.org/10.1145/2740908.2742833.
45. Joel Scheuner, Philipp Leitner, Jü rgen Cito, and Harald Gall. 2014. Cloud workbench-infrastructure-ascode based cloud benchmarking. In Proceedings of the 6th IEEE International Conference on Cloud Computing Technology and Science (CloudCom'14).
46. Nicolas Serrano, Gorka Gallardo, and Josune Hernantes. 2015. Infrastructure as a service and cloud technologies. IEEE Software 32, 2, 30-36. DOI:http://dx.doi.org/10.1109/MS.2015.43.
47. Ignacio Silva-Lepe, Revathi Subramanian, Isabelle Rouvellou, Thomas Mikalsen, Judah Diament, and Arun Iyengar. 2008. SOAlive service catalog: A simplified approach to describing, discovering and composing situational enterprise services. In Proceedings of the 6th International Conference on Service-Oriented Computing (ICSOC'08). Springer, Berlin, 422-437. DOI:http://dx.doi.org/10.1007/978-3-540-89652-4-32.
48. Will Sobel, Shanti Subramanyam, Akara Sucharitakul, Jimmy Nguyen, Hubert Wong, Arthur Klepchukov, Sheetal Patil, Armando Fox, and David Patterson. 2008. Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0. In Cloud Computing and Its Applications.
49. Venkatanathan Varadarajan, Thawan Kooburat, Benjamin Farley, Thomas Ristenpart, and Michael M. Swift. 2012. Resource-freeing attacks: Improve your cloud performance (at your neighbor's expense). In Proceedings of the 2012 ACM Conference on Computer and Communications Security (CCS'12). ACM, New York, NY, 281-292. DOI:http://dx.doi.org/10.1145/2382196.2382228.
50. Edward Walker. 2008. Benchmarking Amazon EC2 for high-performance scientific computing. LOGIN 33, 5, 18-23.
51. Guohui Wang and T. S. Eugene Ng. 2010. The impact of virtualization on network performance of Amazon EC2 data center. In Proceedings of the 29th Conference on Information Communications (INFOCOM'10). IEEE Press, Piscataway, NJ, 1163-1171. http://dl.acm.org/citation.cfmid=1833515.1833691.
52. ZhilinWang, Xinhuai Tang, and Xiangfeng Luo. 2011. Policy-based SLA-aware cloud service provision framework. In Proceedings of the 7th International Conference on Semantics Knowledge and Grid (SKG'11). 114-121.
53. Jiawei Wen, Lei Lu, Giuliano Casale, and Evgenia Smirni. 2015. Less can be more: Micro-managing VMs in Amazon EC2. In Proceedings of the 8th IEEE International Conference on Cloud Computing (CLOUD). 317-324. DOI:http://dx.doi.org/10.1109/CLOUD.2015.50.
54. Rostyslav Zabolotnyi, Philipp Leitner, Waldemar Hummer, and Schahram Dustdar. 2015. JCloudScale: Closing the Gap Between IaaS and PaaS. ACM Transactions on Internet Technology 15, 3, Article 10 (July 2015), 20 pages. DOI:http://dx.doi.org/10.1145/2792980.