On a catalogue of metrics for evaluating commercial cloud services

Proceedings - IEEE/ACM International Workshop on Grid Computing

Volumn , Issue , 2012, Pages 164-173

  Li, Zheng ^a   O'Brien, Liam ^b   Zhang, He ^c   Cai, Rainbow ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b CSIRO   (Australia)

^c UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Catalogue; Cloud computing; Cloud services evaluation; Commercial cloud service; Evaluation metrics

Indexed keywords

BENCHMARK SELECTION; CATALOGUE; CLOUD SERVICES; DIFFERENT SERVICES; EVALUATION METRICS; SYSTEMATIC LITERATURE REVIEW;

CLOUD COMPUTING; DISTRIBUTED DATABASE SYSTEMS; WEB SERVICES;

GRID COMPUTING;

EID: 84869054424     PISSN: 15505510     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/Grid.2012.15     Document Type: Conference Paper

References (58)

1. R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, "Cloud Computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility," Future Gener. Comp. Sy., vol. 25, no. 6, Jun. 2009, pp. 599-616.
2. R. Prodan and S. Ostermann, "A survey and taxonomy of Infrastructure as a Service and Web hosting Cloud providers," Proc. 10th IEEE/ACM Int. Conf. Grid Computing (Grid 2009), IEEE Computer Society, Oct. 2009, pp. 17-25.
3. W. Sobel, S. Subramanyam, A. Sucharitakul, J. Nguyen, H. Wong, A. Klepchukov, S. Patil, A. Fox, and D. Patterson, "Cloudstone: Multiplatform, multi-language benchmark and measurement tools for web 2.0," Proc. 1st Workshop on Cloud Computing and Its Applications (CCA 2008), Oct. 2008, pp. 1-6.
4. S. Akioka and Y. Muraoka, "HPC benchmarks on Amazon EC2," Proc. 24th Int. IEEE Conf. Advanced Information Networking and Applications Workshops (WAINA 2010), IEEE Computer Society, Apr. 2010, pp. 1029-1034.
5. A. Li, X. Yang, S. Kandula, and M. Zhang, "CloudCmp: Comparing public Cloud providers," Proc. 10th Annu. Conf. Internet Measurement (IMC 2010), ACM Press, Nov. 2010, pp. 1-14.
6. J. Dejun, G. Pierre, and C.-H. Chi, "EC2 performance analysis for resource provisioning of service-oriented applications," Proc. 2009 Int. Conf. Service-Oriented Computing (ICSOC/ServiceWave 2009), Springer-Verlag, Nov. 2009, pp. 197-207.
7. J. Wang, P. Varman, and C. Xie, "Avoiding performance fluctuation in cloud storage," Proc. 17th Annu. IEEE Int. Conf. High Performance Computing (HiPC 2010), IEEE Computer Society, Dec. 2010, pp. 1-9.
8. Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, "Case study for running HPC applications in public clouds," Proc. 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp.395-401.
9. S. Garfinkel, "An evaluation of Amazon's Grid Computing services: EC2, S3 and SQS," Sch. Eng. Appl. Sci., Harvard Univ., Canbridge, MA, Tech. Rep. TR-08-07, Jul. 2007.
10. P. Barringer and T. Bennett, "Availability is not equal to Reliability," Barringer & Associates, Inc., Jul. 21, 2003. [online]. Available: http://www.barringer1.com/ar.htm.
11. S. Islam, K. Lee, A. Fekete, and A. Liu, "How a consumer can measure elasticity for Cloud platforms," Proc. 3rd ACM/SPEC Int. Conf. Performance Engineering (ICPE 2012), ACM Press, Apr. 2012, to appear.
12. A. Iosup, N. Yigitbasi, and D. Epema, "On the performance variability of production Cloud services," Delft Univ. Technol., Netherlands, Tech. Rep. PDS-2010-002, Jan. 2010.
13. D. Chiu and G. Agrawal, "Evaluating caching and storage options on the Amazon Web Services Cloud," Proc. 11th ACM/IEEE Int. Conf. Grid Computing (Grid 2010), IEEE Computer Society, Oct. 2010, pp. 17-24.
14. J. E. Kee, "At what price? Benefit-cost analysis and cost-effectiveness analysis in program evaluation," The Evaluation Exchange, vol. 5, no. 2&3, 1999, pp. 4-5.
15. J. Li, M. Humphrey, D. Agarwal, K. Jackson, C. van Ingen, and Y. Ryu, "eScience in the Cloud: A MODIS satellite data reprojection and reduction pipeline in the Windows Azure platform," Proc. 23rd IEEE Int. Symp. Parallel and Distributed Processing (IPDPS 2010), IEEE Computer Society, Apr. 2010, pp. 1-10.
16. N. Yigitbasi, A. Iosup, D. Epema, and S. Ostermann, "C-Meter: A framework for performance analysis of computing Clouds," Proc. 9th IEEE/ACM Int. Symp. Cluster Computing and the Grid (CCGRID 2009), IEEE Computer Society, May 2009, pp. 472-477.
17. P. Bientinesi, R. Iakymchuk, and J. Napper, "HPC on competitive Cloud resources," in Handbook of Cloud Computing, B. Furht and A. Escalante, Eds. New York: Springer-Verlag, 2010, pp. 493-516.
18. Z. Hill and M. Humphrey, "A quantitative analysis of high performance computing with Amazon's EC2 infrastructure: The death of the local cluster?," Proc. 10th IEEE / ACM Int. Conf. Grid Computing (Grid 2009), IEEE Computer Society, Oct. 2009, pp. 26-33.
19. T. A. Hjeltnes and B. Hansson, Cost Effectiveness and Cost Efficiency in E-learning, Trondheim, Norway: The TISIP Foundation, 2005.
20. Z. Hill, J. Li, M. Mao, A. Ruiz-Alvarez, and M. Humphrey, "Early observations on the performance of Windows Azure," Proc. 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp.367-376.
21. C. Binnig, D. Kossmann, T. Kraska, and S. Loesing, "How is the weather tomorrow? Towards a benchmark for the Cloud," Proc. 2nd Int. Workshop on Testing Database Systems (DBTest 2009) in conjunction with ACM SIGMOD/PODPS Int. Conf. Management of Data (SIGMOD/PODS 2009), ACM Press, Jun. 2009, pp. 1-6.
22. H. Liu and S. Wee, "Web server farm in the Cloud: Performance evaluation and dynamic architecture," Proc. 1st Int. Conf. Cloud Computing (CloudCom 2009), Springer-Verlag, Dec. 2009, pp. 369-380.
23. S. Hazelhurst, "Scientific computing using virtual high-performance computing: a case study using the Amazon elastic computing cloud," Proc. 2008 Annu. Research Conf. South African Institute of Computer Scientists and Information Technologists (SAICSIT 2008), ACM Press, Oct. 2008, pp. 94-103.
24. G. Juve, E. Deelman, K. Vahi, G. Mehta, B. Berriman, B. P. Berman, and P. Maechling, "Scientific workflow applications on Amazon EC2," Proc. Workshop on Cloud-based Services and Applications in conjunction with the 5th IEEE Int. Conf. e-Science (e-Science 2009), IEEE Computer Society, Dec. 2009, pp. 59-66.
25. S. Garfinkel, "Commodity grid computing with Amazon's S3 and EC2," ;Login, vol. 32, no. 1, Feb. 2007, pp. 7-13.
26. H. Zhang and M. Ali Babar, "An empirical investigation of systematic reviews in software engineering," Proc. 5th Int. Symp. Empirical Software Engineering and Measurement (ESEM 2011), IEEE Computer Society, 1-10.
27. B. A. Kitchenham, "Guidelines for performance systematic literature reviews in software engineering," Keele Univ. and Univ. Durham, UK, EBSE Tech. Rep. EBSE-2007-01, Ver. 2.3, Jul. 2007.
28. Q. Zhang, L. Cheng, and R. Boutaba, "Cloud computing: State-of-the-art and research challenges," J. Internet Serv. Appl., vol. 1, no. 1, May 2010, pp. 7-18.
29. D. Kossmann, T. Kraska, and S. Loesing, "An evaluation of alternative architectures for transaction processing in the Cloud," Proc. 2010 ACM SIGMOD Int. Conf. Management of Data (SIGMOD 2010), ACM Press, Jun. 2010, pp. 579-590.
30. K. R. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H. J. Wasserman, and N. J. Wright, "Performance analysis of high performance computing applications on the Amazon Web services Cloud," Proc. 2nd IEEE Int. Conf. Cloud Computing Technology and Science (CloudCom 2010), IEEE Computer Society, Nov. - Dec. 2010, pp. 159-168.
31. R. K. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. New York, NY: Wiley Computer Publishing, John Wiley & Sons, Inc., May 1991.
32. M. S. Obaidat and N. A. Boudriga, Fundamentals of Performance Evaluation of Computer and Telecommjnication Systems. Hoboken, New Jersey: John Wiley & Sons, Inc., Jan. 2010.
33. M. D. de Assunção, A. di Costanzo, and R. Buyya, "Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters," Proc. 18th ACM Int. Symp. High Performance Distributed Computing (HPDC 2009), ACM Press, Jun. 2009, pp. 141-150.
34. B. F. Cooper, A. Silberstein, E. Tam, R. Ramakrishnan, and R. Sears, "Benchmarking Cloud serving systems with YCSB," Proc. ACM Symp. Cloud Computing (SoCC 2010), Jun. 2010, pp. 143-154.
35. M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "A view of Cloud computing," Commun. ACM, vol. 53, no. 4, Apr. 2010, pp. 50-58.
36. D. Kossmann and T. Kraska, "Data management in the Cloud: Promises, state-of-the-art, and open questions," Datenbank Spektr., vol. 10, no. 3, Nov. 2010, pp. 121-129.
37. J. Wilkening, A. Wilke, N. Desai, and F. Meyer, "Using Clouds for metagenomics: A case study," Proc. 2009 IEEE Int. Conf. Cluster Computing (Cluster 2009), IEEE Computer Society, Aug.-Sept. 2009, pp. 1-6.
38. M. R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, "Amazon S3 for science grids: a viable solution?," Proc. 2008 Int Workshop on Data-Aware Distributed Computing (DADC 2008), ACM Press, Jun. 2008, pp. 55-64.
39. D. Kondo, B. Javadi, P. Malecot, F. Cappello, and D. P. Anderson, "Cost-benefit analysis of Cloud Computing versus desktop grids," Proc. 23rd IEEE Int. Symp. Parallel and Distributed Processing (IPDPS 2009), IEEE Computer Society, May 2009, pp. 1-12.
40. A. Luckow and S. Jha, "Abstractions for loosely-coupled and ensemble-based simulations on Azure," Proc. 2nd IEEE Int. Conf. Cloud Computing Technology and Science (CloudCom 2010), IEEE Computer Society, Nov.-Dec. 2010, pp. 550-556.
41. A. G. Carlyle, S. L. Harrell, and P. M. Smith, "Cost-effective HPC: The community or the Cloud?," Proc. 2nd IEEE Int. Conf. Cloud Computing Technology and Science (CloudCom 2010), IEEE Computer Society, Nov.-Dec. 2010, pp. 169-176.
42. S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D. H. J. Epema, "A performance analysis of EC2 Cloud computing services for scientific computing," Proc. 1st Int. Conf. Cloud Computing (CloudComp 2009), Springer-Verlag, Oct. 2009, pp. 115-131.
43. G. Wang and T. S. E. Ng, "The impact of virtualization on network performance of Amazon EC2 data center," Proc. 29th Annu. IEEE Int. Conf. Computer Communications (IEEE INFOCOM 2010), IEEE Computer Society, Mar. 2010, pp. 1-9.
44. C. Evangelinos and C. N. Hill, "Cloud computing for parallel scientific HPC applications: Feasibility of running coupled atmosphere-ocean climate models on Amazon's EC2," Proc. 1st Workshop on Cloud Computing and Its Applications (CCA 2008), Oct. 2008, pp. 1-6.
45. L. Zhao, A. Liu, and J. Keung, "Evaluating Cloud platform architecture with the CARE framework," Proc. 17th Asia Pacific Software Engineering Conference (APSEC 2010), IEEE Computer Society, Nov.-Dec. 2010, pp. 60-69.
46. C. Baun and M. Kunze, "Performance measurement of a private cloud in the OpenCirrus™ testbed," Proc. 4th Workshop on Virtualization in High-Performance Cloud Computing (VHPC 2009) in conjunction with 15th Int. European Conf. Parallel and Distributed Computing (Euro-Par 2009), Springer-Verlag, Aug. 2009, pp. 434-443.
47. J. Schad, J. Dittrich, and J.-A. Quiané-Ruiz, "Runtime measurements in the Cloud: observing, analyzing, and reducing variance," VLDB Endowment, vol. 3, no. 1-2, Sept. 2010, pp. 460-471.
48. J. J. Rehr, F. D. Vila, J. P. Gardner, L. Svec, and M. Prange, "Scientific Computing in the Cloud," Comput. Sci. Eng., vol. 12, no. 3, May-Jun. 2010, pp. 34-43.
49. V. Stantchev, "Performance evaluation of Cloud computing offerings," Proc. 3rd Int. Conf. Advanced Engineering Computing and Applications in Sciences (ADVCOMP 2009), IEEE Computer Society, Oct. 2009, pp. 187-192.
50. T. Dalman, T. Dörnemann, E. Juhnke, M. Weitzel, M. Smith, W. Wiechert, K. Nöh, and B. Freisleben, "Metabolic flux analysis in the Cloud," Proc. IEEE 6th Int. Conf. e-Science (e-Science 2010), IEEE Computer Society, Dec. 2010, pp. 57-64.
51. E. Walker, "Benchmarking Amazon EC2 for high-performance scientific computing," ;Login, vol. 33, no. 5, Oct. 2008, pp. 18-23.
52. W. Lu, J. Jackson, and R. Barga, "AzureBlast: A case study of developing science applications on the Cloud," Proc. 1st Workshop on Scientific Cloud Computing (ScienceCloud 2010) in conjunction with 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp. 413-420.
53. M. Brantner, D. Florescu, D. Graf, D. Kossmann, and T. Kraska, "Building a database on S3," Proc. 2008 ACM SIGMOD Int. Conf. Management of Data (SIGMOD 2008), ACM Press, Jun. 2008, pp. 251-264.
54. E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, "The cost of doing science on the cloud: The Montage example," Proc. 2008 Int. Conf. High Performance Computing, Networking, Storage and Analysis (SC 2008), IEEE Computer Society, Nov. 2008, pp. 1-12.
55. D. P. Wall, P. Kudtarkar, V. A. Fusaro, R. Pivovarov, P. Patil, and P. J. Tonellato, "Cloud computing for comparative genomics," BMC Bioinformatics, vol. 11:259, May 2010.
56. J. Stokes, "The PC is order, the Cloud is chaos," Wired Cloudline, Wired.com, Dec. 19, 2011. [online]. Available: http://www.wired.com/ cloudline/2011/12/the-pc-is-order/.
57. Z. Li, L. O'Brien, R. Cai, and H. Zhang, "Towards a taxonomy of performance evaluation of commercial Cloud services," Proc. 5th Int. Conf. Cloud Computing (IEEE CLOUD 2012), IEEE Computer Society, Jun. 2012, to appear.
58. C. Brooks, "Cloud computing benchmarks on the rise," SearchCloudComputing, TechTarget.com, Jun. 10, 2010. [online]. Available: http://searchcloudcomputing.techtarget.com/news/1514547/Cloud-computing- benchmarks-on-the-rise.