On the conceptualization of performance evaluation of IaaS services

IEEE Transactions on Services Computing

Volumn 7, Issue 4, 2014, Pages 628-641

  Li, Zheng ^a   O'Brien, Liam ^b   Zhang, He ^c   Cai, Rainbow ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b GEOSCIENCE AUSTRALIA   (Australia)

^c NANJING UNIVERSITY   (China)

Author keywords

Conceptual model; Conceptualization; Infrastructure as a service (iaas); Performance evaluation; Taxonomy

Indexed keywords

BLUEPRINTS; COST BENEFIT ANALYSIS; COST EFFECTIVENESS; TAXONOMIES;

COMPUTING PARADIGM; CONCEPTUAL MODEL; CONCEPTUALIZATION; DELIVERY MODELS; DIRECTION OF IMPROVEMENT; EVALUATION EXPERIMENTS; NATURAL LANGUAGES; PERFORMANCE EVALUATION;

INFRASTRUCTURE AS A SERVICE (IAAS);

EID: 84919764907     PISSN: 19391374     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSC.2013.39     Document Type: Article

References (51)

1. S. Akioka and Y. Muraoka, "HPC Benchmarks on Amazon EC2," in Proc. 24th Int. IEEE Conf. WAINA, Apr. 2010, pp. 1029-1034.
2. G.D. Alesandre, Updated App Engine Pricing FAQ!, Google Groups, June 2011. [Online]. Available: https://groups.google.com/forum/#!msg/google-appengine/Hluog1-a3n4/uFMhaBWhVi8J.
3. J. Antony, Design of Experiments for Engineers and Scientists. Burlington, MA, USA: Butterworth-Heinemann, Nov. 2003.
4. M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, "A View of Cloud Computing," Commun. ACM, vol. 53, no. 4, pp. 50-58, Apr. 2010.
5. P. Barringer and T. Bennett, "Availability is not equal to reliability," Barringer & Assoc., Inc., Humble, TX, USA, July 2003. [Online]. Available: http://www.barringer1.com/ar.htm.
6. C. Baun and M. Kunze, "Performance Measurement of a Private Cloud in the Opencirrus Testbed," in Proc. 4th Workshop VHPC Conjunction 15th Int. Euro-Par Conf. Distrib. Comput., Aug. 2009, pp. 434-443.
7. P. Bientinesi, R. Iakymchuk, and J. Napper, "HPC on Competitive Cloud Resources," in Handbook of Cloud Computing, B. Furht and A. Escalante, Eds. New York, NY, USA: Springer-Verlag, 2010, pp. 493-516.
8. C. Binnig, D. Kossmann, T. Kraska, and S. Loesing, "How is the Weather Tomorrow? Towards a Benchmark for the Cloud," in Proc. 2nd Int. Workshop DBTest Syst. Conjunction Int. Conf. ACM SIGMOD/PODPS, June 2009, pp. 1-6.
9. A.B. Bondi, "Characteristics of Scalability and Their Impact on Performance," in Proc. 2nd Int. WOSP, Sept. 2000, pp. 195-203.
10. M. Bubak, M. Kasztelnik, M. Malawski, J. Meizner, P. Nowakowski, and S. Varma, "Evaluation of Cloud Providers for VPH Applications," in Proc. 13th IEEE/ACM Int. Symp. CCGrid, May 2013, pp. 200-201.
11. R. Buyya, C.S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, "Cloud Computing and Emerging IT Platforms: Vision, Hype, Reality for Delivering Computing as the 5th Utility," Future Gen. Comput. Syst., vol. 25, no. 6, pp. 599-616, June 2009.
12. A.G. Carlyle, S.L. Harrell, and P.M. Smith, "Cost-Effective HPC: The Community or the Cloud?" in Proc. 2nd IEEE Int. Conf. CloudCom Technol. Sci., Nov./Dec. 2010, pp. 169-176.
13. Amazon Elastic Cloud Computing (EC2), Jan. 2014. [Online]. Available: http://aws.amazon.com/ec2/.
14. D. Chiu and G. Agrawal, "Evaluating Caching and Storage Options on the Amazon Web Services Cloud," in Proc. 11th ACM/ IEEE Int. Conf. GRID Comput., Oct. 2010, pp. 17-24.
15. E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, "The Cost of Doing Science on the Cloud: The Montage Example," in Proc. Int. Conf. High Perform. Comput., Netw., Storage Anal. SC, Nov. 2008, pp. 1-12.
16. J. Dejun, G. Pierre, and C.-H. Chi, "EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications," in Proc. ICSOC/ServiceWave, Nov. 2009, pp. 197-207.
17. P.J. Fortier and H.E. Michel, Computer Systems Performance Evaluation and Prediction. Burlington, MA, USA: Digital Press, July 2003.
18. S. Garfinkel, "An evaluation of Amazon's grid computing services: EC2, S3 and SQS," School Eng. Appl. Sci., Harvard Univ., Cambridge, MA, USA, Tech. Rep. TR-08-07, July 2007.
19. Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, "Case Study for Running HPC Applications in Public Clouds," in Proc. 19th ACM Int. Symp. HPDC, June 2010, pp. 395-401.
20. Z. Hill and M. Humphrey, "A Quantitative Analysis of High Performance Computing with Amazon's EC2 Infrastructure: The Death of the Local Cluster?" in Proc. 10th IEEE/ACM Int. Conf. GRID Comput., Oct. 2009, pp. 26-33.
21. Z. Hill, J. Li, M. Mao, A. Ruiz-Alvarez, and M. Humphrey, "Early Observations on the Performance of Windows Azure," in Proc. 19th ACM Int. Symp. HPDC, June 2010, pp. 367-376.
22. D. Harris, "Watch Out, World: IBM Finally Offers a Real Cloud," in GigaOM - Tech News, Analysis and Trends, Apr. 2011. [Online]. Available: http://gigaom.com/cloud/watch-out-world-ibm-finally-offers-a-real-cloud/.
23. D. Harris, "What Google Compute Engine Means for Cloud Computing," in GigaOM - Tech News, Analysis and Trends, June 2012. [Online]. Available: http://gigaom.com/cloud/what-google-compute-engine-means-for-cloud-computing/.
24. A. Iosup, N. Yigitbasi, and D. Epema, "On the Performance Variability of Production Cloud Services," Delft Univ. Technol., Delft, The Netherlands, Tech. Rep. PDS-2010-002, Jan. 2010.
25. K.R. Jackson, L. Ramakrishnan, K. Muriki, S. Canon, S. Cholia, J. Shalf, H.J. Wasserman, and N.J. Wright, "Performance Analysis of High Performance Computing Applications on the Amazon Web Services Cloud," in Proc. 2nd IEEE Int. Conf. CloudCom Technol. Sci., Nov./Dec. 2010, pp. 159-168.
26. R.K. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, Modeling. Hoboken, NJ, USA: Wiley, May 1991.
27. G. Juve, E. Deelman, K. Vahi, G. Mehta, B. Berriman, B.P. Berman, and P. Maechling, "Scientific Workflow Applications on Amazon EC2," in Proc. Workshop Cloud-Based Services Appl. 5th IEEE Int. Conf. e-Sci., Dec. 2009, pp. 59-66.
28. B.A. Kitchenham and S. Charters, "Guidelines for performance systematic literature reviews in software engineering," Keele Univ. and Univ. Durham, Keele/Durham, U.K., EBSE Tech. Rep. EBSE-2007-01, July 2007, ver. 2.3.
29. D. Kondo, B. Javadi, P. Malecot, F. Cappello, and D.P. Anderson, "Cost-Benefit Analysis of Cloud Computing Versus Desktop Grids," in Proc. 23rd IEEE IPDPS, May 2009, pp. 1-12.
30. J.-Y. Le Boudec, Performance Evaluation of Computer and Communication Systems. Lausanne, Switzerland: EFPL Press, Feb. 2011.
31. J. Li, M. Humphrey, D. Agarwal, K. Jackson, C. van Ingen, and Y. Ryu, "eScience in the Cloud: A MODIS Satellite Data Reprojection and Reduction Pipeline in the Windows Azure Platform," in Proc. 23rd IEEE IPDPS, Apr. 2010, pp. 1-10.
32. Z. Li, L. O'Brien, R. Cai, and H. Zhang, "Towards a Taxonomy of Performance Evaluation of Commercial Cloud Services," in Proc. 5th Int. Conf. IEEE CLOUD Comput., June 2012, pp. 344-351.
33. Z. Li, L. O'Brien, H. Zhang, and R. Cai, "On a Catalogue of Metrics for Evaluating Commercial Cloud Services," in Proc. 13th ACM/IEEE Int. Conf. GRID Comput., Sept. 2012, pp. 164-173.
34. A. Li, X. Yang, S. Kandula, and M. Zhang, "CloudCmp: Comparing Public Cloud Providers," in Proc. 10th Annu. IMC, Nov. 2010, pp. 1-14.
35. Z. Li, H. Zhang, L. O'Brien, R. Cai, and S. Flint, "On Evaluating Commercial Cloud Services: A Systematic Review," J. Syst. Softw., vol. 86, no. 9, pp. 2371-2393, Sept. 2013.
36. A. Luckow and S. Jha, "Abstractions for Loosely-Coupled and Ensemble-Based Simulations on Azure," in Proc. 2nd IEEE Int. Conf. CloudCom Technol. Sci., Nov./Dec. 2010, pp. 550-556.
37. S.J. Mellor, A.N. Clark, and T. Futagami, "Model-Driven Development - Guest Editor's Introduction," IEEE Softw., vol. 20, no. 5, pp. 14-18, Sept./Oct. 2003.
38. R. Miller, "A Look Inside Amazon's Data Centers," Data Center Knowledge, Lawrenceville, NJ, USA, June 2011. [Online]. Available: http://www.datacenterknowledge.com/archives/2011/06/09/a-look-inside-amazons-data-centers/.
39. D.C. Montgomery, Design and Analysis of Experiments 7th ed., Hoboken, NJ, USA: Wiley, Jan. 2009.
40. J. Napper and P. Bientinesi, "Can Cloud Computing Reach the Top500?" in Proc. Combined Workshops UCHPC-MAW, May 2009, pp. 17-20.
41. OMG Unified Modeling Language (OMG UML), Infrastructure, ver. 2.4.1, Object Management Group, Needham, MA, USA, Aug. 2011. [Online]. Available: http://www.omg.org/spec/UML/2.4.1/Infrastructure.
42. S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D.H.J. Epema, "A Performance Analysis of EC2 Cloud Computing Services for Scientific Computing," in Proc. 1st Int. Conf. CloudComp, Oct. 2009, pp. 115-131.
43. R. Prodan and S. Ostermann, "A Survey and Taxonomy of Infrastructure as a Service and Web Hosting Cloud Providers," in Proc. 10th IEEE/ACM Int. Conf. GRID Comput., Oct. 2009, pp. 17-25.
44. W. Sobel, S. Subramanyam, A. Sucharitakul, J. Nguyen, H. Wong, A. Klepchukov, S. Patil, A. Fox, and D. Patterson, "Cloudstone: Multi-Platform, Multi-Language Benchmark and Measurement Tools for Web 2.0," in Proc. 1st Workshop CCA, Oct. 2008, pp. 1-6.
45. V. Stantchev, "Performance Evaluation of Cloud Computing Offerings," in Proc. 3rd Int. Conf. ADVCOMP Appl. Sci., Oct. 2009, pp. 187-192.
46. J. Stokes, "The PC is Order, the Cloud is Chaos," in Wired Cloudline, Dec. 2011. [Online]. Available: http://www.wired.com/cloudline/2011/12/the-pc-is-order/.
47. W.A. Thompson, "On the Foundations of Reliability," Technometrics, vol. 23, no. 1, pp. 1-13, Feb. 1981.
48. R. Tudoran, A. Costan, G. Antoniu, and L. Bougé, "A Performance Evaluation of Azure and Nimbus Clouds for Scientific Applications," in Proc. 2nd Int. Workshop CloudCP Conjunction ACM SIGOPS EuroSys 2012, Apr. 2012, pp. 1-6.
49. N. Yigitbasi, A. Iosup, D. Epema, and S. Ostermann, "C-meter: A Framework for Performance Analysis of Computing Clouds," in Proc. 9th IEEE/ACM Int. Symp. CCGRID, May 2009, pp. 472-477.
50. H. Zhang and M.A. Babar, "An Empirical Investigation of Systematic Reviews in Software Engineering," in Proc. 5th Int. Symp. ESEM, Sept. 2011, pp. 1-10.
51. Q. Zhang, L. Cheng, and R. Boutaba, "Cloud Computing: State-of-the-Art and Research Challenges," J. Internet Services Appl., vol. 1, no. 1, pp. 7-18, May 2010.