A factor framework for experimental design for performance evaluation of commercial cloud services

CloudCom 2012 - Proceedings: 2012 4th IEEE International Conference on Cloud Computing Technology and Science

Volumn , Issue , 2012, Pages 169-176

  Li, Zheng ^a   O'Brien, Liam ^b   Zhang, He ^a   Cai, Rainbow ^b  

^a CSIRO   (Australia)

^b AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

Cloud Computing; commercial Cloud services; experimental design; factor frame work; performance evaluation

Indexed keywords

CLOUD SERVICES; COMPUTING DOMAIN; EVALUATION EXPERIMENTS; EVALUATION STUDY; EXPERIMENTAL FACTORS; FRAME-WORK; PERFORMANCE EVALUATION; SERVICE IMPROVEMENT;

CLOUD COMPUTING; DESIGN OF EXPERIMENTS; DISTRIBUTED DATABASE SYSTEMS; EXPERIMENTS; STATISTICS;

WEB SERVICES;

EID: 84874268719     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CloudCom.2012.6427525     Document Type: Conference Paper

References (33)

1. J. Antony, Design of Experiments for Engineers and Scientists. Burlington, MA: Butterworth-Heinemann, Nov. 2003.
2. C. Baun and M. Kunze, "Performance Measurement of a Private Cloud in the OpenCirrusTM Testbed," Proc. 4th Workshop on Virtualization in High-Performance Cloud Computing (VHPC 2009) in conjunction with 15th Int. European Conf. Parallel and Distributed Computing (Euro-Par 2009), Springer-Verlag, Aug. 2009, pp. 434-443.
3. P. Bientinesi, R. Iakymchuk, and J. Napper, "HPC on competitive Cloud resources," Handbook of Cloud Computing, B. Furht and A. Escalante, eds., New York: Springer-Verlag, pp. 493-516, 2010.
4. C. Binnig, D. Kossmann, T. Kraska, and S. Loesing, "How is the weather tomorrow? Towards a benchmark for the Cloud," Proc. 2nd Int. Workshop on Testing Database Systems (DBTest 2009) in conjunction with ACM SIGMOD/PODPS Int. Conf. Management of Data (SIGMOD/PODS 2009), ACM Press, Jun. 2009, pp. 1-6.
5. D. Chiu and G. Agrawal, "Evaluating caching and storage options on the Amazon Web services Cloud," Proc. 11th ACM/IEEE Int. Conf. Grid Computing (Grid 2010), IEEE Computer Society, Oct. 2010, pp. 17-24.
6. E. Deelman, G. Singh, M. Livny, B. Berriman, and J. Good, "The cost of doing science on the Cloud: The montage example," Proc. 2008 Int. Conf. High Performance Computing, Networking, Storage and Analysis (SC 2008), IEEE Computer Society, Nov. 2008, pp. 1-12.
7. J. Dejun, G. Pierre, and C.-H. Chi, "EC2 Performance Analysis for Resource Provisioning of Service-Oriented Applications," Proc. 2009 Int. Conf. Service-Oriented Computing (ICSOC/ServiceWave 2009), Springer-Verlag, Nov. 2009, pp. 197-207.
8. T. Dyb?a, B. A. Kitchenham, and M. Jørgensen, "Evidencebased software engineering for practitioners," IEEE Softw., vol. 22, no. 1, Jan. 2005, pp. 58-65.
9. P.J. Fortier and H.E. Michel, Computer Systems Performance Evaluation and Prediction. Burlington, MA: Digital Press, Jul. 2003.
10. S. Garfinkel, "Commodity grid computing with Amazon's S3 and EC2," ;Login, vol. 32, no. 1, Feb. 2007, pp. 7-13.
11. S. Garfinkel, "An evaluation of Amazon's grid computing services: EC2, S3 and SQS," Sch. Eng. Appl. Sci., Harvard Univ., Canbridge, MA, Tech. Rep. TR-08-07, Jul. 2007.
12. Q. He, S. Zhou, B. Kobler, D. Duffy, and T. McGlynn, "Case study for running HPC applications in public Clouds," Proc. 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp. 395-401.
13. Z. Hill and M. Humphrey, "A quantitative analysis of high performance computing with Amazon's EC2 infrastructure: The death of the local cluster?," Proc. 10th IEEE/ACM Int. Conf. Grid Computing (Grid 2009), IEEE Computer Society, Oct. 2009, pp. 26-33.
14. Z. Hill, J. Li, M. Mao, A. Ruiz-Alvarez, and M. Humphrey, "Early observations on the performance of Windows Azure," Proc. 19th ACM Int. Symp. High Performance Distributed Computing (HPDC 2010), ACM Press, Jun. 2010, pp. 367-376.
15. A. Iosup, S. Ostermann, N. Yigitbasi, R. Prodan, T. Fahringer, and D.H.J. Epema, "Performance analysis of Cloud computing services for many-tasks scientific computing," IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 6, Jun. 2011, pp. 931-945.
16. A. Iosup, N. Yigitbasi, and D. Epema, "On the performance variability of production Cloud services," Delft Univ. Technol., Netherlands, Tech. Rep. PDS-2010-002, Jan. 2010.
17. S.L. Jackson, Research Methods and Statistics: A Critical Thinking Approach, 4th ed. Belmont, CA: Wadsworth Publishing, Mar. 2011.
18. R.K. Jain, The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling. New York, NY: Wiley Computer Publishing, John Wiley & Sons, Inc., May 1991.
19. G. Juve, E. Deelman, K. Vahi, G. Mehta, B. Berriman, B.P. Berman, and P. Maechling, "Scientific workflow applications on Amazon EC2," Proc. Workshop on Cloudbased Services and Applications in conjunction with the 5th IEEE Int. Conf. e-Science (e-Science 2009), IEEE Computer Society, Dec. 2009, pp. 59-66.
20. J.-Y. Le Boudec, Performance Evaluation of Computer and Communication Systems. Lausanne, Switzerland: EFPL Press, Feb. 2011.
21. Z. Li, L. O'Brien, R. Cai, and H. Zhang, "Towards a taxonomy of performance evaluation of commercial Cloud services," Proc. 5th Int. Conf. Cloud Computing (IEEE CLOUD 2012), IEEE Computer Society, Jun. 1012, pp. 344-351.
22. Z. Li, L. O'Brien, H. Zhang, and R. Cai, "On a catalogue of metrics for evaluating commercial Cloud services," Proc. 13th ACM/IEEE Int. Conf. Grid Computing (Grid 2012), IEEE Computer Society, Sept. 2012, pp. 164-173.
23. Z. Li, L. O'Brien, H. Zhang, and R. Cai, "On a taxonomybased conceptual model of performance evaluation of Infrastructure as a Service," submitted to IEEE Trans. Serv. Comput..
24. A. Li, X. Yang, S. Kandula, and M. Zhang, "Cloud-Cmp: Comparing public Cloud providers," Proc. 10th Annu. Conf. Internet Measurement (IMC 2010), ACM Press, Nov. 2010 pp. 1-14.
25. A. Luckow and S. Jha, "Abstractions for loosely-coupled and ensemble-based simulations on Azure," Proc. 2nd IEEE Int. Conf. Cloud Computing Technology and Science (CloudCom 2010), IEEE Computer Society, Nov./Dec. 2010, pp. 550-556.
26. S.J. Mellor, A.N. Clark, and T. Futagami, "Model-driven development-guest editor's introduction," IEEE Softw., vol. 20, no. 5, Sept./Oct. 2003, pp. 14-18.
27. J. Napper and P. Bientinesi, "Can Cloud computing reach the top500?," Proc. Combined Workshops on UnConventional High Performance Computing Workshop plus Memory Access Workshop (UCHPC-MAW 2009), ACM Press, May 2009, pp. 17-20.
28. S. Ostermann, A. Iosup, N. Yigitbasi, R. Prodan, T. Fahringer, and D.H.J. Epema, "A performance analysis of EC2 Cloud computing services for scientific computing," Proc. 1st Int. Conf. Cloud Computing (CloudComp 2009), Springer-Verlag, Oct. 2009, pp. 115-131.
29. M.R. Palankar, A. Iamnitchi, M. Ripeanu, and S. Garfinkel, "Amazon S3 for science grids: A viable solution?," Proc. 2008 Int. Workshop on Data-aware Distributed Computing (DADC 2008), ACM Press, Jun. 2008, pp. 55-64.
30. R. Prodan and S. Ostermann, "A survey and taxonomy of Infrastructure as a Service and Web hosting Cloud providers," Proc. 10th IEEE/ACM Int. Conf. Grid Computing (Grid 2009), IEEE Computer Society, Oct. 2009, pp. 17-25.
31. W. Sobel, S. Subramanyam, A. Sucharitakul, J. Nguyen, H. Wong, A. Klepchukov, S. Patil, A. Fox, and D. Patterson, "Cloudstone: Multi-platform, multi-language benchmark and measurement tools for Web 2.0," Proc. 1st Workshop on Cloud Computing and Its Applications (CCA 2008), Oct. 2008, pp. 1-6.
32. V. Stantchev, "Performance evaluation of Cloud computing offerings," Proc. 3rd Int. Conf. Advanced Engineering Computing and Applications in Sciences (ADVCOMP 2009), IEEE Computer Society, Oct. 2009, pp. 187-192.
33. J. Stokes, "The PC is order, the Cloud is chaos," Wired Cloudline, Wired.com, available at http://www.wired.com/cloudline/2011/12/the- pc-is-order/, Dec. 2011.