On evaluating commercial Cloud services: A systematic review

Journal of Systems and Software

Volumn 86, Issue 9, 2013, Pages 2371-2393

  Li, Zheng ^a   Zhang, He ^b   O'Brien, Liam ^c   Cai, Rainbow ^a   Flint, Shayne ^a  

^a AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

^b NANJING UNIVERSITY   (China)

^c GEOSCIENCE AUSTRALIA   (Australia)

Author keywords

Cloud; Cloud service evaluation; Computing; Systematic literature review

Indexed keywords

CLOUDS; COMPUTATION THEORY; DISTRIBUTED DATABASE SYSTEMS; ENGINEERING RESEARCH; SOFTWARE ENGINEERING; WEB SERVICES;

CLOUD SERVICES; COMPUTING; EVALUATION MODELING; EVIDENCE BASED SOFTWARE ENGINEERINGS (EBSE); RESEARCH OPPORTUNITIES; STATE OF THE PRACTICE; SYSTEMATIC LITERATURE REVIEW; SYSTEMATIC LITERATURE REVIEW (SLR);

DISTRIBUTED COMPUTER SYSTEMS;

EID: 84881480488     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2013.04.021     Document Type: Article

References (42)

1. Alesandre, G.D.; 2011. Updated App Engine pricing FAQ!, https://groups.google.com/forum/#!msg/google-appengine/Hluog1-a3n4/uFMhaBWhVi8J.
2. M.S. Ali, M.A. Babar, L. Chen, and K.J. Stol A systematic review of comparative evidence of aspect-oriented programming Information and Software Technology 52 2010 871 887
3. Amazon, 2011. High performance computing (HPC) on AWS, http://aws.amazon.com/hpc-applications/.
4. M. Armbrust, A. Fox, R. Griffith, A.D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia A view of Cloud computing Communications of the ACM 53 2010 50 58
5. M.A. Babar, and H. Zhang Systematic literature reviews in software engineering: preliminary results from interviews with researchers Proceedings of the 3rd International Symposium on Empirical Software Engineering and Measurement (ESEM 2009), IEEE Computer Society, Lake Buena Vista Florida, USA 2009 346 355
6. P. Brereton, B.A. Kitchenham, D. Budgen, M. Turner, and M. Khalil Lessons from applying the systematic literature review process within the software engineering domain Journal of Systems and Software 80 2007 571 583
7. Brooks, C.; 2010. Cloud computing benchmarks on the rise, http://searchcloudcomputing.techtarget.com/news/1514547/Cloud-computing- benchmarks-on-the-rise.
8. D. Budgen, B.A. Kitchenham, S.M. Charters, M. Turner, P. Brereton, and S.G. Linkman Presenting software engineering results using structured abstracts: a randomised experiment Empirical Software Engineering 13 2008 435 468
9. R. Buyya, C.S. Yeo, S. Venugopal, J. Broberg, and I. Brandic Cloud computing and emerging IT platforms: vision, hype, and reality for delivering computing as the 5th utility Future Generation Computer Systems 25 2009 599 616
10. O. Dieste, A. Grimán, and N. Juristo Developing search strategies for detecting relevant experiments Empirical Software Engineering 14 2009 513 539
11. T. Dybå, B.A. Kitchenham, and M. Jørgensen Evidence-based software engineering for practitioners IEEE Software 22 2005 58 65
12. D.G. Feitelson Experimental computer science Communications of the ACM 50 2007 24 26
13. M. Ferdman, A. Adileh, O. Kocberber, S. Volos, M. Alisafaee, D. Jevdjic, C. Kaynak, A.D. Popescu, A. Ailamaki, and B. Falsafi Clearing the Clouds: a study of emerging scale-out workloads on modern hardware Proceedings of the 17th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS 2012), ACM Press, London England, UK 2012 37 48
14. I. Foster, Y. Zhao, I. Raicu, and S. Lu Cloud computing and Grid computing 360-degree compared Proceedings of the Workshop on Grid Computing Environments (GCE08) in conjunction with the 2008 International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2008), IEEE Computer Society Austin, TX 2008 1 10
15. Google, 2012. Google Compute Engine, http://cloud.google.com/products/ compute-engine.html.
16. S.M. Habib, S. Ries, and M. Mühlhäuser Cloud computing landscape and research challenges regarding trust and reputation Proceedings of the 2010 Symposia and Workshops on Ubiquitous, Autonomic and Trusted Computing (UIC-ATC 2010), IEEE Computer Society Xi'an, China 2010 410 415
17. Harris, D.; 2011. Watch out, world: IBM finally offers a real Cloud, http://gigaom.com/cloud/watch-out-world-ibm-finally-offers-a-real-cloud/.
18. Harris, D.; 2012. What google compute engine means for cloud computing, http://gigaom.com/cloud/what-google-compute-engine-means-for-cloud-computing/.
19. S. Islam, K. Lee, A. Fekete, and A. Liu How a consumer can measure elasticity for Cloud platforms Proceedings of the 3rd joint WOSP/SIPEW International Conference on Performance Engineering (ICPE 2012), ACM Press Boston, USA 2012 85 96
20. R.K. Jain The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling 1991 Wiley Computer Publishing, John Wiley & Sons, Inc. New York, NY
21. B.A. Kitchenham, and S. Charters Guidelines for Performing Systematic Literature Reviews in Software Engineering Technical Report EBSE 2007-001 2007 Keele University and Durham University Joint Report
22. B.A. Kitchenham, Z. Li, and A. Burn Validating search processes in systematic literature reviews Proceedings of the 1st International Workshop on Evidential Assessment of Software Technologies (EAST 2011) in conjunction with ENASE 2011, SciTePress Beijing, China 2011 3 9
23. D. Kossmann, and T. Kraska Data management in the Cloud: promises, state-of-the-art, and open questions Datenbank Spektr 10 2010 121 129
24. B.C. Lewis, and A.E. Crews The evolution of benchmarking as a computer performance evaluation technique MIS Quarterly 9 1985 7 16
25. Z. Li, L. O'Brien, R. Cai, and H. Zhang Towards a taxonomy of performance evaluation of commercial Cloud services Proceedings of the 5th IEEE International Conference on Cloud Computing (CLOUD 2012), IEEE Computer Society, Honolulu, Hawaii USA 2012 344 351
26. Z. Li, L. O'Brien, H. Zhang, and R. Cai A factor framework for experimental design for performance evaluation of commercial cloud services Proceedings of the 4th IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2012), IEEE Computer Society, Taipei Taiwan 2012 169 176
27. Z. Li, L. O'Brien, H. Zhang, and R. Cai On a catalogue of metrics for evaluating commercial Cloud services Proceedings of the 13th ACM/IEEE International Conference on Grid Computing (GRID 2012), IEEE Computer Society, Beijing China 2012 164 173
28. L.B. Lisboa, V.C. Garcia, D. Lucrédio, E.S. de Almeida, S.R. de Lemos Meira, and R.P. de Mattos Fortes A systematic review of domain analysis tools Information and Software Technology 52 2010 1 13
29. L. Major, T. Kyriacou, and O.P. Brereton Systematic literature review: teaching novices programming using robots Proceedings of the 15th Annual Conference on Evaluation and Assessment in Software Engineering (EASE 2011), IEEE Computer Society Durham, UK 2011 21 30
30. Miller, R.; 2011. A look inside Amazon's data centers, http://www.datacenterknowledge.com/archives/2011/06/09/a-look-inside-amazons- data-centers/.
31. D.C. Montgomery Design and Analysis of Experiments 7th ed. 2009 John Wiley & Sons, Inc. Hoboken, NJ
32. M.S. Obaidat, and N.A. Boudriga Fundamentals of Performance Evaluation of Computer and Telecommjnication Systems 2010 John Wiley & Sons, Inc.; Hoboken New Jersey
33. R. Prodan, and S. Ostermann A survey and taxonomy of Infrastructure as a Service and Web hosting Cloud providers Proceedings of the 10th IEEE/ACM International Conference on Grid Computing (GRID 2009), IEEE Computer Society, Banff Alberta, Canada 2009 17 25
34. B.P. Rimal, E. Choi, and I. Lumb A texonomy and survey of Cloud computing systems Proceedings of the 5th International Joint Conference on INC, IMS and IDC (NCM 2009), IEEE Computer Society Seoul, Korea 2009 44 51
35. P. Runeson, and M. Höst Guidelines for conducting and reporting case study research in software engineering Empirical Software Engineering 14 2009 131 164
36. SearchCloudComputing, 2010. Top 10 Cloud computing providers of 2010. http://searchcloudcomputing.techtarget.com/feature/Top-10-cloud-computing- providers.
37. Stokes, J.; 2011. The PC is order, the Cloud is chaos, http://www.wired.com/insights/2011/12/the-pc-is-order/.
38. V.T.K. Tran, K. Lee, A. Fekete, A. Liu, and J. Keung Size estimation of Cloud migration projects with Cloud Migration Point (CMP) Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement (ESEM 2011), IEEE Computer Society Banff, Canada 2011 265 274
39. H. Zhang, and M. Ali Babar On searching relevant studies in software engineering 14th International Conference on Evaluation and Assessment in Software Engineering (EASE'10) BCS, Keele, England 2010
40. H. Zhang, and M.A. Babar An empirical investigation of systematic reviews in software engineering Proceedings of the 5th International Symposium on Empirical Software Engineering and Measurement (ESEM 2011), IEEE Computer Society Banff, Canada 2011 1 10
41. H. Zhang, M.A. Babar, and P. Tell Identifying relevant studies in software engineering Information and Software Technology 53 2011 625 637
42. Q. Zhang, L. Cheng, and R. Boutaba Cloud computing: State-of-the-art and research challenges Journal of Internet Services and Applications 1 2010 7 18