Computing at massive scale: Scalability and dependability challenges

Proceedings - 2016 IEEE Symposium on Service-Oriented System Engineering, SOSE 2016

Volumn , Issue , 2016, Pages 386-397

  Yang, Renyu ^a   Xu, Jie ^a,b  

^a BEIHANG UNIVERSITY   (China)

^b UNIVERSITY OF LEEDS   (United Kingdom)

Author keywords

Cloud computing; Computing at Scale; Dependability; Performance; Scalability; Service engineering

Indexed keywords

BIG DATA; CLOUD COMPUTING; DISTRIBUTED COMPUTER SYSTEMS; QUALITY OF SERVICE; SCALABILITY; SYSTEMS ENGINEERING;

COMPUTING AT SCALE; DEPENDABILITY; DISTRIBUTED COMPUTING ENVIRONMENT; PERFORMANCE; PERFORMANCE BOTTLENECKS; RESOURCE UTILIZATIONS; SERVICE ENGINEERING; SERVICES AND APPLICATIONS;

INFORMATION MANAGEMENT;

EID: 84978952293     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/SOSE.2016.73     Document Type: Conference Paper

References (67)

1. http://www.cnbc.com/2015/11/10/alibaba-handles-1-billion-in-8-minutes-of-sales-through-alipay-on-singles-day.html
2. http://www.businesswire.com/news/home/20151111006351/en/Alibaba-Group-Generated-USD-14.3-Billion-GMV
3. ODPS: https://www.aliyun.com/product/odps/
4. Docker Project. https://www.docker.io/, 2014.
5. Kubernetes. http://kubernetes.io, Aug. 2014.
6. S. Herbst-Murphy. Clearing and Settlement of Interbank Card Transactions: A MasterCard Tutorial for Federal Reserve Payments Analysts.
7. A. McAfee and B. Erik. Big data: The management revolution. Harvard Business Review, 10 2012.
8. Google Cluster Data V2 (2011). [Online] Available: http://code.google.com/p/googleclusterdata/wiki/ClusterData2011-1
9. (2008) Amazon suffers u.s. outage on friday internet. [Online]. Available: http://news.cnet.com/
10. R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic, "Cloud computing and emerging IT platforms: Vision, hype, and reality for delivering computing as the 5th utility," in Future Gener. Comput. Syst., vol. 25, pp. 599-616, 2009.
11. Z. Zheng, J. Zhu, and M. R. Lyu. Service-generated big data and big data-as-a-service: an overview. In Proceedings of IEEE Big Data, 2013
12. B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R.Das. Modeling and synthesizing task placement constraints in Google compute clusters. In Proceedings of ACM SoCC, 2011
13. C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of ACM SoCC, 2012
14. I. S. Moreno, P. Garraghan, P. Townend, and J. Xu. An approach for characterizing workloads in Google cloud to derive realistic resource utilization models. In Proceedings of IEEE SOSE 2013.
15. I. S. Moreno, P. Garraghan, P. Townend, and J. Xu. Analysis, modeling and simulation of workload patterns in a large-scale utility cloud[J]., IEEE Transactions on Cloud Computing, 2014
16. P. Garraghan, I. S. Moreno, P. Townend, and J. Xu. An analysis of failure-related energy waste in a large-scale cloud environment, in IEEE Transactions on Emerging Topics in Computing, 2014
17. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. in ACM SIGOPS Operating Systems Review. ACM, 2007, 41(3).
18. J. Dean and S. Ghemawat. MapReduce: simplified data processing on large clusters [J]. In Communications of the ACM, 2008, 51(1).
19. R. K. Sahoo, M. S. Squillante, A. Sivasubramaniam, and Y. Zhang. Failure data analysis of a large-scale heterogeneous server environment. In Proceedings of IEEE DSN 2004.
20. K. V. Vishwanath and N. Nagappan. Characterizing cloud computing hardware reliability. In Proceedings of ACM SoCC, 2010, (pp. 193-204).
21. F. Dinu and T. Ng. Understanding the effects and implications of compute node related failures in hadoop. In Proceedings of ACM HPDC, 2012.
22. A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr. Basic concepts and taxonomy of dependable and secure computing. In IEEE Transactions on Dependable and Secure Computing(TDSC),2004.
23. B. Randell and J. Xu, "The evolution of the recovery block concept," Software Fault Tolerance, 1995.
24. A. Avizienis, "The methodology of n-version programming," Software fault tolerance, 1995.
25. M. R. Lyu et al., Handbook of software reliability engineering, 1996
26. Z. Wen, J. Cala, P. Watson, and A. Romanovsky. Cost Effective, Reliable, and Secure Workflow Deployment over Federated Clouds, in Proceedings of IEEE Cloud, 2015
27. Z. Wen, J. Cala, and P. Watson. A scalable method for partitioning workflows with security requirements over federated clouds. In Proceedings of IEEE CloudCom, 2014
28. Z. Zhang, C. Li, Y. Tao, R. Yang, H. Tang, and J. Xu. Fuxi: a fault-tolerant resource management and job scheduling system at internet scale. In Proceedings of the VLDB Endowment, 2014
29. E. Boutin, J. Ekanayake, W. Lin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou. Apollo: scalable and coordinated scheduling for cloud-scale computing. In Proceedings of USENIX OSDI, 2014
30. K. Karanasos, S. Rao, C. Curino, C. Douglas, K. Chaliparambil, G. M.Fumarola, S. Heddaya, R. Ramakrishnan, and S. Sakalanaga. Mercury: Hybrid Centralized and Distributed Scheduling in Large Shared Clusters. In Proceedings of USENIX ATC, 2015
31. A. Toshniwal, S. Taneja, A. Shukla, K. Ramasamy, J. M. Patel, S. Kulkarni, J. Jackson, K. Gade, M. Fu, J. Donham, et al. Storm@twitter. In Proceedings of the ACM SIGMOD, 2014
32. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica. Sparrow: distributed, low latency scheduling. In Proceedings of ACM SOSP,2013
33. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica. Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center. In Proceedings of the USENIX NSDI, 2011
34. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J.Franklin, S. Shenker, and I. Stoica. Resilient distributed datasets: A fault-tolerant abstraction for in-memory cluster computing. In Proceedings of the USENIX NSDI, 2012
35. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes. Omega: flexible, scalable schedulers for large compute clusters. In Proceedings of the ACM EuroSys, 2013
36. A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes. Large-scale cluster management at Google with Borg. In Proceedings of ACM EuroSys, 2015
37. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, et al. "Apache hadoop yarn: Yet another resource negotiator." In Proceedings of the ACM SoCC, 2013.
38. C. Delimitrou and C. Kozyrakis. Quasar: Resource-efficient and qos-aware cluster management. In Proceedings of ACM ASPOLOS, 2014
39. 2PS: a Dependable Data Provisioning Service in Multi-Tenants Cloud Environments, In Proceedings of IEEE HASE, 2016.
40. I. S. Moreno, R. Yang, J. Xu and T. Wo. Improved energy-efficiency in cloud datacenters with interference-aware virtual machine placement. In Proceedings of the IEEE ISADS, 2013
41. R. Yang, I. S. Moreno, J. Xu and T. Wo. T. An analysis of performance interference effects on energy-efficiency of virtualized cloud environments. In Proceedings of the IEEE CloudCom, 2013
42. Y. Wang, R. Yang, T. Wo, W. Jiang and C. Hu. Improving utilization through dynamic VM resource allocation in hybrid cloud environment. In Proceedings of the IEEE ICPADS 2014
43. P. Garraghan, P. Townend and J. Xu. An empirical failure-analysis of a large-scale cloud computing environment. In Proceedings of IEEE HASE 2014
44. P. Garraghan, P. Townend and J. Xu. An analysis of the server characteristics and resource utilization in google cloud. In Proceedings of IEEE IC2E, 2013
45. T. Akidau, A. Balikov, K. Bekiroglu, S. Chernyak, J. Haberman, R. Lax, S. McVeety, D. Mills, P. Nordstrom, and S. Whittle. MillWheel: fault-tolerant stream processing at internet scale. In Proceedings of the VLDB Endowment, 2013
46. S. Melnik, A. Gubarev, J. J. Long, G. Romer, S. Shivakumar, M. Tolton, and T. Vassilakis. Dremel: interactive analysis of web-scale datasets [J]. In Proceedings of the VLDB Endowmen, 2010
47. A. Thusoo, J. S. Sarma, N. Jain, Z. Shao, P. Chakka, S. Anthony, H. Liu, P. Wyckoff, and R. Murthy. Hive: a warehousing solution over a map-reduce framework. In Proceedings of the VLDB Endowment, 2009
48. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski. Pregel: a system for large-scale graph processing. In Proceedings of the ACM SIGMOD, 2010
49. Y. Low, D. Bickson, J. Gonzalez, C. Guestrin, A. Kyrola, and J. M.Hellerstein. Distributed GraphLab: a framework for machine learning and data mining in the cloud. In Proceedings of the VLDB Endowment 2012
50. B. Saha, H. Shah, S. Seth, G. Vijayaraghavan, A. Murthy, and C. Curino. Apache Tez: A unifying framework for modeling and building data processing applications. In Proceedings of ACM SIGMOD, 2015
51. L. Cui, J. Li, T. Wo, B. Li, R. Yang, Y. Cao and J. Huai. HotRestore: a fast restore system for virtual machine cluster. In Proceedings of USENIX LISA, 2014
52. Y. Huang, R. Yang, L. Cui, T. Wo, C. Hu and B. Li. VMCSnap: Taking Snapshots of Virtual Machine Cluster with Memory Deduplication. In Proceedings of IEEE SOSE, 2014
53. J. Li, J. Zheng, L. Cui and R. Yang. ConSnap: Taking continuous snapshots for running state protection of virtual machines. In Proceedings of IEEE ICPADS, 2014
54. A. Moody, G. Bronevetsky, K. Mohror, and B. R. De Supinski, Design, modeling, and evaluation of a scalable multi-level check-pointing system, In Proceedings of IEEE SC, 2010
55. L. A. Barroso, J. Clidaras, and U. Hölzle, "The datacenter as a computer: An introduction to the design of warehouse-scale machines." Morgan & Claypool Publishers, 2013.
56. J. Dean and L. A. Barroso. The tail at scale. In Communications of the ACM, 56(2), 2013.
57. C. Wang, K. Schwan, V. Talwar, G. Eisenhauer, L. Hu, M. Wolf, "A Flexible Architecture Integrating Monitoring and Analytics for Managing Large-scale Datacenters", in Proceedings of ACM ICAC, 2011
58. B. Mauren. Fail at scale. In Communications of the ACM, 58(11), 2015.
59. R. Love, "Kernet Korner: Intro to Inotify", Linux Journal, 139( 8), 2005.
60. R. Ihaka,R. Gentleman,"R: a Language for Data Analysis and Graphic", Journal of Computational Graph Statistics, 1996.
61. Y. Zhang ,R. Yang, T. Wo, C. Hu, J. Kang and L. Cui. CloudAP: Improving the QoS of Mobile Applications with Efficient VM Migration. In Proceedings of IEEE HPCC, 2013
62. M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies. The case for vm-based cloudlets in mobile computing. In IEEE Pervasive Computing, 2009
63. N. Fernando, S. W. Loke, and W. Rahayu. Mobile cloud computing: A survey. In Future Generation Computer Systems, 2013
64. X. Chen, C.-D. Lu, and K. Pattabiraman. Failure analysis of jobs in compute clouds: A google cluster case study. In Proceedings of IEEE ISSRE, 2014
65. A. Rosa, L. Y. Chen, and W. Binder. Understanding the Dark Side of Big Data Clusters: an Analysis beyond Failures. In Proceedings of IEEE DSN, 2015
66. B. H. Sigelman, L. A. Barroso, M. Burrows, P. Stephenson, M. Plakal, D. Beaver, S. Jaspan, and C. Shanbhag. Dapper, a large-scale distributed systems tracing infrastructure. Technical report, Google, 2010
67. H. Mi, H. Wang, Y. Zhou, M. R. Lyu, and H. Cai. Toward fine-grained, unsupervised, scalable performance diagnosis for production cloud computing systems. In IEEE Transactions on Parallel and Distributed Systems, 24(6), 2013