Awan: Locality-aware resource manager for geo-distributed data-intensive applications

Proceedings - 2016 IEEE International Conference on Cloud Engineering, IC2E 2016: Co-located with the 1st IEEE International Conference on Internet-of-Things Design and Implementation, IoTDI 2016

Volumn , Issue , 2016, Pages 32-41

  Jonathan, Albert ^a   Chandra, Abhishek ^a   Weissman, Jon ^a  

^a UNIVERSITY OF MINNESOTA   (United States)

Author keywords

Cloud Computing; Data Intensive; Edge Cloud; Geo distributed; Resource Management; Scheduling

Indexed keywords

CLOUD COMPUTING; CLUSTER COMPUTING; DATA HANDLING; INFORMATION MANAGEMENT; MANAGERS; SCHEDULING; TURNAROUND TIME;

COMPUTING FRAMEWORKS; DATA INTENSIVE; DATA-INTENSIVE APPLICATION; DISTRIBUTED SERVERS; EDGE CLOUDS; GEO-DISTRIBUTED; MANAGEMENT MECHANISMS; RESOURCE MANAGEMENT;

DISTRIBUTED COMPUTER SYSTEMS;

EID: 84978034019     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IC2E.2016.15     Document Type: Conference Paper

References (34)

1. Amazon web services. [Online]. Available: http://aws.amazon.com
2. Microsoft azure. [Online]. Available: https://azure.microsoft.com
3. M. Ryden, K. Oh, A. Chandra, and J. Weissman, Nebula: Distributed edge cloud for data intensive computing, " in Cloud Engineering (IC2E), 2014 IEEE International Conference on. IEEE, 2014, pp. 57-66.
4. S. Ibrahim, H. Jin, B. Cheng, H. Cao, S. Wu, and L. Qi, Cloudlet: towards mapreduce implementation on virtual machines, " in Proceedings of the 18th ACM international symposium on High performance distributed computing. ACM, 2009, pp. 65-66.
5. D. P. Anderson, Boinc: A system for public-resource computing, and storage, " in Grid Computing, 2004. Proceedings. Fifth IEEE/ACM International Workshop on. IEEE, 2004, pp. 4-10.
6. Hadoop. [Online]. Available: https://hadoop.apache.org/
7. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, Dryad: distributed data-parallel programs from sequential building blocks, " in ACM SIGOPS Operating Systems Review, vol. 41, no. 3. ACM, 2007, pp. 59-72.
8. G. Malewicz, M. H. Austern, A. J. Bik, J. C. Dehnert, I. Horn, N. Leiser, and G. Czajkowski, Pregel: a system for large-scale graph processing, " in Proceedings of the 2010 ACM SIGMOD International Conference on Management of data. ACM, 2010, pp. 135-146.
9. M. Zaharia, M. Chowdhury, M. J. Franklin, S. Shenker, and I. Stoica, Spark: cluster computing with working sets, " in Proceedings of the 2nd USENIX conference on Hot topics in cloud computing, vol. 10, 2010, p. 10.
10. B. He, M. Yang, Z. Guo, R. Chen, B. Su, W. Lin, and L. Zhou, Comet: batched stream processing for data intensive distributed computing, " in Proceedings of the 1st ACM symposium on Cloud computing. ACM, 2010, pp. 63-74.
11. B. Heintz, C. Wang, A. Chandra, and J. Weissman, Cross-phase optimization in mapreduce, " in Cloud Engineering (IC2E), 2013 IEEE International Conference on. IEEE, 2013, pp. 338-347.
12. A. Rabkin, M. Arye, S. Sen, V. S. Pai, and M. J. Freedman, Aggregation, and degradation in jetstream: Streaming analytics in the wide area, " in 11th USENIX Symposium on Networked Systems Design, and Implementation (NSDI 14). USENIX Association, 2014, pp. 275-288.
13. Q. Pu, G. Ananthanarayanan, P. Bodik, S. Kandula, A. Akella, P. Bahl, and I. Stoica, Low latency geo-distributed data analytics, " in Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication. ACM, 2015, pp. 421-434.
14. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica, Mesos: A platform for fine-grained resource sharing in the data center." in NSDI, vol. 11, 2011, pp. 22-22.
15. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth, et al., Apache hadoop yarn: Yet another resource negotiator, " in Proceedings of the 4th annual Symposium on Cloud Computing. ACM, 2013, p. 5.
16. M. Schwarzkopf, A. Konwinski, M. Abd-El-Malek, and J. Wilkes, Omega: flexible, scalable schedulers for large compute clusters, " in Proceedings of the 8th ACM European Conference on Computer Systems. ACM, 2013, pp. 351-364.
17. B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak, and M. Bowman, Planetlab: an overlay testbed for broad-coverage services, " ACM SIGCOMM Computer Communication Review, vol. 33, no. 3, pp. 3-12, 2003.
18. J. Dean, and S. Ghemawat, Mapreduce: simplified data processing on large clusters, " Communications of the ACM, vol. 51, no. 1, pp. 107-113, 2008.
19. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, Delay scheduling: a simple technique for achieving locality, and fairness in cluster scheduling, " in Proceedings of the 5th European conference on Computer systems. ACM, 2010, pp. 265-278.
20. R. Wolski, Dynamically forecasting network performance using the network weather service, " Cluster Computing, vol. 1, no. 1, pp. 119-132, 1998.
21. J. Kim, A. Chandra, and J. Weissman, Open: Passive network performance estimation for data-intensive applications, " Dept. of CSE, Univ. of Minnesota, Tech. Rep, pp. 08-041, 2008.
22. B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar, Native client: A sandbox for portable, untrusted x86 native code, " in Security, and Privacy, 2009 30th IEEE Symposium on. IEEE, 2009, pp. 79-93.
23. M. J. Litzkow, M. Livny, and M. W. Mutka, Condor-A hunter of idle workstations, " in Distributed Computing Systems, 1988., 8th International Conference on. IEEE, 1988, pp. 104-111.
24. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese, Wanalytics: Analytics for a geo-distributed data-intensive world, " in Conference on Innovative Data Systems Research (CIDR 2015), 2015.
25. C.-C. Hung, L. Golubchik, and M. Yu, Scheduling jobs across geodistributed datacenters, " in Proceedings of the Sixth ACM Symposium on Cloud Computing. ACM, 2015, pp. 111-124.
26. J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, et al., Spanner: Googles globally distributed database, " ACM Transactions on Computer Systems (TOCS), vol. 31, no. 3, p. 8, 2013.
27. Z. Wu, M. Butkiewicz, D. Perkins, E. Katz-Bassett, and H. V. Madhyastha, Spanstore: Cost-effective geo-replicated storage spanning multiple cloud services, " in Proceedings of the Twenty-Fourth ACM Symposium on Operating Systems Principles. ACM, 2013, pp. 292-308.
28. A. Bessani, R. Mendes, T. Oliveira, N. Neves, M. Correia, M. Pasin, and P. Verissimo, Scfs: a shared cloud-backed file system, " in Proc. of the 2014 USENIX Annual Technical Conference, 2014.
29. K. Oh, A. Raghavan, A. Chandra, and J. Weissman, Redefining data locality for cross-data center storage, " BigSystem 2015: 2nd International Workshop on Software-Defined Ecosystems, 2015.
30. S. Agarwal, J. Dunagan, N. Jain, S. Saroiu, A. Wolman, and H. Bhogan, Volley: Automated data placement for geo-distributed cloud services." in NSDI, 2010, pp. 17-32.
31. A. Verma, L. Pedrosa, M. Korupolu, D. Oppenheimer, E. Tune, and J. Wilkes, Large-scale cluster management at google with borg, " in Proceedings of the Tenth European Conference on Computer Systems. ACM, 2015, p. 18.
32. E. Boutin, J. Ekanayake, W. Lin, B. Shi, J. Zhou, Z. Qian, M. Wu, and L. Zhou, Apollo: scalable, and coordinated scheduling for cloud-scale computing, " in Proc. USENIX Symp. on Operating Systems Design, and Implementation (OSDI), 2014.
33. Torque resource manager. [Online]. Available: http://www. adaptivecomputing.com/products/open-source/torque
34. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, Quincy: fair scheduling for distributed computing clusters, " in Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. ACM, 2009, pp. 261-276.