Flutter: Scheduling tasks closer to data across geo-distributed datacenters

Proceedings - IEEE INFOCOM

Volumn 2016-July, Issue , 2016, Pages

  Hu, Zhiming ^a   Li, Baochun ^b   Luo, Jun ^a  

^a NANYANG TECHNOLOGICAL UNIVERSITY   (Singapore)

^b UNIVERSITY OF TORONTO   (Canada)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; APPLICATION PROGRAMS; ARTIFICIAL INTELLIGENCE; DATA HANDLING; FLUTTER (AERODYNAMICS); INDUCTIVE LOGIC PROGRAMMING (ILP); INTEGER PROGRAMMING; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATHEMATICAL TRANSFORMATIONS; NONLINEAR PROGRAMMING; SCHEDULING ALGORITHMS;

DATA-PARALLEL APPLICATIONS; DESIGN AND IMPLEMENTS; INTEGER LINEAR PROGRAMMING; LINEAR PROGRAMMING SOLVERS; MULTI-NATIONAL CORPORATIONS; NONLINEAR PROGRAMS; TASK-SCHEDULING ALGORITHMS; TOTALLY UNIMODULAR;

BIG DATA;

EID: 84983296861     PISSN: 0743166X     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/INFOCOM.2016.7524469     Document Type: Conference Paper

References (25)

1. A. Vulimiri, C. Curino, B. Godfrey, J. Padhye, and G. Varghese, "Global Analytics in the Face of Bandwidth and Regulatory Constraints," in Proc. of USENIX NSDI, 2015
2. K. Kloudas, M. Mamede, N. Preguiça, and R. Rodrigues, "Technical Report: Pixida: Optimizing Data Parallel Jobs in Bandwidth-Skewed Environments," Tech. Rep., 2015
3. Q. Pu, G. Ananthanarayanan, P. Bodik, S. Kandula, A. Akella, V. Bahl, and I. Stoica, "Low Latency Analytics of Geo-distributed Data in the Wide Area," in Proc. of ACM SIGCOMM, 2015
4. Y.-K. Kwok and I. Ahmad, "Static Scheduling Algorithms for Allocating Directed Task Graphs to Multiprocessors," ACM Computing Surveys, vol. 31, no. 4, pp. 406-471, 1999
5. M. Zaharia, M. Chowdhury, T. Das, A. Dave, J. Ma, M. McCauley, M. J. Franklin, S. Shenker, and I. Stoica, "Resilient Distributed Datasets: A Fault-tolerant Abstraction for In-memory Cluster Computing," in Proc. of USENIX NSDI, 2012
6. K. Ousterhout, R. Rasti, S. Ratnasamy, S. Shenker, B.-G. Chun, and V. ICSI, "Making Sense of Performance in Data Analytics Frameworks," in Proc. of USENIX NSDI, 2015, pp. 293-307
7. R. Meyer, "A Class of Nonlinear Integer Programs Solvable by a Single Linear Program," SIAM Journal on Control and Optimization, vol. 15, no. 6, pp. 935-946, 1977
8. "Breeze: a numerical processing library for Scala." [Online]. Available: https://github.com/scalanlp/breeze
9. "Scala." [Online]. Available: http://www.scala-lang.org
10. "Hadoop." [Online]. Available: https://hadoop.apache.org
11. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay Scheduling: a Simple Technique for Achieving Locality and Fairness in Cluster Scheduling," in Proc. of ACM Eurosys, 2010, pp. 265-278
12. L. Page, S. Brin, R. Motwani, and T. Winograd, "The PageRank Citation Ranking: Bringing Order to the Web." 1999
13. J. E. Gonzalez, R. S. Xin, A. Dave, D. Crankshaw, M. J. Franklin, and I. Stoica, "Graphx: Graph Processing in a Distributed Dataflow Framework," in Proc. of USENIX OSDI, 2014, pp. 599-613
14. J. Leskovec, K. J. Lang, A. Dasgupta, and M. W. Mahoney, "Community Structure in Large Networks: Natural Cluster Sizes and the Absence of Large Well-Defined Clusters," Internet Mathematics, vol. 6, no. 1, pp. 29-123, 2009
15. A. Vulimiri, C. Curino, B. Godfrey, K. Karanasos, and G. Varghese, "WANalytics: Analytics for a Geo-distributed Data-intensive World," in Proc. of CIDR, 2015
16. Y. Kwon, M. Balazinska, B. Howe, and J. Rolia, "Skewtune: mitigating skew in mapreduce applications," in Proc. of ACM SIGMOD, 2012, pp. 25-36
17. C. Jayalath, J. Stephen, and P. Eugster, "From the Cloud to the Atmosphere: Running MapReduce Across Data Centers," IEEE Transactions on Computers, vol. 63, no. 1, pp. 74-87, 2014
18. V. K. Vavilapalli, A. C. Murthy, C. Douglas, S. Agarwal, M. Konar, R. Evans, T. Graves, J. Lowe, H. Shah, S. Seth et al., "Apache Hadoop Yarn: Yet Another Resource Negotiator," in Proc. of ACM SoCC, 2013
19. B. Hindman, A. Konwinski, M. Zaharia, A. Ghodsi, A. D. Joseph, R. H. Katz, S. Shenker, and I. Stoica, "Mesos: A Platform for Fine-Grained Resource Sharing in the Data Center." in Proc. of USENIX NSDI, 2011
20. K. Ousterhout, P. Wendell, M. Zaharia, and I. Stoica, "Sparrow: Distributed, Low Latency Scheduling," in Proc. of ACM SOSP, 2013, pp. 69-84
21. X. Ren, G. Ananthanarayanan, A. Wierman, and M. Yu, "Hopper: Decentralized Speculation-aware Cluster Scheduling at Scale," in Proc. of ACM SIGCOMM, 2015
22. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: Fair Scheduling for Distributed Computing Clusters," in Proc. of the ACM SIGOPS, 2009, pp. 261-276
23. X. Zhang, Z. Zhong, S. Feng, B. Tu, and J. Fan, "Improving Data Locality of MapReduce by Scheduling in Homogeneous Computing Environments," in IEEE 9th International Symposium on Parallel and Distributed Processing with Applications (ISPA), 2011, pp. 120-126
24. M. Hammoud and M. F. Sakr, "Locality-aware Reduce Task Scheduling for MapReduce," in IEEE Conference on Cloud Computing Technology and Science (CloudCom), 2011, pp. 570-576
25. J. Xie, S. Yin, X. Ruan, Z. Ding, Y. Tian, J. Majors, A. Manzanares, and X. Qin, "Improving Mapreduce Performance through Data Placement in Heterogeneous Hadoop Clusters," in IEEE International Symposium on Parallel &Distributed Processing, Workshops and Phd Forum, 2010, pp. 1-9