End-to-End Optimization for Geo-Distributed MapReduce

IEEE Transactions on Cloud Computing

Volumn 4, Issue 3, 2016, Pages 293-306

  Heintz, Benjamin ^a   Chandra, Abhishek ^a   Sitaraman, Ramesh K ^b,c   Weissman, Jon ^a  

^a University of Minnesota   (United States)

^b The Manning College of Information and Computer Sciences   (United States)

^c AKAMAI TECHNOLOGIES   (United States)

Author keywords

Batch processing systems; distributed systems; parallel systems

Indexed keywords

BATCH DATA PROCESSING; HETEROGENEOUS NETWORKS; ITERATIVE METHODS;

BATCH PROCESSING SYSTEM; DATA-INTENSIVE APPLICATION; DISTRIBUTED COMPUTATIONS; DISTRIBUTED SYSTEMS; EXECUTION ENVIRONMENTS; HETEROGENEOUS COMPUTING; MODEL-DRIVEN OPTIMIZATION; PARALLEL SYSTEM;

PARALLEL PROCESSING SYSTEMS;

EID: 84986550276     PISSN: None     EISSN: 21687161     Source Type: Journal    
DOI: 10.1109/TCC.2014.2355225     Document Type: Article

References (31)

1. J. Baker, C. Bond, J. C. Corbett, J. J. Furman, A. Khorlin, J. Larson, J.-M. Leon, Y. Li, A. Lloyd, and V. Yushprakh, "Megastore: Providing scalable, highly available storage for interactive services, " in Proc. Conf. Innovative Data Syst. Res., 2011, pp. 223-234.
2. J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, R. Wang, and D. Woodford, "Spanner: Google's globallydistributed database, " in Proc. 10th USENIX Conf. Operating Syst. Design Implementation, 2012, pp. 251-264.
3. E. Nygren, R. Sitaraman, and J. Sun, "The akamai network: A platform for high-performance internet applications, " ACM SIGOPS Operating Syst. Rev., vol. 44, no. 3, pp. 2-19, 2010.
4. T. Hey, S. Tansley, and K. Tolle, Eds., The Fourth Paradigm: Data- Intensive Scientific Discovery. Redmond, WA, USA: Microsoft Research, 2009.
5. D. Patterson and J. Gray, "A conversation with Jim Gray, " Queue, vol. 1, no. 4, pp. 8-17, Jun. 2003.
6. A. Rabkin, M. Arye, S. Sen, V. Pai, and M. J. Freedman, "Making every bit count in wide-area analytics, " in Proc. 14th USENIX Conf. Hot Topics Oper. Syst., 2013, p. 6.
7. J. Dean and S. Ghemawat, "MapReduce: Simplified data processing on large clusters, " in Proc. 6th Symp. Operating Syst. Des. Implementation, 2004, pp. 137-149.
8. M. Cardosa, C. Wang, A. Nangia, A. Chandra, and J. Weissman, "Exploring MapReduce efficiency with highly-distributed data, " in Proc. 2nd Int. Workshop MapReduce Its Appl., 2011, pp. 27-33.
9. Hadoop [Online]. Available: http://hadoop.apache.org
10. B. Chun, D. Culler, T. Roscoe, A. Bavier, L. Peterson, M. Wawrzoniak, and M. Bowman, "PlanetLab: An overlay testbed for broadcoverage services, " ACM SIGCOMM, vol. 33, no. 3, pp. 3-12, 2003.
11. H. Herodotou, F. Dong, and S. Babu, "No one (cluster) size fits all: Automatic cluster sizing for data-intensive analytics, " in Proc. ACM 2nd ACM Symp. Cloud Comput., 2011, pp. 18:1-18:14.
12. V. Jalaparti, H. Ballani, P. Costa, T. Karagiannis, and A. Rowstron, "Bridging the tenant-provider gap in cloud services, " in Proc. 3rd ACM Symp. Cloud Comput., 2012, pp. 6:1-6:14.
13. A. Verma, N. Zea, B. Cho, I. Gupta, and R. H. Campbell, "Breaking the MapReduce stage barrier, " in Proc. IEEE Int. Conf. Cluster Comput., 2010, pp. 235-244.
14. B. Heintz, A. Chandra, and R. K. Sitaraman, "Optimizing MapReduce for highly distributed environments, " Dept. Comput. Sci. Eng., Univ. Minnesota, Minneapolis, MN, USA, Tech. Rep. TR 12-003, Feb. 2012.
15. Project Gutenberg [Online]. Available: http://www.gutenberg. org/
16. M. Arlitt and T. Jin, "Workload characterization of the 1998 World Cup web site, " HP Labs, Palo Alto, CA, USA, Tech. Rep. HPL- 1999-35R1, Sep. 1999.
17. B. Palanisamy, A. Singh, L. Liu, and B. Jain, "Purlieus: localityaware resource allocation for MapReduce in a cloud, " in Proc. Int. Conf. High Performance Comput., Netw., Storage Anal., 2011, pp. 58:1-58:11.
18. M. Hammoud, M. S. Rehman, and M. F. Sakr, "Center-of-Gravity reduce task scheduling to lower MapReduce network traffic, " in Proc. IEEE Cloud, 2012, pp. 49-58.
19. H. Gadre, I. Rodero, and M. Parashar, "Investigating MapReduce framework extensions for efficient processing of geographically scattered datasets, " in Proc. ACM SIGMETRICS, 2011, pp. 116-118.
20. S. Kim, J. Won, H. Han, H. Eom, and H. Y. Yeom, "Improving Hadoop performance in intercloud environments, " ACM SIGMETRICS Perform. Eval. Rev., vol. 39, no. 3, pp. 107-109, 2011.
21. M. Mattess, R. Calheiros, and R. Buyya, "Scaling MapReduce applications across hybrid clouds to meet soft deadlines, " in Proc. IEEE 27th Int. Conf. Adv. Inf. Netw. Appl., Mar. 2013, pp. 629-636.
22. Y. Luo, Z. Guo, Y. Sun, B. Plale, J. Qiu, and W. W. Li, "A hierarchical framework for cross-domain MapReduce execution, " in Proc. 2nd Int. Workshop Emerging Comput. Methods Life Sci., 2011, pp. 15- 22.
23. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica, "Improving MapReduce performance in heterogeneous environments, " in Proc. 8th USENIX Conf. Operating Syst. Des. Implementation, 2008, pp. 29-42.
24. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, and B. Saha, "Reining in the outliers in map-reduce clusters using mantri, " in Proc. USENIX Conf. Operating Syst. Des. Implementation, 2010, pp. 265-278.
25. F. Ahmad, S. T. Chakradhar, A. Raghunathan, and T. N. Vijaykumar, "Tarazu: Optimizing MapReduce on heterogeneous clusters, " in Proc. 17th Int. Conf. Archit. Support Program. Languages Operating Syst., 2012, pp. 61-74.
26. T. Sandholm and K. Lai, "MapReduce optimization using dynamic regulated prioritization, " in Proc. 11th Int. Joint Conf. Meas. Model. Comput. Syst., 2009, pp. 299-310.
27. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: Fair scheduling for distributed computing clusters, " in Proc. ACM SIGOPS 22nd Symp. Operating Syst. Principles, 2009, pp. 261-276.
28. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica, "Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling, " in Proc. 5th Eur. Conf. Comput. Syst., 2010, pp. 265-278.
29. R. Wolski, N. T. Spring, and J. Hayes, "The network weather service: A distributed resource performance forecasting service for metacomputing, " Future Generation Comput. Syst., vol. 15, pp. 757- 768, 1999.
30. J. Kim, A. Chandra, and J. Weissman, "Passive network performance estimation for large-scale, data-intensive computing, " IEEE Trans. Parallel Distrib. Syst., vol. 22, no. 8, pp. 1365-1373, Aug. 2011.
31. Q. He, C. Dovrolis, and M. Ammar, "On the predictability of large transfer tcp throughput, " in Proc. ACM SIGCOMM, 2005, pp. 145- 156.