Tarazu: Optimizing MapReduce on heterogeneous clusters

International Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS

Volumn , Issue , 2012, Pages 61-74

  Ahmad, Faraz ^a   Chakradhar, Srimat T ^b   Raghunathan, Anand ^a   Vijaykumar, T N ^a  

^a Purdue University   (United States)

^b NEC LABORATORIES AMERICA   (United States)

Author keywords

cluster scheduling; heterogeneous clusters; load imbalance; MapReduce; shuffle

Indexed keywords

CLUSTER SCHEDULING; HETEROGENEOUS CLUSTERS; LOAD IMBALANCE; MAP-REDUCE; SHUFFLE;

COMMUNICATION; COMPUTER HARDWARE; COMPUTER PROGRAMMING; PARALLEL ARCHITECTURES; SCHEDULING;

OPTIMIZATION;

EID: 84858784772     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2150976.2150984     Document Type: Conference Paper

References (38)

1. R. Agrawal and R. Srikant. Fast algorithms for mining association rules. Proceedings of 20th Intl. Conference on Very Large Data Bases, VLDB, 1994.
2. Amazon EC2. http://aws.amazon.com/ec2.
3. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, B. Saha, and E. Harris. Reining in the outliers in Map-Reduce clusters using Mantri. In Proceedings of the Usenix Symposium on Operating Systems Design and Implementation (OSDI), 2010.
4. D. G. Andersen, J. Franklin, M. Kaminsky, A. Phanishayee, L. Tan, and V. Vasudevan. FAWN: A Fast Array of Wimpy Nodes. In Proceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2009.
5. Apache Mahout: Scalable machine learning and data mining. http://mahout.apache.org.
6. A. C. Arpaci-Dusseau, R. H. Arpaci-Dusseau, D. E. Culler, J. M. Hellerstein, and D. A. Patterson. High-performance sorting on networks of workstations. In Proceedings of the SIGMOD International Conference on Management of Data, pages 243-254, Tucson,Arizona, May 1997.
7. Calxeda, Inc. http://www.calxeda.com.
8. Q. Chen, D. Zhang, M. Guo, Q. Deng, and S. Guo. SAMR: A Selfadaptive MapReduce Scheduling Algorithm in Heterogeneous Environment. In Proceedings of the International Conference on Computer and Information Technology (CIT), 2010.
9. B.-G. Chun, G. Iannaccone, G. Iannaccone, R. Katz, G. Lee, and L. Niccolini. An Energy Case for Hybrid Datacenters. In SOSP Workshop on Power Aware Computing and Systems (HotPower), 2009.
10. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Commun. ACM, pages 107-113, Jan. 2008.
11. Facebook Hive. http://hadoop.apache.org/hive.
12. A. Ghodsi, M. Zaharia, B. Hindman, A. Konwinski, S. Shenker, and I. Stoica. Dominant Resource Fairness: Fair allocation of multiple resource types. In Proceedings of the 8th USENIX conference on Networked systems design and implementation(NSDI), 2011.
13. A. Greenberg, J. R. Hamilton, N. Jain, S. Kandula, C. Kim, P. Lahiri, D. A. Maltz, P. Patel, and S. Sengupta. VL2: A scalable and flexible data center network. In Proceedings of the SIGCOMM conference on Data Communication, pages 51-62, 2009.
14. Hadoop. http://lucene.apache.org/hadoop/.
15. J. Hamilton. When Very Low-Power, Low-Cost Servers Don't Make Sense. In http://perspectives.mvdirona.com/2010/05/18/ WhenVeryLowPowerLowCostServersDontMakeSense.aspx, 2010.
16. B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: A MapReduce framework on graphics processors. In Proceedings of the 17th international conference on Parallel Architectures and Compilation Techniques, pages 260-269, 2008.
17. HP Labs. Project Moonshot. In http://www.hp.com/hpinfo/newsroom/press- kits/2011/MoonshotInfrastructure/index.html, 2011.
18. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys '07: Proceedings of the 2nd SIGOPS/EuroSys European Conference on Computer Systems, 2007.
19. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg. Quincy: Fair scheduling for distributed computing clusters. In Proceedings of the 22nd symposium on Operating systems principles, SOSP, pages 261-276, USA, 2009.
20. V. Janapa Reddi, B. C. Lee, T. Chilimbi, and K. Vaid. Web search using mobile cores: Quantifying and mitigating the price of efficiency. In Proceedings of the International Symposium on Computer Architecture (ISCA), 2010.
21. J. Hartigan. Clustering Algorithms. Wiley, 1975.
22. A. Krioukov, P. Mohan, S. Alspaugh, L. Keys, D. E. Culler, and R. H. Katz. NapSAC: Design and implementation of a power-proportional web cluster. ACM Computer Communication Review, 41(1), 2011.
23. W. Lang, J. M. Patel, and S. Shankar. Wimpy node clusters: What about non-wimpy workloads? In Proceedings of the International Workshop on Data Management on New Hardware (DaMoN), 2010.
24. K. Lim, P. Ranganathan, J. Chang, C. Patel, T. Mudge, and S. Reinhardt. Understanding and Designing New Server Architectures for Emerging Warehouse-Computing Environments. In Proceedings of the 35th Annual International Symposium on Computer Architecture (ISCA), pages 315-326, 2008.
25. M. D. Linderman, J. D. Collins, H. Wang, and T. H. Meng. Merge: A programming model for heterogeneous multi-core systems. In Proceedings of the 13th international conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), 2008.
26. Netflix movies data. http://www.netflixprize.com/download.
27. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: a not-so-foreign language for data processing. In Proceedings of the 2008 international conference on Management Of Data, SIGMOD, pages 1099-1110, 2008.
28. M. M. Rafique, N. Ravi, S. Cadambi, S. T. Chakradhar, and A. R. Butt. Power Management for Heterogeneous Clusters: An Experimental Study. In Proceedings of the IEEE International Conference on Green Computing, 2011.
29. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis. Evaluating MapReduce for Multi-core and Multiprocessor Systems. In 13th International Symposium on High Performance Computer Architecture, HPCA, pages 13-24, 2007.
30. SeaMicro, Inc. http://www.seamicro.com.
31. A. Vahdat, M. Al-Fares, N. Farrington, R. N. Mysore, G. Porter, and S. Radhakrishnan. Scale-Out Networking in the Data Center. IEEE Micro, 30:29-41, July 2010.
32. D. Weld. Lecture notes on MapReduce (based on Jeff Dean's slides). http://rakaposhi.eas.asu.edu/cse494/notes/s07-map-reduce.ppt, 2007.
33. X-RIME: Hadoop based large scale social network analysis. http://xrime.sourceforge.net/.
34. H.-C. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Map-reduce-merge: Simplified relational data processing on large clusters. In Proceedings of the SIGMOD international conference on Management Of Data, 2007.
35. Y. Yu, M. Isard, D. Fetterly, M. Budiu, U. Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A System for General-Purpose Distributed Data-Parallel Computing Using a High-Level Language. In Proceedings of International Symposium on Operating System Design and Implementation (OSDI), 2008.
36. M. Zaharia, D. Borthakur, J. Sen Sarma, K. Elmeleegy, S. Shenker, and I. Stoica. Delay Scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of the 5th European conference on Computer systems, EuroSys '10, pages 265-278, 2010.
37. M. Zaharia, A. Konwinski, A. Joseph, R. Katz, and I. Stoica. Improving MapReduce performance in heterogeneous environments. In Proceedings of the Usenix Symposium on Operating Systems Design and Implementation (OSDI), 2008.
38. Hadoop rebalancer. http://hadoop.apache.org/common/docs/r0.17.2/hdfs- user-guide.html#Rebalancer.