STEAMEngine: Driving MapReduce provisioning in the cloud

18th International Conference on High Performance Computing, HiPC 2011

Volumn , Issue , 2011, Pages

  Cardosa, Michael ^a   Narang, Piyush ^a   Chandra, Abhishek ^a   Pucha, Himabindu ^b   Singh, Aameek ^b  

^a UNIVERSITY OF MINNESOTA   (United States)

^b IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CLUSTERING ALGORITHMS;

BUILDING BLOCKES; DISTRIBUTED DATA ANALYSIS; ENERGY; MAP-REDUCE; PROVISIONING FRAMEWORK; RESOURCE REQUIREMENTS; RUNTIME COSTS; RUNTIMES; SPATIO-TEMPORAL; VIRTUAL CLUSTERS;

MAPREDUCE;

EID: 84863411965     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/HiPC.2011.6152649     Document Type: Conference Paper

References (31)

1. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in Proc. of OSDI, 2004.
2. "Hadoop," http://hadoop.apache.org.
3. "Amazon EMR," http://aws.amazon.com/elasticmapreduce/.
4. M. Zaharia, A. Konwinski, A. D. Joseph, R. H. Katz, and I. Stoica, "Improving mapreduce performance in heterogeneous environments," in Proceedings of OSDI, 2008.
5. "Amazon EC2," http://aws.amazon.com/ec2.
6. M. Steinder, I. Whalley, D. Carrera, I. Gaweda, and D. M. Chess, "Server virtualization in autonomic management of heterogeneous workloads," in Integrated Network Management, 2007, pp. 139-148. (Pubitemid 47485167)
7. J. S. Chase, D. C. Anderson, P. N. Thakar, A. M. Vahdat, and R. P. Doyle, "Managing energy and server resources in hosting centers," SIGOPS Oper. Syst. Rev., vol. 35, no. 5, 2001.
8. Y. Chen, A. Das, W. Qin, A. Sivasubramaniam, Q. Wang, and N. Gautam, "Managing server energy and operational costs in hosting centers," in Proceedings of ACM SIGMETRICS, 2005.
9. A. Verma, P. Ahuja, and A. Neogi, "pMapper: Power and Migration Cost Aware Placement of Applications in Virtualized Systems," in ACM Middleware, 2008.
10. T. Wood, P. Shenoy, A. Venkataramani, and M. Yousif, "Blackbox and Gray-box Strategies for Virtual Machine Migration," in Proceedings of NSDI, 2007.
11. G. Khanna, K. Beaty, G. Kar, and A. Kochut, "Application performance management in virtualized server environments," in Proceedings of 10th IEEE/IFIP Network Ops and Management Symp. (NOMS 2006), 2006.
12. A. Singh, M. Korupolu, and D. Mohapatra, "Server-storage virtualization: Integration and load balancing in data centers," in Proceedings of IEEE/ACM Supercomputing, 2008.
13. G. Ananthanarayanan, S. Kandula, A. Greenberg, I. Stoica, Y. Lu, and B. Saha, "Reining in the outliers in map-reduce clusters," in Proceedings of OSDI, 2010.
14. J. G. Koomey, "Worldwide electricity used in data centers," Environmental Research Letters, vol. 3, no. 3, 2008.
15. C. L. Belady, "In the data center, power and cooling costs more than the it equipment it supports," Electronics Cooling Magazine, vol. 13, no. 1, pp. 24-27, 2007.
16. T. Burd, T. Pering, A. Stratakos, and R. Brodersen, "A Dynamic Voltage-Scaled Microprocessor System," in IEEE ISSCC, 2000.
17. N. Tolia, Z. Wang, M. Marwah, C. Bash, P. Ranganathan, and X. Zhu, "Delivering Energy Proportionality with Non Energy-Proportional Systems-Optimizing the Ensemble," in USENIX HotPower, 2008.
18. E. M. Elnozahy, M. Kistler, and R. Rajamony, "Energy-efficient server clusters," in Proceedings of the 2nd Workshop on Power-Aware Computing Systems, 2002.
19. M. Cardosa, A. Singh, H. Pucha, and A. Chandra, "Exploiting Spatio-Temporal Tradeoffs for Energy Efficient MapReduce in the Cloud," Dept. of CSE, Univ. of Minnesota, Tech. Rep. 10-008, Apr. 2010.
20. "Scheduling in hadoop," http://www.cloudera.com/blog/tag/ scheduling/.
21. M. Isard, V. Prabhakaran, J. Currey, U. Wieder, K. Talwar, and A. Goldberg, "Quincy: fair scheduling for distributed computing clusters," in Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles (SOSP), 2009.
22. T. Sandholm and K. Lai, "Mapreduce optimization using dynamic regulated prioritization," in ACM SIGMET-RICS/Performance, 2009.
23. N. Chohan, C. Castillo, M. Spreitzer, M. Steinder, A. Tantawi, and C. Krintz, "See spot run: Using spot instances for mapreduce workflows," in Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud), 2010.
24. J. Leverich and C. Kozyrakis, "On the energy (in)efficiency of hadoop clusters," in Proceedings of the Workshop on Power Aware Computing and Systems (HotPower), 2009.
25. Y. Chen, L. Keys, and R. H. Katz, "Towards Energy Efficient MapReduce," EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2009-109, Aug 2009.
26. T. Condie, N. Conway, P. Alvaro, J. M. Hellerstein, K. Elmeleegy, and R. Sears, "Mapreduce online," in Proceedings of NSDI, 2010.
27. S. Babu, "Towards automatic optimization of mapreduce programs," in ACM SOCC, 2010.
28. A. Ganapathi, Y. Chen, A. Fox, R. H. Katz, and D. A. Patterson, "Statistics-driven workload modeling for the cloud," in Proceedings of 5th International Workshop on Self Managing Database Systems (SMDB), 2010.
29. K. Morton, A. Friesen, M. Balazinska, and D. Grossman, "Estimating the progress of mapreduce pipelines," in Proceedings of IEEE ICDE, 2010.
30. D. G. Feitelson and L. Rudolph, "Parallel job scheduling: Issues and approaches," Springer-Verlag Lecture Notes In Computer Science, vol. 949, 1995.
31. D. G. Feitelson, "Job scheduling in multiprogrammed parallel systems," IBM Research Report, vol. RC 87657, 1997