Resource provisioning framework for MapReduce jobs with performance goals

Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Volumn 7049 LNCS, Issue , 2011, Pages 165-186

  Verma, Abhishek ^a   Cherkasova, Ludmila ^b   Campbell, Roy H ^a  

^a UNIVERSITY OF ILLINOIS AT URBANA CHAMPAIGN   (United States)

^b HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

APPLICATION PERFORMANCE; CAPACITY PLANNING; COMPLETION TIME; DATA MINING TASKS; DATA SETS; JOB COMPLETION; LARGE SCALE DATA; MAP-REDUCE; NODE FAILURE; PROFILING TOOLS; REALISTIC APPLICATIONS; RESOURCE PROVISIONING; SCALING RULES; SERVICE PROVIDER; SPAM DETECTION; WORST CASE SCENARIO; EFFICIENT CAPACITIES; LARGE-SCALE DATA PROCESSING; PERSONALIZED ADVERTISINGS; PROVISIONING FRAMEWORK;

CLOUD COMPUTING; CLUSTER COMPUTING; MIDDLEWARE; DATA MINING; DISTRIBUTED COMPUTER SYSTEMS;

DATA HANDLING;

EID: 83755196778     PISSN: 03029743     EISSN: 16113349     Source Type: Book Series    
DOI: 10.1007/978-3-642-25821-3_9     Document Type: Conference Paper

References (26)

1. Dean, J., Ghemawat, S.: MapReduce: Simplified data processing on large clusters. Communications of the ACM 51(1), 107-113 (2008)
2. O'Malley, O., Murthy, A.: Winning a 60 second dash with a yellow elephant (2009)
3. White, T.: Hadoop: The Definitive Guide, Page 6. Yahoo Press
4. Graham, R.: Bounds for certain multiprocessing anomalies. Bell System Tech. Journal 45, 1563-1581 (1966)
5. Ko, S., Hoque, I., Cho, B., Indranil Gupta, I.: On Availability of Intermediate Data in Cloud Computations. In: Proceedings of HotOS 2009 (2009)
6. Kwak, H., Lee, C., Park, H., Moon, S.: What is Twitter, a social network or a news media?. In: Proc. of Intl. Conference on World Wide Web, pp. 591-600. ACM (2010)
7. Apache, Capacity Scheduler Guide (2010), http://hadoop.apache.org/common/ docs/r0.20.1/capacity scheduler.html
8. Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., Stoica, I.: Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In: Proc. of EuroSys, pp. 265-278. ACM (2010)
9. Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., Goldberg, A.: Quincy: fair scheduling for distributed computing clusters. In: SOSP. ACM (2009)
10. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. ACM SIGOPS OS Review 41(3) (2007)
11. Sandholm, T., Lai, K.: Dynamic Proportional Share Scheduling in Hadoop. In: Frachtenberg, E., Schwiegelshohn, U. (eds.) JSSPP 2010. LNCS, vol. 6253, pp. 110-131. Springer, Heidelberg (2010)
12. Wolf, J., Rajan, D., Hildrum, K., Khandekar, R., Kumar, V., Parekh, S., Wu, K.-L., Balmin, A.: FLEX: A Slot Allocation Scheduling Optimizer for MapReduce Workloads. In: Gupta, I., Mascolo, C. (eds.) Middleware 2010. LNCS, vol. 6452, pp. 1-20. Springer, Heidelberg (2010)
13. Polo, J., Carrera, D., Becerra, Y., Torres, J., Ayguadé, E., Steinder, M., Whalley, I.: Performance-driven task co-scheduling for mapreduce environments. In: 12th IEEE/IFIP Network Operations and Management Symposium (2010)
14. Ganapathi, A., Chen, Y., Fox, A., Katz, R., Patterson, D.: Statistics-driven workload modeling for the cloud. In: Proceedings of SMDB (2010)
15. Morton, K., Balazinska, M., Grossman, D.: ParaTimer: a progress indicator for MapReduce DAGs. In: Proceedings of SIGMOD, pp. 507-518. ACM (2010)
16. Olston, C., Reed, B., Srivastava, U., Kumar, R., Tomkins, A.: Pig latin: a not-so-foreign language for data processing. In: Proceedings of SIGMOD, pp. 1099-1110. ACM (2008)
17. Morton, K., Friesen, A., Balazinska, M., Grossman, D.: Estimating the progress of MapReduce pipelines. In: Proceedings of ICDE, pp. 681-684. IEEE (2010)
18. Phan, L., Zhang, Z., Loo, B., Lee, I.: Real-timeMapReduce Scheduling. In: Technical Report No. MS-CIS-10-32, University of Pennsylvania (2010)
19. Zaharia, M., Konwinski, A., Joseph, A., Katz, R., Stoica, I.: Improving mapreduce performance in heterogeneous environments. In: OSDI (2008)
20. Verma, A., Cherkasova, L., Campbell, R.: ARIA: Automatic Resource Inference and Allocation for MapReduce Environments. In: Proc. of ICAC (2011)
21. Verma, A., Cherkasova, L., Campbell, R.: SLO-Driven Right-Sizing and Resource Provisioning of MapReduce Jobs. In: Proc. of Workshop on Large Scale Distributed Systems and Middleware, LADIS (2011)
22. Tan, J., Pan, X., Kavulya, S., Marinelli, E., Gandhi, R., Narasimhan, P.: Kahuna: Problem Diagnosis for MapReduce-based Cloud Computing Environments. In: NOMS (2010)
23. Tan, J., Kavulya, S., Gandhi, R., Narasimhan, P.: Visual, Log-Based Causal Tracing for Performance Debugging of MapReduce Systems. In: ICDCS, pp. 795-806. IEEE (2010)
24. Ananthanarayanan, G., Kandula, S., Greenberg, A., Stoica, I., Lu, Y., Saha, B., Harris, E.: Reining in the Outliers in Map-Reduce Clusters using Mantri. In: OSDI (2010)
25. Intel, Optimizing Hadoop* Deployments, (2010), http://communities. intel.com/docs/DOC-4218
26. Kambatla, K., Pathak, A., Pucha, H.: Towards optimizing hadoop provisioning in the cloud. In: Proc. of the First Workshop on Hot Topics in Cloud Computing (2009)