MapReduce optimization using regulated dynamic prioritization

SIGMETRICS/Performance'09 - Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems

Volumn 37, Issue 1, 2009, Pages 299-310

  Sandholm, Thomas ^a   Lai, Kevin ^a  

^a HEWLETT PACKARD LABORATORIES   (United States)

Author keywords

DISC; MapReduce; Proportional share; Resource allocation; Workflow optimization

Indexed keywords

COMPLETION TIME; COST BENEFIT RATIO; DIFFERENT SERVICES; DISC; JOB EXECUTION; JOB SCHEDULING; PRIORITIZATION; PROPORTIONAL-SHARE; REAL APPLICATIONS; SHARED DATA; SYSTEM USE; VIRTUAL MACHINES; WORKFLOW OPTIMIZATION;

OPTIMIZATION; PLANNING; RESOURCE ALLOCATION;

COMPUTER RESOURCE MANAGEMENT;

EID: 70449652584     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1555349.1555384     Document Type: Conference Paper

References (43)

1. K. Arrow. Aspects of the theory of risk-bearing. Helsinki: Yrjo Jahnsson Lectures, 1965.
2. A. AuYoung, L. Grit, J. Wiener, and J. Wilkes. Service contracts and aggregate utility functions. In Proceedings of the IEEE International Symposium on High Performance Distributed Computing (HPDC), June 2006.
3. R. Avnur and J. M. Hellerstein. Eddies: Continuously adaptive query processing. In ACM SIGMOD: International Conference on Management of Data, 2007.
4. P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris, A. Ho, R. Neugebauer, I. Pratt, and A. Warfield. Xen and the art of virtualization. In Proceedings of the ACM Symposium on Operating Systems Principles, 2003.
5. R. E. Bryant. Data-intensive supercomputing: The case for DISC. Technical Report CMU-CS-07-128, Carnegie Mellon University, 2007.
6. K. Cardona, J. Secretan, M. Georgiopoulos, and G. Anagnostopoulos. A grid based system for data mining using MapReduce. Technical Report TR-2007-02, AMALTHEA, 2007.
7. B. N. Chun, P. Buonadonna, A. AuYoung, C. Ng, D. C. Parkes, J. Shneidman, A. C. Snoeren, and A. Vahdat. Mirage: A microeconomic resource allocation system for SensorNet testbeds. In Proceedings of the 2nd IEEE Workshop on Embedded Networked Sensors, 2005.
8. B. N. Chun and D. E. Culler. Market-based proportional resource sharing for clusters. Technical Report CSD-1092, University of California at Berkeley, Computer Science Division, January 2000.
9. B. N. Chun and D. E. Culler. User-centric performance analysis of market-based cluster batch schedulers. In Proceedings of the 2nd IEEE International Symposium on Cluster Computing and the Grid, 2002.
10. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In Symposium on Operating System Design and Implementation, 2004.
11. M. Feldman, K. Lai, and L. Zhang. A price-anticipating resource allocation mechanism for distributed shared clusters. In Proceedings of the ACM Conference on Electronic Commerce, 2005.
12. http://hadoop.apache.org/core, 2008.
13. G. Hardin. The tragedy of the commons. Science, 162:1243-1248, 1968.
14. B. He, W. Fang, Q. Luo, N. K. Govindaraju, and T. Wang. Mars: a MapReduce framework on graphics processors. In PACT '08: Proceedings of the 17th international conference on Parallel architectures and compilation techniques, pages 260-269, New York, NY, USA, 2008. ACM.
15. D. Irwin, J. Chase, and L. Grit. Balancing risk and reward in market-based task scheduling. In International Symposium on High Performance Distributed Computing, 2004.
16. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys '07: Proceedings of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems 2007, pages 59-72, New York, NY, USA, 2007. ACM.
17. P. R. Jelenkovic, X. Kang, and J. Tan. Adaptive and scalable comparison scheduling. In ACM SIGMETRICS'07: International Conference on Measurement and Modeling of Computer Systems, pages 215-226, 2007.
18. E. Jensen, C. Locke, and H. Tokuda. A time-driven scheduling model for real-time operating systems. In IEEE Real-Time Systems Symposium , pages 112-122, 1985.
19. K. Lai, L. Rasmusson, E. Adar, S. Sorkin, L. Zhang, and B. A. Huberman. Tycoon: an implemention of a distributed market-based resource allocation system. Multiagent and Grid Systems, 1(3):169-182, Aug. 2005.
20. D. Logothetis and K. Yocum. Ad-hoc data processing in the cloud. Proc. VLDB Endow., 1(2):1472-1475, 2008.
21. N. Moroney, P. Obrador, and G. Beretta. Lexical image processing. In Proceedings of the 16th IS&T/SID Color Imaging Conference, pages 268-273, 2008.
22. C. Olston. Pig: Web-scale processing. http://www.cs.cmu.edu/~olston/pig. ppt, 2008.
23. C. Olston, B. Reed, A. Silberstein, and U. Srivastava. Automatic optimization of parallel dataflow programs. In USENIX Annual Technical Conference, 2008.
24. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig latin: A not-so-foreign language for data processing. In SIGMOD 2008: ACM SIGMOD/PODS Conference, 2008.
25. C. H. Papadimitriou. Algorithms, games, and the Internet. In Symposium on Theory of Computing, 2001.
26. L. Peterson, T. Anderson, D. Culler, , and T. Roscoe. Blueprint for Introducing Disruptive Technology into the Internet. In First Workshop on Hot Topics in Networking, 2002.
27. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming Journal Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure, 13(4):227-298, 2003.
28. F. I. Popovici and J. Wilkes. Profitable services in an uncertain world. In SC05: Proceedings of Supercomputing, 2005.
29. J. Pratt. Risk aversion in the small and in the large. Econometrica, 32:122-136, 1964.
30. C. Ranger, R. Raghuraman, A. Penmetsa, G. Bradski, and C. Kozyrakis. Evaluating MapReduce for multi-core and multiprocessor systems. In HPCA'07: IEEE 13th International Symposium on High Performance Computer Architecture , pages 13-24, 2007.
31. T. Sandholm. Statistical methods for computational markets. Doctoral Thesis ISRN SU-KTH/DSV/R-08/6-SE. Royal Institute of Technology, Stockholm, 2008.
32. T. Sandholm and K. Lai. A statistical approach to risk mitigation in computational markets. In Proceedings of the ACM International Symposium on High Performance Distributed Computing (HPDC), June 2007.
33. T. Sandholm, K. Lai, J. Andrade, and J. Odeberg. Market-based resource allocation using price prediction in a high performance computing grid for scientific applications. In Proceedings of the IEEE International Symposium on High Performance Distributed Computing (HPDC), June 2006.
34. P. G. Selinger, M. M. Astrahan, D. D. Chamberlin, R. A. Lorie, and T. G. Price. Access path selection in a relational database management system. In SIGMOD 1979: ACM SIGMOD International Conference on the Management of Data, 1979.
35. M. A. Shah, J. M. Hellerstein, S. Chandrasekaran, and M. J. Franklin. Flux: An adaptive partitioning operator for continuous query systems. Technical Report UCB/CSD-2-1205, U. C. Berkley, 2002.
36. M. Stonebraker, P. M. Aoki, W. Litwin, A. Pfeffer, A. Sah, J. Sidell, C. Staelin, and A. Yu. Mariposa: a wide-area distributed database system. The VLDB Journal, 5(1):048-063, 1996.
37. S. V. Valvag and D. Johansen. Oivos: Simple and efficient distributed data processing. High Performance Computing and Communications, 2008. HPCC '08. 10th IEEE International Conference on, pages 113-122, Sept. 2008.
38. M. Wachs, M. Abd-El-Malek, E. Thereska, and G. R. Ganger. Argon: performance insulation for shared storage servers. In FAST'07: 5th USENIX Conference on File and Storage Technologies, 2007.
39. C. A. Waldspurger and W. E. Weihl. Lottery scheduling: Flexible proportional-share resource management. In Operating Systems Design and Implementation, pages 1-11, 1994.
40. A. Wierman and M. Nuyens. Scheduling despite inexact job-size information. In ACM SIGMETRICS'08: International Conference on Measurement and Modeling of Computer Systems, pages 25-36, 2008.
41. J. Wolfe, A. Haghighi, and D. Klein. Fully distributed EM for very large datasets. In ICML '08: Proceedings of the 25th international conference on Machine learning, pages 1184-1191, New York, NY, USA, 2008. ACM.
42. M. Zaharia, A. Konwinski, A. D. Joseph, R. Katz, and I. Stoica. Improving MapReduce performance in heterogeneous environments. In OSDI'08: 8th USENIX Symposium on Operating Systems Design and Implementation , 2008.
43. L. Zhang. The efficiency and fairness of a fixed budget resource allocation game. In International Colloquium on Automata, Languages and Programming, pages 485-496, 2005.