Adaptive MapReduce using situation-aware mappers

ACM International Conference Proceeding Series

Volumn , Issue , 2012, Pages 420-431

  Vernica, Rares ^a   Balmin, Andrey ^b   Beyer, Kevin S ^b   Ercegovac, Vuk ^b  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b IBM ALMADEN RESEARCH CENTER   (United States)

Author keywords

Algorithms; H.2.4 Database Management : Systems query processing; Performance

Indexed keywords

ADAPTIVE SAMPLING; ADAPTIVE TECHNIQUE; EXPERIMENTAL EVALUATION; GLOBAL STATE; MAP-REDUCE; MULTIPLE DATA; PERFORMANCE; PERFORMANCE IMPROVEMENTS; PERFORMANCE STABILITY; RUNTIMES; SITUATION-AWARE; START-UP COSTS; SYSTEMS-QUERY PROCESSING;

ALGORITHMS; APPLICATION PROGRAMMING INTERFACES (API); FAULT TOLERANCE;

DATABASE SYSTEMS;

EID: 84863511023     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2247596.2247646     Document Type: Conference Paper

References (36)

1. Apache Hadoop, http://hadoop.apache.org.
2. B. F. Cooper et al., "Building a cloud for Yahoo!" IEEE Data Eng. Bull., vol. 32, no. 1, pp. 36-43, 2009.
3. J. Dean and S. Ghemawat, "MapReduce: a flexible data processing tool," Commun. ACM, vol. 53, no. 1, pp. 72-77, 2010.
4. A. Gates et al., "Building a highlevel dataflow system on top of MapReduce: the Pig experience," PVLDB, vol. 2, no. 2, pp. 1414-1425, 2009.
5. A. Thusoo et al., "Hive - a petabyte scale data warehouse using Hadoop," in ICDE, 2010, pp. 996-1005.
6. Y. Bu and et al., "Haloop: Efficient iterative data processing on large clusters," PVLDB, vol. 3, no. 1, pp. 285-296, 2010.
7. T. Condie et al., "MapReduce online," in NSDI, 2010, pp. 313-328.
8. J. Dittrich et al., "Hadoop++: Making a yellow elephant run like a cheetah (without it even noticing)," PVLDB, vol. 3, no. 1, pp. 518-529, 2010.
9. D. Jiang et al., "The performance of MapReduce: an in-depth study," PVLDB, vol. 3, no. 1, pp. 472-483, 2010.
10. T. Nykiel et al., "MRShare: sharing across multiple queries in MapReduce," PVLDB, vol. 3, no. 1, pp. 494-505, 2010.
11. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: distributed data-parallel programs from sequential building blocks," in EuroSys, 2007, pp. 59-72.
12. V. R. Borkar, M. J. Carey, R. Grover, N. Onose, and R. Vernica, "Hyracks: A flexible and extensible foundation for data-intensive computing," in ICDE, 2011.
13. M. Zaharia et al., "Spark: cluster computing with working sets," in HotCloud, USENIX workshop on Hot topics in cloud computing, 2010.
14. D. Battré et al., "Nephele/PACTs: a programming model and execution framework for web-scale analytical processing," in SoCC, 2010, pp. 119-130.
15. D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand, "Ciel: a universal execution engine for distributed data-flow computing," in NSDI, 2011.
16. Jaql, http://www.jaql.org.
17. S. Babu, "Towards automatic optimization of MapReduce programs," in SoCC, 2010, pp. 137-142.
18. A. Shatdal et al., "Adaptive parallel aggregation algorithms," in SIGMOD Conf., 1995, pp. 104-114.
19. D. J. DeWitt, J. F. Naughton, and D. A. Schneider, "Parallel sorting on a shared-nothing architecture using probabilistic splitting," in PDIS, 1991, pp. 280-291.
20. D. A. Schneider et al., "A performance evaluation of four parallel join algorithms in a shared-nothing multiprocessor environment," in SIGMOD Conf., 1989, pp. 110-121.
21. Apache ZooKeeper http://hadoop.apache.org/zookeeper.
22. P. Hunt et al., "ZooKeeper: wait-free coordination for internet-scale systems," in USENIX Conf., 2010.
23. M. Zaharia et al., "Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling," in EuroSys, 2010, pp. 265-278.
24. J. Wolf et al., "FLEX: a slot allocation scheduling optimizer for MapReduce workloads," in Middleware, to appear, 2010.
25. J. Cieslewicz and K. A. Ross, "Adaptive aggregation on chip multiprocessors," in VLDB, 2007, pp. 339-350.
26. S. Blanas et al., "A comparison of join algorithms for log processing in MapReduce," in SIGMOD Conf., 2010, pp. 975-986.
27. R. Vernica et al., "Efficient parallel set-similarity joins using MapReduce," in SIGMOD Conf., 2010.
28. O. O'Malley and A. C. Murthy, "Winning a 60 second dash with a yellow elephant," Yahoo! Inc., Tech. Rep., 2009.
29. "Skewed join in Pig,"http://wiki.apache.org/pig/ PigSkewedJoinSpec.
30. A. Abouzeid et al., "HadoopDB: an architectural hybrid of MapReduce and dbms technologies for analytical workloads," PVLDB, vol. 2, no. 1, pp. 922-933, 2009.
31. A. Pavlo et al., "A comparison of approaches to large-scale data analysis," in SIGMOD Conf., 2009, pp. 165-178.
32. C. Olston et al., "Automatic optimization of parallel dataflow programs," in USENIX Conf., 2008, pp. 267-273.
33. B. Li, E. Mazur, Y. Diao, A. McGregor, and P. Shenoy, "A platform for scalable one-pass analytics using mapreduce,"sIGMOD 2011.
34. M. Kitsuregawa et al., "Application of hash to data base machine and its architecture," New Generation Comput., vol. 1, no. 1, pp. 63-74, 1983.
35. D. J. DeWitt, J. F. Naughton, D. A. Schneider, and S. Seshadri, "Practical skew handling in parallel joins," in VLDB, 1992, pp. 27-40.
36. J. L. Wolf et al., "New algorithms for parallelizing relational database joins in the presence of data skew," IEEE Trans. Knowl. Data Eng., vol. 6, no. 6, pp. 990-997, 1994.