Towards scalable one-pass analytics using MapReduce

IEEE International Symposium on Parallel and Distributed Processing Workshops and Phd Forum

Volumn , Issue , 2011, Pages 1102-1111

  Mazur, Edward ^a   Li, Boduo ^a   Diao, Yanlei ^a   Shenoy, Prashant ^a  

^a Biologically Inspired Neural and Dynamical Systems Laboratory   (United States)

Author keywords

Data streams; MapReduce; Parallel data processing; Performance analysis

Indexed keywords

ANALYTICAL QUERIES; ARCHITECTURAL CHANGES; COMPUTATION MODEL; CURRENT LIMITATION; DATA SETS; DATA STREAM; DATA-INTENSIVE APPLICATION; DESIGN REQUIREMENTS; EMPIRICAL PERFORMANCE; EXISTING SYSTEMS; INITIAL DESIGN; INTEGRAL PART; LARGE DATASETS; MAP-REDUCE; ONE-PASS; ONLINE AGGREGATIONS; PARALLEL DATA PROCESSING; PERFORMANCE ANALYSIS; PROGRAMMING MODELS; STREAM PROCESSING;

BATCH DATA PROCESSING; COMPUTER PROGRAMMING; DATA HANDLING; DISTRIBUTED PARAMETER NETWORKS; SEARCH ENGINES;

PARALLEL PROCESSING SYSTEMS;

EID: 83455229796     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2011.251     Document Type: Conference Paper

References (36)

1. D. J. Abadi, Y. Ahmad, et al. The design of the Borealis stream processing engine. In CIDR, 277-289, 2005.
2. A. Abouzeid, K. Bajda-Pawlikowski, et al. Hadoopdb: An architectural hybrid of mapreduce and dbms technologies for analytical workloads. PVLDB, 2(1):922-933, 2009.
3. A. Arasu, et al. The CQL continuous query language: semantic foundations and query execution. VLDB J., 15(2): 121-142, 2006.
4. V. Borkar, M. Carey, et al. Hyracks: a flexible and extensible foundation for data-intensive computing. In ICDE, 2011.
5. S. Chandrasekaran, O. Cooper, et al. TelegraphCQ: Continuous dataflow processing for an uncertain world. In CIDR, 2003.
6. T. Condie, N. Conway, et al. Mapreduce online. In NSDI, 21-21, 2010.
7. J. Dean and S. Ghemawat. Mapreduce: simplified data processing on large clusters. In OSDI, 10-10, 2004.
8. D. DeWitt and J. Gray. Parallel database systems: the future of high performance database systems. Commun. ACM, 35(6):85-98, 1992.
9. D. J. DeWitt, R. H. Gerber, et al. Gamma - a high performance dataflow database machine. In VLDB, 228-237, 1986.
10. E. Friedman, P. Pawlowski, et al. Sql/mapreduce: A practical approach to self-describing, polymorphic, and parallelizable user-defined functions. PVLDB, 2(2):1402-1413, 2009.
11. B. Gedik, H. Andrade, et al. SPADE: The System S declarative stream processing engine. In SIGMOD, 2008.
12. P. K. Gunda, L. Ravindranath, et al. Nectar: automatic management of data and computation in data centers. In OSDI, 2010.
13. P. J. Haas and J. M. Hellerstein. Ripple joins for online aggregation. In SIGMOD, 287-298, 1999.
14. Hive project. http://hadoop.apache.org/hive/.
15. M. Isard, M. Budiu, et al. Dryad: distributed data-parallel programs from sequential building blocks. In EuroSys, 59-72, 2007.
16. C. Jermaine, A. Dobra, et al. A disk-based join with probabilistic guarantees. In SIGMOD, 563-574, 2005.
17. D. Jiang, B.C. Ooi, et al. The Performance of MapReduce: An In-depth Study. In VLDB, 472-483, 2010.
18. V. Kumar, H. Andrade, et al. DEDUCE: at the intersection of MapReduce and stream processing. In EDBT, 657-662, 2010.
19. Y. Kwon, M. Balazinska, et al. Skew-resistant parallel processing of feature-extracting scientific user-defined functions. In SoCC, 2010.
20. E. Mazur, B. Li, Y. Diao, and P. Shenoy. Towards scalable one-pass analytics using mapreduce. Technical report, UMass Amherst, 2010. http://cs.umass.edu/~mazur/tech-onepassmr.pdf.
21. K. Morton, M. Balazinska, et al. Paratimer: a progress indicator for mapreduce dags. In SIGMOD, 507-518, 2010.
22. L. Neumeyer, B. Robbins, et al. S4: distributed stream computing platform. In KDCloud, 2010.
23. C. Olston, B. Reed, et al. Pig latin: a not-so-foreign language for data processing. In SIGMOD, 1099-1110, 2008.
24. A. Pavlo, E. Paulson, et al. A comparison of approaches to large-scale data analysis. In SIGMOD, 165-178, 2009.
25. L. D. Shapiro. Join processing in database systems with large main memories. ACM Trans. Database Syst., 11(3):239-264, 1986.
26. R. Soulé, M. Hirzel, et al. A universal calculus for stream processing languages. In ESOP, 2010.
27. W. Thies, M. Karczmarek, et al. Stream It: a language for streaming applications. In CC, 179-196, 2002.
28. W. Thies, M. Karczmarek, et al. Teleport messaging for distributed stream programs. In PPoPP, 2005.
29. W. Thies and S. Amarasinghe. An empirical characterization of stream programs and its implications for language and compiler design. In PACT, 2010.
30. A. Thusoo, J. S. Sarma, et al. Hive - a warehousing solution over a map-reduce framework. PVLDB, 2(2):1626-1629, 2009.
31. F. Tian and D. J. DeWitt. Tuple routing strategies for distributed eddies. In VLDB, 333-344, 2003.
32. S. Wang and E. Rundensteiner. Scalable stream join processing with expensive predicates: workload distribution and adaptation by time-slicing. In EDBT, 299-310, 2009.
33. T. White. Hadoop: The Definitive Guide. O'Reilly Media, Inc., 2009.
34. Y. Yu, P. K. Gunda, et al. Distributed aggregation for data-parallel computing: interfaces and implementations. In SOSP, 247-260, 2009.
35. M. Zaharia, A. Konwinski, et al. Improving MapReduce Performance in Heterogeneous Environments. In OSDI, 2008.
36. Q. Zou, H. Wang, et al. From a stream of relational queries to distributed stream processing. In VLDB, 2010.