FlumeJava: Easy, efficient data-parallel pipelines

ACM SIGPLAN Notices

Volumn 45, Issue 6, 2010, Pages 363-375

  Chambers, Craig ^a   Raniwala, Ashish ^a   Perry, Frances ^a   Adams, Stephen ^a   Henry, Robert R ^a   Bradshaw, Robert ^a   Weizenbaum, Nathan ^a  

^a GOOGLE INC   (United States)

Author keywords

Data parallel programming; Java; MapReduce

Indexed keywords

DATA PARALLEL; DATA REPRESENTATIONS; DATA-FLOW GRAPHS; DATA-PARALLEL PROGRAMMING; EXECUTION STRATEGIES; HIGH-LEVEL ABSTRACTION; JAVA; JAVA LIBRARY; MAP-REDUCE; OPTIMIZED OPERATIONS; PARALLEL DATA; PARALLEL OPERATIONS; REAL-WORLD;

ABSTRACTING; COMPUTER SOFTWARE SELECTION AND EVALUATION; DATA FLOW ANALYSIS; JAVA PROGRAMMING LANGUAGE; OPTIMIZATION; PIPELINES;

PARALLEL PROGRAMMING;

EID: 77957570704     PISSN: 15232867     EISSN: None     Source Type: Journal    
DOI: 10.1145/1809028.1806638     Document Type: Conference Paper

References (20)

1. Cascading. http://www.cascading.org.
2. Hadoop. http://hadoop.apache.org.
3. Pig. http://hadoop.apache.org/pig.
4. R. Chaiken, B. Jenkins, P.-Å . Larson, B. Ramsey, D. Shakib, S.Weaver, and J. Zhou. SCOPE: Easy and efficient parallel processing of massive data sets. Proceedings of the VLDB Endowment (PVLDB), 1(2), 2008.
5. F. Chang, J. Dean, S. Ghemawat,W. C. Hsieh, D. A.Wallach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. Bigtable: A distributed storage system for structured data. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2006.
6. J. Dean. Experiences with MapReduce, an abstraction for large-scale computation. In Parallel Architectures and Compilation Techniques (PACT), 2006.
7. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51, no.1, 2008.
8. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2004.
9. S. Ghemawat, H. Gobioff, and S.-T. Leung. The Google file system. In ACM Symposium on Operating Systems Principles (SOSP), 2003.
10. R. H. Halstead Jr. New ideas in parallel Lisp: Language design, implementation, and programming tools. In Workshop on Parallel Lisp, 1989.
11. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: Distributed data-parallel programs from sequential building blocks. In EuroSys, 2007.
12. J. R. Larus. C++: A large-grain, object-oriented, data-parallel programming language. In Languages and Compilers for Parallel Computing (LCPC), 1992.
13. C. Lasser and S. M. Omohundro. The essential Star-lisp manual. Technical Report 86.15, Thinking Machines, Inc., 1986.
14. E. Meijer, B. Beckman, and G. Bierman. LINQ: Reconciling objects, relations and XML in the .NET framework. In ACM SIGMOD International Conference on Management of Data, 2006.
15. R. S. Nikhil and Arvind. Implicit Parallel Programming in pH. Academic Press, 2001.
16. C. Olston, B. Reed, U. Srivastava, R. Kumar, and A. Tomkins. Pig Latin: A not-so-foreign language for data processing. In ACM SIGMOD International Conference on Management of Data, 2008.
17. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming, 13(4), 2005.
18. J. R. Rose and G. L. Steele Jr. C&z.ast: An extended C language. In C++ Workshop, 1987.
19. H.-c. Yang, A. Dasdan, R.-L. Hsiao, and D. S. Parker. Map-reducemerge: Simplified relational data processing on large clusters. In ACM SIGMOD International Conference on Management of Data, 2007.
20. Y. Yu, M. Isard, D. Fetterly, M. Budiu, Ú . Erlingsson, P. K. Gunda, and J. Currey. DryadLINQ: A system for general-purpose distributed dataparallel computing using a high-level language. In USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2008.