Composing and executing parallel data-flow graphs with shell pipes

Proceedings of the 4th Workshop on Workflows in Support of Large-Scale Science, WORKS '09, in Conjunction with SC 2009

Volumn , Issue , 2009, Pages

  Walker, Edward ^a   Xu, Weijia ^a   Chandar, Vinoth ^b  

^a UNIVERSITY OF TEXAS AT AUSTIN   (United States)

^b ORACLE CORPORATION   (United States)

Author keywords

Coordination languages; Data flow processing; Parallel processing

Indexed keywords

COORDINATION LANGUAGE; COORDINATION LAYER; DATA-FLOW COMPUTATION; DATAFLOW; HIGH THROUGHPUT; LANGUAGE EXTENSIONS; LEGACY COMPONENT; MICRO AND MACRO; PARALLEL COMPUTATION; PARALLEL DATA; PARALLEL PROCESSING;

DATA FLOW ANALYSIS; GRAPHIC METHODS; LINGUISTICS; MANAGEMENT; PARALLEL FLOW; PIPE; QUERY LANGUAGES; THROUGHPUT;

DATA PROCESSING;

EID: 74249107667     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1645164.1645175     Document Type: Conference Paper

References (34)

1. Dennis. M. Ritchie, "The Evolution of the Unix Time-Sharing System", AT&T Laboratories Technical Journal, vol. 6(2) (Oct. 1984), 1577-1593, Oct. 1984.
2. R. P. Lai, "In Praise of Scripting: Real Programming Pragmatism", IEEE Computer, 22-26, July 2008.
3. L. Prechelt, "An Empirical Comparison of Seven Programming Languages", IEEE Computer, vol. 33(10), 23-29, Oct. 2000.
4. st Century", IEEE Computer, 23-30, Mar. 1998.
5. G. Kahn, "The Semantics of a Simple Language for Parallel Programming", Information Processing, vol. 4, 471-475, 1974.
6. G. Kahn and D. B. MacQueen, "Coroutines and Networks of Parallel Processes", Information Processing, vol. 7, 993-998, 1997.
7. T. M. Parks, Bounded Scheduling of Process Networks, PhD Thesis, University of California at Berkeley, 1995.
8. Edward Lee, "Dataflow Process Networks", Proceedings of IEEE, vol. 83(5) (May 1995), 773-801, 1995.
9. D. Gelernter and N. Carriero, "Coordination Languages and their Significance", Comm. ACM, 35(2), 97-107, 1992.
10. G. Papadopoulos, and F. Arbab, Coordination Models and Languages, Advances in Computers - The Engineering of Large Systems, vol. 46, Academic Press, 329-400, 1998.
11. Edward Lee et.al. The Ptolemy Project. (online) http://ptolemy.eecs. berkeley.edu/
12. J. P. Hartmann. 2007. CMS Pipelines Explained. (online) http://vm.marist.edu/~pipeline/pipjarg.pdf
13. B. Gailer. Python-pipelines: A Python Implementation of Hartmann (CMS) Pipelines. (online) http://code.google.com/p/python-pipelines/
14. th International Workshop on High-Level Programming Models and Supportive Environments (HIPS'2004), 12-21, 2004.
15. D. Jones, "Windows PowerShell: Rethinking the Pipeline", Microsoft TechNet Magazine, July 2007.
16. Yahoo!Pipes. (online) http://pipes.yahoo.com/pipes/
17. th Symposium on Operating System Design and Implementation (OSDI'04), (2004), 137-150, 2004.
18. H. Chih Yang, A. Dasdan, R-L. Hsiao, and D. S. Parker, "Map-Reduce-Merge: Simplified Relational Data Processing on Large Clusters", in Proc. of SIGMOD International Conf. on Management of Data, 2007.
19. th International Symposium of High-Performance Computer Architecture (HPCA), Feb. 2007.
20. E. Walker, and T. Minyard, and J. Boisseau, "GridShell: A Login Shell for Orchestrating and Coordinating Applications in a Grid Enabled Environment", Proc. of International Conference of Computing, Communications and Control Technologies, 182-187, Austin, TX, August 2004.
21. ApectShell: Aspect-oriented scripting shells. (online) http://www.tacc.utexas.edu/~ewalker/gridshell/GridShell.htm
22. Apache Hadoop. (online) http://hadoop.apache.org/core/
23. R. Pike, S. Dorward, R. Griesemer, and S. Quinlan, "Interpreting the Data: Parallel Analysis with Sawzall", Scientific Programming 13(4), 2005.
24. C. Olston, B. Reed, U. Srivastave, R. Kumar, and A. Tomkins, "Pig Latin: A not so Foreign Language for Data Processing", in Proc. of International Conf. on Management of Data (Industrial Track), 2008.
25. th Symposium on operating System Design and Iplementation (OSDI'08), 2008.
26. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks", in Proc. of European Conference on Computer Systems (EuroSys), 2007.
27. J. Carnahan, and D. DeCoste, "Pipelets: A Framework for Distributed Computation", W4: Learning in Web Search, 2005.
28. R. Chaiken, B. Jenkins, P-A. Larson, B. Ramsey, D. Shakib, S. Weaver, and J. Zhou, "SCOPE: Easy and Efficient Parallel Processing of Massive Data Sets", in Proc. of International Conference of Very Large Data Bases (VLDB), 2008.
29. TeraSort Benchmark. (online) http://www.hpl.hp.com/hosted/sortbenchmark/
30. Project Gutenberg. (online) http://www.gutenberg.org
31. Dataflow Graphs in System Shells. (online) http://sites.google.com/site/ ewalker544/dataflowshell
32. TOP500 Supercomputing Sites (online), http://www.top500.org.
33. David Kitchin, Adrian Quark, William Cook, and Jayadev Misra, "The Orc Programming Language", in Proc. FMOODS/FORTE, Springer Verlag, LNCS 5522, 1-25, 2009
34. J. Karges, O. Ritter, and S. Suhai, "Design and Implementation of a Parallel Pipe", Operating System Reviews, 31(2), 64-94, 1997.