Dryad: Distributed data-parallel programs from sequential building blocks

Operating Systems Review (ACM)

Volumn , Issue , 2007, Pages 59-72

  Isard, Michael ^a   Budiu, Mihai ^a   Yu, Yuan ^a   Birrell, Andrew ^a   Fetterly, Dennis ^a  

^a MICROSOFT RESEARCH   (United States)

Author keywords

Cluster computing; Concurrency; Dataflow; Distributed programming

Indexed keywords

COMMUNICATION CHANNELS (INFORMATION THEORY); DATA FLOW ANALYSIS; DATA PROCESSING; GRAPH THEORY; PARALLEL ALGORITHMS; SCHEDULING; CLUSTER COMPUTING; COMPUTER SYSTEM RECOVERY; PARALLEL ARCHITECTURES; PROGRAM PROCESSORS;

CLUSTER COMPUTING; SEQUENTIAL BUILDING BLOCKS;

MULTIPROCESSING PROGRAMS; COMPUTER APPLICATIONS;

APPLICATION DEVELOPERS; BUILDING BLOCKES; CPU CORES; DATA CENTERS; DATA PARALLEL; DATA-FLOW GRAPHS; DATAFLOW; DISTRIBUTED DATA; DISTRIBUTED EXECUTION; DISTRIBUTED PROGRAMMING; EXECUTION ENGINE; MULTI CORE; MULTIPLE COMPUTERS; RUNTIMES; SEQUENTIAL PROGRAMS; SHARED MEMORIES; SINGLE COMPUTER; SMALL CLUSTERS;

EID: 34548041192     PISSN: 01635980     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1272996.1273005     Document Type: Conference Paper

References (38)

1. Global grid forum, http://www.gridforum.org/.
2. Intel IXP2XXX product line of network processors. http://www.intel.com/ design/network/products/npfamily/ixp2xxx.htm.
3. Intel platform 2015. http://www.intel.com/technology/architecture/ platform2015/.
4. The LINQ project, http://msdn.microsoft.com/netframework/future/linq/.
5. Open MPI. http://www.open-mpi.org/.
6. SQL Server Integration Services. http://www.microsoft.com/sql/ technologies/integration/default.mspx.
7. Thomas E. Anderson, David E. Culler, David A. Patterson, and NOW Team. A case for networks of workstations: NOW. IEEE Micro, pages 54-64, February 1995.
8. Remzi H. Arpaci-Dusseau. Run-time adaptation in River. Transactions on Computer Systems (TOCS), 21(1):36-86, 2003.
9. Özalp Babaoǧlu, Lorenzo Alvisi, Alessandro Amoroso, Renzo Davoli, and Luigi Alberto Giachini. Paralex: an environment for parallel programming in distributed systems, pages 178-187, New York, NY, USA, 1992. ACM Press.
10. Magdalena Balazinska, Hari Balakrishnan, Samuel Madden, and Mike Stonebraker. Fault-Tolerance in the Borealis Distributed Stream Processing System. In ACM SIGMOD, Baltimore, MD, June 2005.
11. Tom Barclay, Robert Barnes, Jim Gray, and Prakash Sundaresan. Loading databases using dataflow parallelism. SIGMOD Rec., 23(4):72-83, 1994.
12. Chaitanya Baru and Gilles Fecteau. An overview of DB2 parallel edition. In SIGMOD '95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data, pages 460-462, New York, NY, USA, 1995. ACM Press.
13. Guy E. Blelloch. Programming parallel algorithms. Communications of the ACM (CACM), 39(3):85-97, 1996.
14. Robert D. Blumofe, Christopher F. Joerg, Bradley Kuszmaul, Charles E. Leiserson, Keith H. Randall, and Yuli Zhou. Cilk: An efficient multithreaded runtime system. In ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming (PPoPP), pages 207-216, Santa Barbara, California, July 19-21 1995.
15. Eylon Caspi, Michael Chu, Randy Huang, Joseph Yeh, Yury Markovskiy, André DeHon, and John Wawrzynek. Stream computations organized for reconfigurable execution (SCORE): Introduction and tutorial. In FPL, Lecture Notes in Computer Science. Springer Verlag, 2000.
16. Jeff Dean and Sanjay Ghemawat. MapReduce: Simplified data processing on large clusters. In Proceedings of the 6th Symposium on Operating Systems Design and Implementation (OSDI), pages 137-150, December 2004.
17. D. DeWitt, S. Ghandeharizadeh, D. Schneider, H. Hsiao, A. Bricker, and R. Rasmussen. The GAMMA database machine project. IEEE Transactions on Knowledge and Data Engineering, 2(1), 1990.
18. David DeWitt and Jim Gray. Parallel database systems: The future of high performance database processing. Communications of the ACM, 36(6), 1992.
19. D. C. DiNucci and R. G. Babb II. Design and implementation of parallel programs with LGDF2. In Digest of Papers from Compcon '89, pages 102-107, 1989.
20. Armando Fox, Steven D. Gribble, Yatin Chawathe, Eric A. Brewer, and Paul Gauthier. Cluster-based scalable network services. In A CM Symposium on Operating Systems Principles (SOSP), pages 78-91, New York, NY, USA, 1997. ACM Press.
21. Sanjay Ghemawat, Howard Gobioff, and Shun-Tak Leung. The Google file system. In SOSP '03: Proceedings of the nineteenth A CM symposium on Operating systems principles, pages 29-43, New York, NY, USA, 2003. ACM Press.
22. Goetz Graefe. Encapsulation of parallelism in the Volcano query processing system. In SIGMOD '90: Proceedings of the 1990 ACM SIGMOD international conference on Management of data, pages 102-111, New York, NY, USA, 1990. ACM Press.
23. J. Gray, A.S. Szalay, A. Thakar, P. Kunszt, C. Stoughton, D. Slutz, and J. Vandenberg. Data mining the SDSS SkyServer database. In Distributed Data and Structures 4: Records of the 4th International Meeting, pages 189-210, Paris, France, March 2002. Carleton Scientific, also as MSR-TR-2002-01.
24. Jim Gray and Alex Szalay. Science in an exponential world. Nature, 440(23), March 23 2006.
25. J.-H. Hwang, M. Balazinska, A. Rasin, U. Çetintemel, M. Stonebraker, and S. Zdonik. A comparison of stream-oriented high-availability algorithms. Technical Report TR-03-17, Computer Science Department, Brown University, September 2003.
26. Ujval Kapasi, William J. Dally, Scott Rixner, John D. Owens, and Brucek Khailany. The Imagine stream processor. In Proceedings 2002 IEEE International Conference on Computer Design, pages 282-288, September 2002.
27. Eddie Kohler, Robert Morris, Benjie Chen, John Jannotti, and M. Frans Kaashoek. The Click modular router. ACM Transactions on Computer Systems, 18(3):263-297, 2000.
28. James Larus and Michael Parkes. Using cohort scheduling to enhance server performance. In Usenix Annual Technical Conference, June 2002.
29. Orlando Loques, Julius Leite, and Enrique Vinicio Carrera E. P-RIO: A modular parallel-programming environment. IEEE Concurrency, 6(1):47-57, 1998.
30. William Mark, Steven Glanville, Kurt Akeley, and Mark J. Kilgard. Cg: A system for programming graphics hardware in a C-like language. ACM Transactions on Graphics, 22(3):896-907, 2003.
31. P. Newton and J.C. Browne. The CODE 2.0 graphical parallel programming language, pages 167-177, Washington, D. C., United States, July 1992.
32. Ken Phillips. SenSage ESA. SC Magazine, March 1 2006.
33. Rob Pike, Sean Dorward, Robert Griesemer, and Sean Quinlan. Interpreting the data: Parallel analysis with Sawzall. Scientific Programming, 13(4):277-298, 2005.
34. Mehul A. Shah, Joseph M. Hellerstein, and Eric Brewer. Highly available, fault-tolerant, parallel dataflows. In SIGMOD '04: Proceedings of the 2004 ACM SIGMOD international conference on Management of data, pages 827-838, New York, NY, USA, 2004. ACM Press.
35. V. S. Sunderam. PVM: a framework for parallel distributed computing. Concurrency: Pract. Exper., 2(4):315-339, 1990.
36. David Tarditi, Sidd Puri, and Jose Oglesby. Accelerator: using data-parallelism to program GPUs for general-purpose uses. In International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), Boston, MA, October 2006. also as MSR-TR-2005-184.
37. Douglas Thain, Todd Tannenbaum, and Miron Livny. Distributed computing in practice: The Condor experience. Concurrency and Computation: Practice and Experience, 17(2-4):323-356, 2005.
38. P.W. Trinder, H-W. Loidl, and R.F. Pointon. Parallel and distributed Haskells. Journal of Functional Programming, 12(4&5):469-510, 2002.