Swift/T: Large-scale application composition via distributed-memory dataflow processing

Proceedings - 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing, CCGrid 2013

Volumn , Issue , 2013, Pages 95-102

  Wozniak, Justin M ^a   Armstrong, Timothy G ^b   Wilde, Michael ^a   Katz, Daniel S ^a   Lusk, Ewing ^a   Foster, Ian T ^a,b  

^a ARGONNE NATIONAL LABORATORY   (United States)

^b THE UNIVERSITY OF CHICAGO   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CONCURRENT PROGRAMMING; DISTRIBUTED MEMORY; ENABLING TECHNOLOGIES; HIGH-END COMPUTING; INDEPENDENT COMPONENTS; LARGE-SCALE APPLICATIONS; OPTIMIZING COMPILERS; SCIENTIFIC APPLICATIONS;

COMPUTER PROGRAMMING; GRID COMPUTING; OPTIMIZATION; SUPERCOMPUTERS;

SCALABILITY;

EID: 84881276723     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGrid.2013.99     Document Type: Conference Paper

References (36)

1. J. Evans and A. Rzhetsky, "Machine science, " Science, vol. 329, no. 5990, pp. 399-400, 2010.
2. S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi, "Optimization by simulated annealing, " Science, vol. 220, no. 4598, pp. 671-680, 1983.
3. A. Mahajan, S. Leyffer, and C. Kirches, "Solving mixed-integer nonlinear programs by QP-Diving, " Mar. 2012, preprint ANL/MCS-P2071- 0312.
4. J. W. Jones, G. Hoogenboom, P. Wilkens, C. Porter, and G. Tsuji, Eds., Decision Support System for Agrotechnology Transfer Version 4.0: Crop Model Documentation. University of Hawaii, 2003.
5. M. Hategan, J. Wozniak, and K. Maheshwari, "Coasters: uniform resource provisioning and access for scientific computing on clouds and grids, " in Proc. Utility and Cloud Computing, 2011.
6. M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster, "Swift: A language for distributed parallel scripting, " Par. Comp., vol. 37, pp. 633-652, 2011.
7. J. K. Ousterhout, "Scripting: higher level programming for the 21st century, " Computer, vol. 31, no. 3, pp. 23- 30, Mar. 1998.
8. D. Beazley, "Automated scientific software scripting with SWIG, " Future Generation Computer Systems, vol. 19, no. 5, pp. 599-609, 2003.
9. V. Sarkar and J. Hennessy, "Partitioning parallel programs for macrodataflow, " in LFP'86. New York, NY, USA: ACM, 1986, pp. 202-211.
10. S. Tasirlar and V. Sarkar, "Data-Driven Tasks and their implementation, " in Int'l Conf. on Parallel Processing (ICPP) 2009, Sep. 2011, pp. 652- 661.
11. D. G. Murray, M. Schwarzkopf, C. Smowton, S. Smith, A. Madhavapeddy, and S. Hand, "CIEL: a universal execution engine for distributed data-flow computing, " in Proc. NSDI 2011, 2011.
12. S. Evripidou, G. Gao, J.-L. Gaudiot, and V. Sarkar, Eds., 1st Workshop on Data-Flow Execution Models for Extreme Scale Computing: DFM'11. IEEE Computer Society, Oct. 2011.
13. P. Cicotti, "Tarragon: a programming model for latency-hiding scientific computation, " Ph.D. dissertation, U. California, San Diego, 2011.
14. G. Bosilca, A. Bouteiller, A. Danalis, T. Herault, P. Lemarinier, and J. Dongarra, "DAGuE: A generic distributed DAG engine for high performance computing, " U. Tennessee, Tech. Rep. ICL-UT-10-01, 2012Apr. 2010.
15. D. Orozco, E. Garcia, R. Pavel, R. Khan, and G. Gao, "TIDeFlow: The time iterated dependency flow execution model, " in 1st Workshop Data- Flow Execution Models for Extreme Scale Computing: DFM'11, Oct. 2011, pp. 1-9.
16. R. G. Minnich, C. L. Janssen, S. Krishnamoorthy, A. Marquez, M. Gokhale, P. Sadayappan, E. Van Hensbergen, J. McKie, and J. Appavoo, "Fault oblivious eXascale whitepaper, " in Proc. 1st Int'l Workshop Runtime and Operating Systems for Supercomputers: ROSS'11. New York: ACM, 2011, pp. 17-24.
17. H. Kaiser, M. Brodowicz, and T. Sterling, "ParalleX: An advanced parallel execution model for scaling-impaired applications, " in Int'l Conf. Parallel Processing Workshops (ICPPW) 2009, Sep. 2009, pp. 394-401.
18. E. L. Lusk, S. C. Pieper, and R. M. Butler, "More scalability, less pain: A simple programming model and its implementation for extreme computing, " SciDAC Review, vol. 17, pp. 30-37, Jan. 2010.
19. J. Dinan, S. Krishnamoorthy, D. B. Larkins, J. Nieplocha, and P. Sadayappan, "Scioto: A framework for global-view task parallelism, " Int'l Conf. on Parallel Processing, pp. 586-593, 2008.
20. G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, and W. Vogels, "Dynamo: Amazon's highly available key-value store, " SIGOPS Oper. Syst. Rev., vol. 41, pp. 205-220, Oct. 2007.
21. B. Fitzpatrick, "Distributed caching with memcached, " Linux J., vol. 2004, pp. 5-, Aug. 2004.
22. S. Sanfilippo and P. Noordhuis, "Redis, " http://redis.io/.
23. J. T. Feo, D. C. Cann, and R. R. Oldehoeft, "A report on the Sisal language project, " J. Parallel and Distributed Computing, vol. 10, no. 4, pp. 349- 366, 1990.
24. K. R. Traub, "A compiler for the MIT tagged-token dataflow architecture, " Massachusetts Institute of Technology, Cambridge, MA, Tech. Rep., 1986.
25. R. L. Bocchino Jr., V. S. Adve, S. V. Adve, and M. Snir, "Parallel programming must be deterministic by default, " in Workshop Hot Topics in Parallelism: HotPar'09. Berkeley, CA: USENIX Association, 2009.
26. J. M. Wozniak, T. G. Armstrong, E. L. Lusk, D. S. Katz, M. Wilde, and I. T. Foster, "Turbine: A distributed memory data flow engine for many-task applications, " in Int'l Workshop Scalable Workflow Enactment Engines and Technologies (SWEET) 2012, 2012.
27. W. Gropp, E. Lusk, N. Doss, and A. Skjellum, "A high-performance, portable implementation of the MPI message passing interface standard, " Par. Comp., vol. 22, no. 6, pp. 789-828, 1996.
28. S. Zuckerman, J. Suetterlein, R. Knauerhase, and G. R. Gao, "Using a "codelet" program execution model for exascale machines, " in Workshop Adaptive Self-Tuning Comp. Syst. for the Exaflop Era, 2011. New York: ACM, 2011, pp. 64-69.
29. H. C. Baker, Jr. and C. Hewitt, "The incremental garbage collection of processes, " in Proc. of 1977 Symp. on Artificial Intelligence and Programming Languages. New York: ACM, 1977, pp. 55-59.
30. D. M. Beazley, "SWIG: An easy to use tool for integrating scripting languages with C and C++, " in Proceedings of the 4th conference on USENIX Tcl/Tk Workshop, 1996- Volume 4. Berkeley, CA, USA: USENIX Association, 1996.
31. J. McFarland, "FortWrap web site, " http://fortwrap. sourceforge.net.
32. T. G. Armstrong, J. M. Wozniak, M. Wilde, K. Maheshwari, D. S. Katz, M. Ripeanu, E. L. Lusk, and I. T. Foster, "ExM: High level dataflow programming for extreme-scale systems, " in Workshop Hot Topics in Parallelism: HotPar'12, 2012.
33. L. Yu, C. Moretti, A. Thrasher, S. Emrich, K. Judd, and D. Thain, "Harnessing parallelism in multicore clusters with the All-Pairs, Wavefront, and Makeflow abstractions, " Cluster Computing, vol. 13, pp. 24-256, 2010.
34. T. G. Armstrong, Z. Zhang, D. S. Katz, M. Wilde, and I. T. Foster, "Scheduling many-task workloads on supercomputers: Dealing with trailing tasks, " in Workshop on Many-Task Computing on Grids and Supercomputers, 2010.
35. S. Al-Kiswany, A. Gharaibeh, and M. Ripeanu, "The case for a versatile storage system, " SIGOPS Oper. Sys. Rev., vol. 44, no. 1, pp. 10-14, Mar. 2010.
36. E. Vairavanathan, S. Al-Kiswany, L. Costa, M. Ripeanu, Z. Zhang, D. Katz, and M. Wilde, "Workflow-aware storage system: An opportunity study, " in Proc. CCGrid, 2012.