Many-task computing for grids and supercomputers

2008 Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS 2008

Volumn , Issue , 2008, Pages

  Raicu, Ioan ^a   Foster, Ian T ^a,b,c   Zhaoyo, Yong ^d  

^a THE UNIVERSITY OF CHICAGO   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

^d MICROSOFT   (United States)

Author keywords

High Performance Computing; High Throughput Computing; HPC; HTC; Many task Computing; MTC

Indexed keywords

HIGH PERFORMANCE COMPUTING; HIGH THROUGHPUT COMPUTING; HPC; HTC; MANY-TASK COMPUTING; MTC;

MESSAGE PASSING; SUPERCOMPUTERS;

THROUGHPUT;

EID: 67649240403     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/MTAGS.2008.4777912     Document Type: Conference Paper

References (70)

1. A. Gara, et al. "Overview of the Blue Gene/L system architecture", IBM Journal of Research and Development 49(2/3), 2005
2. IBM BlueGene/P(BG/P), http://www.research.ibm.com/bluegene/, 2008
3. J. Ousterhout, "Scripting: Higher Level Programming for the 21st Century", IEEE Computer, March 1998
4. Y. Zhao, I. Raicu, I. Foster. "Scientific Workflow Systems for 21st Century e-Science, New Bottle or New Wine?", IEEE Workshop on Scientific Workflows 2008
5. J. Dean, S. Ghemawat. "MapReduce: Simplified data processing on large clusters." In OSDI, 2004
6. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, I. Raicu, T. Stef-Praun, M. Wilde. "Swift: Fast, Reliable, Loosely Coupled Parallel Computation", IEEE Workshop on Scientific Workflows 2007
7. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, M. Wilde. "Falkon: A Fast and Lightweight Task Execution Framework", IEEE/ACM SC, 2007
8. E. Deelman et al. "Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems", Scientific Programming Journal 13(3), 219-237, 2005 (Pubitemid 43022937)
9. I. Raicu, Y. Zhao, I. Foster, A. Szalay. "Accelerating Large-Scale Data Exploration through Data Diffusion", ACM International Workshop on Data-Aware Distributed Computing 2008
10. M. Isard, M. Budiu, Y. Yu, A. Birrell, D. Fetterly. "Dryad: Distributed Data-Parallel Programs from Sequential Building Blocks", European Conference on Computer Systems (EuroSys), 2007
11. R. Pike, S. Dorward, R. Griesemer, S. Quinlan. "Interpreting the Data: Parallel Analysis with Sawzall", Scientific Programming Journal, Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure 13(4), pp. 227-298, 2005
12. M. Livny, J. Basney, R. Raman, T. Tannenbaum. "Mechanisms for High Throughput Computing", SPEEDUP Journal 1(1), 1997
13. M. Flynn. "Some Computer Organizations and Their Effectiveness", IEEE Trans. Comput. C- 21, pp. 948, 1972
14. D. Thain, T. Tannenbaum, M. Livny, "Distributed Computing in Practice: The Condor Experience", Concurrency and Computation: Practice and Experience 17( 2-4), pp. 323-356, 2005 (Pubitemid 40285781)
15. J. Appavoo, V. Uhlig, A. Waterland. "Project Kittyhawk: Building a Global-Scale Computer", ACM Sigops Operating System Review, 2008
16. Z. Zhang, A. Espinosa, K. Iskra, I. Raicu, I. Foster, M. Wilde. "Design and Evaluation of a Collective I/O Model for Loosely-coupled Petascale Programming", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008
17. Top500, June 2008, http://www.top500.org/lists/2008/06, 2008
18. T. Hey, A. Trefethen. "The data deluge: an esicence perspective", Gid Computing: Making the Global Infrastructure a Reality, Wiley, 2003
19. SDSS: Sloan Digital Sky Survey, http://www.sdss.org/, 2008
20. CERN's Large Hadron Collider, http://lhc.web.cern.ch/lhc, 2008
21. GenBank, http://www.psc.edu/general/software/packages/genbank2008
22. European Molecular Biology Laboratory, http://www.embl.org, 2008
23. C. Catlett, et al. "TeraGrid: Analysis of Organization, System Architecture, and Middleware Enabling New Types of Applications", HPC 2006
24. Open Science Grid (OSG), http://www.opensciencegrid.org/, 2008
25. F. Schmuck and R. Haskin, "GPFS: A Shared- Disk File System for Large Computing Clusters", FAST 2002
26. A. Szalay, A. Bunn, J. Gray, I. Foster, I. Raicu. "The Importance of Data Locality in Distributed Computing Applications", NSF Workflow Workshop 2006
27. J. Gray. "Distributed Computing Economics", Technical Report MSR-TR-2003-2024, Microsoft Research, Microsoft Corporation, 2003
28. A. Bialecki, M. Cafarella, D. Cutting, O. O'Malley. "Hadoop: A Framework for Running Applications on Large Clusters Built of Commodity Hardware", http://lucene.apache.org/hadoop/, 2005
29. D.P. Anderson, "BOINC: A System for Public- Resource Computing and Storage", IEEE/ACM Workshop on Grid Computing, 2004
30. J. Cope, M. Oberg, H.M. Tufo, T. Voran, M. Woitaszek. "High Throughput Grid Computing with an IBM Blue Gene/L", Cluster 2007
31. A. Peters, A. King, T. Budnik, P. McCarthy, P. Michaud, M. Mundy, J. Sexton, G. Stewart. "Asynchronous Task Dispatch for High Throughput Computing for the eServer IBM Blue Gene® Supercomputer", Parallel and Distributed Processing (IPDPS), 2008
32. IBM Corporation. "High-Throughput Computing (HTC) Paradigm", IBM System Blue Gene Solution: Blue Gene/P Application Development, IBM RedBooks, 2008
33. N. Desai. "Cobalt: An Open Source Platform for HPC System Software Research", Edinburgh BG/L System Software Workshop, 2005
34. J. Frey, T. Tannenbaum, I. Foster, M. Frey, S. Tuecke. "Condor-G: A Computation Management Agent for Multi-Institutional Grids", Cluster Computing, 2002
35. Swift Workflow System, www.ci.uchicago.edu/swift, 2008
36. C. Catlett et al., "TeraGrid: Analysis of Organization, System Architecture, and Middleware Enabling New Types of Applications", HPC and Grids in Action, ed. Lucio Grandinetti, IOS Press Advances in Parallel Computing series, Amsterdam, 2007
37. SiCortex, http://www.sicortex.com/, 2008
38. G.v. Laszewski, M. Hategan, D. Kodeboyina. "Java CoG Kit Workflow", in I.J. Taylor, E. Deelman, D.B. Gannon, and M. Shields, eds., Workflows for eScience, pp. 340-356, 2007
39. J.C. Jacob, et al. "The Montage Architecture for Grid-Enabled Science Processing of Large, Distributed Datasets", Earth Science Technology Conference 2004
40. The Functional Magnetic Resonance Imaging Data Center, http://www.fmridc.org/, 2007
41. T. Stef-Praun, B. Clifford, I. Foster, U. Hasson, M. Hategan, S. Small, M. Wilde and Y. Zhao. "Accelerating Medical Research using the Swift Workflow System", Health Grid , 2007
42. Y. Zhao, I. Raicu, I. Foster, M. Hategan, V. Nefedova, M. Wilde. "Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments", Grid Computing Research Progress, Nova Pub. 2008
43. D.T. Moustakas et al. "Development and Validation of a Modular, Extensible Docking Program: DOCK 5", J. Comput. Aided Mol. Des. 20, pp. 601-619, 2006 (Pubitemid 46023925)
44. D. Hanson. "Enhancing Technology Representations within the Stanford Energy Modeling Forum (EMF) Climate Economic Models", Energy and Economic Policy Models: A Reexamination of Fundamentals, 2006
45. T. Stef-Praun, G. Madeira, I. Foster, R. Townsend. "Accelerating solution of a moral hazard problem with Swift", e-Social Science, 2007
46. B. Bode, D.M. Halstead, R. Kendall, Z. Lei, W. Hall, D. Jackson. "The Portable Batch Scheduler and the Maui Scheduler on Linux Clusters", Usenix, 4th Annual Linux Showcase & Conference, 2000
47. W. Gentzsch, "Sun Grid Engine: Towards Creating a Compute Power Grid", 1st International Symposium on Cluster Computing and the Grid, 2001
48. I. Raicu, I. Foster, A. Szalay, G. Turcu. "AstroPortal: A Science Gateway for Large-scale Astronomy Data Analysis", TeraGrid Conference 2006
49. G.B. Berriman, et al., "Montage: a Grid Enabled Engine for Delivering Custom Science-Grade Image Mosaics on Demand", SPIE Conference on Astronomical Telescopes and Instrumentation. 2004
50. J.C. Jacob, et al. "The Montage Architecture for Grid-Enabled Science Processing of Large, Distributed Datasets", Earth Science Technology Conference 2004
51. US National Virtual Observatory (NVO), http://www.us-vo.org/index.cfm, 2008
52. ASC/Alliances Center for Astrophysical Thermonuclear Flashes, http://www.flash.uchicago.edu/website/home/, 2008
53. KEGG's Ligand Database: http://www.genome.ad.jp/kegg/ligand.html, 2008
54. PL protein library, http://protlib.uchicago.edu/, 2008
55. NIST Chemistry WebBook database, http://webbook.nist.gov/chemistry/, 2008
56. S.F. Altschul, W. Gish, W. Miller, E.W. Myers, D.J. Lipman. "Basic Local Alignment Search Tool", J Mol Biol 215 (3): 403-410, 1990
57. Computational Neuroscience Applications Research Infrastructure, http://www.ci.uchicago.edu/wiki/bin/view/CNA RI/WebHome, 2008
58. The Functional Magnetic Resonance Imaging Data Center, http://www.fmridc.org/, 2007
59. J. Dean and S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters", Symposium on Operating System Design and Implementation (OSDI'04), 2004
60. C. Moretti, J. Bulosan, D. Thain, and P. Flynn. "All-Pairs: An Abstraction for Data-Intensive Cloud Computing", IPDPS 2008
61. D. Bernholdt, S. Bharathi, D. Brown, K. Chanchio, M. Chen, A. Chervenak, L. Cinquini, B. Drach, I. Foster, P. Fox, J. Garcia, C. Kesselman, R. Markel, D. Middleton, V. Nefedova, L. Pouchard, A. Shoshani, A. Sim, G. Strand, and D. Williams, "The Earth System Grid: Supporting the Next Generation of Climate Modeling Research", Proceedings of the IEEE, 93 (3), p 485-495, 2005 (Pubitemid 40851204)
62. A. Szalay, A. Bunn, J. Gray, I. Foster, I. Raicu. "The Importance of Data Locality in Distributed Computing Applications", NSF Workflow Workshop 2006
63. I. Raicu, Z. Zhang, M. Wilde, I. Foster, P. Beckman, K. Iskra, B. Clifford. "Towards Loosely-Coupled Programming on Petascale Systems", IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis (SuperComputing/SC08), 2008
64. I. Raicu, Y. Zhao, I. Foster. "IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008", co-located with IEEE/ACM Supercomputing 2008, http://dsl.cs.uchicago.edu/MTAGS08/, 2008
65. M. Pierce, I. Raicu, R. Pordes, J. McGee, D. Repasky. "How to Run One Million Jobs (Megajobs08)", Bird-of-Feather Session at IEEE/ACM Supercomputing 2008, http://gridfarm007.ucs.indiana.edu/megajobBOF/index.php/ Main-Page, 2008
66. Y. Gu, R. Grossman. "Exploring Data Parallelism and Locality in Wide Area Networks", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008
67. E.V. Hensbergen, Ron Minnich. "System Support for Many Task Computing", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008
68. L. Hui, Y. Huashan, L. Xiaoming. "A Lightweight Execution Framework for Massive Independent Tasks", IEEE Workshop on Many- Task Computing on Grids and Supercomputers (MTAGS08) 2008
69. A.T. Thor, G.V. Zaruba, D. Levine, K. De, T.J. Wenaus. "ViGs: A Grid Simulation and Monitoring Tool for Grid Workflows", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008
70. E. Afgan, P. Bangalore. "Embarrassingly Parallel Jobs Are Not Embarrassingly Easy to Schedule on the Grid", IEEE Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS08) 2008