Middleware support for many-task computing

Cluster Computing

Volumn 13, Issue 3, 2010, Pages 291-314

  Raicu, Ioan ^a   Foster, Ian ^b,c   Wilde, Mike ^b,c   Zhang, Zhao ^b   Iskra, Kamil ^b,c   Beckman, Peter ^b,c   Zhao, Yong ^d   Szalay, Alex ^e   Choudhary, Alok ^a   Little, Philip ^f   Moretti, Christopher ^f   Chaudhary, Amitabh ^f   Thain, Douglas ^f  

^a NORTHWESTERN UNIVERSITY   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

^d MICROSOFT   (United States)

^e JOHNS HOPKINS UNIVERSITY   (United States)

^f UNIVERSITY OF NOTRE DAME   (United States)

Author keywords

Computing; Data intensive distributed computing; Falkon; High performance computing; High throughput; Loosely coupled applications; Many task computing; Petascale; Swift

Indexed keywords

DISTRIBUTED COMPUTING; FALKON; HIGH-PERFORMANCE COMPUTING; HIGH-THROUGHPUT; PETASCALE; TASK COMPUTING;

FILE ORGANIZATION; GRID COMPUTING; MANAGEMENT; MIDDLEWARE; SCALABILITY; SCHEDULING; SUPERCOMPUTERS;

THROUGHPUT;

EID: 77955096331     PISSN: 13867857     EISSN: 15737543     Source Type: Journal    
DOI: 10.1007/s10586-010-0132-9     Document Type: Article

References (73)

1. Gara, A., et al.: Overview of the Blue Gene/L system architecture, IBM J. Res. Develop. 49(2/3) (2005).
2. IBM BlueGene/P, http://www. research. ibm. com/bluegene/ (2008).
3. Ousterhout, J.: Scripting: higher level programming for the 21st century. Computer 31, 23-30 (1998).
4. Zhao, Y., Raicu, I., Foster, I.: Scientific workflow systems for 21st century e-science, new bottle or new wine? In: IEEE Workshop on Scientific Workflows (2008).
5. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: USENIX OSDI04 (2004).
6. Zhao, Y., Hategan, M., Clifford, B., Foster, I., von Laszewski, G., Raicu, I., Stef-Praun, T., Wilde, M.: Swift: fast, reliable, loosely coupled parallel computation. In: IEEE Workshop on Scientific Workflows (2007).
7. Raicu, I., Zhao, Y., Dumitrescu, C., Foster, I., Wilde, M.: Falkon: a Fast and Light-weight tasK executiON framework. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC07 (2007).
8. Deelman, E., et al.: Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci. Program. J. 13(3), 219-237 (2005).
9. Raicu, I., Foster, I., Zhao, Y.: Many-task computing for grids and supercomputers. In: IEEE Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS08 (2008).
10. Raicu, I., Zhao, Y., Foster, I., Szalay, A.: Accelerating large-scale data exploration through data diffusion. In: ACM International Workshop on Data-Aware Distributed Computing (2008).
11. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Eur. Conf. Comput. Syst. (2007).
12. Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: parallel analysis with Sawzall. Sci. Program. J. 13(4), 227-298 (2005). Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure.
13. Livny, M., Basney, J., Raman, R., Tannenbaum, T.: Mechanisms for high throughput computing, SPEEDUP J. 1(1) (1997).
14. Foster, I., Kesselman, C. (eds.): The Grid: Blueprint for a Future Computing Infrastructure, Chapter 2: Computational Grids. Morgan Kaufmann, San Mateo (1999).
15. Foster, I., Kesselman, C., Tuecke, S.: The anatomy of the grid. Int. J. Supercomput. Appl. 15, 200-222 (2001).
16. Hey, T., Trefethen, A.: The data deluge: an e-science perspective. In: Gid Computing: Making the Global Infrastructure a Reality (2003).
17. Catlett, C., et al.: TeraGrid: analysis of organization, system architecture, and middleware Enabling New Types of Applications. In: HPC (2006).
18. Open Science Grid (OSG). http://www. opensciencegrid. org/ (2008).
19. Szalay, A., Bunn, A., Gray, J., Foster, I., Raicu, I.: The importance of data locality in distributed computing applications. In: NSF Workflow Workshop (2006).
20. Gray, J.: Distributed computing economics. Technical Report MSR-TR-2003-24, Microsoft Research, Microsoft Corp. (2003).
21. Raicu, I., Zhang, Z., Wilde, M., Foster, I., Beckman, P., Iskra, K., Clifford, B.: Towards loosely-coupled programming on petascale systems. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis, SuperComputing/SC08 (2008).
22. SiCortex. http://www. sicortex. com/ (2008).
23. IBM Blue Gene team: overview of the IBM Blue Gene/P project. IBM J. Res. Develop. 52(1/2), 199-220 (2008).
24. Raicu, I., Foster, I., Zhao, Y., Little, P., Moretti, C., Chaudhary, A., Thain, D.: The quest for scalable support of data intensive workloads in distributed systems. In: ACM HPDC09 (2009).
25. Raicu, I., Zhao, Y., Foster, I., Szalay, A.: A data diffusion approach to large-scale scientific exploration. In: Microsoft eScience Workshop at RENCI (2007).
26. Raicu, I.: Harnessing grid resources with data-centric task farms. Technical Report, University of Chicago (2007).
27. Zhang, Z., Espinosa, A., Iskra, K., Raicu, I., Foster, I., Wilde, M.: Design and evaluation of a collective I/O model for loosely-coupled petascale programming. In: IEEE Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS08 (2008).
28. Raicu, I., Foster, I., Zhao, Y., Szalay, A., Little, P., Moretti, C., Chaudhary, A., Thain, D.: Towards data intensive many-task computing. In: Data Intensive Distributed Computing: Challenges and Solutions for Large-Scale Information Management. IGI Global, Hershey (2009).
29. Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurr. Comput. Pract. Exp. 17(2-4), 323-356 (2005).
30. Robinson, E., DeWitt, D. J.: Turning cluster management into data management: a system overview. In: Conference on Innovative Data Systems Research (2007).
31. Bode, B., Halstead, D. M., Kendall, R., Lei, Z., Hall, W., Jackson, D.: The portable batch scheduler and the Maui scheduler on Linux clusters. In: Usenix, Linux Showcase & Conference (2000).
32. Zhou, S.: LSF: Load sharing in large-scale heterogeneous distributed systems. In: Workshop on Cluster Computing (1992).
33. Gentzsch, W.: Sun grid engine: towards creating a compute power grid. In: 1st International Symposium on Cluster Computing and the Grid (2001).
34. Bialecki, A., Cafarella, M., Cutting, D., O'Malley, O.: Hadoop: a framework for running applications on large clusters built of commodity hardware. http://lucene. apache. org/hadoop/ (2005).
35. Anderson, D. P.: BOINC: a system for public-resource computing and storage. In: IEEE/ACM International Workshop on Grid Computing (2004).
36. Frey, J., Tannenbaum, T., Foster, I., Frey, M., Tuecke, S.: Condor-G: a computation management agent for multi-institutional grids. Cluster Comput 5, 237-246 (2002).
37. Banga, G., Druschel, P., Mogul, J. C.: Resource containers: a new facility for resource management in server systems. In: Symposium on Operating Systems Design and Implementation (1999).
38. Stankovic, J. A., Ramamritham, K., Niehaus, D., Humphrey, M., Wallace, G.: The spring system: integrated support for complex real-time systems. Real-Time Syst. 16(2/3), 97-125 (1999).
39. Mehta, G., Kesselman, C., Deelman, E.: Dynamic deployment of VO-specific schedulers on managed resources. Technical Report, USC ISI (2006).
40. Walker, E., Gardner, J. P., Litvin, V., Turner, E. L.: Creating personal adaptive clusters for managing scientific tasks in a distributed computing environment. In: Workshop on Challenges of Large Applications in Distributed Environments (2006).
41. Singh, G., Kesselman, C., Deelman, E.: Performance impact of resource provisioning on workflows. Technical Report, USC ISI (2006).
42. Anderson, D. P., Korpela, E., Walton, R.: High-performance task distribution for volunteer computing. In: IEEE International Conference on e-Science and Grid Technologies (2005).
43. Cope, J., et al.: High throughput grid computing with an IBM Blue Gene/L. In: Cluster (2007).
44. ® Supercomputer. In: Parallel and Distributed Processing, IPDPS, (2008).
45. IBM coorporation: High-throughput computing (HTC) paradigm. In: IBM System Blue Gene Solution: Blue Gene/P Application Development, IBM RedBooks (2008).
46. Desai, N.: Cobalt: an open source platform for HPC system software research. In: Edinburgh BG/L System Software Workshop (2005).
47. Zhao, Y., Raicu, I., Foster, I., Hategan, M., Nefedova, V., Wilde, M.: Realizing fast, scalable and reliable scientific computations in grid environments. In: Grid Computing Research Progress. Nova Publisher, New York (2008).
48. Swift Workflow System. www. ci. uchicago. edu/swift (2008).
49. Laszewski, G. v., Hategan, M., Kodeboyina, D.: Java CoG kit workflow. In: Taylor, I. J., Deelman, E., Gannon, D. B., Shields, M. (eds.): Workflows for eScience, pp. 340-356. Springer, Berlin (2007).
50. Foster, I.: Globus toolkit version 4: software for service-oriented systems. In: Conference on Network and Parallel Computing (2005).
51. The Globus Security Team. Globus toolkit version 4: grid security infrastructure: a standards perspective. Technical Report, Argonne National Laboratory, MCS (2005).
52. Feller, M., Foster, I., Martin, S.: GT4 GRAM: a functionality and performance study. In: TeraGrid Conference (2007).
53. Podlipnig, S., Böszörmenyi, L.: A survey of Web cache replacement strategies. ACM Comput. Surv. 35(4), 374-398 (2003).
54. Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M., Dumitrescu, C., Raicu, I., Foster, I.: The globus striped Grid FTP framework and server. In: ACM/IEEE SC05 (2005).
55. GKrell M. http://members. dslextreme. com/users/billw/gkrellm/gkrellm. html (2008).
56. Walker, E., Earl, D. J., Deem, M. W.: How to run a million jobs in six months on the NSF teraGrid. In: TeraGrid Conference (2007).
57. Raicu, I., Dumitrescu, C., Foster, I.: Dynamic resource provisioning in grid environments. In: TeraGrid Conference (2007).
58. ANL/UC TeraGrid site details. http://www. uc. teragrid. org/tg-docs/tg-tech-sum. html (2007).
59. Schmuck, F., Haskin, R.: GPFS: a shared-disk file system for large computing clusters. In: FAST (2002).
60. Moretti, C., Bulosan, J., Thain, D., Flynn, P.: All-pairs: an abstraction for data-intensive cloud computing. In: IPDPS (2008).
61. Thain, D., Moretti, C., Hemmes, J.: Chirp: a practical global file system for cluster and grid computing. J. Grid Comput 7, 51-72 (2008).
62. The Functional Magnetic Resonance Imaging Data Center. http://www. fmridc. org/ (2007).
63. NIST Chemistry WebBook Database. http://webbook. nist. gov/chemistry/ (2008).
64. Moustakas, D. T., et al.: Development and validation of a modular, extensible docking program: DOCK 5. J. Comput. Aided Mol. Des. 20, 601-619 (2006).
65. KEGG's Ligand Database. http://www. genome. ad. jp/kegg/ligand. html (2008).
66. Hanson, D.: Enhancing technology representations within the stanford energy modeling forum (EMF) climate economic models, energy and economic policy models: a reexamination of fundamentals (2006).
67. Raicu, I., Foster, I., Szalay, A., Turcu, G.: AstroPortal: a science gateway for large-scale astronomy data analysis. In: TeraGrid Conference (2006).
68. Raicu, I., Foster, I., Szalay, A.: Harnessing grid resources to enable the dynamic analysis of large astronomy datasets. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC06 (2006).
69. SDSS: Sloan Digital Sky Survey. http://www. sdss. org/ (2008).
70. Jacob, J. C., et al.: The montage architecture for grid-enabled science processing of large, distributed datasets. In: Earth Science Technology Conference (2004).
71. Katz, D., Berriman, G., Deelman, E., Good, J., Jacob, J., Kesselman, C., Laity, A., Prince, T., Singh, G., Su, M.: A comparison of two methods for building astronomical image mosaics on a grid. In: Proceedings of the 7th Workshop on High Performance Scientific and Engineering Computing, HPSEC-05 (2005).
72. Pham, Q. T., Balkir, A. S., Tie, J., Foster, I., Wilde, M., Raicu, I.: Data intensive scalable computing on TeraGrid: A comparison of MapReduce and Swift. In: TeraGrid Conference, TG08 (2008).
73. Stevens, R.: The LLNL/ANL/IBM collaboration to develop BG/P and BG/Q. DOE ASCAC Report (2006).