AME: An anyscale Many-Task Computing engine

WORKS'11 - Proceedings of the 6th Workshop on Workflows in Support of Large-Scale Science, Co-located with SC'11

Volumn , Issue , 2011, Pages 137-146

  Zhang, Zhao ^a   Katz, Daniel S ^b   Ripeanu, Matei ^c   Wilde, Michael ^b   Foster, Ian ^b  

^a THE UNIVERSITY OF CHICAGO   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

^c UNIVERSITY OF BRITISH COLUMBIA   (Canada)

Author keywords

Data management; Load balancing; Many Task Computing; Scheduling; Supercomputer systems

Indexed keywords

COMPUTING ENGINES; FILE TRANSFERS; FILESYSTEM; INTER-TASK PARALLELISMS; LOAD-BALANCING; MANY-TASK COMPUTING; NEW APPLICATIONS; PETASCALE; PROCESSING CAPABILITY; REAL APPLICATIONS; RESOURCE PROVISIONING; SYNTHETIC BENCHMARK; TASK DISPATCHING;

DATA HANDLING; INFORMATION MANAGEMENT; MIDDLEWARE; PARALLEL ARCHITECTURES; SCHEDULING; SUPERCOMPUTERS;

BENCHMARKING;

EID: 84863265497     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2110497.2110513     Document Type: Conference Paper

References (17)

1. S. Al-Kiswany, A. Gharaibeh, and M. Ripeanu. The case for a versatile storage system. SIGOPS Oper. Syst. Rev., 44:10-14, March 2010.
2. B. Allcock, J. Bester, J. Bresnahan, A. L. Chervenak, I. Foster, C. Kesselman, S. Meder, V. Nefedova, D. Quesnel, and S. Tuecke. Data management and transfer in high-performance computational grid environments. Parallel Comput., 28:749-771, May 2002. (Pubitemid 34801987)
3. D. Borthakur. HDFS architecture. http://hadoop.apache.org/hdfs/docs/ current/hdfsdesign.pdf.
4. P. H. Carns, W. B. Ligon, III, R. B. Ross, and R. Thakur. PVFS: a parallel file system for linux clusters. In Proceedings of the 4th annual Linux Showcase & Conference - Volume 4, pages 28-28, Berkeley, CA, USA, 2000. USENIX Association.
5. S. Donovan, G. Huizenga, A. J. Hutton, A. J. Hutton, C. C. Ross, C. C. Ross, L. Symposium, L. Symposium, L. Symposium, M. K. Petersen, W. O. Source, and P. Schwan. Lustre: Building a file system for 1,000-node clusters, 2003.
6. J. Frey, T. Tannenbaum, M. Livny, I. Foster, and S. Tuecke. Condor-G: A computation management agent for multi-institutional grids. Cluster Computing, 5:237-246, 2002.
7. K. Iskra, J. W. Romein, K. Yoshii, and P. Beckman. ZOID: I/O-forwarding infrastructure for petascale architectures. In Proc. of 13th ACM SIGPLAN Symp. on Principles and Practice of Parallel Programming, PPoPP '08, pages 153-162, New York, NY, USA, 2008. ACM.
8. D. S. Katz, J. C. Jacob, G. B. Berriman, J. Good, A. C. Laity, E. Deelman, C. Kesselman, and G. Singh. A comparison of two methods for building astronomical image mosaics on a grid. In Proc. 2005 Intl. Conf. on Parallel Proc. Workshops, pages 85-94, 2005. (Pubitemid 43904143)
9. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde. Falkon: a Fast and Light-weight tasK executiON framework. In Proc. IEEE/ACM Supercomputing 2007, pages 1-12, 2007.
10. F. Schmuck and R. Haskin. GPFS: A shared-disk file system for large computing clusters. In In Proceedings of the 2002 Conference on File and Storage Technologies FAST, pages 231-244, 2002.
11. I. Stoica, R. Morris, D. Karger, M. F. Kaashoek, and H. Balakrishnan. Chord: A scalable peer-to-peer lookup service for internet applications. In Proceedings of the ACM SIGCOMM '01 Conference, August 2001.
12. D. Thain, C. Moretti, and J. Hemmes. Chirp: a practical global filesystem for cluster and grid computing. Journal of Grid Computing, 7(1):51-72, 2009.
13. R. Thakur, W. Gropp, and E. Lusk. Data sieving and collective I/O in ROMIO. Symp. on Frontiers of Massively Par. Proc., page 182, 1999.
14. M. Wilde, I. Foster, K. Iskra, P. Beckman, Z. Zhang, A. Espinosa, M. Hategan, B. Clifford, and I. Raicu. Parallel scripting for applications at the petascale and beyond. Computer, 42:50-60, 2009.
15. M. Wilde, M. Hategan, J. M. Wozniak, B. Clifford, D. S. Katz, and I. Foster. Swift: A language for distributed parallel scripting. Parallel Computing, pages 633-652, September 2011.
16. J. M. Wozniak and M. Wilde. Case studies in storage access by loosely coupled petascale applications. In Proc. 4th Annual Workshop on Petascale Data Storage, pages 16-20, 2009.
17. Z. Zhang, A. Espinosa, K. Iskra, I. Raicu, I. Foster, and M. Wilde. Design and evaluation of a collective I/O model for loosely coupled petascale programming. In Proceedings of Many-Task Computing on Grids and Supercomputers, 2008, pages 1-10, 2008.