The quest for scalable support of data-intensive workloads in distributed systems

Proc. 18th ACM International Symposium on High Performance Distributed Computing, HPDC 09, Co-located with the 2009 International Symposium on High Performance Distributed Computing Conf., HPDC'09

Volumn , Issue , 2009, Pages 21-29

  Raicu, Ioan ^a   Foster, Ian T ^a,b,c   Zhao, Yong ^d   Little, Philip ^e   Moretti, Christopher M ^e   Chaudhary, Amitabh ^e   Thain, Douglas ^e  

^a THE UNIVERSITY OF CHICAGO   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c ARGONNE NATIONAL LABORATORY   (United States)

^d MICROSOFT   (United States)

^e University of Notre Dame   (United States)

Author keywords

Data diffusion; Data management; Data aware scheduling; Falkon

Indexed keywords

80 GB/S; ABSTRACT MODELS; ACTIVE STORAGE; BEST MODEL; DATA DIFFUSION; DATA LOCALITY; DATA-INTENSIVE APPLICATION; DATA-INTENSIVE WORKLOADS; DEDICATED HARDWARE; DISTRIBUTED SYSTEMS; EMPIRICAL ANALYSIS; HIGH COSTS; HIGH THROUGHPUT; LARGE DATASETS; ORDERS OF MAGNITUDE; PARALLEL FILE SYSTEM; PERFORMANCE IMPROVEMENTS; REAL-WORLD PERFORMANCE; RESOURCE CHARACTERISTIC; SCHEDULING POLICIES; STORAGE RESOURCES;

COMPUTER SCIENCE; DIE CASTING; DIFFUSION; DISTRIBUTED COMPUTER SYSTEMS; THROUGHPUT;

SCHEDULING;

EID: 70449623522     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1551609.1551642     Document Type: Conference Paper

References (34)

1. A. Szalay, J. Bunn, J. Gray, I. Foster, I. Raicu. "The Importance of Data Locality in Distributed Computing Applications," NSF Workflow Workshop 2006
2. J. Gray. "Distributed Computing Economics," Technical Report MSR-TR-2003-24, Microsoft Research, 2003
3. S. Ghemawat, H. Gobioff, S.T. Leung. "The Google File System," ACM SOSP 2003, pp. 29-43
4. J. Dean, S. Ghemawat. "MapReduce: Simplified Data Processing on Large Clusters," OSDI 2004
5. I. Raicu, Y. Zhao, I. Foster, A. Szalay. "Accelerating Largescale Data Exploration through Data Diffusion," ACM Workshop on Data-Aware Distributed Comp. 2008
6. S. Podlipnig, et al. "A Survey of Web Cache Replacement Strategies," ACM Computing Surveys, 2003
7. R. Lancellotti, et al. "A Scalable Architecture for Cooperative Web Caching," Web Engineering Workshop 2002
8. R. Hasan, et al. "A Survey of Peer-to-Peer Storage Techniques for Distributed File Systems," ITCC 2005
9. W. Xiaohui, et al. "Implementing Data Aware Scheduling in Gfarm Using LSF Scheduler Plugin Mechanism," GCA05, 2005
10. P. Fuhrmann. "dCache, the Commodity Cache," MSST 2004
11. C. Moretti, et al. "All-Pairs: An Abstraction for Data-Intensive Cloud Computing," IPDPS 2008
12. D. Thain, et al. "Chirp: A Practical Global Filesystem for Cluster and Grid Computing," JGC, Springer, 2008
13. I. Raicu, et a. "Falkon: A Fast and Light-weight tasK executiON Framework," IEEE/ACM SC 2007
14. G. Banga, et al. "Resource Containers: A New Facility for Resource Management in Server Systems," OSDI 1999
15. I. Raicu, Z. Zhang, M. Wilde, I. Foster, P. Beckman, K. Iskra, B. Clifford. "Toward Loosely Coupled Programming on Petascale Systems," IEEE SC 2008
16. A. Bialecki, et al. "Hadoop: A Framework for Running Applications on Large Clusters Built of Commodity Hardware," http://lucene.apache.org/ hadoop/, 2005
17. M. Feller, et al. "GT4 GRAM: A Functionality and Performance Study," TeraGrid Conference 2007
18. W. Allcock, J. Bresnahan, R. Kettimuthu, M. Link, C. Dumitrescu, I. Raicu, I. Foster. "The Globus Striped GridFTP Framework and Server," ACM/IEEE SC, 2005
19. P. Cao, et al. "Cost-Aware WWW Proxy Caching Algorithms," USENIX Symposium on Internet Technologies and Systems, 1997
20. I. Raicu, I. Foster, Y. Zhao, A. Szalay, P. Little, C. Moretti, A. Chaudhary, D. Thain. "Towards Data Intensive Many-Task Computing," under review at Data Intensive Distributed Computing: Challenges and Solutions for Large-Scale Information Management, 2009
21. I. Raicu, I. Foster, A. Szalay, G. Turcu. "AstroPortal: A Science Gateway for Large-scale Astronomy Data Analysis," TeraGrid Conf. 2006
22. E. Torng. "A Unified Analysis of Paging and Caching," Algorithmica 20, 175-200, 1998
23. ANL/UC TeraGrid Site Details, http://www.uc.teragrid.org/tg-docs/tg-tech- sum.html, 2007
24. F. Schmuck, R. Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters," FAST 2002
25. T. Kosar. "A New Paradigm in Data Intensive Computing: Stork and the Data-Aware Schedulers," IEEE CLADE 2006
26. X. Wei, et al. "Integrating Local Job Scheduler - LSF with Gfarm," ISPA05, vol. 3758/2005, 2005
27. F. Chang, et al. "Bigtable: A Distributed Storage System for Structured Data," USENIX OSDI 2006
28. Y. Zhao, M. Hategan, B. Clifford, I. Foster, G. von Laszewski, I. Raicu, T. Stef-Praun, M. Wilde. "Swift: Fast, Reliable, Loosely Coupled Parallel Computation," IEEE Workshop on Scientific Workflows 2007
29. Y. Zhao, I. Raicu, I. Foster, M. Hategan, V. Nefedova, M. Wilde. "Realizing Fast, Scalable and Reliable Scientific Computations in Grid Environments," Grid Computing Research Progress, Nova Pub. 2008
30. R. Grossman, Y. Gu. "Data Mining Using High Performance Clouds: Experimental Studies Using Sector and Sphere," ACM KDD 2008
31. Y. Gu, et al. "Distributing the Sloan Digital Sky Survey Using UDT and Sector," e-Science 2006
32. K. Pruhs, et al. "Online Sscheduling," Handbook of Scheduling: Algorithms, Models, and Performance Analysis, 2004
33. S. Irani. "Randomized Weighted Caching with Two Page Weights," Algorithmica, 32:4, 624-640, 2002
34. X. Zhang, A. Espinosa, K. Iskra, I. Raicu, I. Foster, M. Wilde. "Design and Evaluation of a Collective I/O Model for Loosely-coupled PetascaleProgramming," IEEE MTAGS 2008