File-access patterns of data-intensive workflow applications and their implications to distributed filesystems

HPDC 2010 - Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

Volumn , Issue , 2010, Pages 746-755

  Shibata, Takeshi ^a   Choi, Sung Jun ^a   Taura, Kenjiro ^a  

^a UNIVERSITY OF TOKYO   (Japan)

Author keywords

Distributed filesystems; Workflow applications

Indexed keywords

ACCESS PATTERNS; DATA INTENSIVE; DATA SHARING; DISTRIBUTED FILESYSTEMS; FILE ACCESS; FILE SYSTEMS; GRID ENVIRONMENTS; NATURAL LANGUAGE PROCESSING; REAL WORLD DATA; REAL-WORLD; WEB DATA; WORK-FLOWS; WORKFLOW APPLICATIONS; WORKFLOW EXECUTION;

COMPUTATIONAL LINGUISTICS; DATA HANDLING; DATA TRANSFER; DISTRIBUTED COMPUTER SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; SCHEDULING;

MANAGEMENT;

EID: 78650014089     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1851476.1851585     Document Type: Conference Paper

References (31)

1. D. Thain, T. Tannenbaum, and M. Livny, "Distributed computing in practice: the condor experience." Concurrency - Practice and Experience, vol. 17, no. 2-4, pp. 323-356, 2005.
2. E. Deelman, G. Singh, M.-H. Su, J. Blythe, Y. Gil, C. Kesselman, G. Mehta, K. Vahi, G. B. Berriman, J. Good, A. Laity, J. C. Jacob, and D. S. Katz., "Pegasus: a framework for mapping complex scientific workflows onto distributed systems," Scientific Programming Journal, vol. 13, no. 3, pp. 219-237, 2005.
3. T. Oinn, M. Addis, J. Ferris, D. Marvin, M. Senger, M. Greenwood, T. Carver, K. Glover, M. R. Pocock, A. Wipat, and P. Li, "Taverna: a tool for the composition and enactment of bioinformaticsworkflows," Bioinformatics, vol. 20, no. 17, pp. 3045-3054, 2004.
4. J. Dean and S. Ghemawat, "Mapreduce: Simplified data processing on large clusters," in Proceedings of the 6th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2004, p. 137 50.
5. M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly, "Dryad: Distributed data-parallel programs from sequential building blocks," in Proceedings of the 2007 Eurosys Conference, 2007.
6. A. Mandal, K. Kennedy, C. Koelbel, G. Marin, J. Mellor-Crummey, B. Liu, and L. Johnsson, "Scheduling strategies for mapping application workflows onto the grid," in High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium on, July 2005, pp. 125-134.
7. H. Topcuoglu, S. Hariri, and M.-Y. Wu, "Performance-effective and low-complexity task scheduling for heterogeneous computing," Parallel and Distributed Systems, IEEE Transactions on, vol. 13,no. 3, pp. 260-274, Mar 2002.
8. Z. Yu and W. Shi, "An adaptive rescheduling strategy for grid workflow applications," in Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE International, March 2007, pp. 1-8.
9. E. Ilavarasan and P. Thambidurai, "Low complexity performance effective task scheduling algorithm for heterogeneous computing environments," Journal of Computer Science, vol. 3, no. 2, pp. 94-103, 2007.
10. R. Sakellariou and H. Zhao, "A hybrid heuristic for dag scheduling on heterogeneous systems," in Parallel and Distributed Processing Symposium, 2004. Proceedings. 18th International, April 2004, pp. 111-.
11. W. Guo, W. Sun, W. Hu, and Y. Jin, "Resource allocation strategies for data-intensive workflow-based applications in optical grids," in Communication systems, 2006. ICCS 2006. 10th IEEE Singapore International Conference on, Oct. 2006, pp. 1-5.
12. J. Blythe, S. Jain, E. Deelman, Y. Gil, K. Vahi, A. Mandal, and K. Kennedy, "Task scheduling strategies for workflow-based applications in grids," Cluster Computing and the Grid, IEEE International Symposium on, vol. 2, pp. 759-767, 2005.
13. H. Abbasi, M. Wolf, G. Eisenhauer, S. Klasky, K. Schwan, and F. Zheng, "Datastager: scalable data staging services for petascale applications," in HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 39-48.
14. T. Kosar and M. Livny, "Stork: making data placement a first class citizen in the grid," in Distributed Computing Systems, 2004. Proceedings. 24th International Conference on, 2004, pp. 342-349.
15. I. Raicu, Y. Zhao, C. Dumitrescu, I. Foster, and M. Wilde, "Falkon: a fast and light-weight task execution framework," in SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing. New York, NY, USA: ACM, 2007, pp. 1-12.
16. G. Khanna, N. Vydyanathan, U. Catalyurek, T. Kurc, S. Krishnamoorthy, P. Sadayappan, and J. Saltz, "Task scheduling and file replication for data-intensive jobs with batch-shared i/o," in High Performance Distributed Computing, 2006 15th IEEE International Symposium on, 0-0 2006, pp. 241-252.
17. I. Raicu, I. T. Foster, Y. Zhao, P. Little, C. M. Moretti, A. Chaudhary, and D. Thain, "The quest for scalable support of data-intensive workloads in distributed systems," in HPDC '09: Proceedings of the 18th ACM international symposium on High performance distributed computing. New York, NY, USA: ACM, 2009, pp. 207-216.
18. K. Ranganathan and I. Foster, "Decoupling computation and data scheduling in distributed data-intensive applications," in High Performance Distributed Computing, 2002. HPDC-11 2002. Proceedings. 11th IEEE International Symposium on, 2002, pp. 352-358.
19. T. Kosar, "A new paradigm in data intensive computing: Stork and the data-aware schedulers," in Challenges of Large Applications in Distributed Environments, 2006 IEEE, 0-0 2006, pp. 5-12.
20. F. Dong and S. G. Akl, "Scheduling algorithms for grid computing: State of the art and open problems," School of Computing, Queen's University, Tech. Rep., January 2006.
21. S. Bharathi, A. Chervenak, E. Deelman, G. Mehta, M.-H. Su, and K. Vahi, "Characterization of scientific workflows," in Workows in Support of Large-Scale Science, 2008. WORKS 2008. Third Workshop on, Nov. 2008, pp. 1-10.
22. B. Pawlowski, C. Juszczak, P. Staubach, C. Smith, D. Lebel, and D. Hitz, "Nfs version 3 - design and implementation," in In Proceedings of the Summe USENIX Conference, 1994, pp. 137-152.
23. F. Schmuck and R. Haskin, "Gpfs: A shared-disk file system for large computing clusters," in In Proceedings of the 2002 Conference on File and Storage Technologies (FAST, 2002, pp. 231-244.
24. S. A. Weil, S. A. Brandt, E. L. Miller, D. D. E. Long, and C. Maltzahn, "Ceph: A scalable, high-performance distributed file system," in Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2006, p. 307 20.
25. O. Tatebe, Y. Morita, S. Matsuoka, N. Soda, and S. Sekiguchi, "Grid datafarm architecture for petascale data intensive computing," in Cluster Computing and the Grid, 2002. 2nd IEEE/ACM International Symposium on, May 2002, pp. 102-102.
26. N. Dun, K. Taura, and A. Yonezawa, "Gmount: An ad hoc and locality-aware distributed file system by using ssh and fuse," in 9th IEEE/ACM International Symposium on Cluster Computing and the Grid, 2009, pp. 188-195.
27. "Montage: An astronomical image mosaic engine," http://montage.ipac.caltech.edu/.
28. N. Dun, K. Taura, and A. Yonezawa, "Paratrac: A fine-grained profiler for data-intensive workflows," in to appear; 19th IEEE Symposium on High Performance Distributed Computing, 2010.
29. T. Uno, "A new approach for speeding up enumeration algorithms," in Proceeding of ISAAC98(Lecture Note in Computer Science 1533, Springer-Verlag, Algorithms and Computation), 1998, pp. 287-296.
30. N. K. Daisuke Kawahara and S. Kurohashi, "Japanese case structure analysis by unsupervised construction of a case frame dictionary," in International Conference on Computational Linguistics (COLING2000), 2000, pp. 432-438.
31. R. Sasano, D. Kawahara, and S. Kurohashi, "The effect of corpus size on case frame acquisition for discourse analysis," in the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT2009), 2009, pp. 521-529.