Exploiting lustre file joining for effective collective IO

Proceedings - Seventh IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2007

Volumn , Issue , 2007, Pages 267-274

  Yu, Weikuan ^a   Vetter, Jeffrey ^a   Canon, R Shane ^a   Jiang, Song ^b  

^a OAK RIDGE NATIONAL LABORATORY   (United States)

^b Wayne State University   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BANDWIDTH; DATA STORAGE EQUIPMENT; DATA TRANSFER; FILE ORGANIZATION;

HIERARCHICAL STRIPING; SPLIT WRITING;

PARALLEL PROCESSING SYSTEMS;

EID: 34548361907     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CCGRID.2007.51     Document Type: Conference Paper

References (26)

1. IOzone Filesystem Benchmark, http://www.iozone.org/
2. The Parallel Virtual File System, version 2. http://www.pvfs.org/pvfs2.
3. TOP 500 Supercomputers, http://www.top500.org/.
4. Argonne National Laboratory. ROMIO: A High-Performance, Portable MPI-IO Implementation. http://www-unix.mcs.anl.gov/romio/.
5. A. Ching, A. Choudhary, W. Liao, R. Ross, and W. Gropp. Noncontiguous I/O through PVFS. In Proceedings of the IEEE International Conference on Cluster Computing, Chicago, IL, September 2002.
6. Cluster File System, Inc. Lustre: A Scalable, High Performance File System. http://www.Lustre.org/docs.html.
7. A. M. David Nagle, Denis Serenyi. The Panasas ActiveScale Storage Cluster. Delivering Scalable High Bandwidth Storage. In Proceedings of Supercomputing '04, November 2004.
8. R. Latham, R. Ross, and R. Thakur. The Impact of File Systems on MPI-IO Scalability. In Proceedings of the 11th European PVM/MPI Users' Group Meeting (Euro PVM/MPI2004), pages 87.96, September 2004.
9. J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, and R. Latham. Parallel netCDF: A High Performance Scientific I/O Interface. In Proceedings of the Supercomputing '03, November 2003.
10. W. Liao, A. Ching, K. Coloma, A. Choudhary and L. Ward. Implementation and Evaluation of Client-side File Caching for MPI-IO. In Proceedings of the IEEE International Parallel & Distributed Processing Symposium '07, March 2007.
11. W. Liao, K. Coloma, A. Choudhary, L. Ward, E. Russell, and S. Tideman. In Proceedings of the ymposium on High Performance Distributed Computing 2005.
12. Z. Lin, S. Ethier, T. S. Hahm, and W. M. Tang. Size scaling of turbulent transport in magnetically confined plasmas. Phys. Rev. Lett., 88(19): 195004, Apr 2002.
13. J.-P. Prost, R. Treumann, R. Hedges, B. Jia, and A. Koniges. MPI-IO/GPFS, an Optimized Implementation of MPI-IO on Top of GPFS. In Proceedings of Supercomputing '01, November 2001.
14. R. Ross. Parallel I/O Benchmarking Consortium. http://www-unix.mcs. anl.gov/rross/pio-benchmark/html/.
15. R. Ross, J. Moreira, K. Cupps and W. Pfeiffer. Parallel I/O on the IBM BlueGene/L System.
16. F. Schmuck and R. Haskin. GPFS: A Shared-Disk File System for Large Computing Clusters. In FAST '02, pages 231.244. USENIX, Jan. 2002.
17. O. Tatebe, Y. Morita, S. Matsuoka, N. Soda, S. Sekiguchi, "Grid Datafarm Architecture for Petascale Data Intensive Computing," Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid (CCGrid 2002).
18. O. Tatebe, N. Soda, Y. Morita, S. Matsuoka, S. Sekiguchi, "Gfarm v2: A Grid file system that supports high-performance distributed and parallel data computing," Proceedings of the 2004 Computing in High Energy and Nuclear Physics (CHEP04), Interlaken, Switzerland, September 2004.
19. R. Thakur and A. Choudhary. An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays. Scientific Programming, 5(4):301.317, Winter 1996.
20. R. Thakur, W. Gropp, and E. Lusk. An Abstract-Device Interface for Implementing Portable Parallel-I/O Interfaces. In Proceedings of Frontiers '96: The Sixth Symposium on the Frontiers of Massively Parallel Computation. IEEE Computer Society, Oct 1996.
21. R. Thakur, W. Gropp, and E. Lusk. Data sieving and collective I/O in ROMIO. In Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, pages 182.189. IEEE Computer Society Press, 1999.
22. R. Thakur, W. Gropp, and E. Lusk. On Implementing MPI-IO Portably and with High Performance. In Proceedings of the 6th Workshop on I/O in Parallel and Distributed Systems, pages 23.32. ACM Press, May 1999.
23. The National Center for SuperComputing. HDF Home Page. http://hdf.ncsa.uiuc. com/hdf4.html.
24. The National Center for SuperComputing. HDF5 Home Page. http://hdf.ncsa.uiuc.com/HPD5/.
25. P. Wong and R. F. Van der Wijngaart. NAS Parallel Benchmarks I/O Version 2.4. Technical Report NAS-03-002, Computer Sciences Corporation, NASA Advanced Supercomputing (NAS) Division.
26. W. Yu, Q. Gao, and D. K. Panda. Adaptive Connection Management for Scalable MPI over InfiniBand. In International Parallel and Distributed Processing Symposium, Rhodes Island, Greece, April 2006.