Using subfiling to improve programming flexibility and performance of parallel shared-file I/O

Proceedings of the International Conference on Parallel Processing

Volumn , Issue , 2009, Pages 470-477

  Gao, Kui ^a   Liao, Wei Keng ^a   Nisar, Arifa ^a   Choudhary, Alok ^a   Ross, Robert ^b   Latham, Robert ^b  

^a Northwestern University   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

MPI IO; Parallel netCDF; Subfiling

Indexed keywords

COMPUTATIONAL APPLICATIONS; DATA CONSISTENCY; FILE HEADER; FILE SYSTEMS; GLOBAL ARRAYS; I/O BENCHMARK; I/O PERFORMANCE; PARALLEL I/O; PARALLEL MACHINE; PROBLEM SIZE; PROGRAMMING FLEXIBILITY; PROGRAMMING STYLES; SUB-ARRAYS;

DATA PROCESSING;

PARALLEL PROGRAMMING;

EID: 77951432308     PISSN: 01903918     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICPP.2009.68     Document Type: Conference Paper

References (23)

1. R. Thakur and A. Choudhary. An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays. Journal of Scientific Programming, 5(4):301, Winter 1996.
2. Message Passing Interface Forum. MPI-2: Extensions to the Message-Passing Interface. July 1997. http://www.mpiforum.org/docs/docs.html.
3. R. Ross, R. Latham, W. Gropp, R. Thakur, and B. Toonen. Implementing MPI-IO Atomic Mode Without File System Support. In Proceedings of the 5th IEEE/ACM International Symposium on Cluster Computing and the Grid, May 2005.
4. W. Liao, A. Ching, K. Coloma, A. Choudhary, and L. Ward. An Implementation and Evaluation of Client-Side File Caching for MPI-IO. In Proceedings of the International Parallel and Distributed Processing Symposium, March 2007.
5. HPSS User's Guide, High Performance Storage System Release 7.1, February 2009 (Revision 1.0). http://www.hpsscollaboration. org/hpss
6. Anurag Shankar, IU's TeraGrid-accessible HPSS archival storage system, http://racinfo.indiana.edu/hpc/workshops/includes/HPSS TG slides.ppt.
7. R. Rew, G. Davis. The Unidata netCDF: Softwarefor Scientific Data Access. Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography and Hydrology, February 1990.
8. J. Li, W. Liao, A. Choudhary, Robert Ross, Rajeev Thakur, William Gropp, Rob Latham, Andrew Siegel, Brad Gallagher, and Michael Zingale. Parallel netCDF: A Scientific High- Performance I/O Interface. In Proceedings of the ACM/IEEE Conference on Supercomputing (SC), pages 39-49, November 2003.
9. IEEE/ANSI Std. 1003.1. Portable operating system interface (POSIX)- Part 1: System application program interface (API) [C language], 1996.
10. Cluster File System, Inc. Lustre: A Scalable, High Performance File System. http://www.Lustre.org/docs.html.
11. F. Schmuck and R. Haskin. GPFS: A Shared-Disk File System for Large Computing Clusters. In Proceedings of the File and Storage Technologies (FAST '02), pp. 231-244, Jan. 2002.
12. Sun Microsystems. ZFS: the last word in file systems. http://www.sun.com/2004-0914/feature/, 2004.
13. S.G. Inc. XFS: A High-Performance Journaling Filesystem. http://oss.sgi.com/projects/xfs, 2006.
14. J. del Rosario, R. Bordawekar and A. Choudhary, Improved Parallel I/O via a Two-phase Run-time Access Strategy. In Proceedings of the workshop on I/O in Carallel Computer Systems at IPPS '93, pages 56-60, April 1993.
15. R. Thakur, W. Gropp and E. Lusk. Users Guide for ROMIO: A High-Performance, Portable MPI-IO Implementation. Technical Report ANL/MCS-TM-234, Mathematics and Computer Science Division, Argonne National Laboratory, October 1997.
16. D. Kotz, Disk-directed I/O for MIMD Multiprocessors. In proceedings of th e1994 Symposium on Operating Systems Design and Implementation, pages 61-74, 1994.
17. K. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett. Server-directed collective I/O in Panda. In Proceedings of Supercomputing '95, December 1995.
18. W. K. Liao, K. Coloma, A. Choudhary, L. Ward, E. Russel, and N. Pundit, Scalable Design and Implementations for MPI Parallel Overlapping I/O, IEEE Transactions on Parallel and Distributed Systems, 17 (11) pages1264-1276, 2006.
19. X. Ma, M. Winslett, J. Lee, and S. Yu. Improving MPI-IO Output Performance with Active Buffering Plus Threads. In the International Parallel andDistributed Processing Symposium (IPDPS), April 2003.
20. W. K. Liao, K. Coloma, A. Choudhary, L. Ward, E. Russel, and S. Tideman. Collective Caching: Application-Aware Client-Side File Caching. In Proceedings of the 14th International Symposium on High Performance Distributed Computing (HPDC), July 2005.
21. R. Sankaran, E. R. Hawkes, J. H. Chen, T. Lu and C. K. Law. Direct numerical simulations of turbulent lean premixed combustion. Journal of Physics: Conference Series, 46:38-42, 2006.
22. M. Zingale, ":FLASH I/O Benchmark Routine- Parallel HDF 5,": Mar. 2001, http://flash.uchicago.edu/?zingale/flash benchmark io.
23. B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, and H. Tufo, ":FLASH: An Adaptive Mesh Hydrodynamics Code for Modelling Astrophysical Thermonuclear Flashes,": Astrophysical Journal Suppliment, pp. 131-273, 2000.