Combining I/O operations for multiple array variables in parallel netCDF

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2009, Pages

  Gao, Kui ^a   Liao, Wei Keng ^a   Choudhary, Alok ^a   Ross, Robert ^b   Latham, Robert ^b  

^a Northwestern University   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

MPI IO; Parallel netCDF; Parallel IO

Indexed keywords

APPLICATION PROGRAMMERS; ARRAY VARIABLES; ASYNCHRONOUS I/O; COLLECTIVE I/O; DATA SETS; I/O OPERATIONS; I/O PERFORMANCE; LARGE REGIONS; MULTIPLE ARRAYS; NEW MECHANISMS; PARALLEL I/O; PERFORMANCE DEGRADATION; SCIENTIFIC APPLICATIONS; STORAGE SYSTEMS;

BENCHMARKING; CLUSTER COMPUTING; COMPUTER SCIENCE; TECHNICAL PRESENTATIONS;

DATA STRUCTURES;

EID: 72049125851     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTR.2009.5289153     Document Type: Conference Paper

References (22)

1. R. Rew and G. Davis, "The Unidata netCDF: Softwarefor Scientific Data Access," Sixth International Conference on Interactive Information and Processing Systems for Meteorology, Oceanography and Hydrology, February 1990.
2. R. Rew, G. Davis, S. Emmerson and H. Davies, "NetCDF User's Guide for C," Unidata Program Center, June 1997. http://www.unidata.ucar.edu/ packages/netcdf/guidec/.
3. J. Li, W. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher and M. Zingale, "Parallel netCDF: A Scientific High-Performance I/O Interface," In Proceedings of the ACM/IEEE Conference on Supercomputing (SC), pages 39-49, November 2003.
4. Message Passing Interface Forum, "MPI-2: Extensions to the Message-Passing Interface," 1997, [Online] Available: http://www.mpi-forum. org/docs/docs.html.
5. Message Passing Interface Forum, "MPI: A Message-Passing Interface Standard," 1995. http://www.mpiforum.org/docs/docs.html.
6. R. Thakur and A. Choudhary, "An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays," Journal of Scientific Programming, 5(4):301, Winter 1996.
7. R. Thakur, W. Gropp and E. Lusk, "Users Guide for ROMIO: A High-Performance, Portable MPI-IO Implementation," Technical Report ANL/MCS-TM-234, Mathematics and Computer Science Division, Argonne National Laboratory, October 1997.
8. D. Kotz, "Disk-directed I/O for MIMD Multiprocessors," In proceedings of the 1994 Symposium on Operating Systems Design and Implementation, pages 61-74, 1994.
9. K. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett, "Server-directed collective I/O in Panda," In Proceedings of Supercomputing '95, December 1995.
10. W. K. Liao, K. Coloma, A. Choudhary, L. Ward, E. Russel, and N. Pundit, "Scalable Design and Implementations for MPI Parallel Overlapping I/O," IEEE Transactions on Parallel and Distributed Systems, 17 (11) pages1264-1276, 2006.
11. X. Ma, M. Winslett, J. Lee, and S. Yu, "Improving MPI-IO Output Performance with Active Buffering Plus Threads," In the International Parallel andDistributed Processing Symposium (IPDPS), April 2003.
12. W. K. Liao, K. Coloma, A. Choudhary, L. Ward, E. Russel, and S. Tideman, "Collective Caching: Application-Aware Client-Side File Caching," In Proceedings of the 14th International Symposium on High Performance Distributed Computing (HPDC), July 2005.
13. HDF4 Home Page. The National Center for Supercomputing Applications. http://hdf.ncsa.uiuc.edu/hdf4.html.
14. HDF5 Home Page. The National Center for Supercomputing Applications. http://hdf.ncsa.uiuc.edu/HDF5/.
15. F. Schmuck and R. Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters," In Proceedings of the File and Storage Technologies (FAST '02), pp. 231-244, Jan. 2002.
16. Cluster File System, Inc. "Lustre: A Scalable, High Performance File System," http://www.Lustre.org/docs.html.
17. W. K. Liao, A. Choudhary "Dynamically Adapting File Domain Partitioning Methods for Collective I/O Based on Underlying Parallel File System Locking Protocols," In Proceedings of the ACM/IEEE Conference on Supercomputing (SC), pages 313-344, November 2008.
18. M. Zingale, "FLASH I/O Benchmark Routine - Parallel HDF 5," Mar. 2001, http://flash.uchicago.edu/~zingale/flash-benchmark-io.
19. B. Fryxell, K. Olson, P. Ricker, F. X. Timmes, M. Zingale, D. Q. Lamb, P. MacNeice, R. Rosner, and H. Tufo, "FLASH: An Adaptive Mesh Hydrodynamics Code for Modelling Astrophysical Thermonuclear Flashes," Astrophysical Journal Suppliment, pp. 131-273, 2000.
20. R. Thakur, W. Gropp and E. Lusk, "Data sieving and collective I/O in ROMIO," In Proceedings of the Seventh Symposium on the Frontiers of Massively Parallel Computation, pages 182-189. IEEE Computer Society Press, 1999.
21. R. Thakur, W. Gropp, and E. Lusk, "A Case for Using MPI's Derived Datatypes to Improve I/O Performance," in Proc. of SC98: High Performance Networking and Computing, November 1998.
22. J. del Rosario, R. Bordawekar and A. Choudhary, "Improved Parallel I/O via a Two-phase Run-time Access Strategy," In Proceedings of the workshop on I/O in Parallel Computer Systems at IPPS '93 , pages 56-60, April 1993.