Scalable design and implementations for MPI parallel overlapping I/O

IEEE Transactions on Parallel and Distributed Systems

Volumn 17, Issue 11, 2006, Pages 1264-1276

  Liao, Wei Keng ^a   Coloma, Kenin ^a   Choudhary, Alok ^a   Ward, Lee ^b   Russell, Eric ^b   Pundit, Neil ^b  

^a Northwestern University   (United States)

^b SANDIA NATIONAL LABORATORIES   (United States)

Author keywords

Atomic I O; Cache coherence; File atomicity; File consistency; MPI; MPI I O; Overlapping I O

Indexed keywords

CACHE MEMORY; CLIENT SERVER COMPUTER SYSTEMS; FILE ORGANIZATION; INPUT OUTPUT PROGRAMS; INTERFACES (COMPUTER); SERVERS; WORD PROCESSING;

FILE LOCKING; GRAPH COLORING; MESSAGE PASSING INTERFACE; PORTABLE OPERATING SYSTEM INTERFACE; PROCESS-RANK ORDERING;

PARALLEL PROCESSING SYSTEMS;

EID: 33750100114     PISSN: 10459219     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPDS.2006.163     Document Type: Article

References (31)

1. P. Crandall, R. Aydt, A. Chien, and D. Reed, "Input-Output Characteristics of Scalable Parallel Applications," Supercomputing, Dec. 1995
2. N. Nieuwejaar, D. Kotz, A. Purakayastha, C. Ellis, and M. Best, "File-Access Characteristics of Parallel Scientific Workloads," IEEE Trans. Parallel and Distributed Systems, vol. 7, no. 10, pp. 1075-1089, Oct. 1996.
3. E. Smirni, R. Aydt, A. Chien, and D. Reed, "I/O Requirements of Scientific Applications: An Evolutionary View," Proc. Fifth IEEE Int'l Symp. High Performance Distributed Computing, pp. 49-59, 1996.
4. E. Smirm and D. Reed, "Lessons from Characterizing the Input/Output Behavior of Parallel Scientific Applications," Performance Evaluation: An Int'l J., vol. 33, no. 1, pp. 27-44, June 1998.
5. Message Passing Interface Forum, "MPI-2: Extensions to the Message Passing Interface," http://www.mpi-forum.org/docs/docs.html, July 1997.
6. IEEE/ANSI Std. 1003.1, "Portable Operating System Interface (POSIX)-Part 1: System Application Program Interface (API) [C Language]," 1996.
7. HDFGroup, Hierarchical Data Format, version 5, Nat'l Center for Supercomputing Applications, http://hdf.ncsa.uiuc.edu/HDF5, 2004.
8. The UniData Program Center, Network Common Data Form, Univ. Corporation for Atmospheric Research, http://www.unidata.ucar.edu/software/netcdf, 2004.
9. R. Thakur, W. Gropp, and E. Lusk, "Users Guide for ROMIO: A High-Performance, Portable MPI-IO Implementation," Technical Report ANL/MCS-TM-234, Math. and Computer Science Division, Argonne Nat'l Laboratory, Oct. 1997.
10. IEEE Std. 1003.1-2001, System Interfaces, 2001.
11. "RS/6000 SP Software: Parallel I/O File System," IBM, http://www.rs6000.ibm.com/software/sp-products/piofs.html, 1996.
12. F. Schmuck and R. Haskin, "GPFS: A Shared-Disk File System for Large Computing Clusters," Proc. Conf. File and Storage Technologies (FAST '02), pp. 231-244, Jan. 2002.
13. S.G. Inc., XFS: A High-Performance journaling Filesystem, http://oss.sgi.com/projects/xfs, 2006.
14. A. Tanenbaum and M. van Steen, Distributed Systems - Principles and Paradigms. Prentice Hall, 2002.
15. T. Cortes, S. Girona, and J. Labarta, "Design Issues of a Cooperative Cache with No Coherence Problems," High Performance Mass Storage and Parallel I/O: Technologies and Applications, pp. 259-270, 2001.
16. M. Kallahalla and P. Varman "Optimal Prefetching and Caching for Parallel I/O Systems," Proc. 13th Ann. ACM Symp. Parallel Algorithms and Architectures, pp. 219-228, 2001.
17. T. Kimbrel, P. Cao, E. Felten, A. Karlin, and K. Li, "Integrating Parallel Prefetching and Caching," Proc. ACM SIGMETRICS Conf. Measurement and Modelling of Computer Systems, pp. 262-263, 1996.
18. S. Soltis, T. Ruwart, G. Erickson, K. Preslan, and M. O'Keefe, "The Global File System," High Performance Mass Storage and Parallel I/O: Technologies and Applications, pp. 10-15, 2001.
19. B. Forney, A. Arpaci-Dusseau, and R. Arpaci-Dusseau, "Storage-Aware Caching: Revisiting Caching for Heterogeneous Storage Systems," Proc. First Int'l Conf. File and Storage Technologies (FAST), 2002.
20. S. Shepler, C. Beame, R. Callaghan, M. Eisler, D. Noveck, D. Robinson, and R. Thurlow, "NFS version 4 Protocol," Dec. 2000.
21. P. Carns, W. Ligon, R. Ross, and R. Thakur, "PVFS: A Parallel File System for Linux Clusters," Proc. Third Ann. Linux Showcase and Conf., pp. 317-327, Oct. 2000.
22. R. Thakur, R. Brodawekar, A. Choudhary, R. Pormusamy, and T. Singh, "PASSION: Runtime Library for Parallel I/O," Proc. Scalable Parallel Libraries Conf., pp. 119-128, Oct. 1994.
23. J. del Rosario, R. Brodawekar, and A. Choudhary, "Improved Parallel I/O via a Two-Phase Run-Time Access Strategy," Proc. Workshop I/O in Parallel Computer Systems, pp. 56-70, Apr. 1993.
24. R. Thakur, W. Gropp, and E. Lusk, "On Implementing MPI-IO Portably and with High Performance," Proc. Sixth Workshop I/O in Parallel and Distributed Systems, pp. 23-32, May 1999.
25. M. Garey and D. Johnson, Computers and Intractability: A Guide to the Theory of NP-Completeness. New York: W. H. Freeman, 1979.
26. D. Kotz, "Disk-Directed I/O for MIMD Multiprocessors," ACM Trans. Computer Systems, vol. 15, no. 1, pp. 41-74, Feb. 1997.
27. Sandia National Laboratories, ENFS-Extended NFS, http://www.cs.sandia.gov/cplant/doc/io/ENFSUser_Doc.html, 2004.
28. Sandia National Laboratories, "Computational Plant," http://www.cs.sandia.gov/cplant, 2004.
29. K. Seamons, Y. Chen, P. Jones, J. Jozwiak, and M. Winslett, "Server-Directed Collective I/O in Panda," Supercomputing, Dec. 1995.
30. R. Thakur, W. Gropp, and E. Lusk, "Data Sieving and Collective I/O in ROMIO," Proc. Seventh Symp. Frontiers of Massively Parallel Computation, Feb. 1999.
31. R. Thakur and A. Choudhary, "An Extended Two-Phase Method for Accessing Sections of Out-of-Core Arrays," Scientific Programming, vol. 5, no. 4, pp. 301-317, Feb. 1996.