A transparent collective I/O implementation

Proceedings - IEEE 27th International Parallel and Distributed Processing Symposium, IPDPS 2013

Volumn , Issue , 2013, Pages 297-307

  Yu, Yongen ^a   Wu, Jingjin ^a   Lan, Zhiling ^a   Rudd, Douglas H ^b   Gnedin, Nickolay Y ^c   Kravtsov, Andrey ^b  

^a Illinois Institute of Technology   (United States)

^b UNIVERSITY OF CHICAGO   (United States)

^c NASA FERMILAB ASTROPHYSICS CENTER   (United States)

Author keywords

Collective I O; HPC; I O intensive applications; MPI; One sided communication; Parallel I O; Transparent Collective I O

Indexed keywords

COLLECTIVE I/O; HPC; MPI; ONE-SIDED COMMUNICATIONS; PARALLEL I/O;

DISTRIBUTED PARAMETER NETWORKS; OPTIMIZATION;

BENCHMARKING;

EID: 84884831718     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/IPDPS.2013.36     Document Type: Conference Paper

References (27)

1. P. Crandall, R. Aydt, A. Chien and D. Reed, "Input-Ouput Characteristics of Scalable Parallel Applications," in Proceedings of Supercomputing '95, 1995.
2. N. Nieuwejaar, D. Kotz, A. Purakayastha, C. Ellis and M.Best, "File-Access Characteristics of Parallel Scientific Workloads," IEEE Transactions on Parallel and Distributed Systems, vol. 7, pp. 1075-1089, October 1996. (Pubitemid 126780465)
3. E. Smirni, R, Aydt, A. Chien and D. Reed, "I/O Requirements of Scientific Applications: An Evolutionary View," in Proceedings of the Fifth IEEE International Symposium on High Performance Distributed Computing, 1996, pp. 49-59.
4. J. Lofstead, M. Polte, G. Gibson, S.A. Klasky, K. Schwan, R. Oldfield, M. Wolf and Q. Liu, "Six Degrees of Scientific Data: Reading Patterns for Extreme Scale Science IO," in Proceedings of the 20th international ACM symposium on High-Performance Parallel and Distributed Computing, San Jose, CA, June, 2011.
5. Y. Cui, K.B. Olsen, T.H. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D.K. Panda, A. Chourasia, J. Levesque, S.M. Day and P. Maechling, "Scalable Earthquake Simulation on Petascale Supercomputers," in Proceedings of SC '10, New Orleans, Louisiana, USA, 2010.
6. R. Sankaran, E. Hawkes, J. Chen, T. Lu, and C. Law, "Direct Numerical Simulations of Turbulent Lean Premixed Combustion," Journal of Physics: conference series, vol. 46, pp. 38-42, 2006.
7. R.Thakur, W.Gropp and E.Lusk, "Data Sieving and Collective I/O in ROMIO," in Proc. of the 7th Symposium on the Frontiers of Massively parallel Computation, 1999, pp. 182-189.
8. P. Carns, K. Harms, W. Allcock, C. Bacon, S. Lang, R. Latham and R. Ross, "Understanding and improving computational science storage access through continuous characterization," Mass Storage Systems and Technologies, IEEE / NASA Goddard Conference on, vol. 0, pp. 1-14, 2011.
9. M. Zingale. FLASH I/O Benchmark Routine - Parallel HDF 5. [Online]. http://www.ucolick.org/~zingale/flash-benchmark-io/
10. W. Loewe, R. Klundt. IOR HPC Benchmark. [Online]. http://sourceforge.net/ projects/ior-sio/
11. The Los Alamos National Lab MPI-IO Test. [Online]. http://public.lanl. gov/jnunez/benchmarks/mpiiotest.htm
12. ROMIO: A High-Performance, Portable MPI-IO Implementation. [Online]. http://www.mcs.anl.gov/research/projects/romio/
13. G. Memik, M. T. Kandemir, W. Liao, and A. Choudhary, " Multicollective I/O: A technique for exploiting inter-file access patterns," Trans. Storage, pp. 349-369, Aug. 2006.
14. V. Venkatesan, M. Chaarawi, E. Gabriel, and T. Hoefler, "Design and Evaluation of Nonblocking Collective I/O Operations," in Recent Advances in the Message Passing Inerface (EuroMPI'10), vol. 6960, Santorini, Greece, 2011, pp. 90-98.
15. W. Yu, J. Vetter, "ParColl: Partitioned Collective I/O on the Cray XT," in ICPP '08, Portland, OR, 2008, pp. 562-569.
16. J. Blas, F. Isail, D. Singh, and J. Carretero, "View-based collective I/O for MPI-IO," in Cluster Computing and the Grid, 2008. CCGRID '08, 2008, pp. 409-416.
17. Y. Chen, X. Sun, R. Thakur, P. Roth, W. Gropp, "LACIO: A New Collective I/O strategy for Parallel I/O Systems," in the 25th IEEE Int'l Parallel and Distributed Processing Sympsium(IPDPS2011), 2011.
18. X. Zhang, S. Jiang, and D. Kei, "Making Resonance a Common Case: A High-Performance Implementation of Collective I/O on Parallel FIle Systems," in Parallel & Distributed Processing IPDPS '09, 2009, pp. 1-12.
19. W. Liao, and A. Choudhary, "Dynamically Adapting File Domain Partitioning Methods for Collective I/O Based on Underlying Parallel File System Locking Protocolssed," in High Performance Computing, Networking, Storage and Analysis, SC '08, Austin, TX, 2008.
20. CRAY. [Online]. https://fs.hlrs.de/projects/craydoc/docs/books/S-2490-40/ html-S-2490-40/chapter-g1s9a5n5-oswald-benchmarkresults.html
21. NERSC. [Online]. http://www.nersc.gov/users/data-and-networking/ optimizing-io-performance-for-lustre/
22. K. Coloma, A, Ching, A. Choudhary, W.Liao, R. Ross, R. Thakur and L. Ward, "A New Flexible MPI Collective I/O Implemenation," in n Proceedings of the IEEE Conference on Cluster Computing (Cluster 2006), Barcelona, Spain.
23. Extreme Science and Engineering Discovery Environment. [Online]. https://www.xsede.org/
24. Oracle. Lustre File System. [Online]. http://wiki.lustre.org
25. A. V. Kravtsov, A. A. Klypin and A. M. Khokhlov, "Adaptive Refinement Tree: A new High-resolution N-body code for Cosmological Simulations," in Astrophyc. J. Suppl, 1997, pp. 73-94.
26. A. M. Khokhlov, "Fully Threaded Tree Algorithms for Adaptive Refinement Fluid Dynamics Simulations," Journal of Computational Physics, 1998.
27. Y. Yu, D.H. Rudd, Z. lan, N.Y. Gnedin, A. Kravtsov and J. Wu,, "Improving Parallel IO Performance of Cell-based AMR Cosmology Applications," in IPDPS '12, 2012.