PIDX: Efficient parallel I/O for multi-resolution multi-dimensional scientific datasets

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2011, Pages 103-111

  Kumar, Sidharth ^a   Vishwanath, Venkatram ^b   Carns, Philip ^b   Summa, Brian ^a   Scorzelli, Giorgio ^a   Pascucci, Valerio ^a   Ross, Robert ^b   Chen, Jacqueline ^c   Kolla, Hemanth ^c   Grout, Ray ^d  

^a UNIVERSITY OF UTAH   (United States)

^b ARGONNE NATIONAL LABORATORY   (United States)

^c SANDIA NATIONAL LABORATORIES   (United States)

^d NATIONAL RENEWABLE ENERGY LABORATORY   (United States)

Author keywords

High Performance computing; in situ visualization; Interactive visualization; Parallel file systems; Parallel I O

Indexed keywords

HIGH PERFORMANCE COMPUTING; IN SITU VISUALIZATION; INTERACTIVE VISUALIZATIONS; PARALLEL FILE SYSTEMS; PARALLEL I/O;

CLUSTER COMPUTING; COMBUSTION; COMPUTER SOFTWARE SELECTION AND EVALUATION; FILE ORGANIZATION; LAPTOP COMPUTERS; REAL TIME SYSTEMS; USER INTERFACES; VISUALIZATION;

DATA VISUALIZATION;

EID: 80955123465     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2011.19     Document Type: Conference Paper

References (19)

1. HDF5 home page. http://www.hdfgroup.org/HDF5/.
2. P. Carns, K. Harms, W. Allcock, C. Bacon, R. Latham, S. Lang, and R. Ross. Understanding and improving computational science storage access through continuous characterization. In Proceedings of 27th IEEE Conference on Mass Storage Systems and Technologies (MSST 2011), 2011.
3. J. H. Chen, A. Choudhary, B. de Supinski, M. DeVries, E. R. Hawkes, S. Klasky, W. K. Liao, K. L. Ma, J. M. Crummey, N. Podhorszki, R. Sankaran, S. Shende, and C. S. Yoo. Terascale direct numerical simulations of turbulent combustion using s3d. In Computational Science and Discovery Volume 2, January 2009.
4. A. Ching, A. Choudhary, W. K. Liao, R. Ross, and W. Gropp. Evaluating structured I/O methods for parallel file systems. International Journal of High Performance Computing and Networking, 2:133-145, 2004.
5. T.-c. Chiueh and R. H. Katz. Multi-resolution video representation for parallel disk arrays. In Proceedings of the first ACM international conference on Multimedia, MULTIMEDIA '93, pages 401-409, New York, NY, USA, 1993. ACM.
6. J. M. del Rosario, R. Bordawekar, and A. Choudhary. Improved parallel i/o via a two-phase run-time access strategy. SIGARCH Comput. Archit. News, 21:31-38, December 1993.
7. K. Gao, W. keng Liao, A. Nisar, A. Choudhary, R. Ross, and R. Latham. Using subfiling to improve programming flexibility and performance of parallel shared-file i/o. In Parallel Processing, 2009. ICPP '09. International Conference on, pages 470 -477, sept. 2009.
8. S. Guthe, M. Wand, J. Gonser, and W. Strasser. Interactive rendering of large volume data sets. In Proceedings of the conference on Visualization '02, VIS '02, pages 53-60, Washington, DC, USA, 2002. IEEE Computer Society.
9. S. Kumar, V. Pascucci, V. Vishwanath, P. Carns, R. Latham, T. Peterka, M. Papka, and R. Ross. Towards parallel access of multi-dimensional, multiresolution scientific data. In Proceedings of 2010 Petascale Data Storage Workshop, November 2010.
10. J. Li, W. keng Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale. Parallel netCDF: A high-performance scientific I/O interface. In Proceedings of SC2003: High Performance Networking and Computing, Phoenix, AZ, November 2003. IEEE Computer Society Press.
11. J. Lofstead, S. Klasky, S. K., N. Podhorszki, and C. Jin. Flexible io and integration for scientific codes through the adaptable io system (adios). June 2008.
12. G. Memik, M. T. Kandemir, W. Liao, and A. Choudhary. Multicollective I/O: a technique for exploiting inter-file access patterns. Transactions on Storage, 2(3):349-369, 2006. (Pubitemid 44656448)
13. V. Pascucci and R. J. Frank. Global static indexing for real-time exploration of very large regular grids. In Conference on High Performance Networking and Computing archive proceedings of the ACM/IEEE conference on Supercomputing (CDROM), 2001.
14. V. Pascucci and R. J. Frank. Hierarchical indexing for out-of- core ccess to multi-resolution data. In Technical Report UCRL-JC-140581, Lawrence Livermore National Laboratory, 2001.
15. V. Pascucci, D. Laney, R. J. Frank, G. Scorzelli, L. Linsen, B. Hamann, and F. Gygi. Real-time monitoring of large scientic simulations. In ACM Symposiumon Applied Computing03, ACMPress., 2003.
16. H. Shan, K. Antypas, and J. Shalf. Characterizing and predicting the I/O performance of HPC applications using a parameterized synthetic benchmark. In Proceedings of Supercomputing, November 2008.
17. B. Summa, G. Scorzelli, M. Jiang, P.-T. Bremer, and V. Pascucci. Interactive editing of massive imagery made simple: Turning atlanta into atlantis. ACM Trans. Graph., 30:7:1-7:13, April 2011.
18. R. Thakur, W. Gropp, and E. Lusk. On implementing MPI-IO portably and with high performance. In Proceedings of the Sixth Workshop on Input/Output in Parallel and Distributed Systems, pages 23-32, May 1999.
19. C. Wang, J. Gao, L. Li, and H.-W. Shen. A multiresolution volume rendering framework for large-scale time-varying data visualization. In Volume Graphics, 2005. Fourth International Workshop on, pages 11-223, june 2005.