EDO: Improving read performance for scientific applications through elastic data organization

Proceedings - IEEE International Conference on Cluster Computing, ICCC

Volumn , Issue , 2011, Pages 93-102

  Tian, Yuan ^a   Klasky, Scott ^b   Abbasi, Hasan ^b   Lofstead, Jay ^c   Grout, Ray ^d   Podhorszki, Norbert ^b   Liu, Qing ^b   Wang, Yandong ^a   Yu, Weikuan ^a  

^a AUBURN UNIVERSITY   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

^c SANDIA NATIONAL LABORATORIES   (United States)

^d NATIONAL RENEWABLE ENERGY LABORATORY   (United States)

Author keywords

ADIOS; Data Organization; Parallel I O; Planar Read Patterns; Space Filling Curve

Indexed keywords

ADIOS; DATA ORGANIZATION; PARALLEL I/O; PLANAR READ PATTERNS; SPACE-FILLING CURVE;

CLUSTER COMPUTING;

EID: 80955123472     PISSN: 15525244     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CLUSTER.2011.18     Document Type: Conference Paper

References (47)

1. H. Abbasi, G. Eisenhauer, M. Wolf, and K. Schwan. Datastager: scalable data staging services for petascale applications. In HPDC '09, New York, NY, USA, 2009. ACM.
2. H. Abbasi, J. Lofstead, F. Zheng, K. Schwan, M. Wolf, and S. Klasky. Extending i/o through high performance data services. In Cluster Computing and Workshops, 2009. CLUSTER '09. IEEE International Conference on, pages 1 -10, 31 2009-sept. 4 2009.
3. T. Asano, D. Ranjan, T. Roos, and E. Welzl. Space filling curves and their use in the design of geometric data structures. In Lect. Notes Comput. Sc., volume 911, pages 6-44. 1995.
4. L. Chacón. A non-staggered, conservative, ▽ṡB →= 0, finite-volume scheme for 3D implicit extended magnetohydrodynamics in curvilinear geometries. Computer Physics Communications, 163:143-171, Nov. 2004.
5. C. S. Chang, S. Klasky, J. Cummings, R. Samtaney, A. Shoshani, L. Sugiyama, D. Keyes, S. Ku, G. Park, S. Parker, N. Podhorszki, H. Strauss, H. Abbasi, M. Adams, R. Barreto, G. Bateman, K. Bennett, Y. Chen, E. D. Azevedo, C. Docan, S. Ethier, E. Feibush, L. Greengard, T. Hahm, F. Hinton, C. Jin, A. Khan, A. Kritz, P. Krsti, T. Lao, W. Lee, Z. Lin, J. Lofstead, P. Mouallem, M. Nagappan, A. Pankin, M. Parashar, M. Pindzola, C. Reinhold, D. Schultz, K. Schwan, D. Silver, A. Sim, D. Stotler, M. Vouk, M. Wolf, H. Weitzner, P. Worley, Y. Xiao, E. Yoon, and D. Zorin. Toward a first-principles integrated simulation of tokamak edge plasmas - art. no. 012042. Scidac 2008: Scientific Discovery through Advanced Computing, 125:12042-12042, 2008.
6. J. H. Chen et al. Terascale direct numerical simulations of turbulent combustion using S3D. Comp. Sci. & Disc., 2(1):015001 (31pp), 2009.
7. A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, and S. Tuecke. The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets. J. Netw. Comput. Appl., 23:187-200, 1999.
8. H. Childs. Architectural challenges and solutions for petascale postprocessing. J. Phys., 78(012012), 2007.
9. Cluster File System, Inc. Lustre: A Scalable, High Performance File System. http://www.lustre.org/docs.html.
10. Y. Cui, K. B. Olsen, T. H. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D. K. Panda, A. Chourasia, J. Levesque, S. M. Day, and P. Maechling. Scalable earthquake simulation on petascale supercomputers. In Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis, SC '10, pages 1-20, Washington, DC, USA, 2010. IEEE Computer Society.
11. C. H. Q. Ding and Y. He. Data organization and I/O in a parallel ocean circulation model. In Proc. SC99. Society Press, 1999.
12. C. Docan, F. Zhang, M. Parashar, J. Cummings, N. Podhorszki, and S. Klasky. Experiments with memory-to-memory coupling for end-to-end fusion simulation workflows. Cluster Computing and the Grid, IEEE International Symposium on, 0:293-301, 2010.
13. C. Fan, A. Gupta, and J. Liu. Latin cubes and parallel array access. In Parallel Processing Symposium, 1994. Proceedings., Eighth International, pages 128 -132, apr 1994.
14. G. Ghinita, P. Kalnis, and S. Skiadopoulos. Prive: anonymous location-based queries in distributed mobile systems. In Proceedings of the 16th international conference on World Wide Web, WWW '07, pages 371-380, New York, NY, USA, 2007. ACM.
15. J. Gray et al. The recovery manager of the system R database manager. ACM Comput. Surv., 13(2):223-242, 1981.
16. R. W. Grout, A. Gruber, C. Yoo, and J. Chen. Direct numerical simulation of flame stabilization downstream of a transverse fuel jet in cross-flow. P. Combust. Inst., 2010. In press.
17. D. Hilbert. Ueber die stetige abbildung einer line auf ein flächenstück. Math. Ann., 38:459-460, 1891.
18. Y. Hu, A. Cox, and W. Zwaenepoel. Improving fine-grained irregular shared-memory benchmarks by data reordering. In Proc. SC00, pages 33-33, Nov. 2000.
19. H. V. Jagadish. Linear clustering of objects with multiple attributes. SIGMOD Rec., 19(2):332-342, 1990.
20. S. Klasky, S. Ethier, Z. Lin, K. Martins, D. McCune, and R. Samtaney. Grid -based parallel data streaming implemented for the gyrokinetic toroidal code. In SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing, page 24, Washington, DC, USA, 2003. IEEE Computer Society.
21. S. Kuo, M. Winslett, Y. Cho, J. Lee, and Y. Chen. Efficient input and output for scientific simulations. In In Proceedings of I/O in Parallel and Distributed Systems (IOPADS), pages 33-44. ACM Press, 1999.
22. E. Laure, H. Stockinger, and K. Stockinger. Performance engineering in data grids. Concurr. Comp-Pract. E., 17:4-171, 2005.
23. J. K. Lawder and P. J. H. King. Querying multi-dimensional data indexed using the hilbert space-filling curve. SIGMOD Rec., 30(1):19-24, 2001.
24. J. Li et al. Parallel netCDF: A high-performance scientific I/O interface. In Proc. SC03. ACM, 2003.
25. J. Lofstead et al. Managing variability in the io performance of petascale storage system. In Proc. SC10, New York, NY, USA, 2010. IEEE Press.
26. J. Lofstead, M. Polte, G. Gibson, S. Klasky, K. Schwan, R. Oldfield, and M. Wolf. Six degrees of scientific data: Reading patterns for extreme scale science io. In Proceedings of the 20th International ACM Symposium on High-Performance Parallel and Distributed Computing, to appear, HPDC'11. ACM, 2011.
27. J. Lofstead, F. Zheng, S. Klasky, and K. Schwan. Adaptable, metadata rich IO methods for portable high performance IO. Parallel and Distributed Processing Symposium, International, 0:1-10, 2009.
28. The HDF Group. Hierarchical data format version 5, 2000-2010. http://www.hdfgroup.org/HDF5.
29. O. E. B. Messer, S. W. Bruenn, J. M. Blondin, W. R. Hix, A. Mezzacappa, and C. J. Dirk. Petascale supernova simulation with chimera. Journal of Physics: Conference Series, 78, July, 2007.
30. B. Moon, H. Jagadish, C. Faloutsos, and J. Saltz. Analysis of the clustering properties of the hilbert space-filling curve. IEEE T. Knowl. Data En., 13(1):124-141, 2001. (Pubitemid 32301048)
31. NCCS. http://www.nccs.gov/computing-resources/jaguar/.
32. NetCDF. http://www.unidata.ucar.edu/software/netcdf/.
33. R. Niedermeier and P. Sanders. On the manhattan-distance between points on space-filling mesh-indexings. Technical report, 1996.
34. R. Oldfield, P. Widener, A. Maccabe, L. Ward, and T. Kordenbrock. Efficient data-movement for lightweight i/o. Cluster Computing, IEEE International Conference on, 0:1-9, 2006.
35. V. Pascucci and R. J. Frank. Global static indexing for real-time exploration of very large regular grids. In Proc. SC01, pages 45-45, Nov. 2001.
36. M. Polte et al. ...and eat it too: High read performance in write-optimized HPC I/O middleware file formats. In In Proceedings of Petascale Data Storage Workshop 2009 at Supercomputing 2009, 2009.
37. F. Pretorius. Evolution of Binary Black Hole Spacetimes. Phys. Rev. Lett., 95:121101, 2005.
38. J. A. Rice. Hilbert R-tree: An improved R-tree using fractals. Duxbury Press, Ibrahim Kamel, Christos Faloutsos, 1995.
39. M. Rosenblum and J. K. Ousterhout. The design and implementation of a log-Structured file system. ACM T. Comput. Syst., 10:1-15, 1991.
40. S. Sarawagi and M. Stonebraker. Efficient organization of large multidimensional arrays. In Proc. 10th Int. Conf. on Data Eng., pages 328-336, Houston, TX, 1994.
41. M. Seltzer, K. Bostic, M. K. Mckusick, and C. Staelin. An implementation of a log-Structured file system for UNIX. In USENIX'93, pages 3-3, Berkeley, CA, USA, 1993. USENIX Association.
42. T. Tu et al. Accelerating parallel analysis of scientific simulation data via zazen. In FAST'10, pages 129-142. USENIX Association, 2010.
43. Unidata. http://www.hdfgroup.org/projects/netcdf-4/.
44. R. Y. Wang, T. E. Anderson, and D. A. Patterson. Virtual log based file systems for a programmable disk. In OSDI '99: Proceedings of the third symposium on Operating systems design and implementation, pages 29-43, Berkeley, CA, USA, 1999. USENIX Association.
45. W. X. Wang and et al. Gyro-kinetic simulation of global turbulent transport properties in Tokamak experiments. Physics of Plasmas, 13(9):092505, 2006.
46. W. Yu, J. Vetter, and H. Oral. Performance characterization and optimization of parallel I/O on the cray XT. IPDPS, pages 1-11, April 2008.
47. F. Zheng et al. Predata - preparatory data analytics on peta-scale machines. In IPDPS, Atlanta, GA, April 2010.