MLOC: Multi-level layout optimization framework for compressed scientific data exploration with heterogeneous access patterns

Proceedings of the International Conference on Parallel Processing

Volumn , Issue , 2012, Pages 239-248

  Gong, Zhenhuan ^a,b   Rogers, Terry ^a,b   Jenkins, John ^a,b   Kolla, Hemanth ^c   Ethier, Stephane ^d   Chen, Jackie ^c   Ross, Robert ^e   Klasky, Scott ^b   Samatova, Nagiza F ^a,b  

^a NORTH CAROLINA STATE UNIVERSITY   (United States)

^b OAK RIDGE NATIONAL LABORATORY   (United States)

^c SANDIA NATIONAL LABORATORIES   (United States)

^d PRINCETON PLASMA PHYSICS LABORATORY   (United States)

^e ARGONNE NATIONAL LABORATORY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ACCESS PATTERNS; DATA ANALYTICS; DATA EXPLORATION; DATA LAYOUT OPTIMIZATION; DATA PARTITIONING; DATA SETS; DATA SIZE; EXTREME SCALE; HETEROGENEOUS ACCESS; I/O SYSTEMS; INTEGRAL PART; LAYOUT OPTIMIZATION; MULTIRESOLUTION DATA; PARALLEL FILE SYSTEM; REGIONS OF INTEREST; RUNTIME ENVIRONMENTS; SCIENTIFIC DATA; SCIENTIFIC DATABASE; SCIENTIFIC SIMULATIONS; SIMULATION DATA; SPATIO-TEMPORAL; SPATIO-TEMPORAL DATA; STORAGE CAPABILITY; STORAGE SYSTEMS;

DATA REDUCTION; FILE ORGANIZATION; OPTIMIZATION; SEARCH ENGINES;

DIGITAL STORAGE;

EID: 84871133248     PISSN: 01903918     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICPP.2012.39     Document Type: Conference Paper

References (31)

1. W. X. Wang and et al, "Gyro-kinetic simulation of global turbulent transport properties in Tokamak experiments," Physics of Plasmas, vol. 13, no. 9, p. 092505, 2006.
2. J.H. Chen, A. Choudhary, and et al, "Terascale direct numerical simulations of turbulent combustion using S3D," Computational Science & Discovery, vol. 2, no. 1, p. 015001, 2009.
3. F. Schmuck and R. Haskin, "GPFS: A shared-disk file system for large computing clusters," in First USENIX Conference on File and Storage Technologies (FAST'02), Monterey, CA, Jan. 28-30 2002.
4. S. Donovan, G. H. Ibm, and et al, "Lustre: Building a file system for 1,000-node clusters," in Proceedings of the Linux Symposium, 2003.
5. P. H. Carns, W. B. Ligon, III, and et al, "PVFS: a parallel file system for linux clusters," in Proceedings of the 4th annual Linux Showcase & Conference - Volume 4, Berkeley, CA, USA, 2000, pp. 28-28.
6. J. F. Lofstead, S. Klasky, and et al, "Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)," in Proceedings of the 6th International Workshop on Challenges of Large Applications in Distributed Environments, 2008, pp. 15-24.
7. "HDF5." [Online]. Available: http://www.hdfgroup.org/HDF5/
8. J. Li, W. keng Liao, and et al, "Parallel netCDF: A high-performance scientific I/O interface," in Proceedings of Supercomputing, Nov. 2003.
9. H. Abbasi, M. Wolf, and et al, "DataStager: scalable data staging services for petascale applications," in HPDC '09. New York, NY, USA: ACM, 2009, pp. 39-48.
10. F. Zheng, H. Abbasi, and et al, "PreDatA-preparatory data analytics on peta-scale machines," in IPDPS, Atlanta, GA, April 2010.
11. H. Sagan, Space-Filling Curves. Springer-Verlag, New York, NY.
12. J. K. Lawder and P. King, "Querying multi-dimensional data indexed using the Hilbert space-filling curve," SIGMOD Record, vol. 30, 2001.
13. V. Pascucci, "Global static indexing for real-time exploration of very large regular grids." ACM Press, 2001.
14. K. Wu, "FastBit: An efficient indexing technology for accelerating data-intensive science," Journal of Physics: Conference Series, vol. 16, no. 1, p. 556, 2005.
15. S. Lakshminarasimhan, J. Jenkins, and et al, "ISABELA-QA: Query-driven data analytics over ISABELA-compressed scientific data," in International Conference for High Performance Computing, Networking, Storage and Analysis (SC 2011), Seattle, Washington, 2011.
16. S. Lakshminarasimhan, N. Shah, and et al, "Compressing the incompressible with ISABELA: In-situ reduction of spatio-temporal data," in 17th International European Conference on Parallel and Distributed Computing (Euro-Par 2011), Bordeaux, France, 8 2011.
17. B. Moon, H. V. Jagadish, and et al, "Analysis of the clustering properties of the Hilbert space-filling curve," IEEE Transactions on Knowledge and Data Engineering, vol. 13, p. 2001, 2001.
18. P. Lindstrom and M. Isenburg, "Fast and efficient compression of floating-point data," IEEE Transactions on Visualization and Computer Graphics, vol. 12, pp. 1245-1250, 2006.
19. E. R. Schendel, Y. Jin, and et al, "ISOBAR preconditioner for effective and high-throughput lossless data compression," International Conference on Data Engineering (ICDE), 2012.
20. W.-k. Liao and A. Choudhary, "Dynamically adapting file domain partitioning methods for collective I/O based on underlying parallel file system locking protocols," in Proceedings of the ACM/IEEE conference on Supercomputing (SC), Piscataway, NJ, USA, 2008, pp. 3:1-3:12.
21. S. Lang, P. Carns, and et al, "I/O performance challenges at leadership scale," in Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis, ser. SC '09. New York, NY, USA: ACM, 2009, pp. 40:1-40:12.
22. K. Gao, W.-k. Liao, and et al, "Using subfiling to improve programming flexibility and performance of parallel shared-file I/O," in Proceedings of the 2009 International Conference on Parallel Processing, ser. ICPP '09. Washington, DC, USA: IEEE Computer Society, 2009, pp. 470-477.
23. W. Yu, J. Vetter, and et al, "Exploiting Lustre file joining for effective collective IO," in Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, ser. CCGRID '07. Washington, DC, USA: IEEE Computer Society, 2007, pp. 267-274.
24. X. Zhang, S. Jiang, and K. Davis, "Making resonance a common case: A high-performance implementation of collective I/O on parallel file systems," in Proceedings of the IEEE International Symposium on Parallel&Distributed Processing, Washington, DC, USA, 2009, pp. 1-12.
25. W. Gropp, E. Lusk, and R. Thakur, Using MPI-2: Advanced Features of the Message-Passing Interface. MIT Press, Cambridge, MA, 1999.
26. E. Soroush, M. Balazinska, and D. Wang, "ArrayStore: A storage manager for complex parallel array processing," in Proceedings of the 2011 International Conference on Management of Data, ser. SIGMOD '11. New York, NY, USA: ACM, 2010.
27. Y. Tian, S. Klasky, and et al, "EDO: Improving read performance for scientific applications through elastic data organization." in Proceedings of the 14th International Conference on Scientific and Statistical Database Management, ser. Cluster '11. Austin, TX, USA: IEEE, 2011.
28. P. G. Brown, "Overview of SciDB: Large scale array storage, processing and analysis," in Proceedings of the 2010 International Conference on Management of Data, ser. SIGMOD '10. New York, NY, USA: ACM, 2010, pp. 963-968.
29. M. Bhadkamkar, J. Guerra, and et al, "BORG: block-reORGanization for self-optimizing storage systems," in Proccedings of the 7th conference on File and storage technologies. Berkeley, CA, USA: USENIX Association, 2009, pp. 183-196.
30. W. keng Liao, A. Ching, and et al, "An implementation and evaluation of client-side file caching for MPI-IO," in Proceedings of the 21st International Parallel and Distributed Processing Symposium (IPDPS), Long Beach, California, March 2007.
31. P. Carns, S. Lang, and et al, "Small-file access in parallel file systems," in Proceedings of the 2009 IEEE International Symposium on Parallel& Distributed Processing. Washington, DC, USA: IEEE Computer Society, 2009, pp. 1-11.