An overview of the HDF5 technology suite and its applications

ACM International Conference Proceeding Series

Volumn , Issue , 2011, Pages 36-47

  Folk, Mike ^a   Heber, Gerd ^a   Koziol, Quincey ^a   Pourmal, Elena ^a   Robinson, Dana ^a  

^a HDF Group   (United States)

Author keywords

Data management; Data models; Databases; HDF5

Indexed keywords

COMPUTER PROGRAMMING;

HDF5; ITS APPLICATIONS; PERFORMANCE ENHANCING;

INFORMATION MANAGEMENT;

EID: 79956025832     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1966895.1966900     Document Type: Conference Paper

References (63)

1. http://www.hdfgroup.org/HDF5/doc/index.html.
2. HDF5 User's Guide http://www.hdfgroup.org/HDF5/doc/UG/index.html.
3. HDF5 Reference Manual http://www.hdfgroup.org/HDF5/doc/RM/RM-H5Front. html.
4. Koziol, Q. 2011. HDF5 Encyclopedia of Parallel Computing. To appear.
5. Date, C. J. and Darwen, H. 1998. Foundation for Object/Relational Databases - The Third Manifesto. Addison Wesley.
6. Thomsen, E. 2002. OLAP Solutions: Building Multidimensional Information Systems. Second Edition. Wiley.
7. Brown, P. 2001. Object-Relational Database Development. Informix Press.
8. HDF5 File Format Specification Version 2.0 http://www.hdfgroup.org/HDF5/ doc/H5.format.html.
9. Performance Analysis and Issues http://www.hdfgroup.org/HDF5/doc/H5.user/ Performance.html.
10. Universal File Interface (UFI) http://www.barrodale.com/bcs/universal- file-interface-ufi.
11. HDF5 Tools http://www.hdfgroup.org/HDF5/doc/RM/Tools.html.
12. HDFView http://www.hdfgroup.org/hdf-java-html/hdfview/.
13. MathWorks http://www.mathworks.com/.
14. Mathematica http://www.wolfram.com/.
15. VisIt https://wci.llnl.gov/codes/visit/.
16. EnSight http://www.ensight.com/.
17. HDF5 High-level APIs http://www.hdfgroup.org/HDF5/doc/HL/.
18. Gosink, L. et al. 2005. HDF5-FastQuery: Accelerating Complex Queries on HDF5 Datasets using Fast Bitmap Indices. http://crd.lbl.gov/~kewu/ps/LBNL-59602. pdf.
19. Mainzer, J. and Koziol, Q. 2010. RFC: High-Level HDF5 API routines for HPC Applications. http://www.hdfgroup.uiuc.edu/RFC/HDF5/HPC-High-Level-API/ H5HPC-RFC-2010-09-28.pdf.
20. Folk, M. and Heber, G. and Koziol, Q. 2011. HDF5 Information Set. To appear.
21. W3C. 2010. XQuery 3.0: An XML Query Language. http://www.w3.org/TR/ xquery-30/.
22. Nam, B. and Sussman, A. 2003. Improving Access to Multidimensional Self-describing Scientific Datasets. http://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.78.2998&rep=rep1&type=pdf.
23. Altet, F. and Vilata, I. 2007. OPSI: The indexing system of PyTables 2 Professional Edition. http://www.pytables.org/docs/OPSI-indexes.pdf.
24. Chan, C-Y. and Ioannidis, Y. E. 1998. Bitmap Index Design and Evaluation. SIGMOD. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.87. 1270&rep=rep1&type=pdf.
25. Lahdenmaeki, T. and Leach, M. 2004. Relational Database Index Design and the Optimizers. John Wiley & Sons, Inc.
26. Alekseyenko, A. V. and Lee, C. J. 2007. Nested Containment List (NCList): a new algorithm for accelerating interval query of genome alignment and interval databases. Bioinformatics Vol. 23, No 11, pp. 1386-1393. (Pubitemid 47050521)
27. BioHDF http://www.biohdf.org/.
28. HDF5 Image and Palette Specification. 2007. http://www.hdfgroup.org/HDF5/ doc/HL/RM-H5IM.html.
29. HDF5 Table Specification. 2002. http://www.hdfgroup.org/HDF5/doc/HL/H5TB- Spec.html.
30. HDF5 Dimension Scale Specification and Design Notes. 2005. http://www.hdfgroup.org/HDF5/doc/HL/H5DS-Spec.pdf.
31. Howison, M. et al. 2010. Tuning HDF5 for Lustre File Systems. https://secure.nersc.gov/projects/presentations/HDF5-DonofrioNERSC.pdf.
32. Introduction to the HDF5 Packet Table API. 2005. http://www.hdfgroup.org/ HDF5/doc/HL/H5PT-Intro.html.
33. Informix DataBlades. 2010. http://www-01.ibm.com/software/data/informix/ blades/.
34. Data Cartridge - Oracle Wiki. 2010. http://wiki.oracle.com/page/ Data+Cartridge.
35. Message Passing Interface. 2010. http://www.mcs.anl.gov/research/ projects/mpi/.
36. NASA's Earth Observing System . 2010. http://eospso.gsfc.nasa.gov/.
37. The ECS SDP Toolkit Home Page. 2010. http://newsroom.gsfc.nasa.gov/ sdptoolkit/TKDocuments.html.
38. NESDIS Satellite Information . 2010. http://www.nesdis.noaa.gov/ SatInformation.html.
39. NetCDF (network Common Data Form). 2010. http://www.unidata.ucar.edu/ software/netcdf/.
40. SciDB. Overview of SciDB, Large Scale Array Storage, Processing and Analysis. SIGMOD'10. http://www.scidb.org/download/sigmod691-brown.pdf.
41. Kuehn, J.A. 1996. Faster Libraries for Creating Network- Portable Self-Describing Datasets. Cray User Group.
42. XML Linking Language (XLink) Version 1.1. 2010. http://www.w3.org/TR/ xlink11/.
43. Standardizing the Next Generation of Bioinformatics Software Development with BioHDF. BioHDF BoF SC09. 2009.
44. Halpin, T. 2001. Information Modeling and Relational Databases. Morgan Kaufmann Publishers.
45. The NetCDF-4 Data Model. 2010.
46. The "Classic" NetCDF Data Model. 2010.
47. Barrodale Computing Services. 2010. http://www.barrodale.com/.
48. Aura OMI NO2 Level 3 Global (0.25 deg Grids) Data Product-OMNO2e Version 003. http://disc.sci.gsfc.nasa.gov/Aura/dataholdings/OMI/omno2e-v003.shtml.
49. NASA. OMI Data Products and Data Access. 2010. http://disc.sci.gsfc.nasa. gov/Aura/overview/dataholdings/OMI/index.shtml.
50. NASA. Aura. 2010. http://disc.sci.gsfc.nasa.gov/Aura.
51. Ramapriyan, H.K and Moses, J. 2011. NASA's Earth Science Data Systems - Lessons Learned and Future Directions. To appear.
52. HDF OPeNDAP Project. 2010. http://www.hdfgroup.org/projects/opendap/.
53. OPeNDAP: Open-source Project for a Network Data Access Protocol. 2010. http://www.opendap.org/.
54. Yang, M. and Lee, H. 2009. Using a Friendly OPeNDAP Client Library to access HDF5 data. The 89th AMS annual meeting.
55. Gallagher, J. et al. 2007. The Data Access Protocol - DAP 2.0. ESE-RFC-004.1.1.
56. Kent, W.J., Zweig, A.S., Barber, G., Hinrichs, A.S. and Karolchik, D. 2010. BigWig and BigBed: enabling browsing of large distributed datasets. Bioinformatics, 26, 2204-2207.
57. Li, H. 2011. Tabix: fast retrieval of sequence features from generic TAB-delimited files. Bioinformatics, 27, 718-719.
58. Li, H., Handsaker, B., Wysoker, A., Fennell, T., Ruan, J., Homer, N., Marth, G., Abecasis, G. and Durbin, R. 2009. The Sequence Alignment/Map format and SAMtools. Bioinformatics, 25, 2078-2079.
59. Mason, C.E., Zumbo, P., Sanders, S., Folk, M., Robinson, D., Aydt, R., Gollery, M., Welsh, M., Olson, N.E. and Smith, T.M. 2010. Standardizing the next generation of bioinformatics software development with BioHDF (HDF5). Adv Exp Med Biol, 680, 693-700.
60. Shendure, J. and Ji, H. (2008) Next-generation DNA sequencing. Nat Biotechnol, 26, 1135-1145.
61. Making Science Data Easier to Use with OPeNDAP. 2010. http://wiki.esipfed.org/index.php/Making-Science-Data-Easier-to-Use-with- OPeNDAP#EOS-HDF-Data-Readability-.28without-OPeNDAP.29.
62. Open Data Protocol (OData). 2010. http://www.odata.org/.
63. Kahn, S.D. 2011. On the future of genomic data, Science, 331, 728-729.