Review-7

Review of HDF5 operational readiness:

NASA's Earth Science Data Systems Standards Process Group (SPG) is considering the HDF5 for adoption as a community standard. This is the second review of HDF5, this one focusing on its readiness for operational use. The questions below are provided to guide feedback from data systems, application providers, instrument teams and others. You only need to answer questions applicable to you. Please send comments to ese–rfc–007@spg.gsfc.nasa.gov.

1. Describe in a sentence or two your overall experience related to HDF5 (e.g., science data provider, science data systems, software tools developer, and science data user, etc).

   I am HDF-EOS developer and have used HDF5 in HDF-EOS5 and related tools development such as HDFView's HDF-EOS plug-in, NCEP binary data to hdf converter, etc.

2. Do you currently use or plan to use HDF5 in a production setting? What types of applications do you use with HDF5? Is HDF5 applicable to your applications (e.g., Does it work well with the data types and data manipulations in your application?)

   HDF-EOS5 which is an extension of HDF5 is currently used by EOS AURA instrument teams on a production setting. HDF-EOS5 creates Grid, Swath, Point, Profile, and Zonal Average objects using HDF5. The Aura instrument teams, HDF-EOS5 users or tool developers, use these objects for the satellite data presentation.

3. Why do you choose to use HDF5 over other data formats for your applications?

   Because of its potential in developing EOS objects, i.e. Grid, Swath, Point, and Profile, NASA selected HDF5 as the underlying format for HDF-EOS5 for the AURA mission. We were required to extend HDF4 based HDF-EOS to cover HDF5 and satisfy the development needs of AURA instrument and science teams.

4. Have you or your users encountered any difficulty when using some of the data access or visualization tools (e.g., IDL, GrADS, ..) on HDF-5 data files? If you have, please provide a brief description of your experience.

   Our users usually encounter problems with such tools, since they lack HDF-EOS5 implementation, or if implemented the HDF-EOS5 version in the tools is too old for the data files that are created with the newer versions with enhanced features.

5. Does the performance of HDF5 you have experienced meet your requirements? (e.g., Can it handle the data types in your applications? Does it take a long time to read and write HDF5 files?)

   HDF5 can handle all the data types that are required for the HDF-EOS5 applications. The reading or writing time for HDF-EOS5 files is reasonable.

6. What operational challenges or limitations does HDF5 present? (e.g., Does it take a long time to learn how to use it? Does it require advanced processing power, large amounts of memory, complex configuration, etc)

   HDF5 is a complex package and requires some familiarity with its specifications and concepts before writing HDF5 based libraries or applications. The package offers a wide variety of routines that can be used to create complex objects for probably any application. It has the required balance of routines to develop codes for handling different objects and datasets in HDF-EOS5.

7. What benefits does HDF5 present? Do the benefits of HDF5 outweigh the challenges? (e.g., Does it offer the flexibility you want to package the data types in your applications? Does it facilitate interdisciplinary studies?)

   Unlike simple binary data files the HDF5 data files are independent of the platform in which they have been produced, making it easy to share data with other users that want to read data in different platform, whether in Big-Endian or Little-Endian architecture. HDF5 specifications meet the EOS needs in creation and manipulation of needed EOS objects and datasets. In particular

   - Data users read only the data that they need, not the whole file. Data producers can put images, tables, multidimensional arrays, etc into the same file

   - Users do not need to be concerned with the platform in which the data is produced

   - Its limited primary structures, i.e. groups and datasets, makes the file design simple

   - Ample metadata can be added to the file, groups and dataset, making the file self describing.

   - Data files can be internally compressed using different schemes making better data storage and usage.

   The only challenge with HDF5 usage is its complexity. It is hard to use by those who do not want to spend too much time on reading User's Guides and then attempting to write codes for reading data from HDF data files. One needs to spend some time to learn HDF5 in order to be able to write reasonable codes for reading/writing data from/to HDF files. This seems to be too much work for those who are used to extracting data from simple binary or ASCII files.

8. How much data do/will you provide or archive in HDF5? (number of distinct data products or data sets, total data volume, number of files.)

   We will not provide or archive any data, but HDF-EOS5 data producers (AURA instrument teams) will provide and archive huge amount of data.

9. How many users do you have or expect to have for data in HDF5, and what is your expected user community?

   Our user community is the EOS AURA instrument and science teams as well as individual AURA data users and tool developers.