Review-1

Review of HDF5 operational readiness:

NASA's Earth Science Data Systems Standards Process Group (SPG) is considering the HDF5 for adoption as a community standard. This is the second review of HDF5, this one focusing on its readiness for operational use. The questions below are provided to guide feedback from data systems, application providers, instrument teams and others. You only need to answer questions applicable to you. Please send comments to ese-rfc-007@spg.gsfc.nasa.gov.

1. Describe in a sentence or two your overall experience related to HDF5

   We, are a NASA Mission team, are responsible for all aspects of our data products: software, data products, and tools for users to access our data.

2. Do you currently use or plan to use HDF5 in a production setting? What types of applications do you use with HDF5? Is HDF5 applicable to your applications

   We are using HDF5 regularly in our production. All our products are delivered in HDF5 format.

3. Why do you choose to use HDF5 over other data formats for your applications

   All instrument teams on Aura agreed to a set of common formats to help users access our data with a single set of tools rather than require them to learn many different sets. HDF5 was chosen because we believed it was the version of HDF most likely to be around the longest.

4. Have you or your users encountered any difficulty when using some of the data access or visualization tools (e.g., IDL, GrADS, ..) on HDF-5 data files? If you have, please provide a brief description of your experience.

   Yes, after the release of hdf5.1.6 idl and idl-based tools were unable to read data files created using software built with the new hdf5 until idl released a patch.

5. Does the performance of HDF5 you have experienced meet your requirements? (e.g., Can it handle the data types in your applications? Does it take a long time to read and write HDF5 files?)

   It does take a long time, but not unduly so-our files are quite large.

6. What operational challenges or limitations does HDF5 present? (e.g., Does it take a long time to learn how to use it? Does it require advanced processing power, large amounts of memory, complex configuration, etc)

   HDF5 takes a very long time to learn how to use. Fortunately, we and other data providers supply tools to let users access our data without requiring them to learn its intricacies.

7. What benefits does HDF5 present? Do the benefits of HDF5 outweigh the challenges? (e.g., Does it offer the flexibility you want to package the data types in your applications? Does it facilitate interdisciplinary studies?)

   I am unable to answer this because I have no other formats to compare HDF5 with.

8. How much data do/will you provide or archive in HDF5? (number of distinct data products or data sets, total data volume, number of files.)

   We will provide all users approximately 25 files/day every day over the life of the mission, which will be at least 5 years. The file volumes are approximately 4GB/day. Thus, we will have at least 45000 files and a total archived volume of at least 7.3TB. This does not account for multiple versions or diagnostic products.

9. How many users do you have or expect to have for data in HDF5, and what is your expected user community?

   168 users have registered so far through our website. While it is not possible to estimate how many unregistered users there might be, the number of product orders so far has been 2336.