Review-7

Review of operational suitability of HDF-EOS5:

NASA's Earth Science Data Systems Standards Process Group (SPG) is considering the HDF-EOS5 for adoption as a community standard. This is the second review of HDF-EOS5, this one focusing on its readiness for operational use. The questions below are provided to guide feedback from data systems, application providers, instrument teams and others. You only need to answer questions applicable to you. Please send comments to spg-rfc-008@lists.nasa.gov.

1. Describe in a sentence or two your overall experience related to HDF-EOS5 (e.g., science data provider, science data systems, software tools developer, and science data user, etc).

   I represent a science data system that archives and distributes data in HDF-EOS5.

2. Do you currently use or plan to use HDF-EOS5 in a production setting? What types of applications do you use with HDF-EOS5? Is HDF-EOS5 applicable to your applications (e.g., Does it work well with the data types and data manipulations in your application?)

   Yes, we do use HDF-EOS5 in a production setting. Our Giovanni online analysis totol is the main application we use with HDF-EOS5. However, we convert it to HDF4 before the analysis begins.

3. Why do you choose to use HDF-EOS5 over other data formats for your applications?

   It is a mandated EOSDIS standard, and is the data format that the data comes in from the data providers.

4. Have you or your users encountered any difficulty when using some of the data access or visualization tools (e.g., IDL, GrADS, ..) on HDF-EOS5 data files? If you have, please provide a brief description of your experience.

   Problems are encountered on files which have been SZIP compressed. But this isn't just related to HDF5/HDF-EOS5 it also applies to HDF/HDF-EOS with SZIP compression. IDL still doesn't have any HDF-EOS5 calls, you need to do HDF5 calls instead. GrADS doesn't support HDF5 at all.

5. Does the performance of HDF-EOS5 you have experienced meet your requirements? (e.g., Can it handle the data types in your applications? Does it take a long time to read and write HDF-EOS5 files?)

   Accessing HDF5 files is more efficient that old style HDF. Whenever you open an old HDF file the library seems to build a catalog of all HDF objects (an SDS consists of something like 8 separate HDF objects), with hundreds of SDSes this becomes a problem. HDF5 doesn't do that, so you can quickly pull out an array. Also, HDF5 lets you pull out arbitrary data sections, not just a slab. The problem is in how to code that.

6. What operational challenges or limitations does HDF-EOS5 present? (e.g., Does it take a long time to learn how to use it? Does it require advanced processing power, large amounts of memory, complex configuration, etc).

   By itself, it does not take a long time to learn, but when you consider that from a standing start, a user has to learn both HDF5 and HDF-EOS5, the overall learning curve has proven daunting to some users.

7. What benefits does HDF-EOS5 present? Do the benefits of HDF-EOS5 outweigh the challenges? (e.g., Does it offer the flexibility you want to package the data types in your applications? Does it facilitate interdisciplinary studies?)

   Internal compression and enhanced performance are significant benefits for a data archive like us. However, users have had difficulty with HDF-EOS5. Robust conversion tools to convert to CF-1 compliant netCDF could alleviate some of these challenges.

8. How much data do/will you provide or archive in HDF-EOS5? (number of distinct data products or data sets, total data volume, number of files.)

   We have about 40 products now, with more coming. Total volume is several TB, and number of files is over 170,000 and growing.

9. How many users do you have or expect to have for data in HDF-EOS5, and what is your expected user community?

   We expect (eventually) thousands of users, from the atmospheric chemistry and modeling communities.