Review-8

Review of operational suitability of HDF-EOS5:

NASA's Earth Science Data Systems Standards Process Group (SPG) is considering the HDF-EOS5 for adoption as a community standard. This is the second review of HDF-EOS5, this one focusing on its readiness for operational use. The questions below are provided to guide feedback from data systems, application providers, instrument teams and others. You only need to answer questions applicable to you. Please send comments to spg-rfc-008@lists.nasa.gov.

1. Describe in a sentence or two your overall experience related to HDF-EOS5 (e.g., science data provider, science data systems, software tools developer, and science data user, etc).

   We in the MLS Team are responsible for the final data products of the MLS instrument on the EOS-Aura spacecraft. Along with the other instrument teams we first agreed upon a set of Guidelines based on HDF-EOS5 and further restricting data field names, attributes, units, etc. Then we wrote software based on the HDF5/HDF-EOS5 libraries adhering to these standards. We also perform and publish scientific analysis of these data. We fill each of the roles you list: science data provider, science data system, software tools developer, and science data user.

2. Do you currently use or plan to use HDF-EOS5 in a production setting? What types of applications do you use with HDF-EOS5? Is HDF-EOS5 applicable to your applications (e.g., Does it work well with the data types and data manipulations in your application?)

   Our production software and suite of tools were built to work with HDF-EOS5. We use HDF-EOS5 as the standard format for our own products. We use IDL extensively and created an IDL-based reader for scientists based elsewhere to read our data.

3. Why do you choose to use HDF-EOS5 over other data formats for your applications?

   All four instrument teams agreed on a set of Guidelines for standard scientific products. HDF-EOS5 contains data types very useful for atmospheric science data. This is by design. Another benefit is that tools built for pure HDF5 files can read Hdf-EOS5 files successfully, too.

4. Have you or your users encountered any difficulty when using some of the data access or visualization tools (e.g., IDL, GrADS, ..) on HDF-EOS5 data files? If you have, please provide a brief description of your experience.

   Older versions of IDL are unable to read our current products, which require IDL version 6.1 or later. This was due to an upgrade in HDF5 which was not backward compatible.

5. Does the performance of HDF-EOS5 you have experienced meet your requirements? (e.g., Can it handle the data types in your applications? Does it take a long time to read and write HDF-EOS5 files?)

   HDF-EOS5 was designed partly to incorporate our data types. The performance depends on the tools, in most cases posing no obstacle to reading or writing files.

6. What operational challenges or limitations does HDF-EOS5 present? (e.g., Does it take a long time to learn how to use it? Does it require advanced processing power, large amounts of memory, complex configuration, etc).

   A small number of users wished to avoid purchasing IDL, and the alternative tools are much harder to obtain and use. One user wished to avoid HDF altogether-we directed that user to the Giovanni online data visualization web site http://acdisc.sci.gsfc.nasa.gov/Giovanni/mls/

7. What benefits does HDF-EOS5 present? Do the benefits of HDF-EOS5 outweigh the challenges? (e.g., Does it offer the flexibility you want to package the data types in your applications? Does it facilitate interdisciplinary studies?)

   It is much less flexible than a lower-level format would be, but for that reason it was easier to design a common format shared across different teams. We intended that users may read standard products from four different instruments with a single tool instead of learning and deploying four different tools. That was to be the major overarching benefit. Unfortunately, this has not always been the case. We have heard complaints from users annoyed that a single HDF reader was not available to read the files from all 4 instrument teams.

8. How much data do/will you provide or archive in HDF-EOS5? (number of distinct data products or data sets, total data volume, number of files.)

   For level 2 we have 18 standard products and 1 diagnostic product, each product in its own file. These will be provided for most days over an expected 5 year mission. These 19 files normally fill 158 MB of disk space. Therefore, assuming these figures don't change 1825 days 34675 files 288 GB

9. How many users do you have or expect to have for data in HDF-EOS5, and what is your expected user community?

   We currently have more than 190 registered users. The number of product orders is more than 2300. Both numbers increase every week. Our users, who have cited MLS data in more than nnn papers, come from more than mm countries.