LCI2009-Tutorial
Ruth Aydt
2009 LCI Conference Tutorial HDF5 and netCDF ‐4: Two Solutions for Data Management Problems Based on One File Format Elena Pourmal, Albert Cheng, Ruth Aydt (The HDF Group) Ed Hartnett (Unidata/UCAR) Overview Over the past 20 years, the HDF ( www.hdfgroup.org ) and netCDF ( www.unidata.ucar.edu ) file formats have become de facto standards for storing, managing, and exchanging data in science and engineering communities. P etabytes of data have been written in both formats and used in many endeavors, including climate change modeling, weather prediction, nuclear fusion simulation, non ‐destructive material testing, bioinformatics, and high ‐resolution imaging. Advances in hi gh‐performance computing have made it possible to model and study very complex phenomena in a wide range of scientific fields while producing, accessing, and analyzing gigabytes of complex and diverse data. Efficient data management, including seamless dat a interoperability, is a critical part of the scientific discovery process that presents new challenges to the users and maintainers of scientific data formats. This tutorial introduces HDF5 and netCDF ‐4, a new version of netCDF built on top of HDF5. HDF5 and netCDF ‐4 were created to address data management needs in today’s heterogeneous and quickly evolving high ‐performance computational environments ...