ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems
ExaHDF5 is led by Suren Byna, staff scientist in CRD’s Scientific Data Management Group. Hierarchical Data Format version 5 (HDF5) is the most popular high-level I/O library for scientific applications to write and read data files. The HDF Group released the first version of HDF5 in 1998 and since then it has been used by numerous applications, not only in a wide range of scientific domains, but also in finance, aerospace and more. HDF5 is the most-used library for performing parallel I/O on existing HPC systems at the Department of Energy’s leadership computing facilities.
NASA gives HDF5 software the highest technology readiness level (TRL 9), which is given to actual systems “flight proven” through successful mission operations. In the ECP project, the team is developing various parallel HDF5 features to address efficiency, fault-tolerance,and other challenges posed by data management and parallel I/O on exascale architectures. Many of the funded exascale applications and co-design centers require HDF5 for their I/O, and enhancing the HDF5 software to handle the unique challenges of exascale architectures will play an instrumental role in the success of the ECP.