Hierarchical Data Format (HDF) is a set of

file format A file format is a Computer standard, standard way that information is encoded for storage in a computer file. It specifies how bits are used to encode information in a digital storage medium. File formats may be either proprietary format, pr ...

s (HDF4, HDF5) designed to store and organize large amounts of data. Originally developed at the U.S.

National Center for Supercomputing Applications The National Center for Supercomputing Applications (NCSA) is a unit of the University of Illinois Urbana-Champaign, and provides high-performance computing resources to researchers in the United States. NCSA is currently led by Professor Bill ...

, it is supported by The HDF Group, a non-profit corporation whose mission is to ensure continued development of HDF5 technologies and the continued accessibility of data stored in HDF. In keeping with this goal, the HDF libraries and associated tools are available under a liberal, BSD-like license for general use. HDF is supported by many commercial and non-commercial software platforms and programming languages. The freely available HDF distribution consists of the library, command-line utilities, test suite source, Java interface, and the Java-based HDF Viewer (HDFView). The current version, HDF5, differs significantly in design and

API An application programming interface (API) is a connection between computers or between computer programs. It is a type of software interface, offering a service to other pieces of software. A document or standard that describes how to build ...

from the major legacy version HDF4.

Early history

The quest for a portable scientific data format, originally dubbed AEHOO (All Encompassing Hierarchical Object Oriented format) began in 1987 by the Graphics Foundations Task Force (GFTF) at the National Center for Supercomputing Applications (NCSA). NSF grants received in 1990 and 1992 were important to the project. Around this time

NASA The National Aeronautics and Space Administration (NASA ) is an independent agencies of the United States government, independent agency of the federal government of the United States, US federal government responsible for the United States ...

investigated 15 different file formats for use in the

Earth Observing System The Earth Observing System (EOS) is a program of NASA comprising a series of artificial satellite missions and scientific instruments in Earth orbit designed for long-term global observations of the land surface, biosphere, earth's atmosphere, at ...

(EOS) project. After a two-year review process, HDF was selected as the standard data and information system.

HDF4

HDF4 is the older version of the format, although still actively supported by The HDF Group. It supports a proliferation of different data models, including multidimensional arrays, raster images, and tables. Each defines a specific aggregate data type and provides an

for reading, writing, and organizing the data and metadata. New data models can be added by the HDF developers or users. HDF is self-describing, allowing an application to interpret the structure and contents of a file with no outside information. One HDF file can hold a mix of related objects which can be accessed as a group or as individual objects. Users can create their own grouping structures called "vgroups." The HDF4 format has many limitations. It lacks a clear object model, which makes continued support and improvement difficult. Supporting many different interface styles (images, tables, arrays) leads to a complex API. Support for metadata depends on which interface is in use; ''SD'' (Scientific Dataset) objects support arbitrary named attributes, while other types only support predefined metadata. Perhaps most importantly, the use of 32-bit signed integers for addressing limits HDF4 files to a maximum of 2 GB, which is unacceptable in many modern scientific applications.

HDF5

The HDF5 format is designed to address some of the limitations of the HDF4 library, and to address current and anticipated requirements of modern systems and applications. In 2002 it won an R&D 100 Award. HDF5 simplifies the file structure to include only two major types of object: *Datasets, which are typed multidimensional arrays *Groups, which are container structures that can hold datasets and other groups This results in a truly hierarchical, filesystem-like data format. In fact, resources in an HDF5 file can be accessed using the

POSIX The Portable Operating System Interface (POSIX; ) is a family of standards specified by the IEEE Computer Society for maintaining compatibility between operating systems. POSIX defines application programming interfaces (APIs), along with comm ...

-like syntax ''/path/to/resource''. Metadata is stored in the form of user-defined, named attributes attached to groups and datasets. More complex storage APIs representing images and tables can then be built up using datasets, groups and attributes. In addition to these advances in the file format, HDF5 includes an improved type system, and dataspace objects which represent selections over dataset regions. The API is also object-oriented with respect to datasets, groups, attributes, types, dataspaces and property lists. The latest version of

NetCDF NetCDF (Network Common Data Form) is a set of software libraries and self-describing, machine-independent data formats that support the creation, access, and sharing of array-oriented scientific data. The project homepage is hosted by the Unidat ...

, version 4, is based on HDF5. Because it uses B-trees to index table objects, HDF5 works well for

time series In mathematics, a time series is a series of data points indexed (or listed or graphed) in time order. Most commonly, a time series is a sequence taken at successive equally spaced points in time. Thus it is a sequence of discrete-time data. ...

data such as stock price series, network monitoring data, and 3D meteorological data. The bulk of the data goes into straightforward arrays (the table objects) that can be accessed much more quickly than the rows of an

SQL Structured Query Language (SQL) (pronounced ''S-Q-L''; or alternatively as "sequel") is a domain-specific language used to manage data, especially in a relational database management system (RDBMS). It is particularly useful in handling s ...

database, but B-tree access is available for non-array data. The HDF5 data storage mechanism can be simpler and faster than an SQL

star schema In computing, the star schema or star model is the simplest style of data mart Logical schema, schema and is the approach most widely used to develop data warehouses and dimensional data marts. The star schema consists of one or more fact tables ...

Feedback

Criticism of HDF5 follows from its monolithic design and lengthy specification. *HDF5 does not enforce the use of

UTF-8 UTF-8 is a character encoding standard used for electronic communication. Defined by the Unicode Standard, the name is derived from ''Unicode Transformation Format 8-bit''. Almost every webpage is transmitted as UTF-8. UTF-8 supports all 1,112,0 ...

, so client applications may be expecting ASCII in most places. *Dataset data cannot be freed in a file without generating a file copy using an external tool (h5repack).

Officially supported APIs

* C * C++ *

CLI CLI may refer to: Computing * Call Level Interface, an SQL database management API * Command-line interface, of a computer program * Command-line interpreter or command language interpreter; see List of command-line interpreters * CLI (x86 instruc ...

- .NET * Fortran, Fortran 90 * HDF5 Lite (H5LT) – a light-weight interface for C * HDF5 Image (H5IM) – a C interface for images or rasters * HDF5 Table (H5TB) – a C interface for tables * HDF5 Packet Table (H5PT) – interfaces for C and C++ to handle "packet" data, accessed at high-speeds * HDF5 Dimension Scale (H5DS) – allows dimension scales to be added to HDF5 *

Java Java is one of the Greater Sunda Islands in Indonesia. It is bordered by the Indian Ocean to the south and the Java Sea (a part of Pacific Ocean) to the north. With a population of 156.9 million people (including Madura) in mid 2024, proje ...

References

External links

*{{Official website
What is HDF5?HDF-EOS Tools and Information CenterOpen Navigation Surface
C (programming language) libraries Computer file formats Earth sciences data formats Meteorological data and networks