HDF5NDSource

HDF5NDSource exposes an HDF5 dataset through ProxyNDSource using kerchunk metadata pre-indexing. Individual chunks are fetched on demand and converted to Blosc2-compressed chunks stored in the surrounding Proxy or RemoteArray cache.

The source is assumed immutable (assume_immutable=True). It supports fixed-size boolean, integer, floating-point, complex, and fixed-length string arrays. HDF5 filters such as Blosc2 (via hdf5plugin), gzip, and uncompressed datasets are supported.

Install local support with pip install "blosc2[hdf5]". Remote datasets also need blosc2[fsspec] and the protocol driver, such as s3fs for S3.

blosc2.available_datasets(url, storage_options: dict | None = None) → list[str]

Return all dataset paths within an HDF5 file or reference dictionary.

Parameters:

• url (str, os.PathLike, or dict) – Path or URL to an HDF5 file, a JSON reference file, or an in-memory kerchunk reference dictionary.

• storage_options (dict, optional) – Options passed to fsspec or kerchunk for remote URLs.

Returns:

Sorted list of dataset paths (e.g. ['d0/a0', 'd0/d1/a2']).

Return type:

list[str]

class blosc2.HDF5NDSource(urlpath, dataset: str, *, refs: dict | str | PathLike | None = None, storage_options: dict | None = None, max_concurrency: int = 8, blocks=None, cparams=None, _traffic: Traffic | None = None, _filesystem=None)

Read an immutable HDF5 dataset as Blosc2-compressed logical chunks via kerchunk.

Replacing data beneath the same store identity violates this adapter’s contract and may leave previously converted chunks stale. Peak working memory includes concurrently decoded Zarr chunks and their Blosc2 conversion buffers; max_cache_bytes only limits retained compressed chunks.

Parameters:

• urlpath (str or path-like) – URL or file path to the HDF5 file.

• dataset (str) – Path to the dataset within the HDF5 file (e.g. "d0/d1/a2").

• refs (dict, str, or path-like, optional) – Pre-computed kerchunk reference dictionary or path to a JSON reference file. If omitted, the HDF5 metadata will be scanned using kerchunk.

• storage_options (dict, optional) – Parameters passed to fsspec or kerchunk when accessing remote files.

• max_concurrency (int, optional) – Maximum number of concurrent remote requests.

• blocks (tuple, optional) – Blosc2 block shape for chunk caching.

• cparams (dict or CParams, optional) – Blosc2 compression parameters for chunk conversion.

• _traffic (Traffic, optional) – Traffic monitor instance.

Attributes:

attrs

The user attributes of the remote dataset.

blocks

The block shape of the source.

chunks

The chunk shape of the source.

cparams

The compression parameters of the source.

dtype

The dtype of the source.

meta

The fixed-length metadata of the source.

shape

The shape of the source.

vlmeta

The variable-length metadata of the source.

Methods

aget_chunk(nchunk)	Return the compressed chunk in self asynchronously.
get_chunk(nchunk)	Return the compressed chunk in self.

__init__(urlpath, dataset: str, *, refs: dict | str | PathLike | None = None, storage_options: dict | None = None, max_concurrency: int = 8, blocks=None, cparams=None, _traffic: Traffic | None = None, _filesystem=None)

get_chunk(nchunk: int) → bytes

Return the compressed chunk in self.

Parameters:

nchunk (int) – The unidimensional index of the chunk to retrieve.

Returns:

out – The compressed chunk.

Return type:

bytes object