Save and load¶

`save`(array, urlpath[, contiguous])	Save an array to a file.
`open`(urlpath[, mode, offset, dataset, ...])	Open a persistent SChunk, NDArray, a remote C2Array, RemoteArray, Proxy, a DictStore, EmbedStore, or TreeStore, `CTable`, `RemoteCTable`, or `RemoteStore`.
`load`(urlpath[, offset])	Load a persistent Blosc2 object into memory.
`save_array`(arr, urlpath[, chunksize])	Save a serialized NumPy array to a specified file path.
`load_array`(urlpath[, dparams, storage_options])	Load a serialized NumPy array from a file.
`save_tensor`(tensor, urlpath[, chunksize])	Save a serialized PyTorch or TensorFlow tensor or NumPy array to a specified file path.
`load_tensor`(urlpath[, dparams, storage_options])	Load a serialized PyTorch or TensorFlow tensor or NumPy array from a file.
`from_cframe`(cframe[, copy])	Create a EmbedStore, NDArray, SChunk, BatchArray or ObjectArray instance from a contiguous frame buffer.

blosc2.save(array: NDArray, urlpath: str, contiguous=True, **kwargs: Any) → None[source]¶

Save an array to a file.

Parameters:

array¶ (NDArray) – The array to be saved.
urlpath¶ (str) – The path to the file where the array will be saved, or an fsspec URL to upload it to as a single object. See NDArray.save().
contiguous¶ (bool, optional) – Whether to store the array contiguously.
kwargs¶ (dict, optional) – Keyword arguments that are supported by the save() method.

Examples

>>> import blosc2
>>> import numpy as np
>>> # Create an array
>>> array = blosc2.arange(0, 100, dtype=np.int64, shape=(10, 10))
>>> # Save the array to a file
>>> blosc2.save(array, "array.b2", mode="w")

Open a persistent SChunk, NDArray, a remote C2Array, RemoteArray, Proxy, a DictStore, EmbedStore, or TreeStore, CTable, RemoteCTable, or RemoteStore.

See the Notes section for more info on opening Proxy objects.

Parameters:

urlpath¶ (str | pathlib.Path | URLPath class) – The path where the SChunk (or NDArray) is stored. URLPath class is exclusively a Caterva2 dataset reference, including when its urlbase is omitted and inherited from c2context(); a server names its datasets by root and path rather than by URL. Any URL with a scheme (s3://, gs://, https://, zip://, memory://…) is opened through fsspec; see the Notes section for the limits.
mode¶ (str, optional) –
Persistence mode: ‘r’ means read only (must exist); ‘a’ means read/write (create if it doesn’t exist); ‘w’ means create (overwrite if it exists). Defaults to ‘r’ ( read-only).

Open modes also define the allowed persistence side effects:
- 'r' never writes to the persistent object. It writes a local cache only when cache_dir or cache_path explicitly requests one; query acceleration and other implicit execution caches remain process-local only.
- 'a' and 'w' may persist explicit user-visible changes such as data, metadata, and index maintenance, but execution caches and query memoization still remain process-local only.
offset¶ (int, optional) – An offset in the file where super-chunk or array data is located (e.g. in a file containing several such objects). A nonzero offset in a local file opens the embedded Blosc2 frame directly, bypassing filename-based container format detection.
shared_cache¶ (bool, optional) – Share an on-demand disk cache between processes. Defaults to False. Requires cache_dir and a remote source: a standalone .b2nd URL, a B2Z, HDF5, or Zarr container, or a Caterva2 URLPath class. Requires lazy access, CachePolicy.DISK, and assume_immutable=True. Enables lazy access for every remote source when lazy is omitted or None; explicit lazy=False is rejected. Returns the selected table, group, or array using sparse cache storage and operation-scoped locks. Operations on the same cache serialize. All processes using this cache must enable sharing; use a separate directory from ordinary exclusive caches. The aggregate retained compressed-payload budget defaults to 256 MiB per standalone array or table/store owner; max_cache_bytes=None disables eviction. This does not bound total disk usage or peak RAM. Authenticated Caterva2 users must use separate cache directories; tokens are not persisted.
kwargs¶ (dict, optional) –

lazy: bool or None, optional
None (the default) automatically selects the access mode. True returns a lazy remote object; False requests eager access. Known remote .b2nd arrays and .b2z archives default to lazy access, including when cache_dir= is supplied. Pass lazy=False to download the whole container under cache_dir instead. mmap_mode= or a nonzero offset= forces the eager path. Dataset paths currently require lazy=True and reject explicit False. For an fsspec URL or a Caterva2 URLPath class, return a RemoteArray over the remote array dataset and read the byte ranges a slice touches. B2Z and HDF5 table and group nodes return RemoteCTable and RemoteStore instead. Selected local PyTables/HDF5 tables also return RemoteCTable; local ordinary datasets remain RemoteArray. A slice landing in a small part of a large chunk costs only the blocks it touches when ranges are available; chunks small enough to be one cheap request are still fetched whole. What arrives is kept in memory (defaulting to CachePolicy.MEMORY), under cache_dir, or at the exact cache_path (as CachePolicy.DISK).

max_concurrency: int, optional
For lazy remote arrays and RemoteCTable: how many fetches to run at once, in a thread pool. A slice against an object store is almost entirely round-trip latency, so overlapping the requests is what makes a wide slice bearable. Defaults to 8; pass 1 for a protocol with no latency to hide, where the pool costs about 10 microseconds per chunk and saves nothing. Remote tables overlap independent requests across selected columns; their table-specific temporary buffer settings are available through RemoteCTable, not through this general opener.

cache_dir: str | pathlib.Path, optional
Parent directory for a persistent RemoteArray, RemoteCTable, or RemoteStore cache. For remote inputs it stores fetched chunks and metadata; for local Blosc2, HDF5, and Zarr inputs it stores accessed chunks and converted data. Supplying it selects on-demand, read-only access for local inputs. Local caches assume the source is immutable: call refresh() on the cached handle after replacing it at the same path. No cache is created unless you name one.

cache_path: str | pathlib.Path, optional
Exact file for a persistent array cache (CachePolicy.DISK), for local or remote standalone arrays and array leaves. Tables and groups require cache_dir. Mutually exclusive with cache_dir.

cache_storage: str | pathlib.Path, optional
Deprecated alias for cache_dir. Mutually exclusive with cache_dir and cache_path.

cache_policy: CachePolicy, optional
With lazy=True on a remote source, return a RemoteArray using the requested retention policy (NONE, MEMORY, or DISK). When omitted, passing cache_dir or cache_path defaults to CachePolicy.DISK, while omitting them defaults to CachePolicy.MEMORY.

max_cache_bytes: int or None, optional
With lazy=True, bound retained compressed cache payload for a RemoteArray after each operation. Defaults to 256 MiB for both DISK and MEMORY. Passing None with DISK disables cache eviction (unbounded cache). This does not bound the current operation’s working set or result.

mmap_mode: str, optional
If set, the file will be memory-mapped instead of using the default I/O functions and the mode argument will be ignored. For more info, see blosc2.Storage. Please note that the w+ mode, which can be used to create new files, is not supported here since only existing files can be opened. You can use SChunk.__init__ to create new files.

initial_mapping_size: int, optional
The initial size of the memory mapping. For more info, see blosc2.Storage.

locking: bool, optional
Serialize accesses against other handles and other processes via a sidecar lock file. Enable it when several processes operate on the same container. The locking is advisory (every handle on the container must enable it) and cannot be combined with mmap_mode. The BLOSC_LOCKING environment variable enables it globally. For more info, see blosc2.Storage.

cparams: dict
A dictionary with the compression parameters, which are the same that can be used in the compress2() function. Typesize and blocksize cannot be changed.

dparams: dict
A dictionary with the decompression parameters, which are the same that can be used in the decompress2() function.

storage_options: dict, optional
Parameters passed to the underlying fsspec filesystem when opening an fsspec URL (for instance credentials, endpoint URL, token, client_kwargs, etc.).

path: str, optional
Node path within HDF5, Zarr, or B2Z containers (e.g. path="d0/d1/a2"). B2Z and HDF5 also support table and group paths in immutable containers. Requires lazy=True. None leaves selection unspecified; "" and "/" select the root. Do not combine with a selector embedded in the URL.

dataset: str, optional
Supported alias of path. Both may be supplied if they agree after stripping leading/trailing slashes; conflicting values raise ValueError.

hdf5_index: dict | str | PathLike, optional
Pre-computed native HDF5 index, or a local path or remote fsspec URL to its JSON encoding. It must match the source HDF5 URL and dataset scope. Arrays, PyTables tables, and hierarchy stores are supported.

source_format: {None, “blosc2”, “zarr”, “hdf5”, “b2z”, “parquet”}, optional
Format of a lazy remote source. A .zarr URL path component selects Zarr automatically; a .h5 or .hdf5 path selects HDF5 automatically; a .b2z path selects B2Z automatically. An explicit value supports suffix-free paths. Zarr, HDF5, B2Z and Parquet sources automatically enable lazy=True.

parquet_options: dict, optional
PyArrow ParquetFile reader options for a Parquet source. Conversion options such as columns and max_rows are passed separately.

assume_immutable: bool, optional
With lazy=True, skip remote identity checks before reads. Defaults to True. Local disk caches always assume immutable sources; set to a new path or explicitly rebuild the cache after changing a local source.

Returns:

out – Proxy, DictStore, EmbedStore, TreeStore, CTable, RemoteCTable, or RemoteStore The object found in the path.

Return type:

SChunk, NDArray, C2Array, RemoteArray,

Notes

Returned objects can be used as context managers for API consistency. For objects with an explicit close() implementation, exiting the context will close/flush them. Standalone HDF5 RemoteArray handles close their HDF5 source, so the handle rejects further reads. Other logical handles such as regular SChunk, NDArray, C2Array, standalone non-HDF5 RemoteArray, Proxy, and LazyArray currently treat context exit as a no-op. Store-derived RemoteArray handles release their shared source ownership when closed; other handles from that store remain usable.
If urlpath is a URLPath class instance, mode must be ‘r’ and offset must be 0. By default it returns a C2Array. With lazy=True or shared_cache=True, it returns a RemoteArray (defaulting to CachePolicy.DISK when cache_dir or cache_path is provided, and CachePolicy.MEMORY otherwise). Authenticated users sharing a machine must use separate caches.
fsspec URLs need the fsspec extra (pip install "blosc2[fsspec]") and the driver for the protocol (s3fs, gcsfs…), which fsspec asks for by name when it is missing. Driver and protocol parameters (credentials, endpoint URL, region, etc.) can be passed directly via storage_options. mode != 'r' always raises, as object stores have no rename and no locks. A plain URL read rebuilds the object from a cframe held in memory, so it covers .b2nd, .b2f and .b2e only. Remote .b2z archives default to lazy discovery: table nodes return RemoteCTable, groups return RemoteStore, and external array leaves return RemoteArray. Select a nested node with path= or ::path. Tables and groups use cache_dir for DISK caching and default to MEMORY otherwise; array leaves additionally support cache_path. Use lazy=False, cache_dir=... to download a complete archive and open its root locally. Local table archives return CTable.
Persistent data handling follows a no-hidden-writes rule except for an explicitly self-caching RemoteArray:
- mode='r' is observational only and never mutates the opened object.
- mode='a' permits a DISK RemoteArray to retain remote chunks in its own carrier. Other execution caches are not serialized implicitly.
- mode='w' persists explicit mutations requested by the caller.
If the original object saved in urlpath is a Proxy or a RemoteArray, this function reconstructs sources backed by a persistent local SChunk or NDArray, an fsspec URL, or a remote C2Array. Custom proxy sources must be recreated explicitly because their Python class and runtime state are not stored in the cache.
When opening a LazyExpr keep in mind the note above regarding operands.

Examples

>>> import blosc2
>>> import numpy as np
>>> import os
>>> import tempfile
>>> tmpdirname = tempfile.mkdtemp()
>>> urlpath = os.path.join(tmpdirname, 'b2frame')
>>> storage = blosc2.Storage(contiguous=True, urlpath=urlpath, mode="w")
>>> nelem = 20 * 1000
>>> nchunks = 5
>>> chunksize = nelem * 4 // nchunks
>>> data = np.arange(nelem, dtype="int32")
>>> # Create SChunk and append data
>>> schunk = blosc2.SChunk(chunksize=chunksize, data=data.tobytes(), storage=storage)
>>> # Open SChunk
>>> sc_open = blosc2.open(urlpath=urlpath, mode="r")
>>> for i in range(nchunks):
...     dest = np.empty(nelem // nchunks, dtype=data.dtype)
...     schunk.decompress_chunk(i, dest)
...     dest1 = np.empty(nelem // nchunks, dtype=data.dtype)
...     sc_open.decompress_chunk(i, dest1)
...     np.array_equal(dest, dest1)
True
True
True
True
True

To open the same schunk memory-mapped, we simply need to pass the mmap_mode parameter:

>>> sc_open_mmap = blosc2.open(urlpath=urlpath, mmap_mode="r")
>>> sc_open.nchunks == sc_open_mmap.nchunks
True
>>> all(sc_open.decompress_chunk(i, dest1) == sc_open_mmap.decompress_chunk(i, dest1) for i in range(nchunks))
True

blosc2.load(urlpath: str | Path, offset: int = 0, **kwargs: dict)[source]¶

Load a persistent Blosc2 object into memory.

This is the in-memory counterpart to open(). It opens urlpath in read-only mode and returns a standalone object that is not backed by the original file. For CTable, this dispatches to CTable.load(); for array-like containers it returns an in-memory copy.

Parameters:

urlpath¶ (str | pathlib.Path) – Path to the persistent Blosc2 object.
offset¶ (int, optional) – Offset in the file where the object is located. This is mainly useful for SChunk/NDArray objects embedded in a larger file.
kwargs¶ (dict, optional) – Additional read-time keyword arguments passed to open(), such as dparams.

Returns:

A standalone in-memory Blosc2 object.

Return type:

out

Raises:

TypeError – If the opened object cannot be loaded as a standalone in-memory object.

Examples

>>> import blosc2
>>> import numpy as np
>>> arr = blosc2.asarray(np.arange(10), urlpath="example.b2nd", mode="w")
>>> loaded = blosc2.load("example.b2nd")
>>> loaded.urlpath is None
True
>>> np.array_equal(loaded[:], arr[:])
True
>>> blosc2.remove_urlpath("example.b2nd")

blosc2.save_array(arr: ndarray, urlpath: str, chunksize: int | None = None, **kwargs: dict) → int[source]¶

Save a serialized NumPy array to a specified file path.

Parameters:

arr¶ (np.ndarray) – The NumPy array to be saved.
urlpath¶ (str) – The path for the file where the array will be saved. An fsspec URL (s3://, gs://, memory://…) writes the whole container in one shot; it needs the fsspec extra and the protocol driver installed.
chunksize¶ (int) – The size (in bytes) for the chunks during compression. If not provided, it is computed automatically.
kwargs¶ (dict, optional) – These are the same as the kwargs in SChunk.__init__.

Returns:

out – The number of bytes of the saved array.

Return type:

int

Examples

>>> import numpy as np
>>> a = np.arange(1e6)
>>> serial_size = blosc2.save_array(a, "test.bl2", mode="w")
>>> serial_size < a.size * a.itemsize
True

blosc2.load_array(urlpath: str, dparams: dict | None = None, *, storage_options: dict | None = None) → ndarray[source]¶

Load a serialized NumPy array from a file.

Parameters:

urlpath¶ (str) – The path to the file containing the serialized array.
dparams¶ (dict, optional) – A dictionary with the decompression parameters, which can be used in the decompress2() function.
storage_options¶ (dict, optional) – Parameters passed to the underlying fsspec filesystem when opening an fsspec URL.

Returns:

out – The deserialized NumPy array.

Return type:

np.ndarray

Raises:

TypeError – If urlpath is not in cframe format
RunTimeError – If any other error is detected.

Examples

>>> import numpy as np
>>> a = np.arange(1e6)
>>> serial_size = blosc2.save_array(a, "test.bl2", mode="w")
>>> serial_size < a.size * a.itemsize
True
>>> a2 = blosc2.load_array("test.bl2")
>>> np.array_equal(a, a2)
True

blosc2.save_tensor(tensor: tensorflow.Tensor | torch.Tensor | np.ndarray, urlpath: str, chunksize: int | None = None, **kwargs: dict) → int[source]¶

Save a serialized PyTorch or TensorFlow tensor or NumPy array to a specified file path.

Parameters:

tensor¶ (tensorflow.Tensor, torch.Tensor, or np.ndarray) – The tensor or array to be saved.
urlpath¶ (str) – The file path where the tensor or array will be saved. An fsspec URL (s3://, gs://, memory://…) writes the whole container in one shot; it needs the fsspec extra and the protocol driver installed.
chunksize¶ (int) – The size (in bytes) for the chunks during compression. If not provided, it is computed automatically.
kwargs¶ (dict, optional) – These are the same as the kwargs in SChunk.__init__.

Returns:

out – The number of bytes of the saved tensor or array.

Return type:

int

Examples

>>> import numpy as np
>>> th = np.arange(1e6, dtype=np.float32)
>>> serial_size = blosc2.save_tensor(th, "test.bl2", mode="w")
>>> if not os.getenv("BTUNE_TRADEOFF"):
...     assert serial_size < th.size * th.itemsize
...

blosc2.load_tensor(urlpath: str, dparams: dict | None = None, *, storage_options: dict | None = None) → tensorflow.Tensor | torch.Tensor | np.ndarray[source]¶

Load a serialized PyTorch or TensorFlow tensor or NumPy array from a file.

Parameters:

urlpath¶ (str) – The path to the file where the tensor or array is stored.
dparams¶ (dict, optional) – A dictionary with the decompression parameters, which are the same as those used in the decompress2() function.
storage_options¶ (dict, optional) – Parameters passed to the underlying fsspec filesystem when opening an fsspec URL.

Returns:

out – The unpacked PyTorch or TensorFlow tensor or NumPy array.

Return type:

tensor or ndarray

Raises:

TypeError – If urlpath is not in cframe format
RunTimeError – If some other problem is detected.

Examples

>>> import numpy as np
>>> th = np.arange(1e6, dtype=np.float32)
>>> size = blosc2.save_tensor(th, "test.bl2", mode="w")
>>> if not os.getenv("BTUNE_TRADEOFF"):
...     assert size < th.size * th.itemsize
...
>>> th2 = blosc2.load_tensor("test.bl2")
>>> np.array_equal(th, th2)
True

See also

save_tensor(), pack_tensor()

Create a EmbedStore, NDArray, SChunk, BatchArray or ObjectArray instance from a contiguous frame buffer.

Parameters:

cframe¶ (bytes or str) – The bytes object containing the in-memory cframe.
copy¶ (bool) – Whether to internally make a copy. Default is True. With False the returned object points into cframe’s buffer and keeps a reference to it (the buffer lives for as long as the object does), saving time/memory at the cost of the buffer staying pinned.

Returns:

out – BatchArray, ObjectArray, or: RemoteArray

A new instance of the appropriate type containing the data passed.

Return type:

EmbedStore, NDArray, SChunk,

See also

from_cframe(), from_cframe(), from_cframe()