RemoteArray¶

RemoteArray is a persistable proxy for one remote B2ND, B2Z, Zarr, or HDF5 array. It accepts an fsspec URL or a Caterva2 URLPath class. With disk caching enabled, its B2ND carrier is both the portable descriptor and the bounded compressed-data cache.

The default policy is blosc2.CachePolicy.NONE: each operation reads the remote data it needs and no fetched data is retained afterwards. Saving such an object writes only its source descriptor and array geometry.

remote = blosc2.RemoteArray(
    "s3://public-bucket/dataset.b2nd",
    cache_policy=blosc2.CachePolicy.NONE,
)
remote.save("dataset-reference.b2nd")

A Caterva2 dataset is named with blosc2.URLPath rather than an fsspec URL:

remote = blosc2.RemoteArray(
    blosc2.URLPath(
        "@public/dataset.b2nd",
        urlbase="https://example.org/caterva2",
    )
)

By default, RemoteArray assumes its source is immutable and skips remote identity checks before reads. For a replaceable single-file or Caterva2 source, pass assume_immutable=False to refresh its identity and invalidate stale cached data before each operation.

Zarr URLs use a different contract: a .zarr path component selects ZarrNDSource, or pass source_format="zarr" for a suffix-free path. The URL names one array, including its path inside a hierarchy. Zarr sources are assumed immutable for the lifetime of every cache; replacing data beneath the same URL may mix stale and new chunks. Use a new URL or replace the cache when publishing a new dataset. Mutable Zarr stores are not supported.

remote = blosc2.open(
    "s3://public-bucket/hierarchy.zarr/d0/a1",
    lazy=True,
    storage_options={"anon": True},
)

HDF5 URLs (.h5, .hdf5, or source_format="hdf5") select HDF5NDSource. Datasets within an HDF5 container can be specified via standard slash syntax (.../file.h5/dataset), the double-colon separator (.../file.h5::dataset), or the dataset="dataset" argument. Zarr containers similarly accept all three forms (.../file.zarr/dataset, .../file.zarr::dataset, or dataset="dataset"). HDF5 datasets are read through kerchunk metadata pre-indexing. Like Zarr, HDF5 sources are assumed immutable (assume_immutable=True); mutable HDF5 sources are not supported. Pre-computed kerchunk references can be supplied via refs to avoid remote scanning.

remote = blosc2.open(
    "s3://public-bucket/hierarchy.h5/d0/d1/a2",
    lazy=True,
    storage_options={"profile": "blosc2"},
)
# Equivalent to "s3://public-bucket/hierarchy.h5::d0/d1/a2"
# or blosc2.open("s3://public-bucket/hierarchy.h5", lazy=True, dataset="d0/d1/a2", ...)

B2Z archives¶

An external NDArray inside an immutable .b2z archive can be selected using the same three addressing forms:

remote = blosc2.open(
    "s3://public-bucket/hierarchy.b2z::/d0/a3",
    lazy=True,
    storage_options={"anon": True},
)
values = remote[:10, 0, :5]
# Also accepts hierarchy.b2z/d0/a3 or dataset="d0/a3".

Use source_format="b2z" for suffix-free archive URLs. The dataset is a logical tree key without the member’s .b2nd suffix. The native Blosc2 reader preserves source chunks, blocks, dtype, and compression parameters; no kerchunk, Zarr, or HDF5 dependencies are needed. Install the fsspec extra and the protocol backend.

Opening reads the ZIP directory and selected member’s headers. Directory cost scales with archive member count. An 8 KiB archive tail and 16 KiB member prefix are prefetched to combine small metadata requests; larger directories or headers fall back to exact reads. These temporary buffers are released after opening. Subsequent reads fetch native chunks or blocks by byte range; repeated cache hits perform no remote reads. Reopening a saved carrier rereads archive/frame metadata and resolves the member offset afresh. The optimized reader derives its source stamp from the same metadata response used to obtain archive size. Caches from the initial v10 reader may therefore refetch their contents once after upgrading.

Only unencrypted, ZIP_STORED external NDArray members are supported. Groups, embedded leaves inside embed.b2e, other leaf types, and compressed ZIP members are unsupported. Archives must remain immutable; replacing an archive requires replacing its cache. Authorized sparse attachment (RemoteArray.with_sparse_cache()) is supported for eligible external NDArray leaves, while Caterva2 federation is not supported in this version.

See B2ZNDSource for class details.

Caching and persistence¶

Ephemeral in-memory caching is available through blosc2.CachePolicy.MEMORY. Fetched chunks are kept in RAM, bounded by a finite 256 MiB compressed-payload limit by default (customizable via max_cache_bytes) with automatic LRU eviction.

Persistent caching is available through blosc2.CachePolicy.DISK. Disk caches have a finite 256 MiB compressed-payload bound by default and can take an explicit max_cache_bytes bound, or max_cache_bytes=None for an unbounded cache that never evicts chunks. When bounded, the limit is enforced after an operation completes and therefore does not limit its temporary working set or returned NumPy array.

remote = blosc2.RemoteArray(
    "s3://public-bucket/dataset.b2nd",
    cache_policy=blosc2.CachePolicy.DISK,
    cache_path="dataset-cache.b2nd",
    max_cache_bytes=2 * 2**30,
)

When opening a remote array via blosc2.open() with lazy=True, a RemoteArray is always returned: specifying cache_dir or cache_path configures it with blosc2.CachePolicy.DISK, while omitting them configures it with blosc2.CachePolicy.MEMORY.

By default, RemoteArray.save and RemoteArray.to_cframe include valid warm chunks for DISK proxies; MEMORY proxies always export cold carriers. Pass include_cache=False for a cold carrier without changing the warm original. The cache policy and limit remain in both forms; local paths and authentication data are not serialized.

Pass cache_policy=blosc2.CachePolicy.NONE (or another policy) to either export method to produce a cold carrier with an explicit policy, leaving the live proxy unchanged. Caterva2 servers accept persisted MEMORY carriers under opt-in policy but execute them without retained caching (identical to NONE); older Caterva2 servers reject MEMORY resolution. Use DISK for retained carrier caching on Caterva2. Cold exports must not overwrite the live disk carrier.

fetch() and afetch() prefetch and return the proxy. Eviction may discard requested chunks; materialize(item) returns an independent complete NDArray. The raw cache is incomplete storage for inspection, not a materialized array.

Reads and exports on one handle are serialized. Async methods use worker threads; cancelling an await does not stop a running operation. Separate handles and processes sharing a carrier need external locking. Unreadable cache files are preserved and their opening errors are propagated.

Authentication supplied to a live Caterva2 source is deliberately omitted from the carrier. Caterva2’s first server implementation resolves public HTTPS sources only; client credentials never travel with the proxy.

Open a disk-caching carrier in append mode to let misses populate that same file. Read-only mode can use warm chunks but does not retain misses:

cached = blosc2.open("dataset-cache.b2nd", mode="a")
cached[100:200]

Warning

Resolving an uploaded remote reference makes the receiving server perform an outbound request. Caterva2 installations must reject these references by default unless administrators configure allowed protocols, destinations, credentials, redirects, and resource limits. Client-side URL checks are not a server security boundary.

class blosc2.RemoteArray(urlpath, *, cache_policy=CachePolicy.NONE, cache_path=None, cache_dir=None, max_cache_bytes=<policy default>, max_concurrency: int | None = None, storage_options: dict | None = None, source_format: str | None = None, assume_immutable: bool = True, dataset: str | None = None, refs=None, _carrier=None, _runtime_cache_path=None, _source_descriptor=None, _source_blocks=None, _source_cparams=None, _store_owner=None, _runtime_is_mutable: bool = True)[source]¶

A persistable, optionally self-caching reference to a remote array.

With CachePolicy.DISK, the public constructor uses the persisted B2ND carrier itself as the bounded cache. Server code can instead use with_sparse_cache() to keep a private directory-backed runtime cache beside a portable carrier. With CachePolicy.MEMORY, chunks are retained in process memory up to a bounded size. With CachePolicy.NONE, reads retain no data.

Parameters:

urlpath¶ (str, URLPath, or C2Array) – A B2ND or Zarr array URL, an HDF5 or B2Z container URL with a dataset selection, or a Caterva2 array reference.
cache_policy¶ (CachePolicy) – NONE retains no array data. MEMORY retains compressed chunks in client process memory. DISK retains compressed chunks in the RemoteArray carrier at cache_path or under cache_dir.
cache_path¶ (str or path-like, optional) – Exact persistent cache filename. Only valid with DISK and mutually exclusive with cache_dir.
cache_dir¶ (str or path-like, optional) – Directory in which a source-derived persistent cache filename is made. Only valid with DISK.
max_cache_bytes¶ (int or None, optional) – Post-operation compressed-payload bound. It defaults to 256 MiB for DISK and MEMORY. Passing None with DISK disables cache eviction (unbounded cache). MEMORY requires a finite positive integer. It is not applicable to NONE.
max_concurrency¶ (int, optional) – Maximum number of independent remote fetches in flight.
storage_options¶ (dict, optional) – Parameters passed to the underlying fsspec filesystem when opening an fsspec URL.
source_format¶ ({None, "blosc2", "zarr", "hdf5", "b2z"}, optional) – Format of a URL source, inferred from its container suffix when omitted.
dataset¶ (str, optional) – Array path within an HDF5, Zarr, or B2Z container. B2Z supports external NDArray leaves in immutable archives, e.g. dataset="d0/a3".
assume_immutable¶ (bool, optional) – Skip remote identity checks before reads. Defaults to True. Set to False when the object at the URL may be replaced.

Attributes:

assume_immutable: Whether reads skip remote identity checks.
attrs: The read-only user attributes of the remote array.
blocks
cache: The local container used as cache, or None if caching is disabled.
cache_bytes: Compressed bytes currently retained by the runtime cache.
cache_path: The self-caching carrier path, or None for other policies.
cache_policy: The persisted retention policy.
cache_status: How a persistent disk cache was handled, or None otherwise.
cached_payload_bytes: Compressed resident payload accounting from the attached snapshot.
chunks
cparams
dataset: The dataset path within a container source, or None.
device: Hardware device the array data resides on.
dtype: Get the data type of the Operand.
info: A printable summary of this remote reference.
info_items: The fields shown by info.
is_cache_mutable: Whether the currently opened cache is writable.
max_cache_bytes: The persisted post-operation retained-cache bound.
meta: The fixed-length metalayers of the remote array.
mutable: The export default mutability for future exports.
nbytes: The uncompressed size of the remote array.
ndim: The number of dimensions in the remote array.
runtime_cache_path: The mutable sparse cache directory, when one is attached.
schunk: The underlying carrier’s or cache’s SChunk, or None if unattached.
shape: Get the shape of the Operand.
source: A copy of the credential-free source descriptor.
traffic
urlpath: The remote fsspec URL or credential-free Caterva2 URLPath.
vlmeta: The variable-length metadata of the remote array.

Methods

`afetch`([item, max_concurrency])	Prefetch in a worker thread and return this proxy, like `fetch()`.
`all`([axis, keepdims])	Test whether all array elements along a given axis evaluate to True.
`any`([axis, keepdims])	Test whether any array element along a given axis evaluates to True.
`argmax`([axis, keepdims])	Returns the indices of the maximum values along a specified axis.
`argmin`([axis, keepdims])	Returns the indices of the minimum values along a specified axis.
`cache_contains`([item, nchunk])	Check cached coverage; use `read_cached` for an atomic hit/read.
`close`()	Release this store-derived handle; standalone handles retain their existing lifetime.
`cumulative_prod`([axis, dtype, include_initial])	Calculates the cumulative product of elements in the input array ndarr.
`cumulative_sum`([axis, dtype, include_initial])	Calculates the cumulative sum of elements in the input array ndarr.
`fetch`([item, max_concurrency])	Fetch remote data into the cache container.
`item`()	Copy an element of an array to a standard Python scalar and return it.
`materialize`([item])	Return an independent NDArray containing the requested values.
`max`([axis, keepdims])	Return the maximum along a given axis.
`mean`([axis, dtype, keepdims])	Return the arithmetic mean along the specified axis.
`min`([axis, keepdims])	Return the minimum along a given axis.
`prod`([axis, dtype, keepdims])	Return the product of array elements over a given axis.
`read_cached`([item, nchunk])	Return `(hit, result)` atomically, without fetching a missing block.
`save`(urlpath[, contiguous, include_cache, ...])	Save a carrier; MEMORY exports are cold.
`std`([axis, dtype, ddof, keepdims])	Return the standard deviation along the specified axis.
`sum`([axis, dtype, keepdims])	Return the sum of array elements over a given axis.
`to_cframe`(*[, include_cache, cache_policy, ...])	Export a carrier.
`to_device`(device)	Copy the array from the device on which it currently resides to the specified device.
`trim_cache`(target_bytes, *[, max_chunks])	Evict at most `max_chunks` LRU chunks toward a payload-byte target.
`trim_sparse_cache`(runtime_cache_path, ...[, ...])	Trim an offline private cache without constructing a remote source.
`var`([axis, dtype, ddof, keepdims])	Return the variance along the specified axis.
`where`([value1, value2])	Select `value1` or `value2` values based on `True`/`False` for `self`.
`with_sparse_cache`(urlpath, runtime_cache_path, *)	Attach an authorized remote source to a private sparse disk cache.

aget_chunk
get_chunk

__init__(urlpath, *, cache_policy=CachePolicy.NONE, cache_path=None, cache_dir=None, max_cache_bytes=<policy default>, max_concurrency: int | None = None, storage_options: dict | None = None, source_format: str | None = None, assume_immutable: bool = True, dataset: str | None = None, refs=None, _carrier=None, _runtime_cache_path=None, _source_descriptor=None, _source_blocks=None, _source_cparams=None, _store_owner=None, _runtime_is_mutable: bool = True)[source]¶

__getitem__(item)[source]¶

fetch(item=(), max_concurrency: int | None = None)[source]¶

Fetch remote data into the cache container.

Return this proxy, not a materialized array. Eviction may discard prefetched chunks. Use indexing for values or materialize() for an independent NDArray. Requires MEMORY or DISK caching.

async afetch(item=(), max_concurrency: int | None = None)[source]¶

Prefetch in a worker thread and return this proxy, like fetch().

Requires MEMORY or DISK. Cancelling the await does not interrupt an already running fetch, which retains the operation lock until done.

get_chunk(nchunk: int) → bytes[source]¶

async aget_chunk(nchunk: int) → bytes[source]¶

save(urlpath: str | PathLike, contiguous: bool = True, *, include_cache: bool = True, cache_policy=None, mutable: bool | None = None, **kwargs) → str[source]¶

Save a carrier; MEMORY exports are cold. See to_cframe().

Return the written urlpath.

materialize(item=(), **kwargs)[source]¶

Return an independent NDArray containing the requested values.

The output and temporary NumPy buffer are not bounded by max_cache_bytes. Keyword arguments are forwarded to blosc2.asarray.

to_cframe(*, include_cache: bool = True, cache_policy=None, mutable: bool | None = None) → bytes[source]¶

Export a carrier. Only DISK preserves warm chunks by default.

An explicit cache_policy exports a cold carrier with that policy.

shape¶

dtype¶

ndim¶: The number of dimensions in the remote array.

chunks¶

blocks¶

cparams¶

nbytes¶: The uncompressed size of the remote array.

meta¶: The fixed-length metalayers of the remote array.

attrs¶: The read-only user attributes of the remote array.

info¶: A printable summary of this remote reference.

cache¶: The local container used as cache, or None if caching is disabled.

cache_bytes¶: Compressed bytes currently retained by the runtime cache.

cache_policy¶: The persisted retention policy.

max_cache_bytes¶: The persisted post-operation retained-cache bound.

cache_path¶: The self-caching carrier path, or None for other policies.

cache_status¶: How a persistent disk cache was handled, or None otherwise.

schunk¶: The underlying carrier’s or cache’s SChunk, or None if unattached.

source¶: A copy of the credential-free source descriptor.

traffic¶

urlpath¶: The remote fsspec URL or credential-free Caterva2 URLPath.

dataset¶: The dataset path within a container source, or None.

RemoteMetadataMapping¶

RemoteArray.attrs returns a read-only mapping that fetches array attributes only when they are accessed. Use it like a dictionary, or use attrs[:] to fetch all attributes at once.

class blosc2.RemoteMetadataMapping(data: Mapping | None = None)[source]¶

Read-only dictionary-like mapping of remote array metadata.

Methods

`get`(k[,d])
`items`()
`keys`()
`values`()

copy
getall

get(k[, d]) → D[k] if k in D, else d. d defaults to None.[source]¶

CachePolicy¶

class blosc2.CachePolicy(*values)[source]¶: Retention policy for data read through a remote proxy.