FsspecNDSource

A ByteRangeNDSource that serves the chunks of a Blosc2 frame living behind an fsspec URL, reading each one with a range request instead of transferring the whole container. Everything about the frame format, block granularity included, lives in the base class; this adds the fsspec transport. The URL must name a standalone, contiguous .b2nd NDArray frame. It cannot name an HDF5 dataset, a member inside a .b2z store, a sparse directory container, or a computed array: fsspec provides bytes, not dataset semantics. For the Caterva2 alternative and a capability comparison, see Working with Remote Arrays. For other sources, see ProxyNDSource and ProxySource.

examples/remote/rw-fsspec.py is a runnable walkthrough of this and the other two ways to read an fsspec URL, and of writing one back. examples/remote/concurrent-fsspec.py measures max_concurrency against a filesystem with a simulated round trip, since no protocol that runs offline has latency for the thread pool to hide.

class blosc2.FsspecNDSource(urlpath: str, max_concurrency: int = 8, *, storage_options: dict | None = None, _filesystem=None, _traffic=None)

A ByteRangeNDSource reading its frame through fsspec.

This is what blosc2.open(url, lazy=True) builds; wrap it in a Proxy by hand when the cache belongs at a path of your choosing rather than inside cache_dir:

src = blosc2.FsspecNDSource("s3://bucket/big.b2nd")
a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a")

Parameters:

• urlpath (str) – The fsspec URL of the frame.

• max_concurrency (int, optional) – As in ByteRangeNDSource.

• storage_options (dict, optional) – Parameters passed to the underlying fsspec filesystem.

Attributes:

attrs

The user attributes of the remote frame.

blocks

The block shape of the source.

chunks

The chunk shape of the source.

cparams

The compression parameters of the source.

dtype

The dtype of the source.

has_vlmetalayers

Whether the underlying frame has variable-length metalayers.

meta

Fixed-length metadata of the remote frame.

shape

The shape of the source.

stamp

vlmeta

Variable-length metadata of the remote frame.

Methods

aget_chunk(nchunk)	Same as get_chunk(), but without blocking the caller's event loop.
block_plan(nchunk, nblocks)	The range reads that cover nblocks, near-adjacent ones merged.
chunk_layout(nchunk)	Read where the blocks of a chunk are: its header, bstarts and extents.
chunk_layouts(nchunks)	chunk_layout() for several chunks, in as few requests as they fit.
get_chunk(nchunk)	Return the compressed chunk in self.
invalidate_index()	Forget where the chunks and blocks are, so the next read looks again.
read_range(offset, size)	The bytes at [offset, offset + size) of the frame.
read_ranges(spans)	The bytes of every (offset, size) in spans, in that order.
refresh_identity()	Refresh the identity of the object currently stored at this URL.
wants_blocks(nchunk, nwanted[, wave, nruns])	Whether fetching nwanted blocks of a chunk beats fetching all of it.
written_chunks()	Which chunks of the frame hold content, as a boolean per chunk.

read_range(offset: int, size: int) → bytes

The bytes at [offset, offset + size) of the frame.

The whole of the transport: everything else here is the frame format. Fewer bytes may come back only at the end of the frame; anything else is an error, since the caller has no way to ask for the rest. Must be safe to call from several threads at once, which is what lets Proxy overlap the fetches of one slice.

refresh_identity() → None

Refresh the identity of the object currently stored at this URL.

Object-store implementations expose this through ukey. HTTP needs an explicit metadata request because its fsspec ukey identifies only the URL and options, not the response currently available there. None means that the HTTP server supplied no validator strong enough to justify retaining cached data across operations.