FsspecNDSource

A ByteRangeNDSource that serves the chunks of a Blosc2 frame living behind an fsspec URL, reading each one with a range request instead of transferring the whole container. Everything about the frame format, block granularity included, lives in the base class; this adds the fsspec transport. For other sources, see ProxyNDSource and ProxySource.

examples/ndarray/rw-fsspec.py is a runnable walkthrough of this and the other two ways to read an fsspec URL, and of writing one back. examples/ndarray/concurrent-fsspec.py measures max_concurrency against a filesystem with a simulated round trip, since no protocol that runs offline has latency for the thread pool to hide.

class blosc2.FsspecNDSource(urlpath: str, max_concurrency: int = 8)

A ByteRangeNDSource reading its frame through fsspec.

This is what blosc2.open(url, lazy=True) builds; wrap it in a Proxy by hand when the cache belongs at a path of your choosing rather than inside cache_storage:

src = blosc2.FsspecNDSource("s3://bucket/big.b2nd")
a = blosc2.Proxy(src, urlpath="big-cache.b2nd", mode="a")

Parameters:

• urlpath (str) – The fsspec URL of the frame.

• max_concurrency (int, optional) – As in ByteRangeNDSource.

Attributes:

blocks

The block shape of the source.

chunks

The chunk shape of the source.

cparams

The compression parameters of the source.

dtype

The dtype of the source.

shape

The shape of the source.

stamp

Methods

aget_chunk(nchunk)	Same as get_chunk(), but without blocking the caller's event loop.
block_plan(nchunk, nblocks)	The range reads that cover nblocks, near-adjacent ones merged.
chunk_layout(nchunk)	Read where the blocks of a chunk are: its header, bstarts and extents.
chunk_layouts(nchunks)	chunk_layout() for several chunks, in as few requests as they fit.
get_chunk(nchunk)	Return the compressed chunk in self.
invalidate_index()	Forget where the chunks and blocks are, so the next read looks again.
read_range(offset, size)	The bytes at [offset, offset + size) of the frame.
read_ranges(spans)	The bytes of every (offset, size) in spans, in that order.
wants_blocks(nchunk, nwanted[, wave, nruns])	Whether fetching nwanted blocks of a chunk beats fetching all of it.
written_chunks()	Which chunks of the frame hold content, as a boolean per chunk.

read_range(offset: int, size: int) → bytes

The bytes at [offset, offset + size) of the frame.

The whole of the transport: everything else here is the frame format. Fewer bytes may come back only at the end of the frame; anything else is an error, since the caller has no way to ask for the rest. Must be safe to call from several threads at once, which is what lets Proxy overlap the fetches of one slice.