ProxyNDSource

Interface for NDim sources in Proxy. For example, a NDArray, a HDF5 dataset, etc. For a simpler source, see ProxySource.

class blosc2.ProxyNDSource

Base interface for NDim sources in Proxy.

A source may also serve single blocks rather than whole chunks, which is worth doing when a fetch costs a round trip and a slice touches little of the chunks it lands in. Proxy uses that path when the source has all five of blocks_per_chunk, wants_blocks(nchunk, nwanted), chunk_layout(nchunk), block_plan(nchunk, nblocks) and read_range(offset, size); FsspecNDSource implements them over byte ranges and is the worked example.

Having them is not the same as being able to use them for the dataset in hand: a source that knows, without asking anything, that this one is served whole says so with serves_blocks, and Proxy then keeps to chunks from the start rather than to a per-block bitmap it would never fill. It is read once, when the proxy is built, and a source without it counts as True.

A source whose transport can ask for several ranges at once says so with max_ranges and serves read_ranges(spans) and chunk_layouts(nchunks) as well; Proxy then sends a whole wave of reads as one request. Setting wants_wave says that wants_blocks takes a third argument and is to be given the wave – the fetch that chunk belongs to – since a shared round trip is the wave’s to weigh and not the chunk’s. It is an opt-in of its own, in the same shape as max_ranges and read the same way, so a two-argument wants_blocks keeps being called with two. All are optional, and a source without them is asked one range at a time, exactly as before.

A block read that the transport cannot answer raises NotRanged, and Proxy then fetches the chunks it was after whole.

Attributes:

blocks

The block shape of the source.

chunks

The chunk shape of the source.

cparams

The compression parameters of the source.

dtype

The dtype of the source.

shape

The shape of the source.

Methods

aget_chunk(nchunk)	Return the compressed chunk in self asynchronously.
get_chunk(nchunk)	Return the compressed chunk in self.

async aget_chunk(nchunk: int) → bytes

Return the compressed chunk in self asynchronously.

Parameters:

nchunk (int) – The index of the chunk to retrieve.

Returns:

out – The compressed chunk.

Return type:

bytes object

Notes

This method is optional, and only available if the source has an async aget_chunk method.

abstractmethod get_chunk(nchunk: int) → bytes

Return the compressed chunk in self.

Parameters:

nchunk (int) – The unidimensional index of the chunk to retrieve.

Returns:

out – The compressed chunk.

Return type:

bytes object

abstract property blocks: tuple

The block shape of the source.

abstract property chunks: tuple

The chunk shape of the source.

property cparams: CParams

The compression parameters of the source.

This property is optional and can be overridden if the source has a different compression configuration.

abstract property dtype: dtype

The dtype of the source.

abstract property shape: tuple

The shape of the source.