ByteRangeNDSource

A ProxyNDSource that serves the chunks – and the single blocks – of a Blosc2 frame it can read byte ranges of, instead of transferring the whole container. It knows the frame format and nothing about where the frame lives: subclasses supply read_range(offset, size) and nothing else. FsspecNDSource reads through fsspec, and C2Array reads over HTTP ranges from a Caterva2 server. For other sources, see ProxyNDSource and ProxySource.

class blosc2.ByteRangeNDSource(urlpath: str, max_concurrency: int = 8, traffic: Traffic | None = None)

A Proxy source that serves parts of a remote Blosc2 frame.

The frame stays where it is: only its header, its chunk offsets, and what a slice actually touches ever cross the network.

A chunk large enough to be worth taking apart is read block by block – chunk_layout() fetches the offsets of its blocks, block_plan() turns the wanted ones into as few range reads as they fit in – so a slice landing in a corner of a multi-megabyte chunk costs a few kilobytes. Small chunks, memcpyed ones and run-length ones come whole, since there is nothing to save there; wants_blocks() decides which is which without reading anything.

Everything above is the Blosc2 frame format and nothing else, so a subclass only has to say how to read bytes: read_range() is the one abstract method, and the transport behind it decides nothing about the rest. FsspecNDSource reads them with fsspec, and C2Array reads them over HTTP ranges from a Caterva2 server, carrying its auth cookie.

A subclass sets its transport up first and then calls this constructor, which reads the frame’s header through it – one small read, and everything an open decides: that this is a frame at all, and one holding an NDArray. Where the chunks are waits for the first one anything asks about, so a Proxy over a cache that already holds the slice wanted opens the source without ever fetching its index. It may also set a stamp, anything that names the exact bytes it reads, so that Proxy can tell a cache built from other bytes.

Contiguous frames carrying a b2nd metalayer only, which is what blosc2.asarray() and friends write to a single file. Sparse frames and .b2d stores are directories, and cannot be read this way.

Parameters:

• urlpath (str) – Where the frame is, for error messages and for the caller to read back.

• max_concurrency (int, optional) – How many fetches the enclosing Proxy may run at once. Every chunk or block costs one range request, so against an object store a slice is almost entirely round-trip latency and overlapping the requests is the whole win. Defaults to 8, the same figure Proxy.afetch() uses for remote sources. Pass 1 for a protocol with no latency to hide, where the thread pool costs about 10 microseconds per chunk and saves nothing.

Attributes:

blocks

The block shape of the source.

chunks

The chunk shape of the source.

cparams

The compression parameters of the source.

dtype

The dtype of the source.

shape

The shape of the source.

stamp

Methods

aget_chunk(nchunk)	Same as get_chunk(), but without blocking the caller's event loop.
block_plan(nchunk, nblocks)	The range reads that cover nblocks, near-adjacent ones merged.
chunk_layout(nchunk)	Read where the blocks of a chunk are: its header, bstarts and extents.
chunk_layouts(nchunks)	chunk_layout() for several chunks, in as few requests as they fit.
get_chunk(nchunk)	Return the compressed chunk in self.
invalidate_index()	Forget where the chunks and blocks are, so the next read looks again.
read_range(offset, size)	The bytes at [offset, offset + size) of the frame.
read_ranges(spans)	The bytes of every (offset, size) in spans, in that order.
wants_blocks(nchunk, nwanted[, wave, nruns])	Whether fetching nwanted blocks of a chunk beats fetching all of it.
written_chunks()	Which chunks of the frame hold content, as a boolean per chunk.

async aget_chunk(nchunk: int) → bytes

Same as get_chunk(), but without blocking the caller’s event loop.

This is what makes Proxy.afetch() worth using against an object store, where a slice spanning many chunks is nearly all round-trip latency.

The fetch goes to a worker thread rather than being awaited directly. Awaiting an async filesystem’s coroutine looks like the obvious thing to do and does not work: fsspec drives those on a private event loop of its own, so a client created there and awaited here raises “got Future attached to a different loop” (seen with s3fs). Its blocking API is the supported way in, and it hands off to that same private loop, so the thread parks on a queue rather than on a socket.

block_plan(nchunk: int, nblocks: Sequence[int]) → list[tuple[int, int, tuple]]

The range reads that cover nblocks, near-adjacent ones merged.

Each is (offset, size, members) in frame coordinates, where members says which block each piece of the answer is, as (nblock, offset within the read, size).

chunk_layout(nchunk: int) → tuple[bytes, ndarray, ndarray] | None

Read where the blocks of a chunk are: its header, bstarts and extents.

One range read, of a size known in advance since every chunk holds the same number of blocks. None for a chunk this cannot take apart, which the header is what says:

• a chunk of a single block is its own block, and a memcpyed one stores its blocks raw with no bstarts at all;

• a chunk that is a run of one value is its header and that value, with no blocks in the file: blosc2.full writes those at a real offset, unlike the runs of zeros the frame keeps in the offsets themselves;

• a chunk compressed against a codec dictionary keeps it between bstarts and the streams, and _splice_chunk would drop it while leaving the flag that promises it;

• a variable-length-block chunk does not use the zero-length stream that stands in for a block _splice_chunk does not have;

• a chunk without the extended header keeps its bstarts somewhere else entirely, so reading them at byte 32 would be reading data.

Each of those is then fetched whole, at the cost of the one header read that found out.

chunk_layouts(nchunks: Sequence[int]) → list

chunk_layout() for several chunks, in as few requests as they fit.

One request each unless the transport takes several ranges at once, and none at all for a chunk already read: a fetch asks for layouts only where blocks are missing, but the same chunk comes up again as a slice fills it in.

get_chunk(nchunk: int) → bytes

Return the compressed chunk in self.

Parameters:

nchunk (int) – The unidimensional index of the chunk to retrieve.

Returns:

out – The compressed chunk.

Return type:

bytes object

invalidate_index() → None

Forget where the chunks and blocks are, so the next read looks again.

The frame’s offsets move whenever it is written to: a chunk written into a slot that held no content is appended past the old offsets block, which the new one is then written after. Chunks already placed keep their offsets – that is what makes an append-only fill cheap to read alongside – but the index as a whole has to be read again to see the slot that was filled, and the header with it, since the frame’s length and its payload extent are what the offsets are found through.

Nothing is read here: the next lookup pays for it, so a writer that never reads back spends no request on this at all.

Only for a handle that writes, or that follows a frame someone else is writing. A frame that nobody mutates never needs this.

abstractmethod read_range(offset: int, size: int) → bytes

The bytes at [offset, offset + size) of the frame.

The whole of the transport: everything else here is the frame format. Fewer bytes may come back only at the end of the frame; anything else is an error, since the caller has no way to ask for the rest. Must be safe to call from several threads at once, which is what lets Proxy overlap the fetches of one slice.

read_ranges(spans: Sequence[tuple[int, int]]) → list[bytes]

The bytes of every (offset, size) in spans, in that order.

One request each, unless a transport that can carry several ranges in one overrides this and raises max_ranges to say how many. Nothing else has to change for it: this is the only method a batching transport needs, and everything that reads bytes goes through it.

wants_blocks(nchunk: int, nwanted: int, wave: Mapping[int, int] | None = None, nruns: int | None = None) → bool

Whether fetching nwanted blocks of a chunk beats fetching all of it.

Answered without reading anything, so a chunk that says no costs exactly what it costs today: the number of blocks a slice touches is geometry, and an upper bound on the chunk’s compressed size is already in hand from the frame’s offsets. See the thresholds at the top of this module.

wave is the whole fetch this chunk belongs to, {nchunk: nwanted}, which a transport that batches ranges is asked with; see _wave_saves() for what it is used for and why. nruns is how many ranges those blocks will coalesce into, which is what they cost where every range is its own request.

written_chunks() → ndarray

Which chunks of the frame hold content, as a boolean per chunk.

False only for a chunk that was never written: a frame keeps those in their offset rather than in the file, tagged as uninitialized, which is what blosc2.uninit fills an array with. Everything else is True, a run of zeros included – a writer that stored an all-zero chunk stored something, and the tag says so, which is the whole reason to pre-size an array with uninit rather than with zeros.

One range read of the frame’s offsets, and none at all once they have been read: this is the same index every chunk read goes through. So the progress of an array being filled is legible from the bytes a reader already fetches, without asking the server anything about it.

property blocks: tuple

The block shape of the source.

property chunks: tuple

The chunk shape of the source.

property dtype: dtype

The dtype of the source.

max_ranges = 1

How many ranges one request of this transport may carry.

One means one request each, which is all any object store offers. A server answering multipart/byteranges takes more – see read_ranges() – and then a slice costs a couple of requests rather than a couple per chunk it touches.

serves_blocks = True

That blocks are worth asking this source for, which reading a frame is.

The frame is there to be read in pieces – that is what an open of one settles – so a Proxy over it goes straight to the block path. A source that only sometimes serves blocks (C2Array, whose server may compute the dataset rather than store it) overrides this.

property shape: tuple

The shape of the source.

wants_wave = True

That wants_blocks() takes the wave, and wants to be given it.

The implementation here does, and weighs a shared round trip against what the whole fetch skips. A subclass that overrides wants_blocks with a two-argument one sets this back to False, and is then called with two.

When a range read is refused

A transport that reads byte ranges may be answered with something other than the bytes asked for: a server that now streams the dataset, a server that is too busy to serve it, a body that cannot be taken apart. Those raise blosc2.proxy_source.NotRanged, which a Proxy catches for itself – whatever the fetch is still missing comes as whole chunks – and which a caller reading ranges directly can catch by name.

class blosc2.proxy_source.NotRanged(message: str, status: int | None = None)

A transport that reads byte ranges answered with something other than one.

Raised out of ByteRangeNDSource.read_range() and its neighbours when the answer is not the bytes that were asked for: an HTTP 200 carrying the whole dataset, a busy server, a body that cannot be taken apart. It is not fatal to a fetch – Proxy.fetch() catches it and asks for the chunks it wanted whole, which every source can serve – so what it costs is the block granularity, not the data.

A ValueError, which is what a source that cannot be read in ranges raised before this had a type of its own.

Attributes:

args

transient

Whether asking again could be answered differently.

Methods

add_note(object, /)	Exception.add_note(note) -- add a note to the exception
with_traceback(object, /)	Exception.with_traceback(tb) -- set self.__traceback__ to tb and return self.

property transient: bool

Whether asking again could be answered differently.

A 200 is the dataset itself, streamed, and no amount of asking again will make it a file; a server that is busy or broken says nothing at all about how the dataset is served, and costs no download to ask twice.

class blosc2.proxy_source.PartsMissing(message: str, status: int | None = None)

A multi-range answer did not carry all the bytes that were asked for.

A NotRanged for the caller that only wants to know the read failed, and its own type for the transport, which answers it by asking for one range at a time rather than by giving up on ranges.

Attributes:

args

transient

Whether asking again could be answered differently.

Methods

add_note(object, /)	Exception.add_note(note) -- add a note to the exception
with_traceback(object, /)	Exception.with_traceback(tb) -- set self.__traceback__ to tb and return self.