Proxy¶

Class that implements a proxy (with cache support) of a Python-Blosc2 container.

This can be used to cache chunks of regular data container which follows the ProxySource or ProxyNDSource interfaces.

class blosc2.Proxy(src: ProxySource, urlpath: str | None = None, mode='a', **kwargs: dict)[source]¶

Proxy (with cache support) for an object following the ProxySource interface.

This can be used to cache chunks of a regular data container which follows the ProxySource or ProxyNDSource interfaces.

Attributes:

blocks: The blocks of self or None if the data is not a Blosc2 NDArray
chunks: The chunks of self or None if the data is not a Blosc2 NDArray
cparams: The compression parameters of the cache
device: Hardware device the array data resides on.
dtype: The dtype of self or None if the data is unidimensional
fields: Dictionary with the fields of self.
info: The info of the cache
ndim: Get the number of dimensions of the Operand.
schunk: The SChunk of the cache
shape: The shape of self
traffic: What this proxy has read off its source, or None for a local one.
vlmeta: Get the vlmeta of the cache.

Methods

`afetch`([item, max_concurrency])	Retrieve the cache container with the requested data updated asynchronously.
`argmax`([axis, keepdims])	Returns the indices of the maximum values along a specified axis.
`argmin`([axis, keepdims])	Returns the indices of the minimum values along a specified axis.
`cumulative_prod`([axis, dtype, include_initial])	Calculates the cumulative product of elements in the input array ndarr.
`cumulative_sum`([axis, dtype, include_initial])	Calculates the cumulative sum of elements in the input array ndarr.
`fetch`([item, max_concurrency])	Get the container used as cache with the requested data updated.
`item`()	Copy an element of an array to a standard Python scalar and return it.
`to_device`(device)	Copy the array from the device on which it currently resides to the specified device.
`where`([value1, value2])	Select `value1` or `value2` values based on `True`/`False` for `self`.

Special Methods:

`__init__`(src[, urlpath, mode])	Create a new Proxy to serve as a cache to save accessed chunks locally.
`__getitem__`(item)	Get a slice as a numpy.ndarray using the Proxy.

Constructor¶

__init__(src: ProxySource, urlpath: str | None = None, mode='a', **kwargs: dict)[source]¶

Create a new Proxy to serve as a cache to save accessed chunks locally.

Parameters:

src¶ (ProxySource or ProxyNDSource) – The original container.
urlpath¶ (str, optional) – The urlpath where to save the container that will work as a cache.
mode¶ (str, optional) –
“a” means read/write (create if it doesn’t exist); “w” means create (overwrite if it exists). Default is “a”.

With “a” and an existing urlpath, the cache written by an earlier run is adopted as is, so whatever it already holds is not fetched from the source again. It must be a cache from a proxy over a source of the same geometry (shape, dtype and partitioning); anything else raises rather than being silently reused or overwritten.

A source that can name the exact bytes it reads, as FsspecNDSource does with its stamp (fsspec’s token) and C2Array with the server’s mtime, is checked against that too: a cache built from different bytes raises, even when the geometry still fits. For every other source geometry is all there is to check, so a source whose contents changed underneath while its geometry did not is adopted, and the cache keeps serving what the earlier run fetched; pass mode="w" when that source may have been rewritten.

A proxy that blosc2.open() rebuilds over its own cache never comes through here – the cache is handed to it as it stands – so there a stamp that no longer matches empties the cache instead of raising: it starts as though nothing had been fetched and fills again from the bytes served now. Opened read-only there is nothing to empty, and every read falls through to the source.
kwargs¶ (dict, optional) –
Keyword arguments supported:

vlmeta: dict or None

A dictionary with different variable length metalayers. One entry per metalayer:

key: bytes or str
The name of the metalayer.

value: object
The metalayer object that will be serialized using msgpack.

Any other keyword argument (e.g. contiguous) is forwarded to the cache container constructor (blosc2.empty() or SChunk), so callers can request e.g. a sparse (non-contiguous) cache without resorting to the _cache= escape hatch.

Utility Methods¶

__getitem__(item: slice | list[slice]) → ndarray[source]¶

Get a slice as a numpy.ndarray using the Proxy.

Parameters:: item¶ (slice or list of slices) – The slice of the desired data.
Returns:: out – An array with the data slice.
Return type:: numpy.ndarray

Examples

>>> import numpy as np
>>> import blosc2
>>> data = np.arange(25).reshape(5, 5)
>>> ndarray = blosc2.asarray(data)
>>> proxy = blosc2.Proxy(ndarray)
>>> proxy[0:3, 0:3]
[[ 0  1  2]
[ 5  6  7]
[10 11 12]
[20 21 22]]
>>> proxy[2:5, 2:5]
[[12 13 14]
[17 18 19]
[22 23 24]]

async afetch(item: slice | list[slice] | None = (), max_concurrency: int | None = None) → NDArray | SChunk[source]¶

Retrieve the cache container with the requested data updated asynchronously.

Parameters:

item¶ (slice or list of slices, optional) – If provided, only the chunks intersecting with the specified slices will be retrieved if they have not been already.
max_concurrency¶ (int, optional) – Maximum number of aget_chunk calls to have in flight at once (semaphore-bounded, so a slice spanning thousands of chunks doesn’t fire thousands of concurrent requests at the source). Defaults to 1 (serial, as before) for most sources, and to a higher value for remote sources such as C2Array where concurrency turns N x round-trip latency into roughly 1 x round-trip.

Returns:

out – The local container used to cache the already requested data.

Return type:

NDArray or SChunk

Notes

This method is only available if the ProxySource or ProxyNDSource have an async aget_chunk method.

Whole chunks, even from a source that can serve single blocks: a chunk the cache holds only part of is fetched in full here, where fetch() would ask only for the blocks it is missing.

Examples

>>> import numpy as np
>>> import blosc2
>>> import asyncio
>>> from blosc2 import ProxyNDSource
>>> class MyProxySource(ProxyNDSource):
>>>     def __init__(self, data):
>>>         # If the next source is multidimensional, it must have the attributes:
>>>         self.data = data
>>>         f"Data shape: {self.shape}, Chunks: {self.chunks}"
>>>         f"Blocks: {self.blocks}, Dtype: {self.dtype}"
>>>     @property
>>>     def shape(self):
>>>         return self.data.shape
>>>     @property
>>>     def chunks(self):
>>>         return self.data.chunks
>>>     @property
>>>     def blocks(self):
>>>         return self.data.blocks
>>>     @property
>>>     def dtype(self):
>>>         return self.data.dtype
>>>     # This method must be present
>>>     def get_chunk(self, nchunk):
>>>         return self.data.get_chunk(nchunk)
>>>     # This method is optional
>>>     async def aget_chunk(self, nchunk):
>>>         await asyncio.sleep(0.1) # Simulate an asynchronous operation
>>>         return self.data.get_chunk(nchunk)
>>> data = np.arange(20).reshape(4, 5)
>>> chunks = [2, 5]
>>> blocks = [1, 5]
>>> data = blosc2.asarray(data, chunks=chunks, blocks=blocks)
>>> source = MyProxySource(data)
>>> proxy = blosc2.Proxy(source)
>>> async def fetch_data():
>>>     # Fetch a slice of the data from the proxy asynchronously
>>>     slice_data = await proxy.afetch(slice(0, 2))
>>>     # Note that only data fetched is shown, the rest is uninitialized
>>>     slice_data[:]
>>> asyncio.run(fetch_data())
>>> # Using getitem to get a slice of the data
>>> result = proxy[1:2, 1:3]
>>> f"Proxy getitem: {result}"
Data shape: (4, 5), Chunks: (2, 5)
Blocks: (1, 5), Dtype: int64
[[0 1 2 3 4]
[5 6 7 8 9]
[0 0 0 0 0]
[0 0 0 0 0]]
Proxy getitem: [[6 7]]

fetch(item: slice | list[slice] | None = (), max_concurrency: int | None = None) → NDArray | SChunk[source]¶

Get the container used as cache with the requested data updated.

Parameters:

item¶ (slice or list of slices, optional) – If not None, only the chunks that intersect with the slices in items will be retrieved if they have not been already.
max_concurrency¶ (int, optional) – Maximum number of get_chunk calls to run at once, in a thread pool. Only worth raising for sources whose fetches are dominated by round-trip latency, and only safe for sources whose get_chunk is thread-safe, such as FsspecNDSource. Defaults to the source’s own max_concurrency attribute if it has one, else 1 (serial).

Returns:

out – The local container used to cache the already requested data.

Return type:

NDArray or SChunk

Notes

A source that can serve individual blocks (FsspecNDSource) is asked only for the blocks the slice touches, rather than for whole chunks, whenever that is the cheaper way round – see the thresholds in blosc2.proxy_source. The chunks left in the cache then hold only those blocks, and read as zeros elsewhere until the rest are fetched. A source that stops answering range reads partway (a server that now computes the dataset, or is too busy to serve it from its file) does not fail the fetch: the chunks it was asked for come whole instead.

Examples

>>> import numpy as np
>>> import blosc2
>>> data = np.arange(20).reshape(10, 2)
>>> ndarray = blosc2.asarray(data)
>>> proxy = blosc2.Proxy(ndarray)
>>> slice_data = proxy.fetch((slice(0, 3), slice(0, 2)))
>>> slice_data[:3, :2]
[[0 1]
[2 3]
[4 5]]

property blocks: tuple[int]¶: The blocks of self or None if the data is not a Blosc2 NDArray

property chunks: tuple[int]¶: The chunks of self or None if the data is not a Blosc2 NDArray

property cparams: CParams¶: The compression parameters of the cache

property dtype: dtype¶: The dtype of self or None if the data is unidimensional

property fields: dict¶

Dictionary with the fields of self.

Returns:: fields – A dictionary with the fields of the Proxy.
Return type:: dict