Expression Utilities

A series of utilities are provided to work with expressions in a more convenient way.

Functions

blosc2.evaluate(ex: str, local_dict: dict | None = None, global_dict: dict | None = None, out: Array = None, **kwargs: Any) → Array[source]

Evaluate a string expression using the Blosc2 compute engine.

This is a drop-in replacement for numexpr.evaluate(), but using the Blosc2 compute engine. This allows for:

  1. Use more functionality (e.g. reductions) than numexpr.

  2. Follow casting rules of NumPy more closely.

  3. Use both NumPy arrays and Blosc2 NDArrays in the same expression.

As NDArrays can be on-disk, the expression can be evaluated without loading the whole array into memory (i.e. using an out-of-core approach).

Parameters:

  • ex (str) – The expression to evaluate.

  • local_dict (dict, optional) – The local dictionary to use when looking for operands in the expression. If not provided, the local dictionary of the caller will be used.

  • global_dict (dict, optional) – The global dictionary to use when looking for operands in the expression. If not provided, the global dictionary of the caller will be used.

  • out (blosc2.Array, optional) – The output array where the result will be stored. If not provided, a new NumPy array will be created and returned.

  • kwargs (Any, optional) – Additional arguments to be passed to numexpr.evaluate() function.

Returns:

out – The result of the expression evaluation. If out is provided, the result will be stored in out and returned at the same time.

Return type:

blosc2.Array

Examples

>>> import blosc2
>>> import numpy as np
>>> dtype = np.float64
>>> shape = [3, 3]
>>> size = shape[0] * shape[1]
>>> a = np.linspace(0, 5, num=size, dtype=dtype).reshape(shape)
>>> b = blosc2.linspace(0, 5, num=size, dtype=dtype, shape=shape)
>>> expr = 'a * b + 2'
>>> out = blosc2.evaluate(expr)
>>> out
[[ 2.        2.390625  3.5625  ]
[ 5.515625  8.25     11.765625]
[16.0625   21.140625 27.      ]]
blosc2.get_expr_operands(expression: str) → set[source]

Given an expression in string form, return its operands.

Parameters:

expression (str) – The expression in string form.

Returns:

A set of operands found in the expression.

Return type:

set

blosc2.validate_expr(expr: str) → None[source]

Validate expression for forbidden syntax and valid method names.

Parameters:

expr (str) – The expression to validate.

Return type:

None

Decorators

blosc2.jit(func=None, *, out=None, disable=False, strict=None, **kwargs)[source]

Prepare a function so that it can be used with the Blosc2 compute engine.

The inputs of the function can be any combination of NumPy/NDArray arrays and scalars. By default, the function is traced: it is called once with the NumPy arrays replaced by SimpleProxy objects (NDArray objects are used as is) to record a single expression, which is then what actually gets evaluated. Because tracing only calls the function once, an if/for/ while in the body only ever takes the one path that single call happened to follow — see strict below for when jit instead compiles the function whole, so every branch and loop genuinely runs.

The returned value will be a NDArray if a storage kwarg is provided (e.g. cparams=, chunks=, urlpath= — anything that only makes sense for a compressed/persisted container). Else, the return value will be a NumPy array (if the function returns a NumPy array). Execution-tuning kwargs (jit=, jit_backend=, fp_accuracy=) do not by themselves trigger this — they take effect either way, without changing the return type. If out is provided, the result will be computed and stored in the out array.

Parameters:

  • func (callable) – The function to be prepared for the Blosc2 compute engine.

  • out (np.ndarray, NDArray, optional) – The output array where the result will be stored. On the DSL (control-flow) dispatch route, a NumPy out is filled in place (directly when C-contiguous, else via a copy); an NDArray out is not supported there — use compute(urlpath=..., mode="w") instead.

  • disable (bool, optional) – If True, the decorator is disabled and the original function is returned unchanged. Default is False.

  • strict (bool, optional) –

    Control which evaluation route is used:

    • None (default): if func’s body contains an if/for/while and it compiles as a DSL kernel, dispatch to the DSL route (miniexpr runs the whole function, so branches/loops behave as written); a control-flow function that fails DSL extraction still falls back to tracing, but a subsequent tracing failure is annotated with the DSL extraction error. Functions without control flow always trace, even if they happen to be DSL-valid (tracing is faster for pure elementwise expressions).

    • True: always use the DSL route, raising DSLSyntaxError at decoration time if func’s source cannot be parsed as a DSL kernel. Note the guarantee is exactly that – parsing – and not that the kernel will compile: a function that is DSL-shaped but calls something miniexpr does not implement passes here and fails later, at call time, with a RuntimeError. See the DSL syntax reference for what the grammar accepts. (Unrelated to blosc2.dsl_kernel(), which builds a DSLKernel object rather than an evaluating wrapper.)

      This also works as a pandas engine, which is the only way to reach strict through that entry point: df.apply(f, engine=blosc2.jit(strict=True)).

    • False: always use the tracing route, even if func has control flow (this only works when branches/loops depend on plain Python values, not on traced arrays).

  • **kwargs (dict, optional) – Additional keyword arguments supported by the empty() constructor.

Return type:

wrapper

Notes

  • Although many NumPy functions are supported, some may not be implemented yet. If you find a function that is not supported, please open an issue.

  • out and kwargs parameters are not supported for all expressions (e.g. when using a reduction as the last function). In this case, you can still use the out parameter of the reduction function for some custom control over the output.

  • DSL-route kernels do not support broadcasting: every array argument must share the same shape.

Examples

>>> import numpy as np
>>> import blosc2
>>> @blosc2.jit
... def compute_expression(a, b, c):
...     return np.sum(((a ** 3 + np.sin(a * 2)) > 2 * c) & (b > 0), axis=1)
>>> a = np.arange(20, dtype=np.float32).reshape(4, 5)
>>> b = np.arange(20).reshape(4, 5)
>>> c = np.arange(5)
>>> compute_expression(a, b, c)
array([3, 5, 5, 5])

With strict=True the function is compiled as a DSL kernel, so a real per-element if runs as written – only the matching arm is evaluated:

>>> @blosc2.jit(strict=True)
... def clamp(x):
...     if x < 0.0:
...         out = 0.0
...     else:
...         out = x
...     return out
>>> clamp(np.array([-1.5, 2.0, -0.5]))
array([0., 2., 0.])

The guarantee is that the source parses as DSL, checked at decoration time. A body the grammar does not accept is rejected right away, rather than silently falling back to tracing:

>>> @blosc2.jit(strict=True)
... def not_dsl(x):
...     return np.where(x >= 0, x.mean(), x)
Traceback (most recent call last):
    ...
blosc2.dsl_kernel.DSLSyntaxError: Unsupported call target in DSL ...
blosc2.lazywhere(value1=None, value2=None)[source]

Decorator to apply a where condition to a LazyExpr.