Optimization tips

This page collects small idioms that make a measurable difference in speed or memory (often both). Each one is backed by a small benchmark in bench/optim_tips/, which you can run yourself — see the bench/optim_tips README.

Numbers below were measured on an Apple M4 Pro Mac Mini (macOS, Python 3.14); absolute values will differ on your machine, but the direction and rough magnitude of each effect should not.

Build large arrays with blosc2’s own constructors

Constructors like blosc2.arange(), blosc2.linspace() and blosc2.fromiter() fill an NDArray chunk by chunk, using multiple threads. Building the same array in NumPy first and compressing it with asarray() means holding the whole thing uncompressed in memory at once.

# Avoid: materializes the full array in NumPy first
a = blosc2.asarray(np.linspace(0, 1, N))

# Prefer: fills the NDArray chunk by chunk
a = blosc2.linspace(0, 1, N)

blosc2.linspace() vs asarray(np.linspace())

At 200M float64 elements, the constructor was ~1.6x faster and used ~17x less peak memory, and the memory gap widens with array size — the naive path’s memory is O(N), while the constructor’s stays roughly O(chunk size) for compressible enough data. The same applies to arange() and fromiter().

Benchmark for this tip: tip_01_constructors.py

Generate arrays with DSL kernels

The constructors from the previous tip are not magic: internally, blosc2.arange() is a one-line DSL kernel (start + _flat_idx * step) that blosc2 compiles to native code and evaluates chunk by chunk, using multiple threads. The same machinery — a blosc2.dsl_kernel-decorated function handed to blosc2.lazyudf() — is open to you, for any array whose value is a function of its index.

As a worked example, let’s build a random constructor from scratch: hash the element index (here with a classic integer hash), and every chunk can be filled independently, in parallel, reproducibly for a given seed. Blosc2 ships a proper one these days — see Random Functions — but writing a small version by hand is the clearest way to see what the machinery does.

@blosc2.dsl_kernel
def random_int32(seed):
    x = _flat_idx ^ seed
    x = (((x >> 16) ^ x) * 0x45D9F3B) & 0xFFFFFFFF
    x = (((x >> 16) ^ x) * 0x45D9F3B) & 0xFFFFFFFF
    return (x >> 16) ^ x  # full [-2^31, 2^31) range


# Avoid: materializes the full array with NumPy first
rng = np.random.default_rng(42)
a = blosc2.asarray(
    rng.integers(-(2**31), 2**31, size=N, dtype=np.int32), cparams={"clevel": 0}
)

# Prefer: the kernel fills the NDArray chunk by chunk, in parallel
lazy = blosc2.lazyudf(random_int32, (42,), dtype=np.int32, shape=(N,))
a = lazy.compute(cparams={"clevel": 0})

(clevel=0 because random data is incompressible — don’t pay the codec for nothing.)

DSL random kernel vs asarray(rng.integers())

At 200M int32 elements, the DSL kernel was ~3.6x faster and used ~2x less peak memory than generating with NumPy and compressing via asarray() — the peak is just the result itself, since the full NumPy staging array never exists.

The output passes light uniformity checks against NumPy’s PCG64 (the benchmark script prints them) — good enough for synthetic data, benchmarks and test fixtures, but this hand-rolled hash is not a substitute for a real generator. For statistically rigorous work such as Monte Carlo, use blosc2.random, which gives you NumPy-quality streams (one independent SeedSequence per chunk) and keeps the chunk-parallel generation. The benchmark source explains the hash design and the DSL integer-arithmetic rules it relies on; see also the DSL syntax reference.

Benchmark for this tip: tip_11_dsl_random.py

Let @blosc2.jit compile control flow instead of tracing it

@blosc2.jit normally works by tracing: it calls your function once with proxy operands to record a LazyExpr string, so an if/for/while in the body only ever sees one (traced) path — the rest is silently lost. When the body contains control flow and it fits the DSL grammar, jit detects this at decoration time and instead compiles the whole function with the same miniexpr engine that powers @blosc2.dsl_kernel, so every branch and loop runs as written, once per chunk, on NumPy or NDArray operands directly (no conversion copy).

# Without this detection, jit would call the function once, record only the
# branch that one call happened to take, and reuse it for every pixel — not
# a real per-pixel escape-time loop.
@blosc2.jit
def mandelbrot(cr, ci, max_iter):
    zr, zi, n = 0.0, 0.0, 0
    for _ in range(max_iter):
        if zr * zr + zi * zi > 4.0:
            break
        zr, zi = zr * zr - zi * zi + cr, 2 * zr * zi + ci
        n += 1
    return n  # jit detects the control flow and compiles this kernel whole

Functions without control flow always trace, even when they happen to be DSL-valid: tracing plus vectorized ne_evaluate/miniexpr is faster than whole-kernel miniexpr for pure elementwise expressions, so jit only takes the DSL route when tracing would silently lose branches/loops. Use jit(strict=True) to force the DSL route regardless (raises at decoration time if the function can’t be compiled), or jit(strict=False) to force tracing even with control flow (only correct when branches depend on plain Python values, not on the arrays themselves).

Align your reads with the double partition

blosc2 arrays are partitioned twice: the array is split into chunks (the unit of storage and compression), and each chunk is subdivided into blocks (the unit of decompression, sized to fit CPU caches). A read that lands exactly on a partition boundary decompresses only the chunk or block it needs, while the same-sized read shifted off-grid straddles (and decompresses) extra ones.

You don’t have to pick the partitions yourself: let blosc2 choose them, then read them back from arr.chunks and arr.blocks to place your slice boundaries.

At the chunk level

NDArray.slice() has a fast path when both boundaries land on chunk boundaries: whole chunks are copied as-is, compressed, with no decompression at all. Regular reads also benefit — an aligned chunk-sized read decompresses one chunk instead of two.

arr = blosc2.asarray(data)  # let blosc2 pick the partitions
ch = arr.chunks[0]  # e.g. 1000 for a 16000×2000 float64 array

# Avoid: a chunk-sized read straddling two chunks → decompresses both
arr[ch // 2 : ch // 2 + ch, :]

# Aligned read: on the chunk grid → decompresses exactly one chunk
arr[ch : 2 * ch, :]

# Even better: slice() on chunk boundaries → copies chunks as-is, no decompress
arr.slice((slice(ch, 2 * ch), slice(None)))

At the block level

The same alignment principle applies at the block level: a block-aligned read decompresses exactly the blocks it needs. With auto-chosen blocks this effect is small — blocks are tiny by design — but if you configure larger blocks (say, for better compression ratios), keeping reads on the block grid pays off.

The slice() fast path, however, does not apply here: it only works at chunk boundaries, so slice() of a block-sized region still decompresses and recompresses:

big = blosc2.asarray(data, chunks=(4000, 2000), blocks=(100, 2000))
bl = big.blocks[0]

# Avoid: a block-sized read straddling two blocks → decompresses both
big[bl // 2 : bl // 2 + bl, :]

# Aligned read: on the block grid → decompresses exactly one block
big[bl : 2 * bl, :]

# slice() at block granularity → still decompresses + recompresses the chunk
big.slice((slice(bl, 2 * bl), slice(None)))

Aligned vs unaligned reads, chunk and block level

On a 16000×2000 float64 array, 400 chunk-sized reads aligned with the chunk grid were ~2.4× faster than the same reads shifted half a chunk off-grid, and a chunk-aligned slice() was ~7× faster again (no decompression), i.e. ~17× faster than the unaligned read. At the block level (with larger 1.6 MB blocks), 400 block-sized reads aligned with the block grid were ~1.9× faster. However, slice() at block granularity was no faster at all — slice() has to produce a valid compressed array with its own chunk layout, so it can only skip decompression when both boundaries land on chunk boundaries; at block granularity it still decompresses and recompresses.

Practically: reach for slice() when both boundaries land on chunk boundaries and you want a compressed result. Off that path it is strictly more work than arr[...] — and if you wanted NumPy at the end, you have paid to compress something you are about to throw away.

Benchmark for this tip: tip_02_chunk_aligned_slicing.py

Sorted top-k: stream from FULL indexes

A FULL index stores rows in sorted order. When all you need is the top (or bottom) k rows, you can stream just that slice directly from the index sidecar instead of materialising the full sorted permutation. Both CTable and NDArray expose this.

CTable: sort_by(view=True)

CTable.sort_by(view=True) returns a lightweight sorted view that gathers rows from the parent table on demand. On a FULL-indexed column it streams straight from the index — the table is never actually sorted at all:

t.create_index("temperature", kind=blosc2.IndexKind.FULL)

# Avoid: sorts (and copies) the whole table just to keep 10 rows
top10 = t.sort_by("temperature")[:10]

# Prefer: zero-copy view, streamed from the index
top10 = t.sort_by("temperature", view=True)[:10]

CTable.sort_by(view=True) top-10

On a 20M-row table, the view form took ~99× less time and about 25% less peak memory than a full sort_by(). The larger the table relative to k, the bigger the gap, since the naive path’s cost is dominated by sorting rows you’re about to discard.

Benchmark: tip_03_sort_by_view.py

NDArray: iter_sorted(start=-k)

For 1-D NDArray objects, NDArray.iter_sorted(start=-k) reads just the tail of the index sidecar, avoiding the full permutation that argsort() would materialise:

arr.create_index(kind=blosc2.IndexKind.FULL)

# Avoid: materialises all 20M positions just to keep 10
top10 = arr[arr.argsort()[-10:]]

# Prefer: reads only the last 10 entries from the sidecar
top10 = np.fromiter(arr.iter_sorted(start=-10), dtype=arr.dtype)

Descending and bottom-k work via step=-1:

list(arr.iter_sorted(start=-1, stop=-11, step=-1))  # top-10 descending
list(arr.iter_sorted(stop=10))  # bottom-10 ascending
list(arr.iter_sorted(start=9, stop=None, step=-1))  # bottom-10 descending

NDArray.iter_sorted(start=-10) top-10

On a 20M-element float64 array, iter_sorted(start=-10) was ~137× faster and used ~130× less peak memory than argsort()[-10:] — the latter still materialises the full 20M-element positions array even when backed by a FULL index.

Benchmark: tip_03b_ndarray_iter_sorted.py

Let SUMMARY indexes answer min()/max() directly

When closing a CTable, Blosc2 automatically builds SUMMARY indexes (per-block min/max) for its eligible scalar columns — this is on by default (create_summary_index=True). Column.min()/max() (and argmin()/argmax() inside group_by()) then answer from those precomputed summaries instead of decompressing the column at all.

# create_summary_index=True is the default; closing the table builds the index
with blosc2.CTable(Row, urlpath="t.b2d", mode="w") as t:
    t.extend(data)

t = blosc2.open("t.b2d")
hottest = t["temperature"].max()  # answered from the SUMMARY index

Column.max() with vs without a SUMMARY index

On a 10M-row column, the indexed max() took ~5x less time than without an index, and needed essentially no extra memory — it never touches the compressed column data at all.

The same SUMMARY indexes can also let a selective where() query skip whole blocks, but only when the column’s values are ordered or clustered enough that a block’s min/max range can exclude the predicate entirely. With independently random data every block spans nearly the full value range and there is nothing to skip — so the min()/max() speedup is the one you can always count on.

Benchmark for this tip: tip_04_summary_index_where.py

Reduce columns directly — don’t slice them first

t["col"][:] materializes the whole column as one big NumPy array. If all you want is a reduction, call it on the Column itself: sum(), mean(), min(), … work chunk by chunk and never hold the whole column decompressed at once — while still handling null values and deleted rows correctly.

# Avoid: decompresses the whole column into one NumPy array first
total = t["val"][:].sum()

# Prefer: chunk-wise reduction straight over the compressed column
total = t["val"].sum()

col.sum() vs col[:].sum()

On a 250M-row column, col.sum() was ~1.6x faster, but more importantly it used ~62x less peak memory. For large tables the memory savings alone can be the deciding factor.

Benchmark for this tip: tip_05_column_reduce.py

Filtered reductions: push the predicate down with where=

The previous tip extends to filtered aggregates. The NumPy-style idiom — materialize the value column and the predicate column, build a boolean mask, then reduce — decompresses both columns in full just to keep a fraction of the rows. Column reductions accept a where= predicate instead, which is pushed down into the same chunk-wise scan: no intermediate arrays, no filtered view.

# Avoid: decompresses both full columns just to mask one of them
temp = t["temperature"][:]
reg = t["region"][:]
total = temp[reg == 3].sum()

# Prefer: the filter travels with the chunk-wise reduction
total = t["temperature"].sum(where=t.region == 3)

col.sum(where=...) vs NumPy-style masking

On a 20M-row table, the pushed-down form was ~2.1x faster and used ~7x less peak memory. Predicates can combine several columns too: t["amount"].sum(where=(t.price < 300) & (t.qty > 0)).

Benchmark for this tip: tip_08_where_pushdown.py

Memory-map read-only opens

blosc2.open(path, mmap_mode="r") memory-maps the file instead of going through regular file I/O, so chunks are read directly from the mapped pages — no read syscall per block, and no copy out of the page cache into an intermediate buffer. For workloads that touch many scattered chunks, this adds up.

# Avoid (for read-heavy, scattered access): regular I/O per chunk
arr = blosc2.open(path)

# Prefer: map the file once, read pages directly
arr = blosc2.open(path, mmap_mode="r")

open(mmap_mode='r') vs plain open()

Across 8,000 scattered slice reads, mmap_mode="r" was ~1.2x faster; peak memory was essentially identical for this single-process, single-open workload.

A single warm-cache process, as benchmarked here, is actually the worst case for showing mmap off — the payoff multiplies with several readers on the same file, which is the next tip.

Benchmark for this tip: tip_06_mmap_read.py

Many readers on one file? mmap in every one of them

When several processes read the same blosc2 file concurrently, open it with mmap_mode="r" in each reader. Every regular-I/O access pays a syscall plus a copy from the OS page cache into private buffers, and that per-access overhead compounds under kernel contention — while mmap readers all go straight to a single, shared set of mapped pages. So the speedup grows with the number of readers, and physical memory stays at roughly one copy of the file no matter how many readers attach.

# Avoid: each reader process pays syscalls + private buffer copies
arr = blosc2.open(path)  # reader 1..N

# Prefer: all readers share one set of mapped pages
arr = blosc2.open(path, mmap_mode="r")  # reader 1..N

Concurrent readers: regular I/O vs mmap

With 8 concurrent readers doing random slice reads over a 269 MB array, mmap was ~1.1x faster in wall time and used ~5% less total CPU; with one reader the two are level. Don’t be alarmed if mmap readers look heavier in RSS — each process charges the shared mapped pages it touched, but they exist once physically; per-process private memory is identical in both modes. The full discussion, including the measurement table, is in the Sharing containers across processes guide’s colophon; see that guide too for the multi-reader/NFS/Windows caveats.

That margin was much wider before C-Blosc2 3.3.1, which stopped reopening the frame file on every chunk access: the regular path caught up, mmap did not regress.

Benchmark for this tip: tip_10_mmap_many_readers.py

Skip constraint checks in extend() with validate=False

You can pass a NDArray directly as a column value to CTable.extend(): both the write and the constraint validation happen chunk by chunk, so the array is never fully decompressed — it goes from compressed source to compressed column with only O(chunk) extra memory. Columns with no declared constraints skip validation automatically.

But for a column that does declare constraints (ge=, max_length=, …), validation still has to decompress and check every chunk; if you already know the data is valid, validate=False skips that pass.

# Default: every chunk is decompressed once to check declared constraints
t.extend({"val": src})

# Prefer, for known-good data: skip the constraint checks entirely
t.extend({"val": src}, validate=False)

CTable.extend(validate=False)

Extending a table with a 20M-row NDArray column carrying a ge=0 constraint, validate=False was ~1.4x faster; peak memory was within ~10%, since validation is chunk-wise anyway.

Benchmark for this tip: tip_07_chunked_writes.py