Common Interface for Remote and Local Datasets

Francesc Alted — Wed, 30 Sep 2026 10:45:00 GMT

I am happy to announce that the new Python-Blosc2 4.14 brings a common interface to remote Parquet tables, Zarr and HDF5 arrays, PyTables tables, and indeed, native Blosc2 containers. Most interestingly, a local cache keeps fetched data available for the next request/query; moreover, if cache is persistent, you can reuse the downloaded data throughout different sessions.

TL;DR

One opener for everything: blosc2.open() provides a unified interface across remote Parquet, Zarr, HDF5, PyTables, and native Blosc2 datasets over HTTP, S3, or any fsspec filesystem.
Zero full downloads: Stream only the exact chunks, row groups, or physical columns your queries touch—no need to pull gigabytes of data to inspect a slice.
Smart persistent caching: Passing cache_dir=... automatically caches fetched data locally across sessions and processes, eliminating redundant network roundtrips.
Index-accelerated remote queries: Remote queries automatically exploit persisted PyTables and Blosc2 indexes without fetching entire columns or rebuilding indexes locally.
Ecosystem interoperability: Combine remote arrays across different formats directly in lazy expressions (b + z * h5), or stream Arrow batches into DuckDB and Polars.

To whet your appetite, here is a practical tour using public datasets on Backblaze B2 (which is fully compatible with S3). Here we are going to exercise the HTTP path, but all the protocols supported by fsspec (like S3, GDrive, GCS, SFTP and many others) should be supported too.

Blosc Home Page (Posts about arrow)

Common Interface for Remote and Local Datasets

TL;DR