Roadmap to 4.0#

This document lists the main goals for the C-Blosc2 4.x series. Most of them are features that were originally planned for 3.0 but were deferred (see ROADMAP-TO-3.0.rst for the historical record). Contributions towards any of them are very welcome.

New Features in C-Blosc2#

  • Move lazy expressions to C-Blosc2: right now, lazy expressions are only supported in the Python wrapper (python-blosc2). Moving them to C-Blosc2 would allow other wrappers (Java, R, Julia…) to benefit from this powerful feature.

  • Optimization for multi-socket machines: right now, C-Blosc2 is optimized for single-socket machines. However, in multi-socket machines, memory access is not uniform (NUMA architecture), so optimizations are needed to make sure that every thread is accessing to local memory as much as possible. This would require to use e.g. numactl or libnuma so as to pin threads and memory allocations to the local socket.

  • Support for GPUs: nowadays, GPUs are becoming more and more powerful, and having support for them in C-Blosc2 would be a great addition. The idea is to offload the compression, but most importantly, decompression tasks to the GPU, so that the CPU is free to do other tasks. This would require to use e.g. CUDA or ROCm so as to access to the GPU capabilities.

  • ACID version of the frame format: right now, frames are not ACID compliant (Atomicity, Consistency, Isolation and Durability). This means that if a frame is being written and the process is interrupted (e.g. power failure, crash…), the frame may be left in an inconsistent state. Making frames ACID compliant would require to implement a journaling mechanism so that changes are first written to a journal before being applied to the frame. This would ensure that frames are always in a consistent state, even in the event of a failure. The 3.2 series already added related (but weaker) guarantees: growth-SWMR (single writer, multiple readers) support and optional file locking for multi-process access.

  • Framed version of the TreeStore: the TreeStore is a data structure developed in Python-Blosc2. It is meant to be a hierarchical data structure that allows to store and access data in a tree-like fashion. Making it framed would allow to transfer it more easily between different machines, as well as storing it in memory.

General improvements#

  • Improve the safety of the library: even if we have already made a long way in improving our safety, mainly thanks to the efforts of Nathan Moinvaziri, we take safety seriously, so this is always a work in progress.

  • Checksums: the frame can benefit from having a checksum per every chunk/index/metalayer. This will provide more safety towards frames that are damaged for whatever reason. Also, this would provide better feedback when trying to determine the parts of the frame that are corrupted. Candidates for checksums can be the xxhash32 or xxhash64, depending on the goals (to be decided).

  • Multiple index chunks in frames: right now, only one chunk is allowed for indexing other chunks. Provided the 2GB limit for a chunksize, that means that ‘only’ 256 million of chunks can be stored in a frame. Allowing for more than one index chunk would overcome this limitation.

  • More robust detection of CPU capabilities: although currently this detection is quite sophisticated, the code responsible for that has organically grown for more than 10 years and it is time to come with a more modern and robust way of doing this. google/cpu_features may be a good helper for doing this refactoring.

  • Documentation: utterly important for attracting new users and making the life easier for existing ones. Besides keeping the quality of the API docstrings, more documentation materials should be provided, like tutorials or a book about Blosc (or at least, the beginnings of it).