Blosc Home Page (Posts by Marta Iborra)

User Defined Pipeline for Python-Blosc2

Marta Iborra — Thu, 15 Dec 2022 08:00:20 GMT

The Blosc Development Team is happy to announce that the latest version of Python-Blosc2 allows user-defined Python functions all throughout its compression pipeline: you can use Python for building prefilters, postfilters, filters, codecs for Blosc2 and explore all its capabilities. And if done correctly (by using e.g. NumPy, numba, numexpr...), most of the time you won't even need to translate those into C for speed.

The Blosc2 pipeline

The Blosc2 pipeline includes all the functions that are applied to the data until it is finally compressed (and decompressed back). As can be seen in the image below, during compression the first function that can be applied to the data is the prefilter (if any), then the filters pipeline (with a maximum of six filters) and, last but not least, the codec itself. For decompressing, the order will be the other way around: first the codec, then the filters pipeline and finally the postfilter (if any).

Defining prefilters and postfilters

A prefilter is a function that is applied to the SChunk each time you add data into it (e.g. when appending or updating). It is executed for each data block and receives three parameters: input, output and offset. For convenience, the input and output are presented as NumPy arrays; the former is a view of the input data and the later is an empty NumPy array that must be filled (this is actually what the first filter in the filters pipeline will receive). Regarding the offset, it is an integer which indicates where the corresponding block begins inside the SChunk container.

You can easily set a prefilter to a specific SChunk through a decorator. For example:

schunk = blosc2.SChunk()
@schunk.prefilter(np.int64, np.float64)
def pref(input, output, offset):
    output[:] = input - np.pi + offset

This decorator requires the data types for the input (original data) and output NumPy arrays, which must be of same itemsize.

If you do not want the prefilter to be applied anymore, you can always remove it:

schunk.remove_prefilter("pref")

As for the postfilters, they are applied at the end of the pipeline during decompression. A postfilter receives the same parameters as the prefilter and can be set in the same way:

@schunk.postfilter(np.float64, np.int64)
def postf(input, output, offset):
    output[:] = input + np.pi - offset

In this case, the input data is the one from the buffer returned by the filter pipeline, and the output data type should be the same as the original data (for a good round-trip).

You can also remove postfilters whenever you want:

schunk.remove_postfilter("postf")

Fillers

Before we move onto the next step in the pipeline, we need to introduce the fillers. A filler is similar to a prefilter but with a twist. It is used to fill an empty SChunk and you can pass to it any extra parameter you want, as long as it is a NumPy array, SChunk or Python Scalar. All these extra parameters will arrive to the filler function as a tuple containing just the corresponding block slice for each parameter (except for the scalars, that are passed untouched). To declare a filter, you will also need to pass the inputs along with its data type. For example:

@schunk.filler(((schunk0, dtype0), (ndarray1, dtype1), (py_scalar3, dtype2)), schunk_dtype)
def filler(inputs_tuple, output, offset):
    output[:] = inputs_tuple[0] - inputs_tuple[1] * inputs_tuple[2]

This will automatically append the data to the schunk, but applying the filler function first. After that the schunk would be completely filled, the filler will be de-registered, so the next time you update some data the it would not be executed; a filler is meant to build new SChunk objects from other containers.

User-defined filters and codecs

The main difference between prefilters/postfilters and their filters/codecs counterparts is that the former ones are meant to run for an specific SChunk instance, whereas the later can be locally registered and hence, used in any SChunk.

Let's start describing the user-defined filters. A filter is composed by two functions: one for the compression process (forward), and another one for the decompression process (backward). Such functions receive the input and output as NumPy arrays of type uint8 (bytes), plus the filter meta and the SChunk instance to which the data belongs to. The forward function will fill the output with the modified data from input. The backward will be responsible of reversing the changes done by forward so that the data returned at the end of the decompression should be the same as the one received at the beginning of the compression. Check the drawing below:

So, once we have the pair of forward and backward functions defined, they can be registered locally associating to a filter ID between 160 and 255:

blosc2.register_filter(id, forward, backward)

Now, we can use the user-defined filter in any SChunk instance by choosing the new local ID in the filters pipeline:

schunk.cparams = {"filters": [id], "filters_meta": [meta]}

Regarding the user-defined codecs, they do not differ too much from its filter counterparts. The main difference is that, because their goal is to actually compress data, the corresponding functions (in this case encoder and decoder) will need to return the size in bytes of the compressed/decompressed data respectively. This time the scheme would be:

To register a codec, you name it, assign an ID to it and pass the user-defined functions:

blosc2.register_codec(codec_name, id, encoder, decoder)

And to use it you just use its ID in the cparams:

schunk.cparams = {"codec": id, "codec_meta": meta}

Final words

We have seen how easily you can define your own filters and codecs for the Blosc2 compression pipeline. They are very easy to use because they conveniently wrap input and output data as NumPy arrays. Now, you can start experimenting with different filter/compression algorithms straight from Python. You can even come with a library of such filters/codecs that can be used in all your data pipeline processing. Welcome to compression made easy!

See more examples in the repository.

Find the complete documentation at: https://www.blosc.org/python-blosc2/python-blosc2.html.

This work has been made thanks to a Small Development Grant from NumFOCUS. NumFOCUS is a non-profit organization supporting open code for better science. If you like this, consider giving a donation to them (and if you like our work, you can nominate it to our project too!). Thanks!

Wrapping C-Blosc2 in Python (a beginner's view)

Marta Iborra — Mon, 10 May 2021 07:32:20 GMT

An initial release of the Python wrapper for C-Blosc2 is now available in: https://github.com/Blosc/python-blosc2. In this blog I will try to explain some of the most difficult aspects that I had to learn in doing this and how I solved them.

This work is being made thanks to a grant from the Python Software Foundation.

Python views

At university, the first programming language that I learned was Python. But because programming was new for the majority of the class the subject only covered the basics: basic statements and classes. And although these were easy to understand, the views were unknown to me (until now).

To explain what the views are, let’s suppose we have the following code in Python:

>>> import sys
>>> a = []
>>> b = a
>>> sys.getrefcount(a)
3

The reference count for the object is 3: a, b and the argument passed to sys.getrefcount().

Basically, to avoid making copies of a same variable, Python uses views. Every variable has its counter and until the counter is 0, the variable is not deleted. But that means that two threads cannot access the counter at the same time. Because having a lock for every variable would be inefficient and could produce deadlocks (which means that several threads are waiting for each other), the GIL was created. So GIL was my next thing to learn.

GIL and Cython

GIL stands for Global Interpreter Lock. With a single lock on the interpreter there are no deadlocks. But the execution of any Python program must acquire the interpreter lock, which prevents some programs to take advantage of the multi-threading execution.

When writing C extensions, this lock is very useful because it can be released. Thus, the program can be more efficient (i.e. threads can actually run in parallel). To write a function with the GIL I spent many time reading about it. Unfortunately, nothing seemed to expain what I wanted to do until I found this nice blog from Nicolas Hug in which he explains the 3 rules you have to follow to make Cython release the GIL.

First of all, Cython needs to know which C functions that were imported are thread-safe. This is done by using the nogil statement in the function declaration. Then, inside the function the with nogil statement lets Cython know that this block is going to be executed with the GIL released. But to make that code block safe, there cannot be any Python interaction inside that block.

To understand it better, an example is shown below:

cdef extern from "math_operation.h":
    int add(int a, int b)nogil

cpdef sum(src, dest):
    cdef int len_src = len(src)
    cdef int len_dest = len(dest)
    cdef int result
    with nogil:
        # Code with the GIL released
        result = add(len_src, len_dest)
    # Code with the GIL, any Python interaction can be done here

The function sum returns the result of adding the length of src and dest. As you can see, the function has been defined with the cpdef statement instead of the def. The c lets Cython know that this function can be called with C. So this is necessary when writing a function with the GIL released, otherwise you will be trying to execute a Python program without the GIL (which, as explained previously cannot be done). Notice that len_src and len_dest have also been defined as C integers with the cdef int statement. If not, it would not be possible to work with them with the GIL released (the with nogil block).

On the other hand, the p lets Cython know that this function can be called through Python. This does not have to be done always, only when you want to call that function from Python.

Cython typed memoryviews

One of the main differences between the python-blosc and python-blosc2 API, is that the functions compress_ptr and decompress_ptr are no longer supported. We decided to do so, because the Pickle protocol 5 already makes an optimization of the copies. That way, we could have a similar performance for compress_ptr and decompress_ptr but with the functions pack and unpack.

However, when timing the functions I realised that in the majority of the cases, although the compress function from python-blosc2 was faster than the compress_ptr, the decompress function was slower than the decompress_ptr. Thus I checked the code to see if the speed could somehow be increased.

Originally, the code used the Python Buffer Protocol. which is part of the Python/C API. The Python Buffer Protocol lets you (among other things) obtain a pointer to the raw data of an object. But because it wasn't clear for me wether it needed to do a copy or not we decided to work with Cython typed memoryviews.

Cython typed memoryviews are very similar to Python memory views, but with the main difference that the first ones are a C-level type and therefore they do not have much Python overhead. Because it is a C-level type you have to know the dimension of the buffer from which you want to obtain the typed memoryview as well as its data type.

The shape dimension of the buffer is expressed writing as many : between brackets as dimensions it has. If the memory is allocated contiguously, you can write ::1 instead in the corresponding dimension. On the other hand, the type is expressed as you would do it in Cython. In the following code, you can see an example for a one-dimensional numpy array:

import numpy as np
arr = np.ones((10**6,), dtype=np.double)
cdef double [:] typed_view = arr

However, if you want to define a function that receives an object whose type may be unknown, you will have to create a Python memoryview and then cast it into the type you wish as in the next example:

# Get a Python memoryview from an object
mem_view = memoryview(object)
# Cast that memory view into an unsigned char memoryview
cdef unsigned char[:]typed_view = mem_view.cast('B')

The 'B' indicates to cast the memoryview type into an unsigned char.

But if I run the latter code for a binary Python string, it produces a runtime error. It took me 10 minutes to fix the error adding the const statement to the definition of the Cython typed memoryview (as shown below), but I spent two days trying to understand the error and its solution.

# Get a Python memoryview from an object
mem_view = memoryview(object)
# Cast that memory view into an unsigned char memoryview
cdef const unsigned char[:]typed_view = mem_view.cast('B')

The reason why the const statement fixed it, is that a binary Python string is a read-only buffer. By declaring the typed memoryview to const, Cython is being told that the object from the memory view is a read-only buffer so that it cannot change it.

Conclusions

So far, my experience wrapping C-Blosc2 has had some ups and downs.

One method that I use whenever I learn something new is to write down a summary of what I read. Sometimes is almost a copy (therefore some people may find it useless), but it always works really well for me. It helps me connect the ideas better or to build a global idea of what I have or want to do.

Another aspect I realized when doing this wrapper is that because I am a stubborn person, I usually tend to force myself to try to understand something and get frustrated if I do not. However, I have to recognize that sometimes it is better to forget about it until the next day. Your brain will organize your ideas at night so that you can invest better your time the next morning.

But maybe the most difficult part for me was the beginning, and therefore I have to thank Francesc Alted and Aleix Alcacer for giving me a push into the not always easy world of Python extensions.

Introducing Sparse Frames

Marta Iborra — Mon, 08 Feb 2021 07:32:20 GMT

Overview

The new sparse frame implementation allows the storage of Blosc2 super-chunk data chunks sparsely on-disk, using the filesystem as a key/value storage. This mimics existing formats like bcolz or Zarr.

For the sparse implementation we are making use of the existing contiguous frame, in order to store the metadata and the index for accessing the different chunks. Here you can see the new sparse format compared with the existing contiguous frame:

As can be seen in the image above, the contiguous frame file is made of a header, a chunks section and a trailer. The header contains information needed to decompress the chunks and the trailer contains a user meta data chunk. The chunks section for a contiguous frame is made of all the data chunks plus the index chunk. The latter contains the offset where each chunk begins inside the contiguous frame. All these pieces are stored sequentially, without any empty spaces between them.

However, in a sparse frame the chunks are stored somewhere as independent binary files. But there is still the need to store the information to decompress the chunks as well as a place to store the user meta data. All this goes to the chunks.b2frame, which is actually a contiguous frame file with the difference that its chunks section contains only the index chunk. This index chunk stores the ID of each chunk (an integer from 0 to 2^32-1). The name of the chunk file is built by expressing the chunk ID in hexadecimal, padded with zeros (until 8 characters) and adding the .chunk extension. For example, if the index chunk is 46 (2E in hexadecimal) the chunk file name would be 0000002E.chunk.

Advantages

The big advantage of the sparse frame compared with the contiguous one is avoiding empty spaces resulting when updating a chunk.

To better illustrate this, let's imagine that the set of the data chunks in a contiguous frame is stored like in the Jenga board game tower, a tower built with wood blocks. But in constrast to the genuine Jenga board game, not all the blocks have the same size (the uncompressed size of a the chunks is the same, but not the compressed one):

Above it is shown the initial structure of such a tower. If the yellow piece is updated (changed by another piece) there are two possibilities. The first one is that the new piece fits into the empty space left where the old piece was. In that case, the new piece is put in the previous space without any problem and we have no empty spaces left. However, if the new piece does not fit into the empty space, the new piece has to be placed at the top of the tower (like in the game), leaving an empty space where the old piece was.

On the other hand, the chunks of an sparse frame can be seen as books on a shelf, where each book is a different chunk:

If one needs to update one book with the new, taller edition, one only has to grab the old edition and replace it by the new one. As there is no limit in the height of the books, the yellow book can be replaced with a larger book without creating empty spaces, and making a better use of space.

Example of use

Creating a sparse frame in C-Blosc2 is easy; just specifify the name of the directory where you want to store your chunks and you are done:

blosc2_storage storage = {.urlpath="dir1.b2frame"};
schunk = blosc2_schunk_new(storage);
for (nchunk = 0; nchunk < NCHUNKS; nchunk++) {
    blosc2_schunk_append_buffer(schunk, data, isize);
}

The above will create NCHUNKS of chunks in the "dir.b2frame". After that, you can open and read the frame with:

schunk = blosc2_schunk_open("dir1.b2frame");
for (nchunk = 0; nchunk < NCHUNKS; nchunk++) {
    blosc2_schunk_decompress_chunk(schunk, nchunk, data_dest, isize);
}

Simple and effective.

You can have a look at a more complete example here.

Future work

We think that this implementation opens the door to several interesting possibilities.

For example, by introducing networking code in Blosc2, the chunks could be stored in another machine and accessed remotely. That way, with just the metadata (the contiguous frame) we could access all the data chunks in the sparse frame.

For example, let's suppose that we have a sparse frame with 1 million chunks. The total size of the data chunks from this sparse frame is 10 TB, but the contiguous frame size can be as small as 10 KB. So, with just sending an small object of 10 KB, any worker could access the whole 10 TB of data.

The remote stores could be typical networked key/value databases. The key is the identifier for each element of the database, whereas the value is the information that is associated to each key (similar to a set of unique keys and a set of doors). In this case, the key would be built from the metadata (e.g. a URL) plus the index of the chunk, and the value would be the data chunk itself.

This can lead to a whole new range of applications, where data can be spread in the cloud and workers can access to it by receiving small amounts of serialized buffers (the contiguous frame). This way, arbitrarily large data silos could be created and accessed via the C-Blosc2 library (plus a key/value network store).

Note by Francesc: The implementation of sparse frames has been done by Marta Iborra, who is the main author of this blog too. Marta joined the Blosc team a few months ago as a student, and the whole team is very pleased with the quality of her contribution; we would be thrilled to continue having her among us for the next months (but this requires some budget indeed). If you like where we are headed, please consider making a donation to the Blosc project via the NumFOCUS Foundation: https://blosc.org/pages/donate. Thank you!