Blosc2 Sparse Frame Format

Blosc (as of version 2.0.0) has a Sparse Frame (SFrame for short) format that allows for non-contiguous storage of Blosc2 data chunks on disk.

When creating a sparse frame one must denote the contiguous flag in storage struct as false and provide a name (which represents a directory, but in the future it could be an arbitrary URL) in storage.urlpath for the sframe to be stored. It is recommended to name the directory with the .b2frame (or .b2f for short) extension.

A SFrame is made up of a frame index file and the chunks stored in the same directory on-disk. The frame index file follows the format described in the contiguous frame format document, with the difference that the frame’s chunks section is made up of multiple files (one per chunk). The frame index file name is always chunks.b2frame, and it also contains the metadata for the sframe.

Chunks

The chunks are stored in the directory as binary files. Each chunk file name will be composed of the index of the chunk in hexadecimal written in capital letters with a length of 8 characters padded with zeros with the .chunk extension. As an example, 15 chunks could be named as follows:

00000000.chunk, 00000001.chunk, ··· , 0000000E.chunk, 0000000F.chunk

Each chunk follows the format described in the chunk format document.

Note: The real order of the chunks is in the index chunk and may not follow the order of the names. This can occur when doing an insertion or a reorder. For more information see the Examples section below.

Sidecar lock file

When the optional file locking is enabled (via the locking member of blosc2_stdio_params, passed in the params member of the blosc2_io struct, or globally via the BLOSC_LOCKING environment variable), a small sidecar file named .b2lock appears inside the sframe directory. It is used to serialize accesses from several handles (typically in different processes) to the same sframe: readers share the lock, mutating operations take it exclusively, and it also carries a change counter so that open handles detect mutations made through other handles. See the file-locking.c example for usage.

The sidecar is not part of the sframe format: it carries no frame data, it is safe to delete whenever no process has the sframe open, and it is removed together with the sframe directory. Note that the locking is advisory (it only protects the sframe if every handle enables it) and that it is not supported on network filesystems (NFS).

Examples

Structure example

As shown below, an sframe of 4 chunks will be composed of a directory with each chunk file and the frame file:

dir.b2frame/
│
├── 00000000.chunk
│
├── 00000001.chunk
│
├── 00000002.chunk
│
├── 00000003.chunk
│
└── chunks.b2frame

Insertion example

When doing an insertion in the nth position, in the same position of the index chunk will be the real chunk index which will be the numbers of chunks that there were before inserting the new one. Following the previous example, the content of the directory and the index chunk is shown before and after an insertion in the 2nd position:

Before                                 After

dir.b2frame/                          dir.b2frame/
│                                      │
├── 00000000.chunk                     ├── 00000000.chunk
│                                      │
├── 00000001.chunk                     ├── 00000001.chunk
│                                      │
├── 00000002.chunk                     ├── 00000002.chunk
│                                      │
├── 00000003.chunk                     ├── 00000003.chunk
│                                      │
└── chunks.b2frame                     ├── 00000004.chunk
                                       │
                                       └── chunks.b2frame

Possible index                         New index
chunk content:  [0, 1, 2, 3]           chunk content: [0, 1, 4, 2, 3]

Note that neither the file names nor their contents change, so when accessing the 2nd chunk the 00000004.chunk file will be read.

Reorder example

As in the insertion case, when doing a reorder the chunks names and their contents are not changed, but the content of the index chunk does. When reordering the chunks, a new order list is passed and the index chunk is changed according to that list. Following with the first example of this section, the content of the index chunk is shown before and after reordering:

Before                                 After

Possible index                         New index
chunk content:  [0, 1, 2, 3]           chunk content:  [3, 1, 0, 2]
New order list: [3, 1, 0, 2]