Blosc1 API#
This is the classic API from Blosc1 with 32-bit limited containers.
Main API#
-
void blosc2_init(void)#
Initialize the Blosc library environment.
You must call this previous to any other Blosc call, unless you want Blosc to be used simultaneously in a multi-threaded environment, in which case you can use the blosc2_compress_ctx blosc2_decompress_ctx pair.
The BLOSC_PREFETCH environment variable, set to a number of threads, turns on read-ahead for frames on disk here, and BLOSC_PREFETCH_POPULATE says for which memory-mapped frames it populates the mapping (see blosc2_set_prefetch).
See also
-
void blosc2_destroy(void)#
Destroy the Blosc library environment.
You must call this after to you are done with all the Blosc calls, unless you have not used blosc2_init() before.
See also
-
int blosc1_compress(int clevel, int doshuffle, size_t typesize, size_t nbytes, const void *src, void *dest, size_t destsize)#
Compress a block of data in the
srcbuffer and returns the size of compressed block.Remark
Compression is memory safe and guaranteed not to write
destmore than what is specified indestsize. There is not a minimum forsrcbuffer sizenbytes. Equivalent to blosc2_compress.- Environment variables
-
This function honors different environment variables to control internal parameters without the need of doing that programmatically. Here are the ones supported:
BLOSC_CLEVEL=(INTEGER): This will overwrite the
clevelparameter before the compression process starts.BLOSC_SHUFFLE=[NOSHUFFLE | SHUFFLE | BITSHUFFLE]: This will overwrite the
doshuffleparameter before the compression process starts.BLOSC_DELTA=(1|0): This will call blosc2_set_delta() before the compression process starts.
BLOSC_TYPESIZE=(INTEGER): This will overwrite the
typesizeparameter before the compression process starts.BLOSC_COMPRESSOR=[BLOSCLZ | LZ4 | LZ4HC | ZLIB | ZSTD]: This will call blosc1_set_compressor before the compression process starts.
BLOSC_NTHREADS=(INTEGER): This will call blosc2_set_nthreads before the compression process starts.
BLOSC_SPLITMODE=(ALWAYS | NEVER | AUTO | FORWARD_COMPAT): This will call blosc1_set_splitmode() before the compression process starts.
BLOSC_BLOCKSIZE=(INTEGER): This will call blosc1_set_blocksize before the compression process starts. NOTE: The blocksize is a critical parameter with important restrictions in the allowed values, so use this with care.
BLOSC_NOLOCK=(ANY VALUE): This will call blosc2_compress_ctx under the hood, with the compressor, blocksize and numinternalthreads parameters set to the same as the last calls to blosc1_set_compressor, blosc1_set_blocksize and blosc2_set_nthreads. BLOSC_CLEVEL, BLOSC_SHUFFLE, BLOSC_DELTA and BLOSC_TYPESIZE environment vars will also be honored.
BLOSC_BLOSC1_COMPAT=(ANY VALUE): The chunk is written in the Blosc1 format, as if blosc2_ctx_set_blosc1_format() had been called, so that C-Blosc 1.x can decode it. Also honored together with BLOSC_NOLOCK.
See also
- Parameters:
clevel – The desired compression level and must be a number between 0 (no compression) and 9 (maximum compression).
doshuffle – Specifies whether the shuffle compression preconditioner should be applied or not. BLOSC_NOFILTER means not applying filters, BLOSC_SHUFFLE means applying shuffle at a byte level and BLOSC_BITSHUFFLE at a bit level (slower but may achieve better compression).
typesize – Is the number of bytes for the atomic type in binary
srcbuffer. This is mainly useful for the shuffle preconditioner. For implementation reasons, only a 1 < typesize < 256 will allow the shuffle filter to work. When typesize is not in this range, shuffle will be silently disabled.nbytes – The number of bytes to compress in the
srcbuffer.src – The buffer containing the data to compress.
dest – The buffer where the compressed data will be put, must have at least the size of
destsize.destsize – The size of the dest buffer. Blosc guarantees that if you set
destsizeto, at least, (nbytes+ BLOSC2_MAX_OVERHEAD), the compression will always succeed.
- Returns:
The number of bytes compressed. If
srcbuffer cannot be compressed intodestsize, the return value is zero and you should discard the contents of thedestbuffer. A negative return value means that an internal error happened. This should never happen. If you see this, please report it back together with the buffer data causing this and compression settings.
Warning
The
srcbuffer and thedestbuffer can not overlap.
-
int blosc1_decompress(const void *src, void *dest, size_t destsize)#
Decompress a block of compressed data in
src, put the result indestand returns the size of the decompressed block.Remark
Decompression is memory safe and guaranteed not to write the
destbuffer more than what is specified indestsize. Similar to blosc2_decompress.Remark
In case you want to keep under control the number of bytes read from source, you can call blosc1_cbuffer_sizes first to check whether the
nbytes(i.e. the number of bytes to be read fromsrcbuffer by this function) in the compressed buffer is ok with you.- Environment variables
This function honors different environment variables to control internal parameters without the need of doing that programmatically. Here are the ones supported:
BLOSC_NTHREADS=(INTEGER): This will call blosc2_set_nthreads before the proper decompression process starts.
BLOSC_NOLOCK=(ANY VALUE): This will call blosc2_decompress_ctx under the hood, with the numinternalthreads parameter set to the same value as the last call to blosc2_set_nthreads.
See also
- Parameters:
src – The buffer to be decompressed.
dest – The buffer where the decompressed data will be put.
destsize – The size of the
destbuffer.
- Returns:
The number of bytes decompressed. If an error occurs, e.g. the compressed data is corrupted or the output buffer is not large enough, then a negative value will be returned instead.
Warning
The
srcbuffer and thedestbuffer can not overlap.
-
int blosc1_getitem(const void *src, int start, int nitems, void *dest)#
Get
nitems(oftypesizesize) insrcbuffer starting instart.The items are returned in
destbuffer, which has to have enough space for storing all items.Remark
The function blosc2_getitem is a more complete and secure version.
- Parameters:
src – The compressed buffer from data will be decompressed.
start – The position of the first item (of
typesizesize) from where data will be retrieved.nitems – The number of items (of
typesizesize) that will be retrieved.dest – The buffer where the decompressed data retrieved will be put.
- Returns:
The number of bytes copied to
destor a negative value if some error happens.
-
int16_t blosc2_get_nthreads(void)#
Returns the current number of threads that are used for compression/decompression.
-
int16_t blosc2_set_nthreads(int16_t nthreads)#
Initialize a pool of threads for compression/decompression.
If
nthreadsis 1, then the serial version is chosen and a possible previous existing pool is released (blosc2_free_resources ends the pools that are no longer used). If this is not called,nthreadsis set to 1 internally. The threads are per process; a child process created with fork() starts its own threads on first use (see README_THREADED.rst).- Parameters:
nthreads – The number of threads to use.
- Returns:
The previous number of threads.
-
int blosc2_set_prefetch(int nthreads, int64_t max_ahead, int64_t piece_size)#
Read ahead when reading slices of arrays and super-chunks stored in contiguous frames on disk with the default filesystem or the memory-mapped I/O: b2nd_get_slice_cbuffer, b2nd_get_slice, b2nd_get_orthogonal_selection and blosc2_schunk_get_slice_buffer.
Without read-ahead, a slice reads the blocks it needs one after the other, each with at most the kernel’s read-ahead window per read; on a network file system every such read is a round trip. With read-ahead, the chunks the slice needs (and that are not in the page cache yet) are read by up to
nthreadsbackground threads in pieces of at mostpiece_sizebytes, in the order the slice decompresses them and at mostmax_aheadbytes ahead of it, and each chunk is decompressed as soon as its bytes are in. Of a chunk that a slice (b2nd_get_slice_cbuffer, b2nd_get_slice) covers only in part, only the blocks it needs are read, after the chunk’s header (which says where they are; one round trip more when it is not cached). The data goes to the page cache (memory-mapped reads stay zero-copy); pages that are cached already are not read again (asked with cachestat() on Linux >= 6.5, else mincore()). On Linux the kernel answers that only for files the process owns or may write to (before 6.5: only files it owns); other files are read again even when cached, which is correct but costs a copy.Opening a frame file gets its start and its end (header, trailer and chunk index) in one round trip, also with read-ahead off; with read-ahead on, a file of up to 1 MiB (from the frame’s offset on) is asked for whole.
Reads that are not cached get faster on any storage, most on high-latency ones; 4 threads and the default piece size give most of the speed. The threads mostly wait for the storage, but the kernel spends CPU on every byte they read; README_THREADED.rst has which settings to use when.
On GPFS (IBM Storage Scale) read() does not fill the page cache but a cache of GPFS’s own, so for a frame read through the mmap I/O (BLOSC2_IO_FILESYSTEM_MMAP) the threads populate its mapping there instead (madvise(MADV_POPULATE_READ), Linux >= 5.14), cached pages included: the slice then finds the pages mapped and does not fault on them (builds with glibc >= 2.28 recognize GPFS). The BLOSC_PREFETCH_POPULATE environment variable, read at blosc2_init, does this for the memory-mapped frames on every file system when set to 1, and for none when set to 0.
How many threads: each slice being read gets up to
nthreadsread-ahead threads, and slices that threads of the process read at the same time get theirs each. So a process runs up tonthreadstimes the number of slices it reads at once, at most BLOSC2_PREFETCH_MAX_NTHREADS in all: a program that reads with one thread hasnthreadsof them, one that reads with 8 threads at once up to 8 xnthreads. Decompression threads (nthreadsof blosc2_dparams) are separate and do not count: they decompress a chunk while the read-ahead threads fetch the next ones. Read-ahead threads are started as needed and run until blosc2_destroy (lowering the number does not stop them), each keeping a buffer as large as the largest piece it has read.Off by default; the BLOSC_PREFETCH environment variable, set to a number of threads, turns it on at blosc2_init. The setting and the threads are per process; a child process created with fork() starts its own threads on first use. Supported on POSIX systems; elsewhere (Windows) this has no effect.
- Parameters:
nthreads – Read-ahead threads per slice being read (see above), at most BLOSC2_PREFETCH_MAX_NTHREADS; 0 turns read-ahead off.
max_ahead – Most bytes read ahead of the chunk being decompressed (0 for the default, BLOSC2_PREFETCH_MAX_AHEAD_DEFAULT).
piece_size – Largest read handed to one thread, in bytes (0 for the default, BLOSC2_PREFETCH_PIECE_SIZE_DEFAULT; at least BLOSC2_PREFETCH_MIN_PIECE_SIZE).
- Returns:
The previous number of threads, or a negative error code.
- void(*) blosc_threads_callback (void *callback_data, void(*dojob)(void *), int numjobs, size_t jobdata_elsize, void *jobdata)
Pointer to a callback function that executes
dojob(jobdata + i*jobdata_elsize)fori = 0 to numjobs-1, possibly in parallel threads (but not returning until alldojobcalls have returned).This allows the caller to provide a custom threading backend as an alternative to the default Blosc-managed threads.
callback_datais passed through fromblosc2_set_threads_callback.
-
void blosc2_set_threads_callback(blosc_threads_callback callback, void *callback_data)#
Set the threading backend for parallel compression/decompression to use
callbackto execute work instead of using the Blosc-managed threads.The
callback_dataargument is passed through to the callback.See also
- Parameters:
callback – the callback to use. Passing
NULLuses the default Blosc threading backend.callback_data – the callback data.
Warning
This function is not thread-safe and should be called before any other Blosc function: it affects all Blosc contexts.
-
const char *blosc1_get_compressor(void)#
Get the current compressor that is used for compression.
- Returns:
The string identifying the compressor being used.
-
int blosc1_set_compressor(const char *compname)#
Select the compressor to be used.
The supported ones are “blosclz”, “lz4”, “lz4hc”, “zlib” and “ztsd”. If this function is not called, then “blosclz” will be used.
- Parameters:
compname – The name identifier of the compressor to be set.
- Returns:
The code for the compressor (>=0). In case the compressor is not recognized, or there is not support for it in this build, it returns a -1.
-
void blosc2_set_delta(int dodelta)#
Select the delta coding filter to be used.
This call should always succeed.
- Parameters:
dodelta – A value >0 will activate the delta filter. If 0, it will be de-activated
-
int blosc1_get_blocksize(void)#
Get the internal blocksize to be used during compression.
0 means that an automatic blocksize is computed internally.
- Returns:
The size in bytes of the internal block size.
-
void blosc1_set_blocksize(size_t blocksize)#
Force the use of a specific blocksize.
If 0, an automatic blocksize will be used (the default).
Warning
The blocksize is a critical parameter with important restrictions in the allowed values, so use this with care.
-
void blosc1_set_splitmode(int splitmode)#
Set the split mode.
BLOSC_FORWARD_COMPAT offers reasonably forward compatibility, BLOSC_AUTO_SPLIT is for nearly optimal results (based on heuristics), BLOSC_NEVER_SPLIT and BLOSC_ALWAYS_SPLIT are for the user experimenting when trying to get best compression ratios and/or speed.
If not called, the default mode is BLOSC_FORWARD_COMPAT_SPLIT.
This function should always succeed.
- Parameters:
splitmode – It can take the next values: BLOSC_FORWARD_COMPAT_SPLIT BLOSC_AUTO_SPLIT BLOSC_NEVER_SPLIT BLOSC_ALWAYS_SPLIT
-
int blosc2_free_resources(void)#
Free possible memory temporaries and thread resources.
Use this when you are not going to use Blosc for a long while.
Blosc’s own thread pools (not on Windows) that no context uses any more are kept, a few of them with the temporaries of their threads, so that contexts created later with the same number of threads do not start new threads; this function ends them.
- Returns:
A 0 if succeeds, in case of problems releasing the resources, it returns a negative number.
Compressed buffer information#
-
void blosc1_cbuffer_sizes(const void *cbuffer, size_t *nbytes, size_t *cbytes, size_t *blocksize)#
Get information about a compressed buffer, namely the number of uncompressed bytes (
nbytes) and compressed (cbytes).It also returns the
blocksize(which is used internally for doing the compression by blocks).You only need to pass the first BLOSC_MIN_HEADER_LENGTH bytes of a compressed buffer for this call to work.
Remark
Equivalent to function blosc2_cbuffer_sizes.
This function should always succeed.
- Parameters:
cbuffer – The buffer of compressed data.
nbytes – The pointer where the number of uncompressed bytes will be put.
cbytes – The pointer where the number of compressed bytes will be put.
blocksize – The pointer where the block size will be put.
-
void blosc1_cbuffer_metainfo(const void *cbuffer, size_t *typesize, int *flags)#
Get information about a compressed buffer, namely the type size (
typesize), as well as some internalflags.You can use the
BLOSC_DOSHUFFLE,BLOSC_DOBITSHUFFLE,BLOSC_DODELTAandBLOSC_MEMCPYEDsymbols for extracting the interesting bits (e.g.flags&BLOSC_DOSHUFFLEsays whether the buffer is byte-shuffled or not).This function should always succeed.
- Parameters:
cbuffer – The buffer of compressed data.
typesize – The pointer where the type size will be put.
flags – The pointer of the integer where the additional info is encoded. The
flagsis a set of bits, where the currently used ones are:bit 0: whether the shuffle filter has been applied or not
bit 1: whether the internal buffer is a pure memcpy or not
bit 2: whether the bitshuffle filter has been applied or not
bit 3: whether the delta coding filter has been applied or not
-
void blosc2_cbuffer_versions(const void *cbuffer, int *version, int *versionlz)#
Get information about a compressed buffer, namely the internal Blosc format version (
version) and the format for the internal Lempel-Ziv compressor used (versionlz).This function should always succeed.
- Parameters:
cbuffer – The buffer of compressed data.
version – The pointer where the Blosc format version will be put.
versionlz – The pointer where the Lempel-Ziv version will be put.
-
const char *blosc2_cbuffer_complib(const void *cbuffer)#
Get the compressor library/format used in a compressed buffer.
This function should always succeed.
- Parameters:
cbuffer – The buffer of compressed data.
- Returns:
The string identifying the compressor library/format used.
-
int blosc1_cbuffer_validate(const void *cbuffer, size_t cbytes, size_t *nbytes)#
Checks that the compressed buffer starting at
cbufferof lengthcbytesmay contain valid blosc compressed data, and that it is safe to call blosc1_decompress/blosc1_getitem.On success, returns 0 and sets
nbytesto the size of the uncompressed data. This does not guarantee that the decompression function won’t return an error, but does guarantee that it is safe to attempt decompression.- Parameters:
cbuffer – The buffer of compressed data.
cbytes – The number of compressed bytes.
nbytes – The pointer where the number of uncompressed bytes will be put.
- Returns:
On failure, returns negative value.
Utility functions#
-
int blosc2_compcode_to_compname(int compcode, const char **compname)#
Get the compressor name associated with the compressor code.
- Parameters:
compcode – The code identifying the compressor
compname – The pointer to a string where the compressor name will be put.
- Returns:
The compressor code. If the compressor code is not recognized, or there is not support for it in this build, -1 is returned.
-
int blosc2_compname_to_compcode(const char *compname)#
Get the compressor code associated with the compressor name.
- Parameters:
compname – The string containing the compressor name.
- Returns:
The compressor code. If the compressor name is not recognized, or there is not support for it in this build, -1 is returned instead.
-
const char *blosc2_list_compressors(void)#
Get a list of compressors supported in the current build.
This function does not leak, so you should not free() the returned list.
This function should always succeed.
- Returns:
The comma separated string with the list of compressor names supported.
-
const char *blosc2_get_version_string(void)#
Get the version of Blosc in string format.
- Returns:
The string with the current Blosc version. Useful for dynamic libraries.
-
int blosc2_get_complib_info(const char *compname, char **complib, char **version)#
Get info from compression libraries included in the current build.
- Parameters:
compname – The compressor name that you want info from.
complib – The pointer to a string where the compression library name, if available, will be put.
version – The pointer to a string where the compression library version, if available, will be put.
- Returns:
The code for the compression library (>=0). If it is not supported, this function returns -1.
Warning
You are in charge of the
complibandversionstrings, you should free() them so as to avoid leaks.