ListArray

Overview

ListArray is a row-oriented container for variable-length list cells. It is the natural public container for list-valued blosc2.CTable columns, but it is also useful on its own whenever you want typed, row-addressable list data.

Internally, ListArray uses one of two lower-level backends:

• blosc2.BatchArray for append/scan-oriented workloads

• blosc2.ObjectArray for simpler row-level replacement semantics

Quick example

import blosc2

with blosc2.ListArray(
    item_spec=blosc2.string(max_length=16, nullable=True),
    nullable=True,
    storage="batch",
    urlpath="ingredients.b2b",
    mode="w",
) as arr:
    arr.append(["salt", None, "sugar"])
    arr.append([])
    arr.append(None)

    print(arr[0])
    print(arr[1:])

reopened = blosc2.open("ingredients.b2b", mode="r")
print(type(reopened).__name__)

Note

Returned Python lists are detached values. Mutating them locally does not write back to the container; reassign the whole cell instead.

Batching

Batch storage writes up to 2048 list cells per compressed chunk by default. batch_rows counts outer list cells, not child elements or bytes. A partial final batch is persisted by ListArray.flush(), ListArray.close(), or context-manager exit. Pass a smaller positive value to reduce remote overfetch, usually at the cost of more requests and weaker compression.

batch_rows=None opts into caller-managed boundaries: appends remain pending until a flush, which writes every pending cell as one batch. This can consume substantial memory and make a remote scalar read download a large batch. The setting has no effect with storage="vl". Existing arrays retain their stored batch boundaries when reopened.

Nested values and predicates

Nullability is independent at each level. The following schema accepts a null outer list, null inner lists, and null integers inside an inner list:

nested = blosc2.list(
    blosc2.list(blosc2.int32(nullable=True), nullable=True),
    nullable=True,
)

Membership predicates inspect immediate children and do not flatten nesting:

arr.contains([1, 2])
arr.overlaps([[1, 2], [3]])

On a CTable column these methods return Boolean row predicates that compose with other column expressions. An optional membership index accelerates flat lists of scalar Boolean, numeric, string, or bytes values:

table.create_index("tags", kind="membership")
selected = table[table["tags"].overlaps(["python", "numpy"])]

Null outer lists and empty lists do not match. A nullable child matches an explicit None. Nested-list and struct membership uses the scan path; creating a membership index for those child types is not supported.

With storage="batch", serializer="arrow", ListArray.to_arrow() and ListArray.arrow_slice() decompress stored Arrow blocks without converting their cells to Python lists. Combining blocks may copy Arrow buffers. Slice bounds follow Python semantics, and pending rows are flushed before export.

class blosc2.ListArray(spec: ListSpec | None = None, *, item_spec: SchemaSpec | None = None, nullable: bool = False, storage: str = 'batch', serializer: str = 'msgpack', batch_rows: int | None = 2048, items_per_block: int | None = None, _from_schunk=None, **kwargs: Any)

A row-oriented container for list-valued data.

Backed internally by either blosc2.ObjectArray or blosc2.BatchArray.

Attributes:

attrs

User attributes; the recommended alias for vlmeta.

batch_rows

Target number of rows per persisted batch, if configured.

cbytes

Compressed byte size reported by the backend.

contiguous

Whether the backing store is contiguous on disk.

cparams

Compression parameters of the underlying container.

cratio

Compression ratio reported by the backend.

dparams

Decompression parameters of the underlying container.

info

Human-readable information reporter for this array.

info_items

Items used by info to render this array’s summary.

items_per_block

Maximum number of list cells per internal compressed block.

meta

Fixed-length metadata mapping for the underlying container.

nbytes

Uncompressed byte size reported by the backend.

schunk

Underlying blosc2.SChunk used by the backend.

urlpath

Path of the persistent backing store, or None for memory-only arrays.

vlmeta

Variable-length metadata mapping for the underlying container.

Methods

append(value)	Append one list cell and return the new number of rows.
arrow_slice(start, stop)	Export a contiguous row slice, keeping Arrow-backed blocks in Arrow form.
close()	Flush pending rows and close the logical container.
contains(value)	Return one Boolean per row indicating immediate-child membership.
copy(**kwargs)	Return a copy, optionally with different storage arguments.
extend(values, *[, validate])	Append multiple list cells.
extend_arrow(arrow_array)	Append a PyArrow list array without materializing Python cells.
flush()	Persist any pending rows when using the batch backend.
from_arrow(arrow_array, *[, item_spec, ...])	Build a ListArray from a PyArrow list or chunked list array.
overlaps(values)	Return one Boolean per row when any immediate child is in values.
to_arrow()	Return the data as a PyArrow list array.
to_cframe()	Serialize the underlying container to a contiguous C-frame.

Constructors

__init__(spec: ListSpec | None = None, *, item_spec: SchemaSpec | None = None, nullable: bool = False, storage: str = 'batch', serializer: str = 'msgpack', batch_rows: int | None = 2048, items_per_block: int | None = None, _from_schunk=None, **kwargs: Any) → None

Create a list-valued container.

Parameters may be supplied either as a complete spec or as an item_spec plus list/storage options. Storage-related keyword arguments are passed to blosc2.Storage. Batch storage flushes automatically every batch_rows list cells (2048 by default). Pass None for caller-managed batches; flush(), close(), or context-manager exit persists the partial final batch.

classmethod from_arrow(arrow_array, *, item_spec: SchemaSpec | None = None, nullable: bool = True, storage: str = 'batch', serializer: str = 'msgpack', batch_rows: int | None = 2048, items_per_block: int | None = None, **kwargs: Any) → ListArray

Build a ListArray from a PyArrow list or chunked list array.

Row Interface

__getitem__(index: int | slice | list[int] | tuple[int, ...] | ndarray) → Any

Return one cell or a list of cells selected by index, slice, or mask.

__setitem__(index: int, value: Any) → None

Replace one list cell.

__len__() → int

Return the number of rows.

__iter__() → Iterator[Any]

Iterate over list cells.

Mutation

append(value: Any) → int

Append one list cell and return the new number of rows.

extend(values: Iterable[Any], *, validate: bool = True) → None

Append multiple list cells.

Set validate=False only for trusted values that already match this array’s schema.

flush() → None

Persist any pending rows when using the batch backend.

copy(**kwargs: Any) → ListArray

Return a copy, optionally with different storage arguments.

When cparams is not overridden and there are no unflushed (pending) rows, this uses blosc2.BatchArray.chunk_copy() for batch-storage arrays: compressed chunks are transferred at the C level without any Python-level deserialisation. The speed-up is dramatic for large arrays — copying 24 M rows of GPS-path data takes seconds instead of hours.

For vl-storage arrays, or when cparams is overridden, this falls back to an element-wise copy.

Parameters:

**kwargs – Forwarded to the underlying backend or ListArray constructor. Common options: urlpath, cparams, dparams, contiguous.

Returns:

A new standalone copy.

Return type:

ListArray

close() → None

Flush pending rows and close the logical container.

Context Manager

__enter__() → ListArray

Enter a context manager and return this array.

__exit__(exc_type, exc_val, exc_tb) → bool

Exit a context manager, flushing pending rows.

Public Members

to_arrow()

Return the data as a PyArrow list array.

arrow_slice(start: int, stop: int | None)

Export a contiguous row slice, keeping Arrow-backed blocks in Arrow form.

Bounds follow Python slice semantics. Pending rows are flushed first. Combining multiple blocks may copy buffers, but does not create Python cells.

to_cframe() → bytes

Serialize the underlying container to a contiguous C-frame.

contains(value: Any) → ndarray

Return one Boolean per row indicating immediate-child membership.

Null outer lists and empty lists return false. Nested lists compare structurally and are not flattened.

overlaps(values: Iterable[Any]) → ndarray

Return one Boolean per row when any immediate child is in values.