Working with ListArray

The BatchArray tutorial shows a batch-oriented store for variable-length items. Many datasets are instead row-oriented: every row owns a list of values, such as the sensor readings of one event or the tags of one post, and the lists have different lengths from row to row.

ListArray is the row-oriented container for that shape of data. Every row is a list, and all items in all rows share one declared item type (the item spec). Rows can have any length, including zero. Under the hood a ListArray is backed by a BatchArray (the default "batch" storage) or by an ObjectArray ("vl" storage), but you always address it by row, never by batch.

Creating a ListArray

The item_spec declares the type of the items inside each list. It can be a scalar spec such as blosc2.float64() or blosc2.int32(), a string spec such as blosc2.string(max_length=16), or a schema spec for structured (dataclass) rows. Let’s create an array for floating-point sensor readings:

[1]:
import numpy as np

import blosc2

readings = blosc2.ListArray(item_spec=blosc2.float64())
print("rows:", len(readings))
rows: 0

Appending and extending rows

append() adds one row and extend() adds many. Rows do not need a common length, and an empty list is a perfectly valid row — here the second event simply recorded no readings:

[2]:
readings.append([21.5, 22.0, 21.8])
readings.append([])  # an event with no readings
readings.extend([[19.9], [23.1, 23.4, 23.2, 23.6], [20.0, 20.2]])

print("rows:", len(readings))
print("all rows:", readings[:])
rows: 5
all rows: [[21.5, 22.0, 21.8], [], [19.9], [23.1, 23.4, 23.2, 23.6], [20.0, 20.2]]

Reading rows back

Rows come back as plain Python lists. A ListArray supports integer indexing (negative indices count from the end), slicing, and fancy indexing with a list or a NumPy array of positions:

[3]:
print("first row:      ", readings[0])
print("last row:       ", readings[-1])
print("rows 1 to 3:    ", readings[1:3])
print("rows 0 and 3:   ", readings[[0, 3]])
print("rows 4 and 2:   ", readings[np.array([4, 2])])
first row:       [21.5, 22.0, 21.8]
last row:        [20.0, 20.2]
rows 1 to 3:     [[], [19.9]]
rows 0 and 3:    [[21.5, 22.0, 21.8], [23.1, 23.4, 23.2, 23.6]]
rows 4 and 2:    [[20.0, 20.2], [19.9]]

Nullable rows and string items

Pass nullable=True if a row itself may be missing (None), as opposed to present but empty. String items work through a string spec, and contains() returns a boolean mask over the rows — handy for filtering:

[4]:
tags = blosc2.ListArray(item_spec=blosc2.string(max_length=16), nullable=True)
tags.extend([["alpha", "beta"], None, ["gamma"]])

print("tags:             ", tags[:])
print("rows with 'alpha':", tags.contains("alpha"))
tags:              [['alpha', 'beta'], None, ['gamma']]
rows with 'alpha': [ True False False]

ListArray as a CTable column

A list column can live inside a CTable next to ordinary scalar columns. Declare it in the dataclass schema with blosc2.list(...), then append rows as tuples. You can filter on the scalar columns with where() and read the list column back row by row:

[5]:
from dataclasses import dataclass


@dataclass
class Event:
    name: str = blosc2.field(blosc2.utf8())
    readings: list[float] = blosc2.field(  # noqa: RUF009
        blosc2.list(blosc2.float64())
    )


events = blosc2.CTable(Event)
events.append(("mañana ☀️", [21.5, 22.0]))
events.append(("midday", [23.1, 23.4, 23.2]))
events.append(("evening", []))

print("names:   ", events.name[:])
print("readings:", events.readings[:])
print("midday readings:", events.where("name == 'midday'").readings[:])
names:    ['mañana ☀️' 'midday' 'evening']
readings: [[21.5, 22.0], [23.1, 23.4, 23.2], []]
midday readings: [[23.1, 23.4, 23.2]]

Arrow interop

ListArray.from_arrow() builds an array from an Arrow ListArray (the item spec must match the Arrow list item type), and to_arrow() converts back with the same values:

[6]:
import pyarrow as pa

arrow_arr = pa.array([[1.0, 2.0], [], [3.5]], type=pa.list_(pa.float64()))
from_arrow = blosc2.ListArray.from_arrow(arrow_arr, item_spec=blosc2.float64())

print("from Arrow:", from_arrow[:])
print("round-trip values equal:", from_arrow.to_arrow().to_pylist() == arrow_arr.to_pylist())
from Arrow: [[1.0, 2.0], [], [3.5]]
round-trip values equal: True

ListArray or BatchArray?

  • Choose ``ListArray`` when your data is a sequence of typed lists addressed by row: sensor readings per event, tags per item, tokens per document. It validates items against the item spec, embeds in CTable as a list column, converts to and from Arrow, and offers row-level helpers such as contains() and overlaps().

  • Choose ``BatchArray`` when you store arbitrary Python objects and think in batches — for example chunks of pre-processed payloads — and you do not need a per-item type or row semantics.

Like other Blosc2 containers, a ListArray can be persisted by passing a urlpath; with the default batch storage, rows are flushed in batches of batch_rows (2048 by default), and flush()/close() persist the final partial batch.