Working with ListArray¶
The BatchArray tutorial shows a batch-oriented store for variable-length items. Many datasets are instead row-oriented: every row owns a list of values, such as the sensor readings of one event or the tags of one post, and the lists have different lengths from row to row.
ListArray is the row-oriented container for that shape of data. Every row is a list, and all items in all rows share one declared item type (the item spec). Rows can have any length, including zero. Under the hood a ListArray is backed by a BatchArray (the default "batch" storage) or by an ObjectArray ("vl" storage), but you always address it by row, never by batch.
Creating a ListArray¶
The item_spec declares the type of the items inside each list. It can be a scalar spec such as blosc2.float64() or blosc2.int32(), a string spec such as blosc2.string(max_length=16), or a schema spec for structured (dataclass) rows. Let’s create an array for floating-point sensor readings:
[1]:
import numpy as np
import blosc2
readings = blosc2.ListArray(item_spec=blosc2.float64())
print("rows:", len(readings))
rows: 0
Appending and extending rows¶
append() adds one row and extend() adds many. Rows do not need a common length, and an empty list is a perfectly valid row — here the second event simply recorded no readings:
[2]:
readings.append([21.5, 22.0, 21.8])
readings.append([]) # an event with no readings
readings.extend([[19.9], [23.1, 23.4, 23.2, 23.6], [20.0, 20.2]])
print("rows:", len(readings))
print("all rows:", readings[:])
rows: 5
all rows: [[21.5, 22.0, 21.8], [], [19.9], [23.1, 23.4, 23.2, 23.6], [20.0, 20.2]]
Reading rows back¶
Rows come back as plain Python lists. A ListArray supports integer indexing (negative indices count from the end), slicing, and fancy indexing with a list or a NumPy array of positions:
[3]:
print("first row: ", readings[0])
print("last row: ", readings[-1])
print("rows 1 to 3: ", readings[1:3])
print("rows 0 and 3: ", readings[[0, 3]])
print("rows 4 and 2: ", readings[np.array([4, 2])])
first row: [21.5, 22.0, 21.8]
last row: [20.0, 20.2]
rows 1 to 3: [[], [19.9]]
rows 0 and 3: [[21.5, 22.0, 21.8], [23.1, 23.4, 23.2, 23.6]]
rows 4 and 2: [[20.0, 20.2], [19.9]]
Nullable rows and string items¶
Pass nullable=True if a row itself may be missing (None), as opposed to present but empty. String items work through a string spec, and contains() returns a boolean mask over the rows — handy for filtering:
[4]:
tags = blosc2.ListArray(item_spec=blosc2.string(max_length=16), nullable=True)
tags.extend([["alpha", "beta"], None, ["gamma"]])
print("tags: ", tags[:])
print("rows with 'alpha':", tags.contains("alpha"))
tags: [['alpha', 'beta'], None, ['gamma']]
rows with 'alpha': [ True False False]
ListArray as a CTable column¶
A list column can live inside a CTable next to ordinary scalar columns. Declare it in the dataclass schema with blosc2.list(...), then append rows as tuples. You can filter on the scalar columns with where() and read the list column back row by row:
[5]:
from dataclasses import dataclass
@dataclass
class Event:
name: str = blosc2.field(blosc2.utf8())
readings: list[float] = blosc2.field( # noqa: RUF009
blosc2.list(blosc2.float64())
)
events = blosc2.CTable(Event)
events.append(("mañana ☀️", [21.5, 22.0]))
events.append(("midday", [23.1, 23.4, 23.2]))
events.append(("evening", []))
print("names: ", events.name[:])
print("readings:", events.readings[:])
print("midday readings:", events.where("name == 'midday'").readings[:])
names: ['mañana ☀️' 'midday' 'evening']
readings: [[21.5, 22.0], [23.1, 23.4, 23.2], []]
midday readings: [[23.1, 23.4, 23.2]]
Arrow interop¶
ListArray.from_arrow() builds an array from an Arrow ListArray (the item spec must match the Arrow list item type), and to_arrow() converts back with the same values:
[6]:
import pyarrow as pa
arrow_arr = pa.array([[1.0, 2.0], [], [3.5]], type=pa.list_(pa.float64()))
from_arrow = blosc2.ListArray.from_arrow(arrow_arr, item_spec=blosc2.float64())
print("from Arrow:", from_arrow[:])
print("round-trip values equal:", from_arrow.to_arrow().to_pylist() == arrow_arr.to_pylist())
from Arrow: [[1.0, 2.0], [], [3.5]]
round-trip values equal: True
ListArray or BatchArray?¶
Choose ``ListArray`` when your data is a sequence of typed lists addressed by row: sensor readings per event, tags per item, tokens per document. It validates items against the item spec, embeds in
CTableas a list column, converts to and from Arrow, and offers row-level helpers such ascontains()andoverlaps().Choose ``BatchArray`` when you store arbitrary Python objects and think in batches — for example chunks of pre-processed payloads — and you do not need a per-item type or row semantics.
Like other Blosc2 containers, a ListArray can be persisted by passing a urlpath; with the default batch storage, rows are flushed in batches of batch_rows (2048 by default), and flush()/close() persist the final partial batch.