RemoteCTable¶

RemoteCTable is a read-only blosc2.CTable backed by a remote B2Z archive, a local or remote PyTables/HDF5 table, a Parquet file, or a Caterva2 URLPath table. Fixed-width, shaped, nullable, UTF-8, batch-backed variable-length, batch-backed list, struct/object, and dictionary columns are fetched on demand. Open local PyTables tables through blosc2.open() with path= or a ::table selector. Standalone tables can be opened directly; tables inside a hierarchy can be selected with dataset= or through blosc2.RemoteStore. PyTables/HDF5 sources may supply hdf5_index= as a native index dictionary, local JSON path, or remote fsspec URL. This skips source discovery and does not modify the HDF5 file; see Working with Remote Arrays.

Batch-backed reads transfer and decode whole compressed batches. Dictionary codes remain selective, while the full vocabulary is loaded on first use. ListArray batches contain 2048 list cells by default. Smaller batches reduce overfetch for sparse reads; larger batches usually improve scans and compression. The row limit is not a byte limit, so one unusually large list can still require a large transfer. See ListArray for batching controls.

Nullable list elements, nested lists, and contains/overlaps predicates have the same semantics as local tables. Without an index a predicate scans the remote list batches. A persisted kind="membership" index on a flat scalar list fetches only the compressed posting batches for requested values. If the result projects only other columns, the list payload remains unopened. Indexes on nested lists and structs are not supported. ListArray storage="vl" also remains unavailable through RemoteCTable. Caterva2 API tables support bounded row slices, iteration, and column reads; predicates are unsupported for this source.

Scalar queries automatically use persisted SUMMARY, FULL, PARTIAL, OPSI, and BUCKET indexes. Their sidecars are opened lazily and participate in the outer table’s cache budget and traffic accounting. SUMMARY reads compact min/max records before fetching candidate data blocks; positional indexes use their navigation data to fetch selected value and row-position ranges. Queries retain a correct scan fallback when an index layout or expression is unsupported.

Saving and materializing have different meanings:

with blosc2.RemoteCTable(url) as table:
    table["temperature"][:100]  # warm part of one column
    table.save("reference.b2z")
    table.save("cold-reference.b2z", include_cache=False)
    local = table.materialize(urlpath="complete.b2z")

save() writes the source descriptor, table metadata, and by default only payload already retained in the cache. Missing data is still read from the original source after reopening. For Caterva2 tables, saved payload consists of exact row ranges and projections already read; those selections can also seed a new sparse cache. materialize() returns an independent local table and reads all data needed for it. For Caterva2 API tables, large slices and materialization fetch consecutive bounded row ranges. copy(), to_b2z(), and to_b2d() also create local tables. Caterva2 exposes logical live rows, so either compact setting produces a dense local table. Materializing to urlpath writes batches to disk and publishes only a complete result; in-memory materialization retains the whole result. Fetch batches and the remote cache can still consume substantial memory, especially for variable length columns. Remote Caterva2 predicates and whole-table Arrow, CSV, pandas, NumPy, cframe, and textual conversions remain unsupported; apply those methods to the local result returned by materialize().

Table cache bytes, limits, and traffic are scoped to the shared remote owner and may include sibling leaves. This also applies when a table column is itself a RemoteArray reference to another fsspec or Caterva2 URL: the outer table policy overrides the carrier policy for that handle, and all such columns share one budget. Standalone instances of those arrays keep their original policies. A table selected from a store owns an independent handle, but its columns and views remain borrowed from that table. Refresh a nested table through its root store; a standalone table can call refresh().

Server processes can share one bounded sparse disk cache with blosc2.open(url, cache_dir="shared-cache", shared_cache=True). All processes using that directory must enable sharing. Operations serialize per store, and the aggregate compressed-payload budget defaults to 256 MiB; explicitly pass max_cache_bytes=None for unlimited retention. Use a separate directory from ordinary exclusive caches. The budget does not bound total disk usage or peak RAM.

RemoteCTable.with_sparse_cache(url, runtime_cache_path) remains available for advanced attachment with manifests or seed carriers, with the same default budget and shared-cache implementation.

Parquet cache details¶

Parquet cache_dir stores each accessed physical field and row group as a native CTable directory under the RemoteStore generation’s parquet-groups directory. The common manifest retains the footer, schema, conversion options, source marker, and row-group boundaries. Warm opens reuse this discovery and payload without contacting the source. Cached sources are assumed immutable until refresh() is called. New portable .b2z archives use the common RemoteStore manifest. Local Parquet sources use the same cache layout and may export .b2z references tied to the local source path.

HTTP Parquet sources need byte-range support. A persistent cache also needs a source size and version marker, such as an ETag, modification time, or Backblaze B2 file ID. HTTP read-ahead is disabled by default to avoid fetching unused bytes; an explicit storage_options={"cache_type": "bytes"} restores fsspec buffering. The traffic counter counts fetched ranges; explicitly enabled transport buffering may make actual HTTP transfer different.

Multi-column selections, including table previews, fetch independent Parquet column chunks concurrently using max_concurrency (8 by default). Decoding and cache writes stay on the calling thread. row_buffer_bytes bounds each wave of compressed responses; one physical field larger than the budget runs alone. A physical field with nested leaves is fetched as one unit. Reads still load whole column chunks for the selected row groups, even for a few rows. Use max_concurrency=1 for serial transport.

See Working with Remote Tables for column access, filtering, buffering, reference saving, and materialization examples.

class blosc2.RemoteCTable(urlpath=None, *, dataset=None, path=None, storage_options=None, cache_policy=<policy default>, max_cache_bytes=<policy default>, cache_dir=None, shared_cache=False, hdf5_index=None, max_concurrency=8, metadata_buffer_bytes=8388608, row_buffer_bytes=67108864, source_format=None, parquet_options=None, columns=None, max_rows=None, string_max_length=None, null_storage=None, auto_null_sentinels=True, separate_nested_cols=True, list_serializer='msgpack', blosc2_batch_size=2048, blosc2_items_per_block=None, batch_size=2048, cparams=None, dparams=None, validate=False, _filesystem=None, _filesystem_resolver=None, _batch_validator=None)[source]¶

A read-only CTable whose columns are fetched on demand.

Supported columns include fixed-width, UTF-8, batch-backed variable-length, batch-backed list, struct/object and dictionary columns. Batch reads transfer one whole compressed batch; dictionary decoding loads the full vocabulary on first use.

Independent column requests overlap by default. max_concurrency defaults to 8; use 1 for serial reads. metadata_buffer_bytes (8 MiB) and row_buffer_bytes (64 MiB) bound temporary transport batches, not retained caches or total RAM. An indivisible oversized unit is read alone. These positive-integer settings can also be changed on an open table; views use their base table’s settings. There is no automatic CPU/RAM-based tuning.

blosc2.open accepts max_concurrency but not the table-specific buffer keywords. Use this constructor or the returned table’s settings to tune buffers. Cache policies and max_cache_bytes remain independent.

A RemoteCTable’s cache policy applies to every column read through it, including columns backed by independent RemoteArray carriers. Those columns share the table owner’s cache budget and traffic accounting; their persisted standalone policies are not used or modified by the table.

Caterva2 URLPath tables fetch bounded result cframes by row range and projection. Exact repeated nonempty results share the root owner’s cache; dictionary, nested, timestamp, and nullable representations remain in their CTable frame form instead of being flattened through NumPy rows.

hdf5_index accepts a native index dictionary or a local/remote JSON path for PyTables/HDF5 sources. Supplying one skips HDF5 discovery.

path selects the table within the source. dataset remains a supported alias; when both are supplied they must agree after stripping outer slashes. None leaves selection unspecified; an empty string or slash selects the root. Selector keywords cannot be combined with a selector embedded in the URL.

Attributes:

attrs: Read-only user attributes.
blocks: Block shape shared by the table’s aligned fixed-size columns.
cache_bytes: Compressed payload currently retained at this object’s scope.
cache_policy: Configured payload-retention policy.
cbytes: Total compressed size in bytes (all columns + valid_rows mask).
chunks: Chunk shape shared by the table’s aligned fixed-size columns.
computed_columns: Read-only view of the computed-column definitions.
cratio: Compression ratio for the whole table payload.
indexes: Return a list of blosc2.Index handles for all active indexes.
info: Get information about this table.
info_items: Structured summary items used by info().
is_cache_mutable: Whether the current local cache is writable, not the remote table.
max_cache_bytes: Configured compressed-payload allowance.
max_concurrency
metadata_buffer_bytes
metadata_bytes
mutable: The export default mutability for future reference exports.
nbytes: Total uncompressed size in bytes (all columns + valid_rows mask).
ncols: Total number of columns, including computed (virtual) columns.
nrows
row_buffer_bytes
schema: The compiled schema that drives this table’s columns and validation.
source: Credential-free descriptor for the selected remote object.
traffic: Source-read accounting for this object or its shared owner.
vlmeta: Variable-length metadata attached to this table.

Methods

`add_column`(name, spec[, values])	Add a new column filled from values, or from the default declared in spec.
`add_computed_column`(name, expr, *[, dtype, ...])	Add a read-only virtual column computed from stored columns.
`add_generated_column`(name, *, values[, ...])	Add a stored generated column maintained by the table.
`append`(data)	Append a single row to the table.
`apply`(func, *[, columns, dtype, engine])	Run a row-batch UDF over live column values and materialize the result.
`assign`(**named_exprs)	Return a view with additional computed columns, without copying data.
`close`()	Release resources owned by this handle.
`column_schema`(name)	Return the `CompiledColumn` descriptor for name.
`compact`()	Physically rewrite every column array keeping only live rows.
`compact_index`([col_name, expression, name])	Compact an index, merging any incremental append runs.
`convert_nulls`([columns, to, null_value, inplace])	Convert nullable columns between sentinel and validity-mask storage.
`copy`([compact, urlpath, overwrite, chunks, ...])	Return a new standalone copy of this table.
`cov`()	Return the covariance matrix as a numpy array.
`create_index`([col_name, field, expression, ...])	Build and register an index for a stored column or table expression.
`delete`(ind)	Mark one or more rows as deleted (tombstone deletion).
`describe`()	Print a per-column statistical summary.
`drop_column`(name)	Remove a column from the table.
`drop_computed_column`(name)	Remove a computed column from the table.
`drop_index`([col_name, expression, name])	Remove an index and delete any sidecar files.
`dropna`([subset])	Return a view excluding rows where any column in subset is null.
`extend`(data, *[, validate])	Append multiple rows at once.
`from_arrow`(schema[, batches, urlpath, mode, ...])	Build a `CTable` from an Arrow schema and iterable of record batches.
`from_csv`(path, row_cls, *[, header, sep, ...])	Build a `CTable` from a CSV file.
`from_pandas`(df, row_cls)	Build a `CTable` from a pandas DataFrame.
`from_parquet`(path, *[, columns, batch_size, ...])	Read a Parquet file into a `CTable`.
`group_by`(keys, *[, sort, dropna, engine, ...])	Return a deferred group-by object for this table.
`head`([N])	Return a view of the first N live rows (default 5).
`index`([col_name, expression, name])	Return the index handle for a stored-column or expression target.
`iter_arrow_batches`(*[, columns, batch_size, ...])	Yield live rows as Arrow batches (65,536 rows per batch by default).
`iter_sorted`(cols[, ascending, start, stop, ...])	Iterate rows in sorted order without materializing a full copy.
`load`(urlpath, *[, deserialize])	Load a persistent table from urlpath into RAM.
`materialize`(*[, urlpath, overwrite, ...])	Return an independent local copy of this table or view.
`materialize_computed_column`(name, *[, ...])	Materialize a computed column into a new stored snapshot column.
`open`(urlpath, *[, mode, mmap_mode, deserialize])	Open a persistent CTable from urlpath.
`open_reference`(path, *[, storage_options, ...])	Reopen a saved Parquet RemoteStore archive.
`rebuild_index`([col_name, expression, name])	Drop and recreate an index with the same parameters.
`refresh`()	Reload a standalone table and invalidate its old columns and views.
`refresh_generated_column`(name)	Recompute a stored generated/materialized column from its source columns.
`refresh_generated_columns`(*[, source])	Refresh all generated columns, optionally only those depending on source.
`rename_column`(old, new)	Rename a column.
`sample`(n, *[, seed])	Return a read-only view of n randomly chosen live rows.
`save`([destination, urlpath, include_cache, ...])	Export this table as a portable remote-reference archive.
`schema_dict`()	Return a JSON-compatible dict describing this table's schema.
`select`(cols)	Return a column-projection view exposing only cols.
`slice`(start[, stop, copy])	Return a contiguous range of live (non-deleted) rows.
`sort_by`(cols[, ascending, inplace, view])	Return the table sorted by one or more columns.
`sorted_slice`(col, key, *[, ascending])	Return rows `key` in `col`-sorted order, reading only the slice window.
`tail`([N])	Return a view of the last N live rows (default 5).
`take`(args, *kwargs)	Return a compact table containing rows at the requested positions.
`to_arrow`(args, *kwargs)	Convert all live rows to a `pyarrow.Table`.
`to_b2d`(urlpath, *[, overwrite, compact, ...])	Write this table to a directory-backed store.
`to_b2z`(urlpath, *[, overwrite, compact, ...])	Write this table to a compact `.b2z` container.
`to_cframe`(*[, preserve_sources])	Serialize this table to a bytes buffer (a CFrame).
`to_csv`(args, *kwargs)	Write all live rows to CSV.
`to_pandas`(args, *kwargs)	Convert to a pandas DataFrame.
`to_parquet`(args, *kwargs)	Write Parquet in batches of 65,536 rows by default.
`to_string`(args, *kwargs)	Return a tabular string representation of the table.
`trim_capacity`()	Shrink fixed-width physical storage to the last live row position.
`view`(new_valid_rows)	Return a row-filter view backed by a boolean mask array without copying data.
`where`(args, *kwargs)	Return a row-filtered view matching a boolean predicate.
`with_sparse_cache`(urlpath, runtime_cache_path, *)	Attach a remote CTable to a sparse disk cache shared across processes.

property attrs¶: Read-only user attributes.

property cache_bytes¶: Compressed payload currently retained at this object’s scope.

property cache_policy¶: Configured payload-retention policy.

property cbytes¶: Total compressed size in bytes (all columns + valid_rows mask).

close() → None[source]¶: Release resources owned by this handle.

copy(compact=True, *, urlpath=None, overwrite=False, chunks=None, blocks=None, cparams=None)[source]¶

Return a new standalone copy of this table.

This is the only operation that truly reclaims memory: when compact=True the new table allocates fresh arrays sized exactly to the live row count, discarding all deleted-row gaps and unused capacity.

Parameters:

compact¶ – If True (default), only live (non-deleted) rows are copied. The result is a dense table with no tombstones and no parent dependency — ideal for materialising a filtered view or freeing memory after heavy deletions. If False, all physical slots are copied including deleted gaps, preserving the tombstone state exactly for in-memory copies.
urlpath¶ – Destination path for a persistent copy. The .b2z extension selects a compact zip-backed store; any other path uses a directory-backed store. A .b2d suffix is recommended for directory-backed stores. If None (default), return an in-memory copy.
overwrite¶ – If True, replace an existing persistent destination.
chunks¶ – Chunk size (in items) to use for all scalar columns in the copy. Overrides the chunk size inherited from the source schema. Pass an int for a 1-D chunk or a tuple for multi-dimensional arrays.
blocks¶ – Block size (in items) to use for all scalar columns in the copy. Overrides the block size inherited from the source schema. Pass an int for a 1-D block or a tuple for multi-dimensional arrays. Summary indexes are rebuilt with the new block granularity.
cparams¶ – Compression parameters (codec, clevel, …) to apply to all columns in the copy. Overrides per-column and table-level settings from the source.

head(N=5)[source]¶: Return a view of the first N live rows (default 5).

property is_cache_mutable: bool¶: Whether the current local cache is writable, not the remote table.

property max_cache_bytes¶: Configured compressed-payload allowance.

property mutable: bool¶: The export default mutability for future reference exports.

property nbytes¶: Total uncompressed size in bytes (all columns + valid_rows mask).

classmethod open_reference(path, *, storage_options=None, parquet_options=None)[source]¶: Reopen a saved Parquet RemoteStore archive.

refresh() → None[source]¶

Reload a standalone table and invalidate its old columns and views.

Preserve the table and its cache on discovery/initialization failure. For tables obtained from a RemoteStore, refresh the root store instead.

save(destination: str | PathLike | None = None, *, urlpath: str | PathLike | None = None, include_cache: bool = True, mutable: bool | None = None, overwrite: bool = False) → str[source]¶: Export this table as a portable remote-reference archive.

schema_dict()[source]¶: Return a JSON-compatible dict describing this table’s schema.

select(cols)[source]¶

Return a column-projection view exposing only cols.

The returned object shares the underlying NDArrays with this table (no data is copied). Row filtering and value writes work as usual; structural mutations (add/drop/rename column, append, …) are blocked.

Parameters:

cols¶ –

Ordered list of column names to keep. For tables with nested (dotted) column names, a struct-prefix name automatically expands to all descendant leaves:

t.select(["trip.begin"])   # expands to trip.begin.lon, trip.begin.lat
t.select(["trip"])          # expands to all trip.* leaves

Raises:

KeyError – If any name in cols is not a column of this table (and does not match any struct prefix).
ValueError – If cols is empty.

slice(start, stop=None, /, *, copy=True)[source]¶

Return a contiguous range of live (non-deleted) rows.

The range is given the way range() takes its bounds, either as a single stop (table.slice(stop)), as start/stop integers (table.slice(start, stop)), or as a Python slice (table.slice(slice(start, stop))). Negative bounds count from the end; step is not supported.

Parameters:

start¶ – Range bounds, interpreted as logical positions among the live rows.
stop¶ – Range bounds, interpreted as logical positions among the live rows.
copy¶ – When True (the default, mirroring NDArray.slice()) a compact copy of the range is returned. When False a zero-copy view is returned instead, sharing the parent’s column data (read-only, like head()/tail()).

Returns:

out – The requested rows, re-indexed from 0.

Return type:

CTable

property source¶: Credential-free descriptor for the selected remote object.

tail(N=5)[source]¶: Return a view of the last N live rows (default 5).

take(*args, **kwargs)[source]¶

Return a compact table containing rows at the requested positions.

Indices are interpreted as logical row positions among live rows. The returned table preserves the order of indices and any duplicates, unlike mask-based views.

to_arrow(*args, **kwargs)[source]¶: Convert all live rows to a pyarrow.Table.

to_b2d(urlpath, *, overwrite=False, compact=False, preserve_sources=False)[source]¶

Write this table to a directory-backed store.

Directory-backed CTable stores may use any path that does not end in .b2z; using a .b2d suffix is recommended for clarity. For persistent, non-view .b2z tables opened read-only and compact=False, this uses a fast physical-unpack path: the zip members are extracted as already-compressed leaves. This preserves the physical layout, including deleted rows and spare capacity, and does not recompress columns.

For in-memory tables, views, writable .b2z tables, existing directory-backed tables, or compact=True, this falls back to the logical save() path, materializing only visible/live rows into a new directory-backed store.

Examples

Fast-unpack an existing compact zip store into a directory-backed table:

table = blosc2.CTable.open("data.b2z", mode="r")
table.to_b2d("data.b2d", overwrite=True)
table.close()

Materialize a filtered view into a directory-backed store:

view = table.where(table["score"] > 10)
view.to_b2d("high-score.b2d", overwrite=True)

Force a logical compacted copy, even for a persistent .b2z table:

table.to_b2d("data-compact.b2d", overwrite=True, compact=True)

to_b2z(urlpath, *, overwrite=False, compact=False, preserve_sources=False)[source]¶

Write this table to a compact .b2z container.

.b2z is the compact zip-backed CTable format. For persistent, non-view directory-backed tables and compact=False, this uses a fast physical-pack path: the backing TreeStore directory is zipped with already-compressed leaves stored as-is. This preserves the physical layout, including deleted rows and spare capacity, and does not recompress columns. A .b2d suffix is recommended for directory-backed stores, but not required.

For in-memory tables, views, existing .b2z tables, or compact=True, this falls back to the logical save() path, materializing only visible/live rows into a new .b2z store.

Examples

Fast-pack an existing directory-backed table into a compact zip store:

table = blosc2.CTable.open("data.b2d", mode="r")
table.to_b2z("data.b2z", overwrite=True)
table.close()

Materialize a filtered view into a new compact store:

view = table.where(table["score"] > 10)
view.to_b2z("high-score.b2z", overwrite=True)

Force a logical compacted copy, even for a persistent .b2d table:

table.to_b2z("data-compact.b2z", overwrite=True, compact=True)

to_cframe(*, preserve_sources=False)[source]¶

Serialize this table to a bytes buffer (a CFrame).

This is the Blosc2-bytes counterpart of to_b2z(), mirroring blosc2.NDArray.to_cframe(). The table is packed into an in-memory blosc2.EmbedStore (one entry per column, plus the schema, the _valid_rows mask and any user vlmeta) and serialized to a single bytes object.

Only live rows are serialized: for a view or slice the result is materialized first. The result is fully self-contained and can be rebuilt with blosc2.ctable_from_cframe() without any temp file, which makes it suitable as a transport format (e.g. Caterva2 table slice fetch) including under Pyodide.

Returns:: The serialized table.
Return type:: bytes

See also

blosc2.ctable_from_cframe()

to_csv(*args, **kwargs)[source]¶

Write all live rows to CSV.

Uses Python’s stdlib csv module — no extra dependency required. Fixed-shape ndarray column cells are serialised as JSON arrays for readability and shape safety (e.g. "[1.0, 2.0, 3.0]").

Nulls are written as empty fields, except in a mask-backed text column, where an empty field cannot mean both "" and missing: there a null is written as \N and a valid "" as \E (a value that already looks like either grows a backslash). from_csv() reads both back, so "" and None survive a round trip.

Parameters:

path¶ – Destination file path (created or overwritten). If None (the default), nothing is written and the CSV is returned as a string, like pandas’ DataFrame.to_csv().
header¶ – If True (default), write column names as the first row.
sep¶ – Field delimiter. Defaults to ","; use "\t" for TSV.
encoding¶ – Text encoding of the written file. Defaults to "utf-8" rather than the platform’s, which cannot encode most of what a text column may hold: Python’s default on Windows is cp1252, which rejects anything outside Latin-1. Pass another codec only to satisfy a consumer that requires one.

Returns:

The CSV text when path is None, otherwise None.

Return type:

str or None

to_pandas(*args, **kwargs)[source]¶

Convert to a pandas DataFrame.

Scalar columns become regular DataFrame columns. Fixed-shape ndarray columns become object-dtype columns whose cells hold NumPy arrays of per-row shape item_shape.

A null becomes pandas NA, never the value that stands in for it in storage. A column holding one is therefore given a dtype that can say so – Int64/UInt8/… for integers, boolean for bools, NaN/NaT where the NumPy dtype already has one, and object with None for text, bytes, complex and ndarray cells. A column with no null keeps the dtype it has always had here.

Note

A float column is the one lossy case, in one direction: pandas spells missing as NaN in every float dtype it has (even Float64 folds NaN into NA), so a mask-backed float column, where NaN is an ordinary value, cannot distinguish its nulls from its NaNs once converted. to_arrow() has a validity bitmap and is exact for this case.

Return type:: pandas.DataFrame

Examples

>>> import blosc2
>>> from dataclasses import dataclass
>>> import numpy as np
>>> @dataclass
... class Row:
...     id: int = blosc2.field(blosc2.int64())
...     embedding: object = blosc2.field(blosc2.ndarray((3,), dtype=blosc2.float32()))
>>> t = blosc2.CTable(Row, new_data=[
...     (1, np.array([1, 2, 3], dtype=np.float32)),
...     (2, np.array([4, 5, 6], dtype=np.float32)),
... ])
>>> df = t.to_pandas()
>>> df["id"].tolist()
[1, 2]
>>> df["embedding"].dtype
dtype('O')
>>> np.testing.assert_array_equal(df["embedding"][0], np.array([1, 2, 3], dtype=np.float32))

to_parquet(*args, **kwargs)[source]¶

Write Parquet in batches of 65,536 rows by default.

Unless specified, the row-group size follows the export batch size. Smaller batches reduce temporary memory usage.

to_string(*args, **kwargs)[source]¶

Return a tabular string representation of the table.

By default (max_rows=None, max_width=None) this renders the whole table — every row and every column — like pandas’ DataFrame.to_string(). This is independent of the global blosc2.set_printoptions(); those only affect the truncated str/repr/print view.

Parameters:

max_rows¶ – Maximum number of rows before truncating to a compact head/tail view. None (default) shows all rows; -1 also means all, 0 shows none, a positive int caps it.
max_width¶ – Character budget for column fitting. None (default) or -1 shows all columns; a positive int truncates the middle ones with ... to fit.
show_dimensions¶ – Whether to append a [N rows x M columns] footer. False (default) omits it, matching pandas’ to_string(); True always shows it; "truncate" shows it only when the view is truncated (the behaviour of str/repr).
display_index¶ – Whether to include a pandas-like logical row index column. If None (default), use the global value configured with blosc2.set_printoptions().
index_name¶ – Optional label for the displayed index column.

property traffic¶: Source-read accounting for this object or its shared owner.

property vlmeta¶

Variable-length metadata attached to this table.

Returns a mapping-like proxy that supports item access, iteration, and the [:] bulk getter. Values are serialised via msgpack, so all standard types (int, float, str, bool, list, dict) are supported. The metadata is stored separately from the internal schema metadata and persists through close() / reopen for disk-backed tables.

Examples

>>> import blosc2
>>> import dataclasses
>>> @dataclasses.dataclass
... class Row:
...     x: int = 0
>>> t = blosc2.CTable(Row)
>>> t.attrs["author"] = "Alice"
>>> t.attrs["tags"] = ["alpha", "beta"]
>>> t.attrs["count"] = 42
>>> print(t.attrs["author"])
Alice
>>> print(t.attrs[:])
{'author': 'Alice', 'tags': ['alpha', 'beta'], 'count': 42}
>>> del t.attrs["count"]
>>> for name in t.attrs:
...     print(name, t.attrs[name])
...
author Alice
tags ['alpha', 'beta']

where(*args, **kwargs)[source]¶

Return a row-filtered view matching a boolean predicate.

Signature:

where(expr_result) -> CTable

The predicate can be supplied as a boolean blosc2.LazyExpr, a boolean blosc2.NDArray, a boolean NumPy array, a boolean Column, an unbound col() expression, a blosc2.LazyUDF (including those backed by a blosc2.dsl_kernel()), or a string expression evaluated against this table’s columns. String expressions can reference stored and computed columns directly by name.

The returned object is a CTable view sharing the original column data. The row-selection mask is evaluated immediately and intersected with the table’s current live rows; selected column data is not copied.

Parameters:: expr_result¶ – Boolean predicate selecting rows. Strings are converted to a lazy expression with table columns as operands, e.g. "value * category >= 150". Column objects can also be used in Python expressions, e.g. (t.value * t.category) >= 150.
Returns:: A view over the same columns containing only rows where the predicate is true and the source row is live. When columns is provided, the returned view is additionally projected to that ordered subset of columns.
Return type:: CTable
Raises:: TypeError – If expr_result does not evaluate to a boolean Blosc2/NumPy array or lazy expression.

Examples

Start with a small table:

>>> from dataclasses import dataclass
>>> @dataclass
... class Record:
...     name: str = blosc2.field(blosc2.string(max_length=8))
...     score: int = blosc2.field(blosc2.int64())
...     group: int = blosc2.field(blosc2.int64())
>>> t = blosc2.CTable(
...     Record,
...     new_data=[("a", 30, 1), ("b", 10, 2), ("c", 20, 1), ("d", 40, 2)],
... )

Filter using a string expression:

>>> view = t.where("score > 15")
>>> print(view.base is t, view.nrows)
True 3
>>> slim = t.where("score > 15", columns=["name", "score"])
>>> print(slim.col_names)
['name', 'score']

Compound predicates can use column expressions:

>>> selected = t.where((t.group == 1) & (t.score >= 25))
>>> print(selected["name"][:].tolist())
['a']

Blosc2 lazy functions can be used in column expressions:

view = t.where(((t.value + 2) * blosc2.sin(t.category)) >= 10)

For column names that are not valid Python identifiers, use item access:

view = t.where((t["unit price"] * t["quantity"]) > 100)

For tables with nested (dotted) column names, dotted leaf names and attribute-chain proxies work in both string and expression forms:

view = t.where("trip.begin.lon > -87.7 and payment.fare > 10")
view = t.where(t.trip.begin.lon > -87.7)

Notes

A predicate over a nullable column is three-valued: a comparison against a null is unknown, not false, and &/|/~ combine that third value by Kleene’s rules (see NullableBoolExpr). where() keeps the rows a predicate is true for, so an unknown row is dropped just like a false one – which is what makes t.where(~(t.a > 10)) exclude the nulls rather than select them. Both spellings below agree, and either predicate can be asked which rows it could not answer for (p.is_null()).

Use bitwise operators (&, |, ~) or string expressions for element-wise boolean logic. Python’s logical operators and, or and not cannot be overloaded and therefore do not build lazy column expressions.

Use:

t.where((t.x > 0) & (t.y < 10))
t.where(~t.returned)
t.where("not returned")

not:

t.where((t.x > 0) and (t.y < 10))
t.where(not t.returned)

classmethod with_sparse_cache(urlpath, runtime_cache_path, *, dataset=None, path=None, manifest=None, max_cache_bytes=<policy default>, carrier=None, storage_options=None, max_concurrency=8, metadata_buffer_bytes=8388608, row_buffer_bytes=67108864, _filesystem=None, _filesystem_resolver=None, _batch_validator=None, _source_validator=None, _manifest_validator=None, _max_nodes=None, source_format=None)[source]¶

Attach a remote CTable to a sparse disk cache shared across processes.

path and dataset select the table as in the ordinary constructor. The aggregate compressed-payload budget defaults to 256 MiB; pass max_cache_bytes=None for unlimited retention. For ordinary shared caching, prefer blosc2.open(url, cache_dir=..., shared_cache=True).