Skip to content

yggdrasil.data.types.nested.struct_arrow

struct_arrow

Arrow-side cast helpers for :class:StructType targets.

Materialized entry points, one per source shape:

  • :func:cast_arrow_struct_array — struct → struct (per-child rebuild, missing children defaulted, nested casts threaded through options.copy(source=, target=)).
  • :func:cast_arrow_map_array — map → struct via pc.map_lookup; one lookup per target child.
  • :func:cast_arrow_list_array — list → struct by positional index; out-of-bounds is null.
  • :func:cast_arrow_tabular — Table/RecordBatch column rebuild against the merged schema; missing source columns get defaults.

Streaming entry points (span batch boundaries):

  • :func:cast_arrow_batch_iterator — per-batch tabular cast over an iterable of pa.RecordBatch, with optional streamed rechunking keyed on :attr:CastOptions.byte_size.
  • :func:rechunk_arrow_batches — re-exported from :mod:yggdrasil.arrow.cast (lives there because it's a pure pyarrow util with no struct-cast coupling); kept here for back-compat with existing import paths.

All materialized helpers short-circuit on options.need_cast and rely on the parent :meth:StructType._cast_arrow_array for engine- level dispatch. The iterator helper threads through :func:cast_arrow_tabular per batch, so the same short-circuit holds.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: Iterable[RecordBatch], options: "CastOptions"
) -> Iterator[pa.RecordBatch]

Cast an iterable of pa.RecordBatch and stream-rechunk to target size.

Per-batch cast goes through :func:cast_arrow_tabular (which already short-circuits on schema match — also when options.target is unbound). Source-field binding is deferred to the first batch so callers don't have to peek upstream; the bound options are reused for every subsequent batch.

Sizing knobs options.row_size and options.byte_size drive output rechunking via :func:rechunk_arrow_batches. When neither is set, casted batches pass through unchanged.

:raises TypeError: if any item is not a :class:pa.RecordBatch.