Skip to content

yggdrasil.data.record

record

Single-row materialization with a singleton :class:Schema reference.

A :class:Record is a :class:collections.abc.Mapping over one row's values, keyed by field name. The schema is shared by reference across sibling rows — a stream of N records carries one :class:Schema and N value tuples, not N (Schema, values) pairs. That keeps row materialization cheap when the same schema repeats across millions of rows (the only real use case).

:class:Record is the natural unit returned by :meth:Tabular.read_records and consumed by :meth:Tabular.write_records. Subclasses with a richer row shape (SQL row, Spark Row, etc.) should still satisfy the Mapping contract so callers don't need to know the concrete origin.

Record 

Record(values: RecordValues, schema: 'Schema')

Bases: Mapping[str, Any]

Single row, keyed by field name, sharing a :class:Schema.

The Mapping protocol gives callers record[name], record.get(name, default), in, keys(), values(), items(), and len() without any extra surface. Positional integer access (record[0]) is supported as a convenience for fast-path callers that already know the field index.

Construction:

  • Record((v0, v1, v2), schema) — values aligned with schema.fields order.
  • Record({"a": 1, "b": 2}, schema) — dict re-aligned to schema.fields order; missing keys land as None.

The schema is taken by reference — pass the same :class:Schema instance across a stream of rows to keep allocation flat.

values property 

values: tuple

Positional value tuple, aligned with schema.fields.

schema property 

schema: 'Schema'

The shared :class:Schema. Same instance across sibling rows.

from_arrow_batches classmethod 

from_arrow_batches(
    batches: Iterable[RecordBatch], *, schema: "Schema | None" = None
) -> Iterator["Record"]

Yield :class:Record s from an Arrow-batch stream.

The first batch's schema becomes the singleton :class:Schema all yielded records share, unless one is passed explicitly. Per-row values are materialized via column[i].as_py() — cheap for primitive columns, expensive for nested types. If you only need a few columns, project the batch first.

from_spark_frame classmethod 

from_spark_frame(
    frame: "SparkDataFrame", *, schema: "Schema | None" = None
) -> Iterator["Record"]

Yield :class:Record s from a Spark DataFrame.

Rows stream lazily through frame.toLocalIterator() — the DataFrame is never collected as a whole, so the driver stays memory-bounded for large frames. The :class:Schema is derived once from :meth:Field.from_spark (or taken explicitly) and shared by reference across every yielded record.

Per-row values come from row.asDict(recursive=True) so nested structs land as plain Python dicts / lists, matching the Arrow-batch path's as_py() conventions instead of leaving Spark Row objects in the field values.