Skip to content

yggdrasil.data.schema

schema

StructField — a :class:Field whose dtype is a :class:StructType.

StructField is a thin type-hinting subclass of :class:Field: every schema-shaped operation lives on :class:Field itself. The subclass exists so the constructor can take a list of children directly (StructField([f1, f2])) instead of the dtype-by-dtype shape Field requires.

Schema is exported as a historical alias for StructField.

Field dataclass 

Field(
    name: str,
    dtype: DataType | type[DataType] | DataType,
    nullable: bool = True,
    metadata: dict[bytes | str, bytes | str | object] | None = None,
    tags: dict[bytes | str, bytes | str | object] | None = None,
    default: Any = None,
    parent: "Field | None" = None,
)

Bases: BaseChildrenFields

position property 

position: int | None

Optional 0-based index this field claims in a parent schema.

Stored in :attr:metadata under :data:POSITION_KEY. Used by :meth:select_in_field (and the engine-specific select_in_* helpers) as the last-resort fallback when :attr:name doesn't match a child name in the receiving schema — the receiver's children[position] (or column at position) is then resolved by name and used.

None (the default) leaves position-based lookup disabled, matching the historical name-only resolver.

default_value property 

default_value

Field's default Python value (or the dtype-level default).

Reads :data:DEFAULT_VALUE_KEY from :attr:metadata first; falls back to self.dtype.default_pyobj when the metadata slot is unset. Renamed from default so the constructor classmethod :meth:Field.default can take that name — field.default would otherwise shadow it via descriptor lookup.

media_type property 

media_type: 'Any | None'

:class:MediaType describing how this field's data is stored.

Decodes the b"media_type" metadata key — the mime-string canonical form ("application/vnd.apache.arrow.file", "application/vnd.apache.parquet", …) round-tripped through :meth:MediaType.from_. None when no media-type hint has been stamped.

Populated by :class:Folder._persist_schema so a schema loaded from a folder's .ygg/schema.arrow sidecar tells the reader which on-disk format the rows were last written in (Arrow IPC, Parquet, …) without walking the part files. Schema-level (top-level :class:StructField) is the canonical slot, but the accessor lives on :class:Field so per-column hints (e.g. the response-body field's HTTP Content-Type) can use the same property.

fields property 

fields: list['Field']

Children excluding constraint-only fields.

inner_fields property 

inner_fields: 'OrderedDict[str, Field]'

Compat view of the children as an ordered {name: field} map.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short() -> str

A compact name:dtype header tag — the dtype via :meth:~yggdrasil.data.types.base.DataType.short (recursive for nested types). Used for the column headers in :meth:yggdrasil.io.tabular.Tabular.display.

markers 

markers() -> str

The main schema markers for a preview header, space-joined ("" when none): the key / layout flags (PK / FK / CK / partition / cluster / sorted / IK) and a * for a non-nullable (required) column. The compact cousin of :meth:_pretty_markers.

default classmethod 

default(
    name: str = "",
    dtype: DataType = ObjectType(),
    nullable: bool = True,
    metadata: dict[bytes, bytes] | None = None,
    tags: dict[bytes, bytes] | None = None,
    default: Any = None,
)

Build a default-typed Field (ObjectType() unless overridden).

Convenience constructor for the "I just have a name" path — callers passing a plain string into APIs that expect a :class:Field (e.g. CastOptions(match_by=["id"])) land here. The instance-side default accessor was renamed to :attr:default_value so this name was free for the constructor.

pretty_format 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this field with the header on one line and the dtype below.

Layout is uniform across flat and nested dtypes — every field renders as a single field: 'name' <dtype>{markers} header line, with nested dtypes walking their inner fields inline at level + 1 so the tree reads as a flat list of consistent rows::

field: 'row' struct
  field: 'id' int64 not null [PK]
  field: 'name' string
  field: 'inner' struct
    field: 'age' int64
    field: 'email' string

indent is the per-level step in spaces; level is the current depth. The header carries the dtype kind (struct / list / map for nested, the primitive pretty-format for flat), the not null marker, the bracketed marker group (primary / foreign / constraint key, partition / cluster / sorted, any caller-defined tags, default value), and the comment.

Map dtypes flatten the synthetic entry struct into field: 'key' … / field: 'value' … lines so the key / value framing reads at the same level as a struct's own children rather than under an artificial wrapper.

Examples::

>>> print(field("id", "int64", nullable=False,
...             tags={"primary_key": True}).pretty_format())
field: 'id' int64 not null [PK]

>>> print(field("date", "date32",
...             tags={"partition_by": True}).pretty_format())
field: 'date' date32 [partition]

>>> print(field("user", StructType.from_fields([
...     field("id", "int64"),
...     field("email", "string"),
... ])).pretty_format())
field: 'user' struct
  field: 'id' int64
  field: 'email' string

invalidate_cache 

invalidate_cache(*, cascade: bool = True) -> None

Drop cached engine projections, cascading to ancestors by default.

Public surface over :meth:_invalidate_cache. Callers that mutate the underlying state outside of the with_* mutators (custom DataType subclass that swaps children in place, external code that pokes dtype.fields directly) should call this once to make sure the next to_arrow_field / to_polars_field / to_pyspark_field / *_schema request rebuilds with the new state. With cascade=True (the default) every ancestor reachable via :attr:parent also drops its cache, so a struct's cached arrow schema gets rebuilt after one of its children mutates.

equals 

equals(
    other: Any,
    check_names: bool = True,
    check_dtypes: bool = True,
    check_nullable: bool = True,
    check_metadata: bool = True,
) -> bool

Structural equality check with configurable scope.

Mirrors :meth:DataType.equals. Coerces other to a Field so that callers can pass a pa.Field / dict / etc. without manual conversion. Returns False on coercion failure instead of raising.

  • check_names: compare this field's name and recurse into child field names for nested types. For struct-shaped fields the comparison is order-independent (children matched by name) when check_names is True, mirroring how Arrow schemas are name-keyed.
  • check_dtypes: recurse into the dtype and compare nullable (both are structural, schema-defining attributes).
  • check_metadata: compare this field's metadata and recurse.

set_position 

set_position(value: int | None) -> 'Field'

Set / clear :attr:position on self in place.

Negative values are rejected — positions are forward indices into the parent schema; if you need a last-element fallback, resolve it before calling.

with_position 

with_position(position: int | None) -> 'Field'

Return a copy of this field with :attr:position set / cleared.

check_pandas_metadata 

check_pandas_metadata(source: Any = None) -> 'Field'

Stamp pandas index tags onto child fields from a b"pandas" blob.

pandas carries its DataFrame index layout in the pyarrow b"pandas" schema metadata (index_columns). This reads that blob and marks each matching child as an index level via :meth:with_index_key, so a struct-shaped Field round-trips the index when it later rebuilds a DataFrame.

source is whatever carries the blob — a pa.Schema, a pa.Table, raw bytes / str JSON, or an already-parsed dict. When omitted, falls back to self.metadata[b"pandas"] (which :meth:from_arrow_schema preserves). Mutates and returns self for chaining; a no-op when there's no blob or no string index columns.

PARITY: Python/pandas-only. The TS port has no pandas
counterpart, so there is no mirror for this method.

with_field 

with_field(
    field: "Field | pa.Field | str",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True,
    **kwargs: Any
) -> "Field"

Return self with field appended or merged in.

mode controls collision behavior when a child with the same name already exists. Accepts a :class:Mode member or any alias :meth:Mode.from_ understands.

  • :data:Mode.AUTO / :data:Mode.OVERWRITE — replace the existing child verbatim with field.
  • :data:Mode.APPEND — append a fresh child even if the name collides (struct semantics: last-write-wins for duplicate names; both entries survive in the children tuple).
  • :data:Mode.IGNORE — keep the existing child; drop the incoming.
  • :data:Mode.ERROR_IF_EXISTS — raise :class:ValueError on collision.
  • :data:Mode.UPSERT / :data:Mode.MERGE — :meth:merge_with the existing child against the incoming one (dtype, nullability, metadata), keeping the existing child's identity.

Auto-promotion to struct: when self isn't a struct (a primitive Field, a list/map, …) the call returns a fresh struct Field whose first child is the previous self (renamed to its current name so it's addressable) and whose second child is field. The promoted struct keeps self's name, nullability, and metadata — only the dtype changes.

Bare-string shorthand: self.with_field("price") reads as "make sure a child named 'price' exists." That call goes through :meth:Field.from_any which infers a sensible default dtype.

inplace=True (the default) mutates self and returns it. inplace=False returns a fresh copy.

with_fields 

with_fields(
    fields: "Iterable[Field | pa.Field | str]",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True
) -> "Field"

Apply :meth:with_field for every entry in fields.

Same mode semantics as :meth:with_field; the loop short- circuits :data:Mode.IGNORE once any one collision keeps the existing child (no global "first one wins, drop the rest" gymnastics — collisions are evaluated per name).

Auto-promotes self to a struct on the first call when needed; subsequent fields land on that struct.

autotag 

autotag(tags: dict[AnyStr, AnyStr] | None = None) -> 'Field'

Stamp this field with tags derived from its dtype and name.

Writes Databricks-friendly auto-tags in place:

  • Everything from :meth:DataType.autotag (kind plus dtype detail like unit / tz / precision / scale / signed / iso / srid).
  • nullable for data-quality policies.
  • Name-based heuristics for governance: role=identifier for *_id / *_uuid, role=audit_timestamp for created_at patterns, plus pii / sensitive stamps for columns that obviously carry personal or credential data.

For struct-shaped fields (schemas) primary_key / partition_by / cluster_by entries on this field's metadata get consumed into per-child tags, and each child is autotagged in turn — so schema.autotag() propagates without the caller having to walk children manually.

Returns a new struct-shaped Field for schema-style autotagging, or self for primitive autotagging — both modes also stamp in place so existing f.autotag() chains keep working.

from_field classmethod 

from_field(f: 'Field') -> 'Field'

Lift a :class:Field to cls.

For cls is Field this is identity. For subclasses (e.g. :class:Schema) it normalises the input to the subclass shape — for struct dtypes we keep the children, for non-struct we wrap the field as a single-child struct so the schema-shape contract holds.

from_fields classmethod 

from_fields(
    fields: Iterable["Field | Any"],
    *,
    name: str = DEFAULT_FIELD_NAME,
    nullable: bool = False,
    metadata: dict[bytes | str, bytes | str | object] | None = None,
    tags: dict[bytes | str, bytes | str | object] | None = None
) -> "Field"

Build a struct-shaped instance from a list of fields.

from_spark_column classmethod 

from_spark_column(column: 'ps.Column') -> 'Field'

Build a :class:Field from a pyspark.sql.Column.

Column objects don't expose a typed dtype on the public Python surface — we read the SQL-rendered expression instead and parse that:

  • id — bare reference. Name is id, dtype defers to the fallback (ObjectType) since neither the JVM nor the Spark Connect proxy exposes the underlying schema on a free-standing column.
  • CAST(<expr> AS <dtype>) / CAST(<expr> AS <dtype>) — name follows the inner <expr>'s leaf, dtype reads straight off <dtype> through :meth:DataType.from_str. Covers df["x"].cast("string"), df["x"].astype("decimal(10,2)"), F.col("x").cast(StringType()).
  • <expr> AS <alias> — name follows <alias>, dtype comes from the inner <expr> (recurses, so a cast inside an alias keeps its dtype).
  • Anything else falls back to the full SQL string as the name with :class:ObjectType as the dtype, since we can't infer the dtype of an arbitrary Catalyst expression without binding it through :meth:SparkSession.createDataFrame (which would be a live JVM round trip the caller didn't ask for).

Source of the SQL string, in order:

  1. Classic Spark: column._jc.toString() — the JVM Column.
  2. Spark Connect: column._expr.__repr__() — the proxy doesn't have _jc (accessing it raises PySparkAttributeError(JVM_ATTRIBUTE_NOT_SUPPORTED)) but _expr.__repr__ is exactly what Column.__repr__ wraps as "Column<'<sql>'>".
  3. repr(column) stripped of the Column<'…'> wrapper — last-resort for any future PySpark whose internal slots renamed.

Use :meth:Field.from_spark_field instead when the caller already has the resolved StructField (e.g. from df.schema.fields[i]) — that path keeps the precise dtype without going through the SQL string.

to_dict 

to_dict(dump_parent: bool = False) -> dict[str, Any]

Serialize this field to a JSON-friendly dict.

dump_parent (default False) controls whether :attr:parent — the structural back-pointer to the field this one is nested under — is included. Children are still emitted via the dtype's to_dict (a struct field's dtype carries its members), so dropping parent prevents the recursion that would otherwise echo the whole ancestor chain into every nested field's payload.

to_arrow_field 

to_arrow_field(dump_json: bool = False) -> pa.Field

Project to a :class:pa.Field.

Arrow preserves nested-type structure (struct, list, map) with per-field metadata recursively, so the dtype intent round-trips natively without us stuffing a type_json blob into the metadata. Only callers that need the exact :class:DataType subclass back (e.g. Decimal precision / Timestamp tz / extension types) should pass dump_json=True.

dump_json defaults to False; the cached path is the canonical (no-blob) shape, which is what every internal caller wants now that :meth:from_arrow_field falls back through :meth:DataType.from_arrow_type when the blob is missing.

to_arrow_schema 

to_arrow_schema() -> pa.Schema

Project this field as a top-level :class:pa.Schema.

Struct-shaped fields (including :class:~yggdrasil.data.Schema) unfold their children into the schema's columns; non-struct fields produce a single-column schema with self as that column. The schema-level metadata mirrors self.metadata, plus the field's name / nullable flag re-embedded as b"name" / b"nullable" so :meth:Field.from_arrow_schema can recover them (pa.Schema has no native slot for either).

to_polars_schema 

to_polars_schema() -> 'polars.Schema'

Project this field as a :class:polars.Schema.

Struct-shaped fields unfold into the schema's columns; non-struct fields produce a single-column schema.

to_pyspark_field 

to_pyspark_field() -> 'pst.StructField'

Project to a Spark :class:StructField.

Spark's :class:StructType preserves struct children with their own metadata, so primitive and struct dtypes don't need a type_json round-trip blob. Spark's :class:MapType / :class:ArrayType only carry the element / key+value Spark types and lose any field-level metadata on the way through, so we dump the dtype JSON for those (and only those) to recover the original yggdrasil dtype on read.

to_spark_schema 

to_spark_schema() -> 'pst.StructType'

Project this field as a top-level Spark :class:StructType.

Struct-shaped fields unfold their children into the StructType's fields; non-struct fields produce a single-field StructType.

as_spark 

as_spark() -> 'Field'

Return a Field whose dtype is Spark-compatible.

Stays on the yggdrasil side of the boundary — the result is still a :class:Field, just with :attr:dtype swapped for whatever self.dtype.as_spark() produced (an unsigned int widens to signed, a non-UTC timestamp drops to naive, TimeType becomes StringType, …). When the dtype is already Spark-compatible the same instance is returned, so the call is cheap to make defensively.

Use :meth:to_pyspark_field when you need an actual pyspark.sql.types.StructField.

as_polars 

as_polars() -> 'Field'

Return a Field whose dtype is Polars-compatible.

Mirrors :meth:as_spark for Polars — :attr:dtype is swapped for self.dtype.as_polars() (sub-32-bit floats widen to Float32Type, second-precision timestamps / durations widen to milliseconds, nested types recurse). Already-Polars-compatible fields return self so the call is cheap to make defensively. Use :meth:to_polars_field when you need a real pl.Field.

cast 

cast(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast obj to this field using its native engine.

Routing is by module prefix via :meth:ObjectSerde.module_and_name:

  • pyarrow.* → :meth:cast_arrow
  • polars.* → :meth:cast_polars
  • pandas.* → :meth:cast_pandas
  • pyspark.* → :meth:cast_spark
  • iterator / iterable → recurse per element (lazy generator)
  • everything else → :class:TypeError

self.dtype.type_id == OBJECT is handled by the narrow methods — they pass obj through unchanged because a variant column must never be cast. No redundant guard here.

cast_arrow 

cast_arrow(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pyarrow object — dispatch by shape.

Table/RecordBatch → :meth:cast_arrow_tabular, Array/ChunkedArray → :meth:cast_arrow_array.

cast_polars 

cast_polars(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any polars object — dispatch by shape.

DataFrame/LazyFrame → :meth:cast_polars_tabular, Series → :meth:cast_polars_series, Expr → :meth:cast_polars_expr.

cast_pandas 

cast_pandas(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pandas object — dispatch by shape.

DataFrame → :meth:cast_pandas_tabular + index check, Series → :meth:cast_pandas_series.

cast_spark 

cast_spark(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any spark object — dispatch by shape.

DataFrame → :meth:cast_spark_tabular, Column → :meth:cast_spark_column.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this field.

Object targets passthrough (variant). Otherwise the dtype's struct view owns the per-batch tabular cast and byte_size rechunk — same shape contract as :meth:cast_arrow_tabular, just lazy.

fill_nulls 

fill_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in obj using the native engine — engine + shape detection.

Routes the same way :meth:cast does. See :meth:fill_arrow / :meth:fill_polars / :meth:fill_pandas / :meth:fill_spark for the per-engine behaviour.

fill_arrow 

fill_arrow(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pyarrow object.

Arrays go through :meth:fill_arrow_array_nulls directly. Tables / RecordBatches re-use the tabular cast path with self as the target — a no-op cast that still runs the per-column null-fill via the struct walk.

fill_polars 

fill_polars(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any polars object.

Series / Expr go through :meth:fill_polars_array_nulls — which handles both shapes uniformly (Expr is the lazy counterpart of Series; the fill operator grafts onto each identically). DataFrame / LazyFrame route through :meth:cast_polars_tabular as a self-targeted cast.

fill_pandas 

fill_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pandas object.

fill_spark 

fill_spark(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any spark object.

polars_alias 

polars_alias(obj: Any) -> Any

Rename a polars Series / Expr to match this field's name.

No-op when the target name matches the current name, or when this field only has the sentinel name. Calling defensively is free — zero-cost on the no-rename path.

spark_alias 

spark_alias(obj: Any) -> Any

Rename a Spark Column to match this field's name.

Spark DataFrames aren't handled — renaming a DataFrame requires a projection with named columns, which isn't a single-method operation. Column is the rename target here.

pandas_alias 

pandas_alias(obj: Any) -> Any

Rename a pandas Series to match this field's name.

Pandas has no .alias() — rename is series.name = ..., which mutates. This helper returns the series so it chains like :meth:polars_alias / :meth:spark_alias. DataFrames aren't handled (column rename is a projection, not a single-method op).

finalize_arrow_array 

finalize_arrow_array(
    array: Array | ChunkedArray, *, default_scalar: Scalar | None = None
)

Fill nulls on a pyarrow Array / ChunkedArray.

No alias step: pa.Array / ChunkedArray don't carry a name. Tabular naming lives in the pa.Field that wraps the array in a Table/RecordBatch, which :meth:cast_arrow_tabular handles through the struct walk.

finalize_arrow 

finalize_arrow(obj: Any, *, default_scalar: Scalar | None = None) -> Any

Finalize any pyarrow object — dispatch by shape.

Array/ChunkedArray → fill. Table/RecordBatch → identity.

finalize_polars_series 

finalize_polars_series(series: 'polars.Series', *, default_scalar: Any = None)

Fill nulls, alias a polars Series to the target name.

finalize_polars_expr 

finalize_polars_expr(expr: 'polars.Expr', *, default_scalar: Any = None)

Fill nulls, alias a polars Expr to the target name.

Same as :meth:finalize_polars_series — polars Series and Expr share the fill + alias primitives, so the finalize shape is identical. Separate method for call-site clarity.

finalize_polars 

finalize_polars(
    obj: "polars.Series | polars.Expr | polars.DataFrame | polars.LazyFrame",
    *,
    default_scalar: Any = None
)

Finalize any polars object — dispatch by shape.

Series/Expr → fill + alias. DataFrame/LazyFrame → identity (tabular cast already finalized per-column via the struct walk).

finalize_pandas_series 

finalize_pandas_series(series: 'pd.Series', *, default_scalar: Any = None)

Fill nulls, rename a pandas Series to the target name.

finalize_pandas 

finalize_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any pandas object — dispatch by shape.

Series → fill + rename. DataFrame → identity.

check_pandas_indexes 

check_pandas_indexes(obj: Any) -> Any

Promote columns tagged index_key to the DataFrame index.

Collects children with :attr:index_key set, sorted by :attr:index_key_level, and calls set_index on the DataFrame. __index_level_N__ placeholder names are mapped back to None so the round-trip matches the source.

For a Series whose field is itself tagged index_key, the Series is returned as-is — the caller decides how to attach it as an index.

Passthrough when no children carry the tag or when the object is not a DataFrame.

finalize_spark_column 

finalize_spark_column(
    column: "ps.Column", *, default_scalar: Any = None
) -> "ps.Column"

Fill nulls, alias a Spark Column to the target name.

finalize_spark 

finalize_spark(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any spark object — dispatch by shape.

Column → fill + alias. DataFrame → identity (tabular cast already finalized).

finalize 

finalize(obj: Any, *, default_scalar: Any = None) -> Any

Finalize obj using its native engine — module-prefix dispatch.

Mirrors :meth:cast / :meth:fill_nulls routing.

select 

select(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field with only the selected children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).

drop 

drop(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field without the specified children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).

StructField dataclass 

StructField(*args: Any, **kwargs: Any)

Bases: Field

A :class:Field whose dtype is a :class:StructType.

StructField([f1, f2, ...]) builds a struct field from its children directly — sugar for the equivalent Field(name=..., dtype=StructType(fields=(...,))) chain that :class:Field's constructor accepts. Every schema-shaped method (mapping surface, set operators, engine schema export, autotag, struct-aware equals) is inherited from :class:Field.

StructField(fields, name=..., metadata=..., ...).

fields is the children list — :class:Field instances, :class:pa.Field instances, or anything :meth:Field.from_any accepts. dtype=<StructType> is accepted as an alternative for callers (and the Field.__new__ redirect) that already have a built struct dtype in hand; pass one or the other, not both.

The wide *args / **kwargs shape is here because Field.__new__ redirects struct-shaped Field(...) calls (positional or keyword) to this class and Python then re-enters __init__ on the already-stamped instance with the original Field arguments. The re-init guard below absorbs that pass before any signature parsing runs.

position property 

position: int | None

Optional 0-based index this field claims in a parent schema.

Stored in :attr:metadata under :data:POSITION_KEY. Used by :meth:select_in_field (and the engine-specific select_in_* helpers) as the last-resort fallback when :attr:name doesn't match a child name in the receiving schema — the receiver's children[position] (or column at position) is then resolved by name and used.

None (the default) leaves position-based lookup disabled, matching the historical name-only resolver.

default_value property 

default_value

Field's default Python value (or the dtype-level default).

Reads :data:DEFAULT_VALUE_KEY from :attr:metadata first; falls back to self.dtype.default_pyobj when the metadata slot is unset. Renamed from default so the constructor classmethod :meth:Field.default can take that name — field.default would otherwise shadow it via descriptor lookup.

media_type property 

media_type: 'Any | None'

:class:MediaType describing how this field's data is stored.

Decodes the b"media_type" metadata key — the mime-string canonical form ("application/vnd.apache.arrow.file", "application/vnd.apache.parquet", …) round-tripped through :meth:MediaType.from_. None when no media-type hint has been stamped.

Populated by :class:Folder._persist_schema so a schema loaded from a folder's .ygg/schema.arrow sidecar tells the reader which on-disk format the rows were last written in (Arrow IPC, Parquet, …) without walking the part files. Schema-level (top-level :class:StructField) is the canonical slot, but the accessor lives on :class:Field so per-column hints (e.g. the response-body field's HTTP Content-Type) can use the same property.

fields property 

fields: list['Field']

Children excluding constraint-only fields.

inner_fields property 

inner_fields: 'OrderedDict[str, Field]'

Compat view of the children as an ordered {name: field} map.

to_field 

to_field() -> Field

Identity — StructField IS a Field. Kept for type-narrowing callers.

as_spark 

as_spark() -> 'StructField'

Return a :class:StructField whose dtype is Spark-compatible.

as_polars 

as_polars() -> 'StructField'

Return a :class:StructField whose dtype is Polars-compatible.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short() -> str

A compact name:dtype header tag — the dtype via :meth:~yggdrasil.data.types.base.DataType.short (recursive for nested types). Used for the column headers in :meth:yggdrasil.io.tabular.Tabular.display.

markers 

markers() -> str

The main schema markers for a preview header, space-joined ("" when none): the key / layout flags (PK / FK / CK / partition / cluster / sorted / IK) and a * for a non-nullable (required) column. The compact cousin of :meth:_pretty_markers.

default classmethod 

default(
    name: str = "",
    dtype: DataType = ObjectType(),
    nullable: bool = True,
    metadata: dict[bytes, bytes] | None = None,
    tags: dict[bytes, bytes] | None = None,
    default: Any = None,
)

Build a default-typed Field (ObjectType() unless overridden).

Convenience constructor for the "I just have a name" path — callers passing a plain string into APIs that expect a :class:Field (e.g. CastOptions(match_by=["id"])) land here. The instance-side default accessor was renamed to :attr:default_value so this name was free for the constructor.

pretty_format 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this field with the header on one line and the dtype below.

Layout is uniform across flat and nested dtypes — every field renders as a single field: 'name' <dtype>{markers} header line, with nested dtypes walking their inner fields inline at level + 1 so the tree reads as a flat list of consistent rows::

field: 'row' struct
  field: 'id' int64 not null [PK]
  field: 'name' string
  field: 'inner' struct
    field: 'age' int64
    field: 'email' string

indent is the per-level step in spaces; level is the current depth. The header carries the dtype kind (struct / list / map for nested, the primitive pretty-format for flat), the not null marker, the bracketed marker group (primary / foreign / constraint key, partition / cluster / sorted, any caller-defined tags, default value), and the comment.

Map dtypes flatten the synthetic entry struct into field: 'key' … / field: 'value' … lines so the key / value framing reads at the same level as a struct's own children rather than under an artificial wrapper.

Examples::

>>> print(field("id", "int64", nullable=False,
...             tags={"primary_key": True}).pretty_format())
field: 'id' int64 not null [PK]

>>> print(field("date", "date32",
...             tags={"partition_by": True}).pretty_format())
field: 'date' date32 [partition]

>>> print(field("user", StructType.from_fields([
...     field("id", "int64"),
...     field("email", "string"),
... ])).pretty_format())
field: 'user' struct
  field: 'id' int64
  field: 'email' string

invalidate_cache 

invalidate_cache(*, cascade: bool = True) -> None

Drop cached engine projections, cascading to ancestors by default.

Public surface over :meth:_invalidate_cache. Callers that mutate the underlying state outside of the with_* mutators (custom DataType subclass that swaps children in place, external code that pokes dtype.fields directly) should call this once to make sure the next to_arrow_field / to_polars_field / to_pyspark_field / *_schema request rebuilds with the new state. With cascade=True (the default) every ancestor reachable via :attr:parent also drops its cache, so a struct's cached arrow schema gets rebuilt after one of its children mutates.

equals 

equals(
    other: Any,
    check_names: bool = True,
    check_dtypes: bool = True,
    check_nullable: bool = True,
    check_metadata: bool = True,
) -> bool

Structural equality check with configurable scope.

Mirrors :meth:DataType.equals. Coerces other to a Field so that callers can pass a pa.Field / dict / etc. without manual conversion. Returns False on coercion failure instead of raising.

  • check_names: compare this field's name and recurse into child field names for nested types. For struct-shaped fields the comparison is order-independent (children matched by name) when check_names is True, mirroring how Arrow schemas are name-keyed.
  • check_dtypes: recurse into the dtype and compare nullable (both are structural, schema-defining attributes).
  • check_metadata: compare this field's metadata and recurse.

set_position 

set_position(value: int | None) -> 'Field'

Set / clear :attr:position on self in place.

Negative values are rejected — positions are forward indices into the parent schema; if you need a last-element fallback, resolve it before calling.

with_position 

with_position(position: int | None) -> 'Field'

Return a copy of this field with :attr:position set / cleared.

check_pandas_metadata 

check_pandas_metadata(source: Any = None) -> 'Field'

Stamp pandas index tags onto child fields from a b"pandas" blob.

pandas carries its DataFrame index layout in the pyarrow b"pandas" schema metadata (index_columns). This reads that blob and marks each matching child as an index level via :meth:with_index_key, so a struct-shaped Field round-trips the index when it later rebuilds a DataFrame.

source is whatever carries the blob — a pa.Schema, a pa.Table, raw bytes / str JSON, or an already-parsed dict. When omitted, falls back to self.metadata[b"pandas"] (which :meth:from_arrow_schema preserves). Mutates and returns self for chaining; a no-op when there's no blob or no string index columns.

PARITY: Python/pandas-only. The TS port has no pandas
counterpart, so there is no mirror for this method.

with_field 

with_field(
    field: "Field | pa.Field | str",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True,
    **kwargs: Any
) -> "Field"

Return self with field appended or merged in.

mode controls collision behavior when a child with the same name already exists. Accepts a :class:Mode member or any alias :meth:Mode.from_ understands.

  • :data:Mode.AUTO / :data:Mode.OVERWRITE — replace the existing child verbatim with field.
  • :data:Mode.APPEND — append a fresh child even if the name collides (struct semantics: last-write-wins for duplicate names; both entries survive in the children tuple).
  • :data:Mode.IGNORE — keep the existing child; drop the incoming.
  • :data:Mode.ERROR_IF_EXISTS — raise :class:ValueError on collision.
  • :data:Mode.UPSERT / :data:Mode.MERGE — :meth:merge_with the existing child against the incoming one (dtype, nullability, metadata), keeping the existing child's identity.

Auto-promotion to struct: when self isn't a struct (a primitive Field, a list/map, …) the call returns a fresh struct Field whose first child is the previous self (renamed to its current name so it's addressable) and whose second child is field. The promoted struct keeps self's name, nullability, and metadata — only the dtype changes.

Bare-string shorthand: self.with_field("price") reads as "make sure a child named 'price' exists." That call goes through :meth:Field.from_any which infers a sensible default dtype.

inplace=True (the default) mutates self and returns it. inplace=False returns a fresh copy.

with_fields 

with_fields(
    fields: "Iterable[Field | pa.Field | str]",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True
) -> "Field"

Apply :meth:with_field for every entry in fields.

Same mode semantics as :meth:with_field; the loop short- circuits :data:Mode.IGNORE once any one collision keeps the existing child (no global "first one wins, drop the rest" gymnastics — collisions are evaluated per name).

Auto-promotes self to a struct on the first call when needed; subsequent fields land on that struct.

autotag 

autotag(tags: dict[AnyStr, AnyStr] | None = None) -> 'Field'

Stamp this field with tags derived from its dtype and name.

Writes Databricks-friendly auto-tags in place:

  • Everything from :meth:DataType.autotag (kind plus dtype detail like unit / tz / precision / scale / signed / iso / srid).
  • nullable for data-quality policies.
  • Name-based heuristics for governance: role=identifier for *_id / *_uuid, role=audit_timestamp for created_at patterns, plus pii / sensitive stamps for columns that obviously carry personal or credential data.

For struct-shaped fields (schemas) primary_key / partition_by / cluster_by entries on this field's metadata get consumed into per-child tags, and each child is autotagged in turn — so schema.autotag() propagates without the caller having to walk children manually.

Returns a new struct-shaped Field for schema-style autotagging, or self for primitive autotagging — both modes also stamp in place so existing f.autotag() chains keep working.

from_field classmethod 

from_field(f: 'Field') -> 'Field'

Lift a :class:Field to cls.

For cls is Field this is identity. For subclasses (e.g. :class:Schema) it normalises the input to the subclass shape — for struct dtypes we keep the children, for non-struct we wrap the field as a single-child struct so the schema-shape contract holds.

from_fields classmethod 

from_fields(
    fields: Iterable["Field | Any"],
    *,
    name: str = DEFAULT_FIELD_NAME,
    nullable: bool = False,
    metadata: dict[bytes | str, bytes | str | object] | None = None,
    tags: dict[bytes | str, bytes | str | object] | None = None
) -> "Field"

Build a struct-shaped instance from a list of fields.

from_spark_column classmethod 

from_spark_column(column: 'ps.Column') -> 'Field'

Build a :class:Field from a pyspark.sql.Column.

Column objects don't expose a typed dtype on the public Python surface — we read the SQL-rendered expression instead and parse that:

  • id — bare reference. Name is id, dtype defers to the fallback (ObjectType) since neither the JVM nor the Spark Connect proxy exposes the underlying schema on a free-standing column.
  • CAST(<expr> AS <dtype>) / CAST(<expr> AS <dtype>) — name follows the inner <expr>'s leaf, dtype reads straight off <dtype> through :meth:DataType.from_str. Covers df["x"].cast("string"), df["x"].astype("decimal(10,2)"), F.col("x").cast(StringType()).
  • <expr> AS <alias> — name follows <alias>, dtype comes from the inner <expr> (recurses, so a cast inside an alias keeps its dtype).
  • Anything else falls back to the full SQL string as the name with :class:ObjectType as the dtype, since we can't infer the dtype of an arbitrary Catalyst expression without binding it through :meth:SparkSession.createDataFrame (which would be a live JVM round trip the caller didn't ask for).

Source of the SQL string, in order:

  1. Classic Spark: column._jc.toString() — the JVM Column.
  2. Spark Connect: column._expr.__repr__() — the proxy doesn't have _jc (accessing it raises PySparkAttributeError(JVM_ATTRIBUTE_NOT_SUPPORTED)) but _expr.__repr__ is exactly what Column.__repr__ wraps as "Column<'<sql>'>".
  3. repr(column) stripped of the Column<'…'> wrapper — last-resort for any future PySpark whose internal slots renamed.

Use :meth:Field.from_spark_field instead when the caller already has the resolved StructField (e.g. from df.schema.fields[i]) — that path keeps the precise dtype without going through the SQL string.

to_dict 

to_dict(dump_parent: bool = False) -> dict[str, Any]

Serialize this field to a JSON-friendly dict.

dump_parent (default False) controls whether :attr:parent — the structural back-pointer to the field this one is nested under — is included. Children are still emitted via the dtype's to_dict (a struct field's dtype carries its members), so dropping parent prevents the recursion that would otherwise echo the whole ancestor chain into every nested field's payload.

to_arrow_field 

to_arrow_field(dump_json: bool = False) -> pa.Field

Project to a :class:pa.Field.

Arrow preserves nested-type structure (struct, list, map) with per-field metadata recursively, so the dtype intent round-trips natively without us stuffing a type_json blob into the metadata. Only callers that need the exact :class:DataType subclass back (e.g. Decimal precision / Timestamp tz / extension types) should pass dump_json=True.

dump_json defaults to False; the cached path is the canonical (no-blob) shape, which is what every internal caller wants now that :meth:from_arrow_field falls back through :meth:DataType.from_arrow_type when the blob is missing.

to_arrow_schema 

to_arrow_schema() -> pa.Schema

Project this field as a top-level :class:pa.Schema.

Struct-shaped fields (including :class:~yggdrasil.data.Schema) unfold their children into the schema's columns; non-struct fields produce a single-column schema with self as that column. The schema-level metadata mirrors self.metadata, plus the field's name / nullable flag re-embedded as b"name" / b"nullable" so :meth:Field.from_arrow_schema can recover them (pa.Schema has no native slot for either).

to_polars_schema 

to_polars_schema() -> 'polars.Schema'

Project this field as a :class:polars.Schema.

Struct-shaped fields unfold into the schema's columns; non-struct fields produce a single-column schema.

to_pyspark_field 

to_pyspark_field() -> 'pst.StructField'

Project to a Spark :class:StructField.

Spark's :class:StructType preserves struct children with their own metadata, so primitive and struct dtypes don't need a type_json round-trip blob. Spark's :class:MapType / :class:ArrayType only carry the element / key+value Spark types and lose any field-level metadata on the way through, so we dump the dtype JSON for those (and only those) to recover the original yggdrasil dtype on read.

to_spark_schema 

to_spark_schema() -> 'pst.StructType'

Project this field as a top-level Spark :class:StructType.

Struct-shaped fields unfold their children into the StructType's fields; non-struct fields produce a single-field StructType.

cast 

cast(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast obj to this field using its native engine.

Routing is by module prefix via :meth:ObjectSerde.module_and_name:

  • pyarrow.* → :meth:cast_arrow
  • polars.* → :meth:cast_polars
  • pandas.* → :meth:cast_pandas
  • pyspark.* → :meth:cast_spark
  • iterator / iterable → recurse per element (lazy generator)
  • everything else → :class:TypeError

self.dtype.type_id == OBJECT is handled by the narrow methods — they pass obj through unchanged because a variant column must never be cast. No redundant guard here.

cast_arrow 

cast_arrow(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pyarrow object — dispatch by shape.

Table/RecordBatch → :meth:cast_arrow_tabular, Array/ChunkedArray → :meth:cast_arrow_array.

cast_polars 

cast_polars(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any polars object — dispatch by shape.

DataFrame/LazyFrame → :meth:cast_polars_tabular, Series → :meth:cast_polars_series, Expr → :meth:cast_polars_expr.

cast_pandas 

cast_pandas(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pandas object — dispatch by shape.

DataFrame → :meth:cast_pandas_tabular + index check, Series → :meth:cast_pandas_series.

cast_spark 

cast_spark(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any spark object — dispatch by shape.

DataFrame → :meth:cast_spark_tabular, Column → :meth:cast_spark_column.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this field.

Object targets passthrough (variant). Otherwise the dtype's struct view owns the per-batch tabular cast and byte_size rechunk — same shape contract as :meth:cast_arrow_tabular, just lazy.

fill_nulls 

fill_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in obj using the native engine — engine + shape detection.

Routes the same way :meth:cast does. See :meth:fill_arrow / :meth:fill_polars / :meth:fill_pandas / :meth:fill_spark for the per-engine behaviour.

fill_arrow 

fill_arrow(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pyarrow object.

Arrays go through :meth:fill_arrow_array_nulls directly. Tables / RecordBatches re-use the tabular cast path with self as the target — a no-op cast that still runs the per-column null-fill via the struct walk.

fill_polars 

fill_polars(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any polars object.

Series / Expr go through :meth:fill_polars_array_nulls — which handles both shapes uniformly (Expr is the lazy counterpart of Series; the fill operator grafts onto each identically). DataFrame / LazyFrame route through :meth:cast_polars_tabular as a self-targeted cast.

fill_pandas 

fill_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pandas object.

fill_spark 

fill_spark(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any spark object.

polars_alias 

polars_alias(obj: Any) -> Any

Rename a polars Series / Expr to match this field's name.

No-op when the target name matches the current name, or when this field only has the sentinel name. Calling defensively is free — zero-cost on the no-rename path.

spark_alias 

spark_alias(obj: Any) -> Any

Rename a Spark Column to match this field's name.

Spark DataFrames aren't handled — renaming a DataFrame requires a projection with named columns, which isn't a single-method operation. Column is the rename target here.

pandas_alias 

pandas_alias(obj: Any) -> Any

Rename a pandas Series to match this field's name.

Pandas has no .alias() — rename is series.name = ..., which mutates. This helper returns the series so it chains like :meth:polars_alias / :meth:spark_alias. DataFrames aren't handled (column rename is a projection, not a single-method op).

finalize_arrow_array 

finalize_arrow_array(
    array: Array | ChunkedArray, *, default_scalar: Scalar | None = None
)

Fill nulls on a pyarrow Array / ChunkedArray.

No alias step: pa.Array / ChunkedArray don't carry a name. Tabular naming lives in the pa.Field that wraps the array in a Table/RecordBatch, which :meth:cast_arrow_tabular handles through the struct walk.

finalize_arrow 

finalize_arrow(obj: Any, *, default_scalar: Scalar | None = None) -> Any

Finalize any pyarrow object — dispatch by shape.

Array/ChunkedArray → fill. Table/RecordBatch → identity.

finalize_polars_series 

finalize_polars_series(series: 'polars.Series', *, default_scalar: Any = None)

Fill nulls, alias a polars Series to the target name.

finalize_polars_expr 

finalize_polars_expr(expr: 'polars.Expr', *, default_scalar: Any = None)

Fill nulls, alias a polars Expr to the target name.

Same as :meth:finalize_polars_series — polars Series and Expr share the fill + alias primitives, so the finalize shape is identical. Separate method for call-site clarity.

finalize_polars 

finalize_polars(
    obj: "polars.Series | polars.Expr | polars.DataFrame | polars.LazyFrame",
    *,
    default_scalar: Any = None
)

Finalize any polars object — dispatch by shape.

Series/Expr → fill + alias. DataFrame/LazyFrame → identity (tabular cast already finalized per-column via the struct walk).

finalize_pandas_series 

finalize_pandas_series(series: 'pd.Series', *, default_scalar: Any = None)

Fill nulls, rename a pandas Series to the target name.

finalize_pandas 

finalize_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any pandas object — dispatch by shape.

Series → fill + rename. DataFrame → identity.

check_pandas_indexes 

check_pandas_indexes(obj: Any) -> Any

Promote columns tagged index_key to the DataFrame index.

Collects children with :attr:index_key set, sorted by :attr:index_key_level, and calls set_index on the DataFrame. __index_level_N__ placeholder names are mapped back to None so the round-trip matches the source.

For a Series whose field is itself tagged index_key, the Series is returned as-is — the caller decides how to attach it as an index.

Passthrough when no children carry the tag or when the object is not a DataFrame.

finalize_spark_column 

finalize_spark_column(
    column: "ps.Column", *, default_scalar: Any = None
) -> "ps.Column"

Fill nulls, alias a Spark Column to the target name.

finalize_spark 

finalize_spark(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any spark object — dispatch by shape.

Column → fill + alias. DataFrame → identity (tabular cast already finalized).

finalize 

finalize(obj: Any, *, default_scalar: Any = None) -> Any

Finalize obj using its native engine — module-prefix dispatch.

Mirrors :meth:cast / :meth:fill_nulls routing.

select 

select(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field with only the selected children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).

drop 

drop(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field without the specified children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).