Skip to content

yggdrasil.data.types

types

BJsonType dataclass 

BJsonType(
    byte_size: int | None = None, large: bool = False, view: bool = False
)

Bases: BinaryType

JSON encoded as bytes (binary / packed JSON).

Storage matches :class:BinaryType — pa.binary() / pa.large_binary() / pa.binary_view() / fixed-width pa.binary(n) depending on flags. The wire format is plain UTF-8 JSON unless a downstream consumer interprets it differently (BSON, MessagePack-encoded JSON, etc.); yggdrasil itself only writes UTF-8 JSON when encoding from nested / scalar values.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

Float8Type dataclass 

Float8Type(byte_size: int | None = 1)

Bases: FloatingPointType

1-byte FP8 — the storage tag for ML-framework Float8 variants.

No native Arrow / Polars / Spark equivalent: to_arrow and to_polars widen to 32-bit, to_spark produces FloatType, and :meth:FloatingPointType.as_spark collapses to Float32Type so downstream Spark pipelines see a width Spark can actually represent. Carry the tag through schemas / round- trips at the yggdrasil layer; convert at the boundary.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

IntegerType dataclass 

IntegerType(byte_size: int | None = None, signed: bool = True)

Bases: NumericType

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

reinterpret_pyobj 

reinterpret_pyobj(value: int) -> int

Reinterpret value as this type's two's-complement form.

Mirrors what pyarrow.compute.cast(..., safe=False) does for signed↔unsigned casts at the Arrow level — values are masked to byte_size * 8 bits and read back with this type's signedness. max(uint64) becomes -1 when reinterpreted as int64; -1 becomes max(uint64) reinterpreted the other way.

NullType dataclass 

NullType(byte_size: int | None = None)

Bases: PrimitiveType

All-null column. Identity under merge; no-op on cast.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

NumericType dataclass 

NumericType(byte_size: int | None = None)

Bases: PrimitiveType, ABC

Abstract base for every numeric leaf type.

Sits between :class:PrimitiveType and the concrete numeric leaves. Handles cross-engine empty-string → null normalization (Databricks CSV ingest convention) and bridges sibling integer / float subclasses through merge_with so the specialized fixed-width classes still merge as one family.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

pretty_format abstractmethod 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this dtype with one element per line.

indent is the per-level step in spaces. level is the current depth — the line is prefixed with indent * level spaces. Flat dtypes render as a single line; nested dtypes (struct / list / map) override this to lay each child out on its own line at level + 1.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

PrimitiveType dataclass 

PrimitiveType(byte_size: int | None = None)

Bases: DataType, ABC

Shared scalar-shaped base for every non-nested leaf type.

The only state is byte_size — a physical width hint that flows into Arrow / DDL encoding. Subclasses layer on their own frozen-dataclass fields (signed, precision/scale, unit/tz, ...) and override the cast / autotag / to_* hooks as needed.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

pretty_format abstractmethod 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this dtype with one element per line.

indent is the per-level step in spaces. level is the current depth — the line is prefixed with indent * level spaces. Flat dtypes render as a single line; nested dtypes (struct / list / map) override this to lay each child out on its own line at level + 1.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

SJsonType dataclass 

SJsonType(
    byte_size: int | None = None,
    large: bool = False,
    view: bool = False,
    fixed_size: bool = False,
)

Bases: StringType

JSON encoded as a UTF-8 string (text JSON).

to_arrow() emits pa.string() (or pa.large_string() / pa.string_view() when large / view are set) and Polars / Spark land on their plain string types — there is no ABI-stable native JSON dtype in any of those engines that we want to bind to. The distinction lives at the yggdrasil layer: a column declared sjson round-trips through schema metadata, gets the right cast behavior (encode on the way in, decode on the way out), and shows up as sjson in pretty-printed output.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

TemporalType dataclass 

TemporalType(
    byte_size: int | None = None,
    unit: TimeUnit = TimeUnit.MICROSECOND,
    tz: str | None = None,
)

Bases: PrimitiveType, ABC

Base class for Date / Time / Timestamp / Duration.

Holds shared fields (unit / tz) and cross-engine dispatch logic. Subclasses implement type_id, the handles_* / from_* class methods, and to_arrow / to_polars / to_spark / to_spark_name.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

pretty_format abstractmethod 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this dtype with one element per line.

indent is the per-level step in spaces. level is the current depth — the line is prefixed with indent * level spaces. Flat dtypes render as a single line; nested dtypes (struct / list / map) override this to lay each child out on its own line at level + 1.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

TimestampType dataclass 

TimestampType(
    byte_size: int | None = None,
    unit: TimeUnit = TimeUnit.MICROSECOND,
    tz: Timezone = Timezone.NAIVE,
)

Bases: TemporalType

tz_iana property 

tz_iana: str | None

IANA token for :attr:tz, or None when naive.

Bridge for engine APIs (pa.timestamp, pl.Datetime, Spark) that take a string. New code should prefer self.tz directly — it carries :class:Timezone helpers like is_utc() and utc_offset().

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

ObjectType dataclass 

ObjectType()

Bases: DataType

Variant type — opaque Python object with no fixed schema.

The catch-all for values that don't map cleanly to any structured or primitive type. Cast operations against ObjectType are no-ops: the input passes through untouched, which is what you want for a variant column carrying heterogeneous data.

Engine mapping:

  • Polars: pl.Object
  • pandas: object dtype
  • Arrow: large_binary() (physical stand-in; Arrow has no variant type)
  • Spark: BinaryType() (physical stand-in)

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

autotag 

autotag() -> dict[bytes, bytes]

Return a dict of Databricks-friendly tags derived from this type.

These are auto-tags: they describe shape, not intent. The base output is a single kind key — a lowercase form of type_id ("integer", "string", "timestamp", "array", ...) that tag-based Unity Catalog policies can match on. Subclasses extend this with dtype-specific detail (unit, tz, precision / scale, signed, srid, ...) — always via super().autotag() so the kind key stays present.

Keys are bare (no t: prefix) — prefixing is handled by BaseMetadata.update_tags when these land on a Field.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

DataType dataclass 

DataType()

Bases: BaseChildrenFields, ABC

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

pretty_format abstractmethod 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this dtype with one element per line.

indent is the per-level step in spaces. level is the current depth — the line is prefixed with indent * level spaces. Flat dtypes render as a single line; nested dtypes (struct / list / map) override this to lay each child out on its own line at level + 1.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

autotag 

autotag() -> dict[bytes, bytes]

Return a dict of Databricks-friendly tags derived from this type.

These are auto-tags: they describe shape, not intent. The base output is a single kind key — a lowercase form of type_id ("integer", "string", "timestamp", "array", ...) that tag-based Unity Catalog policies can match on. Subclasses extend this with dtype-specific detail (unit, tz, precision / scale, signed, srid, ...) — always via super().autotag() so the kind key stays present.

Keys are bare (no t: prefix) — prefixing is handled by BaseMetadata.update_tags when these land on a Field.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

NestedType dataclass 

NestedType()

Bases: DataType, ABC

Base class for nested data types (struct, map, array).

Nested types expose their inner fields via children and therefore share a common equals implementation that compares their children pairwise.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

pretty_format abstractmethod 

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this dtype with one element per line.

indent is the per-level step in spaces. level is the current depth — the line is prefixed with indent * level spaces. Flat dtypes render as a single line; nested dtypes (struct / list / map) override this to lay each child out on its own line at level + 1.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

autotag 

autotag() -> dict[bytes, bytes]

Return a dict of Databricks-friendly tags derived from this type.

These are auto-tags: they describe shape, not intent. The base output is a single kind key — a lowercase form of type_id ("integer", "string", "timestamp", "array", ...) that tag-based Unity Catalog policies can match on. Subclasses extend this with dtype-specific detail (unit, tz, precision / scale, signed, srid, ...) — always via super().autotag() so the kind key stays present.

Keys are bare (no t: prefix) — prefixing is handled by BaseMetadata.update_tags when these land on a Field.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

StructType dataclass 

StructType(fields: tuple['Field'] = tuple())

Bases: NestedType

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

autotag 

autotag() -> dict[bytes, bytes]

Return a dict of Databricks-friendly tags derived from this type.

These are auto-tags: they describe shape, not intent. The base output is a single kind key — a lowercase form of type_id ("integer", "string", "timestamp", "array", ...) that tag-based Unity Catalog policies can match on. Subclasses extend this with dtype-specific detail (unit, tz, precision / scale, signed, srid, ...) — always via super().autotag() so the kind key stays present.

Keys are bare (no t: prefix) — prefixing is handled by BaseMetadata.update_tags when these land on a Field.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

DictionaryType dataclass 

DictionaryType(
    byte_size: int | None = None,
    value_type: PrimitiveType = None,
    categories: tuple[Any, ...] = (),
    ordered: bool = False,
)

Bases: PrimitiveType

Dictionary-encoded categorical primitive.

value_type defaults to :class:StringType (the most common case). categories is an iterable of values that get coerced through value_type.convert_pyobj and de-duplicated in first-seen order; an empty tuple means "open dictionary". ordered flips the Arrow ordered flag (and is the only field whose merge requires a category-tuple match).

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

EnumType dataclass 

EnumType(
    byte_size: int | None = None,
    value_type: PrimitiveType = None,
    categories: tuple[Any, ...] = (),
    ordered: bool = False,
    name: str | None = None,
    members: dict[str, Any] | None = None,
)

Bases: DictionaryType

Semantic enum — strict named value set, dictionary-encoded on the wire.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

from_pyenum classmethod 

from_pyenum(hint: Any) -> 'EnumType'

Build an :class:EnumType from a Python enum.Enum subclass.

Promotes to :class:StrEnumType / :class:IntEnumType when every member value matches a typed specialization — gives the round-trip path on DataType.from_pytype a single entry point that lands on the most specific class available.

IntEnumType dataclass 

IntEnumType(
    byte_size: int | None = None,
    value_type: PrimitiveType = (
        lambda: IntegerType(byte_size=8, signed=True)
    )(),
    categories: tuple[Any, ...] = (),
    ordered: bool = False,
    name: str | None = None,
    members: dict[str, Any] | None = None,
)

Bases: EnumType

Integer-valued enum — schema parallel of stdlib enum.IntEnum.

Defaults to signed int64 storage (IntegerType(byte_size=8, signed=True)) which matches Python's native int width. Pass a narrower :class:IntegerType (e.g. Int32Type()) to pin storage when the enum's range is known.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

from_pyenum classmethod 

from_pyenum(hint: Any) -> 'EnumType'

Build an :class:EnumType from a Python enum.Enum subclass.

Promotes to :class:StrEnumType / :class:IntEnumType when every member value matches a typed specialization — gives the round-trip path on DataType.from_pytype a single entry point that lands on the most specific class available.

StrEnumType dataclass 

StrEnumType(
    byte_size: int | None = None,
    value_type: PrimitiveType = StringType(),
    categories: tuple[Any, ...] = (),
    ordered: bool = False,
    name: str | None = None,
    members: dict[str, Any] | None = None,
)

Bases: EnumType

String-valued enum — schema parallel of stdlib enum.StrEnum.

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

from_pyenum classmethod 

from_pyenum(hint: Any) -> 'EnumType'

Build an :class:EnumType from a Python enum.Enum subclass.

Promotes to :class:StrEnumType / :class:IntEnumType when every member value matches a typed specialization — gives the round-trip path on DataType.from_pytype a single entry point that lands on the most specific class available.

UnionType dataclass 

UnionType(members: tuple[DataType, ...] = tuple())

Bases: DataType

DataType wrapper around a Python Union / Optional.

members is the (frozen) tuple of inner :class:DataType arms. NullType membership is what makes the union nullable — the canonical pattern for Optional[T] is UnionType(T, Null).

nullable property 

nullable: bool

True iff at least one member is :class:NullType.

non_null_members property 

non_null_members: tuple[DataType, ...]

The members minus any :class:NullType arms (order preserved).

select_fields 

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short 

short(depth: int = 2) -> str

A compact, single-line type tag — recursive (bounded) for nested types.

Scalars render as i64 / f64 / str / bool / date / ts / dec / bin / json; nested types recurse through their own children — list<i64>, struct<x:i64, y:str>, map<str,f64>, even list<struct<id:i64>> — until depth runs out (then a bare list / struct / map). Built from this type's :attr:type_id + :attr:children, not any one engine. Struct fields are capped and the whole tag is elided past :data:_SHORT_TAG_MAX chars, so a deep schema can't widen a :meth:yggdrasil.io.tabular.Tabular.display header forever.

to_pyhint 

to_pyhint() -> Any

Return the Python type hint that maps back to this DataType.

Cached when :meth:from_pytype stamped an explicit hint on the instance (preserves user-defined dataclasses, enum classes, numpy.int64 and other narrow aliases the canonical reconstruction would collapse). Otherwise falls back to :meth:_default_pyhint, the subclass hook that builds a canonical hint from the dtype's own state.

expand_alias classmethod 

expand_alias(name: str) -> str

Expand a known short-alias prefix.

pa.Table → pyarrow.Table, pl.DataFrame → polars.DataFrame, etc. Returns name unchanged when no registered prefix matches. The table lives at :attr:PYHINT_ALIASES.

strip_annotated staticmethod 

strip_annotated(hint: Any) -> Any

Strip Annotated[T, ...] down to T (recursive).

unwrap_newtype staticmethod 

unwrap_newtype(hint: Any) -> Any

Unwrap a chain of NewType aliases to the base type.

normalize_hint classmethod 

normalize_hint(hint: Any) -> Any

strip_annotated then unwrap_newtype — common preamble.

Every from_pytype / unwrap_* flow needs the hint stripped of Annotated metadata and the NewType chain unwrapped before the real dispatch. Bundled into one classmethod so the ordering ("strip Annotated before NewType, recursive on both") lives in one place.

unwrap_optional classmethod 

unwrap_optional(hint: Any) -> tuple[bool, Any]

Return (is_optional, inner) for Optional[T] / T | None.

Only collapses a Union whose non-None arms reduce to exactly one type — Optional[int] → (True, int), int | None → (True, int). Multi-type unions return (False, hint) so callers can decide how to handle them (the cast registry generally falls through to StringType).

unwrap_nullable_hint classmethod 

unwrap_nullable_hint(hint: Any) -> tuple[Any, bool]

Field-flavoured Optional unwrap: (inner, has_null).

Differences from :meth:unwrap_optional:

  • String hints route through :class:ParsedDataType.from_ so the field name / nullability tag baked into the DSL form (e.g. "int64?") survives.
  • Multi-type unions stay intact — the caller wants (Union[A, B], True), not (Union[A, B], False) — so :class:yggdrasil.data.Field can stamp nullable=True without losing the multi-arm shape.

is_runtime_value staticmethod 

is_runtime_value(x: Any) -> bool

True for runtime values (42, [], MyClass()).

False for type hints — distinguishes convert(42, int) (runtime value with target hint) from convert(int, str) (two hints, no value). Used to gate dispatch decisions in downstream tooling.

resolve_str_annotation classmethod 

resolve_str_annotation(
    s: str, func_globals: Optional[dict[str, Any]] = None
) -> Any

Resolve a string annotation to a real type.

Tries, in order:

  1. eval in func_globals + builtins — picks up local imports and aliases declared in the function's own module.
  2. eval against typing for generic shapes (Optional[int], list[int]) when func_globals doesn't have them in scope.
  3. Alias-prefix expansion via :meth:expand_alias + dotted importlib.import_module lookup so pa.Table / pl.DataFrame resolve without the function's globals having ever imported the alias.

Returns s verbatim when every path fails — callers treat that as "unresolved, skip coercion / leave as-is".

resolve_function_annotations classmethod 

resolve_function_annotations(func: 'Callable[..., Any]') -> dict[str, Any]

Resolve every annotation on func to a real type.

Two-pass best effort so a single unresolvable annotation doesn't blow up the rest:

  1. :func:inspect.get_annotations with eval_str=True — evaluates every annotation in the function's globals + builtins in one shot.
  2. Per-annotation :meth:resolve_str_annotation fallback for entries the fast path left as strings (it silently returns the literal when its eval misses).

Anything still a string after both passes is left untouched — callers treat it as "couldn't resolve, skip coercion" rather than raising.

as_polars 

as_polars() -> 'DataType'

Return a Polars-flavored :class:DataType for this type.

Same shape as :meth:as_spark — stays on the yggdrasil side of the boundary and returns a :class:DataType whose :meth:to_polars lands on a dtype Polars natively represents. Defaults to self; subclasses Polars can't store at their declared width / precision override:

  • Float8Type and Float16Type widen to Float32Type (Polars has no sub-32-bit floats);
  • TimestampType / DurationType with second-precision (unit="s") widen to unit="ms" (Polars supports ms / us / ns only);
  • nested types (ArrayType / MapType / StructType) recurse via as_polars on their child fields.

:class:Field and :class:Schema expose a matching as_polars that delegates to self.dtype.as_polars and re-wraps so callers chain through Field-shaped APIs without dropping back to a plain :class:DataType.

as_spark 

as_spark() -> 'DataType'

Return a Spark-flavored :class:DataType for this type.

as_spark lives on the yggdrasil side of the boundary: it returns a :class:DataType that maps cleanly to a Spark dtype (i.e. one self.to_spark() would round-trip without a widening-time surprise). For types Spark already represents natively (signed ints, Float32 / Float64, Date, String / Binary / Boolean, decimal, naive / UTC timestamps), the default is to return self unchanged.

Subclasses Spark cannot represent natively (IntegerType with signed=False, Float16Type, DurationType, TimeType, non-UTC TimestampType) override this to return the closest Spark-compatible yggdrasil dtype — usually a widened integer, a StringType, or a naive timestamp. Nested types (ArrayType / MapType / StructType) recurse via as_spark on their child fields so the whole tree comes back Spark-compatible in one call.

autotag 

autotag() -> dict[bytes, bytes]

Return a dict of Databricks-friendly tags derived from this type.

These are auto-tags: they describe shape, not intent. The base output is a single kind key — a lowercase form of type_id ("integer", "string", "timestamp", "array", ...) that tag-based Unity Catalog policies can match on. Subclasses extend this with dtype-specific detail (unit, tz, precision / scale, signed, srid, ...) — always via super().autotag() so the kind key stays present.

Keys are bare (no t: prefix) — prefixing is handled by BaseMetadata.update_tags when these land on a Field.

from_pytype classmethod 

from_pytype(hint: Any) -> 'DataType'

Parse a Python annotation into a :class:DataType.

Stamps the resulting instance's _pyhint_cache with the original hint when the canonical reconstruction (:meth:_default_pyhint) wouldn't round-trip — so from_pytype(MyDataclass).to_pyhint() returns MyDataclass itself rather than a generic struct hint, from_pytype(MyEnum).to_pyhint() returns the enum class, and from_pytype(np.int64).to_pyhint() returns np.int64 rather than the canonical int.

cast_arrow_batch_iterator 

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more_options
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this dtype.

Non-struct dtypes promote to a single-column struct via :meth:to_struct and reuse the struct's iterator helper.

cast_polars_expr 

cast_polars_expr(
    series: "polars.Series | polars.Expr",
    options: "CastOptions | None" = None,
    **more_options
) -> "polars.Series | polars.Expr"

Expr-shape passthrough to :meth:cast_polars_series.

:meth:cast_polars_series already dispatches by isinstance — this method exists so callers that know they hold an Expr can name it.

without_null 

without_null() -> 'UnionType'

Return a copy with all :class:NullType members removed.

Preserves the multi-arm shape — even when only one non-null member is left the result is still a :class:UnionType. Use :meth:to_field (or the explicit non_null_members[0]) when you want the un-nested type.

to_field 

to_field(
    name: str = DEFAULT_FIELD_NAME,
    nullable: bool = True,
    metadata: dict[str, Any] | None = None,
) -> "Field"

Wrap into a :class:Field, flattening the union.

Bridge from the union-rich DataType layer to the nullable-flat Field layer:

  • If any member is :class:NullType, drop those arms and set Field.nullable=True regardless of the nullable arg (Null membership is the stronger signal of intent).
  • If only one non-null member remains, un-nest the union and use that member as Field.dtype directly.
  • Otherwise (multi-arm non-null), keep the trimmed union as Field.dtype.

Empty union (no members) and union-of-only-Null both collapse to a :class:Field with NullType() dtype, nullable=True.

DataTypeId

Bases: IntEnum

Stable numeric tag for every :class:DataType subclass.

The integer values are clustered by family so a plain sorted(DataTypeId) walk visits related types together — and so range-based property checks (is_integer / is_temporal / …) read as one inequality rather than a hand-maintained set. Placeholder gaps inside each family leave room for future additions (e.g. INT128, BFLOAT16, TIMETZ) without re-numbering anything else.

Layout:

  • 0–9 — sentinel / opaque (OBJECT, NULL).
  • 10–19 — boolean.
  • 20–39 — integer family (abstract INTEGER + sized signed INT8…INT64 + unsigned UINT8…UINT64).
  • 40–49 — floating-point family (abstract FLOAT + sized FLOAT16…FLOAT64).
  • 50–59 — decimal.
  • 60–69 — temporal (DATE / TIME / TIMESTAMP / DURATION).
  • 70–79 — bytes (BINARY / STRING).
  • 80–99 — extensions (DICTIONARY / ENUM / STR_ENUM / INT_ENUM / UNION / SJSON / BJSON).
  • 100+ — nested (ARRAY / MAP / STRUCT).