Skip to content

yggdrasil.data

data

TimeUnit

Bases: str, Enum

Canonical time-unit token for temporal DataType instances.

Members carry the lowercase short form so the enum is a drop-in string replacement::

TimeType(unit=TimeUnit.MICROSECOND)
TimeType(unit="us")  # equivalent — both store ``"us"``

Use :meth:parse when accepting external input — it canonicalizes aliases ("microseconds", "micros", "µs") to a member and raises :class:ValueError for unknown tokens.

seconds property

seconds: float

Seconds per one of this unit (used for scalar epoch math).

Calendar-style interval units (year_month / day_time / month_day_nano) have no fixed second-count and return float('nan') so comparisons surface the mismatch instead of silently truncating to zero.

order property

order: int

Precision rank — higher = finer.

Used by TemporalType._merge_with_same_id to pick the wider of two units. Calendar interval units sit at rank -1 so they don't outrank fixed-precision ones in normal merges.

is_subsecond property

is_subsecond: bool

True for ms / us / ns.

is_calendar property

is_calendar: bool

True for the variable-length interval forms.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'TimeUnit'

Coerce any Python value into a :class:TimeUnit.

Accepts:

  • :class:TimeUnit (returned as-is);
  • any string the alias table or canonical member values know — s / ms / us / ns / d / interval forms, plurals (microseconds), long forms (millisecond), mixed case, µs, hyphens / spaces;
  • objects exposing a time_unit / unit attribute (Polars Datetime / Duration, PyArrow TimestampType / DurationType / Time32Type / Time64Type); the attribute is re-funneled through from_;
  • None — returns default if supplied, else raises.

default swallows unknown / unparseable input. Without it, unknown tokens raise :class:ValueError and unsupported types raise :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

Return True when :meth:from_ would succeed for value.

Timezone dataclass

Timezone(iana: str)

An immutable wrapper around an IANA timezone identifier.

Instances are created via :meth:from_ (accepts strings, ZoneInfo, datetime.tzinfo, other Timezone objects, or None → UTC) or directly::

tz = Timezone("Europe/Paris")
tz = Timezone.from_("CET")        # → Timezone("Europe/Paris")
tz = Timezone.from_("+01:00")     # → Timezone("Etc/GMT-1")
tz = Timezone.from_(ZoneInfo("Europe/Paris"))

The naive case is represented by :attr:Timezone.NAIVE — a sentinel instance whose iana is the empty string and whose __bool__ is False. Use it instead of None so a tz: Timezone field type can stay non-optional.

key property

key: str

Alias for :attr:iana — matches ZoneInfo.key.

utc_seconds_offset property

utc_seconds_offset: int | None

Fixed UTC offset in seconds, or None for non-fixed zones.

Returns 0 for UTC and its equivalents (Etc/UTC, GMT, Etc/GMT+0, …), the parsed offset for Etc/GMT±N (note IANA's sign-flip — Etc/GMT-3 means UTC+3, so the property reports +10800 not -10800), and None for any zone whose offset depends on DST or for :attr:Timezone.NAIVE. Use :meth:utc_offset instead when you need a DST-aware offset for a specific instant.

Naming follows the "type-suffix unit" convention so utc_seconds_offset makes the unit explicit at the call site — utc_offset returns a :class:datetime.timedelta, utc_offset_hours returns fractional hours, and this property returns whole seconds (or None).

tzinfo property

tzinfo: tzinfo

Return as a stdlib tzinfo (same as to_zoneinfo()).

from_ classmethod

from_(obj: Any, *, default: Any = ...) -> 'Timezone'

Coerce any Python value into a :class:Timezone.

Accepts:

  • :class:Timezone (returned as-is — including Timezone.NAIVE);
  • :class:zoneinfo.ZoneInfo (extracts key);
  • a string — IANA name, alias (CET / EST / Z / GMT / Etc/UTC / +00:00 / …), or fixed offset ("+01:00", "UTC-05", "-0530");
  • datetime.tzinfo instance (ZoneInfo, datetime.timezone, third-party zones) — the key / zone attribute is preferred; otherwise falls back to the fixed UTC offset;
  • timezone-aware datetime / time (extracts tzinfo);
  • objects exposing a tz / time_zone / timezone / iana attribute (PyArrow TimestampType, Polars Datetime, foreign Timezone classes); the attribute is re-funneled through from_;
  • None — returns :attr:UTC for backward compatibility unless default is supplied.

default swallows unknown / unparseable input. Without it, bad input raises :class:ValueError (string parse failure) or :class:TypeError (unsupported value type).

to_zoneinfo cached

to_zoneinfo() -> ZoneInfo

Return the zoneinfo.ZoneInfo for this timezone.

Raises :class:ValueError for :attr:Timezone.NAIVE since there is no concrete zone to materialize.

utc_offset

utc_offset(at: datetime | None = None) -> _dt.timedelta

Return the UTC offset at instant at (default: now).

The result accounts for DST transitions.

Timezone("Europe/Paris").utc_offset(datetime(2024, 7, 1)) datetime.timedelta(seconds=7200) # +02:00 (CEST)

utc_offset_hours

utc_offset_hours(at: datetime | None = None) -> float

Return the UTC offset in fractional hours.

Timezone("Asia/Kolkata").utc_offset_hours() 5.5

is_naive

is_naive() -> bool

Return True for :attr:Timezone.NAIVE — the tz-less sentinel.

is_utc

is_utc() -> bool

Return True if this timezone is UTC (or an equivalent).

is_fixed_offset

is_fixed_offset() -> bool

Return True if this timezone has a fixed UTC offset (no DST).

is_dst

is_dst(at: datetime | None = None) -> bool

Return True if DST is active at instant at (default: now).

Timezone("Europe/Paris").is_dst(datetime(2024, 7, 1)) True Timezone("Europe/Paris").is_dst(datetime(2024, 1, 1)) False

dst_offset

dst_offset(at: datetime | None = None) -> _dt.timedelta

Return the DST adjustment at instant at.

Returns timedelta(0) when DST is not active or for fixed-offset timezones, and for :attr:Timezone.NAIVE.

abbreviation

abbreviation(at: datetime | None = None) -> str

Return the timezone abbreviation (e.g. "CET", "CEST") at at.

Timezone("Europe/Paris").abbreviation(datetime(2024, 7, 1)) 'CEST' Timezone("Europe/Paris").abbreviation(datetime(2024, 1, 1)) 'CET'

distance_to

distance_to(other: 'Timezone', at: datetime | None = None) -> _dt.timedelta

Return the offset difference between self and other at at.

A positive result means other is ahead of self.

Timezone.UTC.distance_to(Timezone.CET, datetime(2024, 7, 1)) datetime.timedelta(seconds=7200)

now

now() -> _dt.datetime

Return the current wall-clock time in this timezone.

The returned datetime is timezone-aware.

today

today() -> _dt.date

Return today's date in this timezone.

localize

localize(naive: datetime) -> _dt.datetime

Stamp a naive datetime with this timezone (no conversion).

Raises:

Type Description
ValueError

If naive is already timezone-aware, or if this is :attr:Timezone.NAIVE.

convert

convert(aware: datetime) -> _dt.datetime

Convert a timezone-aware datetime to this timezone.

Raises:

Type Description
ValueError

If aware is naive, or if this is :attr:Timezone.NAIVE.

midnight

midnight(date: date | None = None) -> _dt.datetime

Return midnight (00:00) in this timezone for date (default: today).

from_polars_type classmethod

from_polars_type(dtype: 'polars.DataType') -> 'Timezone | None'

Extract the timezone from a Polars Datetime dtype.

Returns None if the dtype is not Datetime or has no timezone.

Timezone.from_polars_type(pl.Datetime("us", "Europe/Paris")) Timezone('Europe/Paris')

polars_normalize classmethod

polars_normalize(
    col: "polars.Series | polars.Expr",
    *,
    lazy: bool = True,
    return_value: Literal["iana"] = "iana"
) -> "polars.Series | polars.Expr"

Normalize timezone strings in a Polars column using alias replacement only.

from_arrow_type classmethod

from_arrow_type(dtype: Any) -> 'Timezone | None'

Extract the timezone from a PyArrow TimestampType.

Returns None if the type has no tz attribute or it is empty.

import pyarrow as pa Timezone.from_arrow_type(pa.timestamp("us", tz="Europe/Paris")) Timezone('Europe/Paris')

arrow_timestamp_type

arrow_timestamp_type(unit: str = 'us') -> Any

Return a pa.timestamp(unit, tz=self.iana) type.

Timezone.CET.arrow_timestamp_type("ns") TimestampType(timestamp[ns, tz=Europe/Paris])

all_iana classmethod

all_iana() -> frozenset[str]

Return all IANA timezone identifiers available on this system.

Mode

Bases: IntEnum

is_read_only property

is_read_only: bool

True for read-only modes — i.e. no write disposition.

allows_write property

allows_write: bool

True when the mode admits any write disposition.

readable property

readable: bool

True when the mode admits reads.

Every :class:Mode canonically resolves to a + POSIX form (rb, rb+, wb+, ab+, xb+) — all of those admit reads. Only the strict :data:READ_ONLY rb and :data:IGNORE (which is no-op) deny writes; nothing here denies reads.

writable property

writable: bool

True when the mode admits writes — alias of :attr:allows_write.

appendable property

appendable: bool

True when writes append at EOF rather than at the cursor.

Only :data:APPEND carries POSIX O_APPEND semantics; every other write mode positions writes at the explicit cursor.

os_mode property

os_mode: str

Stdlib :func:open mode string for this :class:Mode.

  • :attr:READ_ONLY"rb"
  • :attr:OVERWRITE / :attr:TRUNCATE"wb+"
  • :attr:APPEND"ab+"
  • :attr:ERROR_IF_EXISTS"xb+"
  • everything else (AUTO, IGNORE, UPSERT, MERGE) → "rb+" (in-place edit; the disposition is enforced higher up).

from_ classmethod

from_(value: Optional[ModeLike], default: Optional[Mode] = None) -> Mode

Normalize value into a :class:Mode.

Accepts:

  • :class:Mode (returned as-is, idempotent).
  • Aliases like "overwrite", "OVERWRITE", "error-if-exists", "replace", "add" — see :data:STR_MAPPING.
  • POSIX-style mode strings — "rb", "wb", "ab+", "x", "r+b" — parsed structurally; any combination of one primary character (r/w/a/x) plus optional b/t/+ flags is accepted.
  • None → returns default if supplied, else :data:Mode.AUTO.

Falls back to :class:ValueError for unrecognized strings. Numeric / non-string non-Mode inputs raise :class:TypeError — the input grammar is "string or enum," not "anything stringifiable."

MimeType dataclass

MimeType(
    name: str,
    value: str,
    extensions: tuple[str, ...] = (),
    magics: tuple[MagicMatcher, ...] = (),
    is_codec: bool = False,
    is_tabular: bool = False,
    is_blob: bool = False,
)

Dataclass MIME descriptor + registries.

  • extensions: dotless, lower-case keys
  • magics: ordered matchers
  • is_codec: compression / wrapper formats
  • is_tabular: row/tabular-ish formats (read into a frame)
  • is_blob: opaque single-file payload — straight byte IO, no row structure (images, pdf, archives, pickle, …). Mutually exclusive with is_tabular; codecs and directory/connector mimes are neither.

get classmethod

get(value: object) -> 'MimeType | None'

Pure lookup — never raises. Returns None on miss.

Kept as the low-level "resolve by name/value" entry point: :meth:from_ / :meth:from_str / :meth:from_magic layer the default-handling contract on top.

parse_many classmethod

parse_many(obj: Any) -> list['MimeType']

Resolve obj to a flat, deduped list of :class:MimeType.

Accepts anything :meth:from_ accepts, plus:

  • iterables (list, tuple, set, generator) of any supported input
  • Accept-header strings: "application/json, text/csv;q=0.8"
  • composite format+codec strings: "application/csv+gzip", "parquet+zstd", "trades.parquet.zst"
  • wildcard strings: "*/*", "text/*", "image/*"
  • None[]

Order: first-seen wins, deduped by identity. For composites, the base format is emitted first, then the codec (codec wraps format).

Never raises on an unresolvable element — unknowns are dropped silently. For strict per-element resolution, call :meth:from_.

from_magic classmethod

from_magic(
    magic: Union[bytes, bytearray, memoryview, IO[bytes], str, Path],
    default: "MimeType | None" = _RAISE,
) -> "MimeType | None"

Resolve by sniffing magic bytes from magic.

Accepts raw bytes/memoryview, an IO, or anything the buffer class can wrap. Reads the first 64 bytes and walks the registered magic matchers in definition order.

:param default: see :class:MimeType class docstring for the shared default-handling contract. :raises ValueError: on a miss when default was not supplied.

from_str classmethod

from_str(value: str, default: 'MimeType | None' = _RAISE) -> 'MimeType | None'

Resolve a :class:str — path-like, bare extension, or mime value.

Tries, in order:

  • Direct lookup against the lower-cased input (covers "text/csv" / "json" / ".csv" without paying a :class:pathlib.Path allocation).
  • If the string looks path-like (contains / or \), take its suffix as an extension key.
  • Fall back to :meth:get (name / mime-value lookup with the application/ / text/ / … prefix stripping).
  • Structural sniff on leading { / [.

:param default: see :class:MimeType class docstring for the shared default-handling contract. :raises ValueError: on a miss when default was not supplied.

MimeTypes

Singleton MIME definitions.

MediaType dataclass

MediaType(mime_type: MimeType, codec: Codec | None = None)

from_many classmethod

from_many(
    mime_types: Iterable[MimeType], default: "MediaType" = ...
) -> "MediaType"

Compose a MediaType from an ordered mime list (e.g. URL extensions).

Two conventions land here:

  • trades.csv.zst["csv", "zst"]: the codec is the outer wrapper (you have to unzst before you can parse csv) → MediaType(CSV, codec=ZSTD).
  • part-xxx.zstd.parquet["zstd", "parquet"]: the format is the outer wrapper and the codec is the parquet page-codec hint baked into the file. Parquet handles the decompression internally → MediaType(PARQUET) with no outer codec; setting one would route the read through a decompressor that doesn't belong on this byte stream.

Order matters: the last mime decides. Last-is-codec promotes the codec to the wrapper slot; last-is-format keeps the format alone and drops earlier codec hints.

from_magic classmethod

from_magic(
    buf: Union[bytes, bytearray, memoryview, IO[bytes]],
    *,
    default: "MediaType" = ...
) -> "MediaType"

Sniff a :class:MediaType from raw bytes.

Two-stage: outer magic identifies the wrapper; if the wrapper is a codec, decompress a head-window and sniff the inner format. See module docstring for the full story.

Accepts bytes / bytearray / memoryview directly. For convenience, also accepts an IO[bytes] — in that case it delegates to :meth:from_io, which manages cursor save/restore.

:param default: Fallback when the outer sniff finds nothing. ... (default) raises :class:ValueError; any other value (including None) is returned as-is.

from_io classmethod

from_io(io_obj: IO[bytes], *, default: 'MediaType' = ...) -> 'MediaType'

Sniff a :class:MediaType from a file-like, seeking freely.

Captures the cursor on entry and restores it on exit, so the caller's stream position is unaffected. Inside the call we seek freely:

  1. seek(0) and read the first :data:_OUTER_PEEK bytes for outer-magic resolution.
  2. If the outer mime is a codec, seek(0) again and drive the codec's streaming decompressor directly to read the first :data:_INNER_PEEK bytes of decoded payload.
  3. Sniff the decoded head for the inner mime.

The codec resolution is one dict lookup (:meth:Codec.from_mime) — the outer magic loop already landed on the registered :class:MimeType singleton, so we reuse it as the key without re-resolving. For streaming codecs (gzip / zstd / lz4 / bz2 / xz / zlib / lzma) the decompressor is opened directly on the IO via :meth:Codec._open_decompress_reader — no IO wrap, no full-buffer materialization. Non-streaming codecs (snappy, brotli) fall back to reading the compressed body and calling :meth:Codec.decompress_bytes.

Decompression errors during the inner sniff are swallowed — the worst case is returning a less-specific MediaType (the outer codec is preserved). A separate caller that wants to validate decompressibility should call :meth:Codec.decompress directly.

:param io_obj: Any file-like with read + seek (stdlib io.BytesIO, an open file, our own IO, etc.). :param default: Fallback when the outer sniff finds nothing. ... (default) raises :class:ValueError; any other value (including None) is returned as-is.

Codec

Bases: ABC

Abstract compression codec.

is_streaming property

is_streaming: bool

True when both compress and decompress have streaming paths.

Callers with large (GiB-scale) inputs should check this before passing them to :meth:compress / :meth:decompress. When False, those methods fall back to materializing the full payload in memory through :meth:compress_bytes / :meth:decompress_bytes.

compress

compress(src: 'IO') -> 'IO'

Compress src into a new :class:IO.

Streams chunk-by-chunk when :meth:_open_compress_writer is available. Otherwise falls back to a full-in-memory bytes roundtrip — callers with multi-GiB inputs should inspect :attr:is_streaming first.

The source cursor is restored on exit.

decompress

decompress(src: 'IO', copy: bool = True) -> 'IO'

Decompress src into a new :class:IO.

Streams chunk-by-chunk when :meth:_open_decompress_reader is available. Otherwise falls back to a full-in-memory bytes roundtrip — callers with multi-GiB compressed inputs should inspect :attr:is_streaming first.

The source cursor is restored on exit.

read_start_end

read_start_end(
    src: "IO | bytes | bytearray | memoryview",
    *,
    n_start: int = 64,
    n_end: int = 64,
    chunk_size: int = _CHUNK
) -> tuple[bytes, bytes]

Return the first n_start and last n_end bytes of the decoded stream.

Streams the decompression and keeps only a bounded amount of state in memory (:math:n\_end + chunk\_size bytes) — safe for very large compressed inputs when the codec supports streaming decompression.

When the codec does NOT expose a streaming decoder, falls back to a full :meth:decompress_bytes call, which materializes the whole uncompressed payload in memory.

from_ classmethod

from_(obj: Any, default: 'Codec | None' = ...) -> 'Codec | None'

Parse an arbitrary input into a Codec instance.

Accepts: - :class:Codec instances (returned as-is). - Short names like "gzip", "zstd" (case-insensitive). - Anything :meth:MimeType.parse can resolve to a codec mime. - None → returns default.

Codecs

Singleton class for accessing all registered codecs.

ByteUnit

Bases: IntEnum

Canonical IEC binary byte-unit token + scalar value.

Each member's value IS the byte count for one unit, so 128 * ByteUnit.MIB reads naturally at the call site and slots into any int field. Use :meth:parse_size when accepting external config / API input — it canonicalizes "128 MB" / "1.5 GiB" / raw integers / ByteUnit members all to a plain integer byte count.

bytes property

bytes: int

Bytes per one of this unit (== member value).

short property

short: str

Short colloquial token: "B" / "KB" / "MB" / ….

iec property

iec: str

Strict IEC token: "B" / "KiB" / "MiB" / ….

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'ByteUnit'

Coerce any Python value into a :class:ByteUnit member.

Accepts:

  • :class:ByteUnit (returned as-is);
  • any string the alias table knows — B / KB / MiB / gigabyte / mixed case / trailing s;
  • an integer matching a member's byte value (1024 → :attr:KIB);
  • None — returns default if supplied, else raises.

default swallows unknown / unparseable input. Without it, unknown tokens raise :class:ValueError and unsupported types raise :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would succeed for value.

parse_size classmethod

parse_size(
    value: Union[int, str, "ByteUnit", None], *, default: Any = ...
) -> int

Coerce a size-like value to an integer byte count.

The single entry point for "config gave me a size, give me bytes." Accepts:

  • :class:int — passed through (must be non-negative);
  • :class:ByteUnit — its scalar value;
  • a quantity string "128 MB" / "1.5 GiB" / "512" — parsed with this enum's IEC conventions;
  • a bare unit string "MiB" — yields one unit (1024**2);
  • None — returns default if supplied, else raises.

Floating-point quantities round to the nearest byte ("1.5 KiB"1536). Negative values raise :class:ValueError.

format classmethod

format(n: int, *, iec: bool = True, precision: int = 1) -> str

Format an integer byte count as a human-readable string.

Picks the largest unit at which n divides cleanly into a scalar ≥ 1, defaulting to IEC tokens ("128 MiB"); pass iec=False for the colloquial short form ("128 MB"). precision controls fractional digits.

pretty classmethod

pretty(
    v: float,
    unit: "ByteUnit | str | None" = None,
    *,
    iec: bool = True,
    precision: int = 1
) -> str

Human-readable rendering of a quantity v expressed in unit.

v is a scalar count of unit (bytes by default); it's scaled to a byte count and handed to :meth:format. The companion to :meth:format for the common "I have N MiB, show it nicely" case::

ByteUnit.pretty(1536)             # "1.5 KiB"
ByteUnit.pretty(8, ByteUnit.MIB)  # "8.0 MiB"
ByteUnit.pretty(1.5, "gb")        # "1.5 GiB"

unit defaults to :attr:B (resolved at call time so the class default is usable inside the method body); pass iec=False for the colloquial short form.

Unit

Unit(symbol: str, factor: float, offset: float = 0.0)

Mixin providing :meth:from_ / :meth:convert / scalar helpers.

Subclasses combine this with :class:Enum and declare members as (symbol, factor_to_canonical, offset_to_canonical) tuples (or (symbol, factor) — offset defaults to 0.0). The canonical member of each family has factor=1.0 and offset=0.0; conversion goes through that canonical pivot.

Extra spellings ("megawatthours"MWH) live in a <Family>._ALIASES dict assigned outside the enum body — the enum metaclass would treat a class-body dict as a member.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

EnergyUnit

EnergyUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Energy unit (canonical = joule).

Members cover SI scale (J / kJ / MJ / GJ / TJ), the watt-hour family that dominates electricity-market ingestion (Wh / kWh / MWh / GWh / TWh), and the imperial / domestic units (BTU / cal / kcal / therm).

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

PowerUnit

PowerUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Power unit (canonical = watt).

SI scale (W / kW / MW / GW / TW) plus mechanical horsepower. Energy-market feeds publish capacity in MW, transmission limits in GW, household-scale appliances in W — having one enum cover all of them is what keeps schema-per-source curated views from re-stringing the unit token at every read.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

MassUnit

MassUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Mass unit (canonical = kilogram).

SI scale (mg / g / kg / t / Mt) plus the imperial pair (lb / oz) commonly used in commodity feeds.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

LengthUnit

LengthUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Length unit (canonical = metre).

SI scale (mm / cm / m / km) plus the common imperial / aviation / maritime tokens (in / ft / yd / mi / nmi).

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

VolumeUnit

VolumeUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Volume unit (canonical = cubic metre).

SI scale (mL / L / m³) plus the gallon variants and the oil barrel — commodity feeds split between US gallons, UK gallons, and the 42-gallon oil barrel, and a curated view that doesn't keep them straight ships wrong totals.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

TemperatureUnit

TemperatureUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Temperature unit (canonical = kelvin).

The one affine family — Celsius and Fahrenheit have non-zero offsets relative to kelvin, so the conversion is canonical = value * factor + offset (not pure scaling). Symbols use the standard degree glyphs (°C / °F) so the curated tables render legibly in BI tools; aliases cover the plain-ascii forms ("C" / "F" / "K").

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

PressureUnit

PressureUnit(symbol: str, factor: float, offset: float = 0.0)

Bases: Unit, Enum

Pressure unit (canonical = pascal).

SI scale (Pa / kPa / MPa / hPa) plus the bar family (bar / mbar), atmosphere, psi, and torr. Weather feeds publish in hPa or mbar, oil & gas in psi or bar, process control in kPa or MPa — one enum keeps them aligned.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Unit'

Coerce value to a member of this unit family.

Accepts: a member (passed through), a known symbol ("MWh" / "°C" / "psi"), the enum member name ("KWH"), or any alias the per-family _ALIASES dict knows ("megawatthours" / "celsius" / "pound"). Lookup is case-insensitive.

Pass default to swallow unknown / unparseable input; without it, unknown strings raise :class:ValueError with a hint listing the valid symbols, and non-string non-member input raises :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would resolve value.

convert classmethod

convert(value: float, source: Any, target: Any) -> float

Convert value from source unit to target unit (scalar).

to_canonical

to_canonical(value: float) -> float

Convert value in self's unit to the family's canonical unit.

from_canonical

from_canonical(value: float) -> float

Convert value in the canonical unit to self's unit.

canonical classmethod

canonical() -> 'Unit'

The family's canonical member (factor=1.0, offset=0.0).

factor_map classmethod

factor_map() -> dict[str, float]

{symbol: factor} for every member + alias, for per-row maps.

offset_map classmethod

offset_map() -> dict[str, float]

{symbol: offset} for every member + alias, for per-row maps.

JoinType

Bases: IntEnum

Canonical join kind for tabular join surfaces.

Use :meth:from_ when accepting external input — it canonicalizes aliases ("left", "LEFT JOIN", "anti", "outer", integer codes) to a member and raises :class:ValueError for unknown tokens.

Pass :attr:arrow / :attr:polars / :attr:sql to engine join APIs — pyarrow's :meth:pa.Table.join and polars' :meth:DataFrame.join accept different spellings of the same concept, and storing the integer code keeps the enum a clean discriminator without binding to either spelling.

arrow property

arrow: str

pyarrow's :meth:pa.Table.join join_type token.

polars property

polars: str

polars' :meth:DataFrame.join how token.

Polars has no built-in right semi / right anti form — those raise :class:NotImplementedError. Swap operands and use the left-side equivalent.

sql property

sql: str

SQL JOIN clause keyword (uppercase, with trailing JOIN).

is_inner property

is_inner: bool

True for :attr:INNER.

is_outer property

is_outer: bool

True for any of the three outer joins.

is_semi property

is_semi: bool

True for :attr:LEFT_SEMI / :attr:RIGHT_SEMI.

is_anti property

is_anti: bool

True for :attr:LEFT_ANTI / :attr:RIGHT_ANTI.

is_cross property

is_cross: bool

True for :attr:CROSS.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'JoinType'

Coerce any Python value into a :class:JoinType.

Accepts:

  • :class:JoinType (returned as-is);
  • any string the alias table or canonical Arrow tokens know — "inner", "left", "left outer", "LEFT JOIN", "anti", "left_anti", "outer", "cross", …; mixed case, hyphens / underscores / spaces all normalize;
  • an integer code matching a member's value (round-trips with int(JoinType.X));
  • None — returns default if supplied, else raises.

default swallows unknown / unparseable input. Without it, unknown tokens raise :class:ValueError and unsupported types raise :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

Return True when :meth:from_ would succeed for value.

Scheme

Bases: str, Enum

Canonical URL-scheme token for a Yggdrasil :class:URLBased subclass.

Subclasses :class:str so a member is interchangeable with its scheme token everywhere a string is expected (url.scheme == Scheme.DBFS works, f"{Scheme.S3}://bucket/key" reads naturally). The lazy-import resolver lives on :meth:path_class.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Scheme'

Coerce value to a :class:Scheme member.

Accepts:

  • :class:Scheme (returned as-is);
  • a scheme string — case-insensitive, trailing :// and whitespace tolerated, common aliases ("s3a" → :attr:S3, "local" → :attr:FILE, "memory" → :attr:MEMORY);
  • None — returns default if supplied, else raises.

default swallows unknown / unparseable input. Without it, unknown tokens raise :class:ValueError and unsupported types raise :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

True when :meth:from_ would succeed for value.

path_class

path_class() -> 'type[URLBased]'

Return the concrete :class:URLBased subclass for this scheme.

Lazy: the backend module is imported on first use, which fires the :meth:URLBased.__init_subclass__ side-effect that registers the class. Subsequent calls hit the per-process cache.

Raises :class:ImportError when the backend's optional dependencies aren't installed (databricks-sdk for the Databricks schemes, boto3 for s3, …) — the message names the missing extra so the caller can install it.

resolve classmethod

resolve(value: Any, *, default: Any = ...) -> 'type[URLBased]'

Shortcut: cls.from_(value).path_class().

Useful where the caller has a raw scheme string (or URL) and just wants the concrete URLBased class. default is forwarded to :meth:from_ for forgiving lookup; it does not catch :class:ImportError from :meth:path_class.

NodeType

Bases: str, Enum

Canonical Databricks node_type_id values.

Member values are the exact strings the Databricks SDK expects. Use :attr:NodeType.DEFAULT for the codebase-wide default node type, the semantic-size aliases (:attr:SMALL / :attr:MEDIUM / :attr:LARGE / :attr:XLARGE) for intent-driven sizing, or the explicit cloud-specific members when you need a precise SKU.

The enum is intentionally not exhaustive — Databricks publishes hundreds of SKUs per cloud and most callers want a small, opinionated set. Coerce caller-provided strings through :meth:to_id / :meth:from_ so unknown SKUs round-trip as plain strings instead of raising.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'NodeType'

Coerce value into a :class:NodeType member.

Accepts:

  • :class:NodeType (returned as-is);
  • any string in :data:_NODETYPE_ALIASES, case-insensitive;
  • a bare member name ("FLEET_XLARGE");
  • None — returns default if supplied, else :attr:NodeType.DEFAULT.

Raises :class:ValueError for unknown strings unless default is supplied. To accept arbitrary SKU strings (without rejecting valid cloud-specific identifiers), call :meth:to_id instead.

to_id classmethod

to_id(
    value: Union[str, "NodeType", None],
    *,
    default: Optional[Union[str, "NodeType"]] = None
) -> str

Coerce value to a Databricks node_type_id string.

Unlike :meth:from_, this is forgiving: arbitrary SKU strings round-trip unchanged so callers can pass cloud-specific identifiers the enum does not enumerate ("r5d.metal", custom marketplace images, …). Aliases are resolved when present.

Resolution order:

  1. NodeType member → its string value.
  2. Alias / member name / value match → the canonical string.
  3. Bare string → trimmed and returned as-is.
  4. Nonedefault if supplied, else :attr:NodeType.DEFAULT.

is_known classmethod

is_known(value: Any) -> bool

True when :meth:from_ would succeed for value.

NodeSpec dataclass

NodeSpec(
    cpu_cores: int,
    ram_gib: float,
    gpu_count: int = 0,
    local_disk_gib: float = 0.0,
    cloud: str = "",
)

Hardware specs for a :class:NodeType member.

The numbers reflect the vendor's published per-VM characteristics for a single worker (or driver) — not the cluster-wide totals. ram_gib is binary GiB (1024 ** 3 bytes), matching every cloud's own documentation convention.

Attributes

cpu_cores Virtual CPU count exposed to Spark. ram_gib Memory available to Spark, in IEC GiB. gpu_count Number of GPUs (0 for non-GPU SKUs). local_disk_gib Local SSD/NVMe storage attached to the instance. 0 when the SKU has no local disk (relies on remote/elastic storage). cloud Cloud family identifier ("fleet" / "aws" / "azure" / "gcp") used by :meth:NodeType.from_cpu_and_ram when prefer is set.

State

Bases: IntEnum

Unified execution state for async statement / job results.

Use :meth:from_ to normalize backend-specific tokens, and the is_* predicates to derive done / failed / started without re-implementing the membership sets per backend.

is_idle property

is_idle: bool

True for :attr:IDLE — built locally, not submitted yet.

is_queued property

is_queued: bool

True for :attr:QUEUED — prepared, sitting in a submission queue.

is_pending property

is_pending: bool

True for :attr:PENDING — handed to a backend, awaiting ack.

is_accepted property

is_accepted: bool

True for :attr:ACCEPTED — acknowledged but not yet running.

is_running property

is_running: bool

True for :attr:RUNNING — actively executing.

is_partial property

is_partial: bool

True for :attr:PARTIAL — partially filled, still active.

is_started property

is_started: bool

True for anything from :attr:RUNNING onward.

Mirrors :attr:StatementResult.started: once the backend has actually started executing, is_started flips and stays True through every terminal state. :attr:ACCEPTED is not started — the venue holds the order, no execution yet.

is_active property

is_active: bool

True for non-terminal states with backend awareness.

Covers :attr:QUEUED through :attr:PARTIAL — anything the caller can reasonably wait on. :attr:IDLE is excluded (nothing has been submitted yet) and every terminal state is excluded (no more transitions).

is_done property

is_done: bool

True for terminal states (:attr:SUCCEEDED, :attr:REJECTED, :attr:FAILED, :attr:CANCELED, :attr:EXPIRED) — no more transitions expected.

is_failed property

is_failed: bool

True for :attr:REJECTED / :attr:FAILED / :attr:CANCELED / :attr:EXPIRED.

Every non-success terminal state counts as failed because each one means "the caller asked for a result and didn't get one": cancellation, rejection, mid-run error, or TTL expiry all leave the operation incomplete from the caller's view, and the per-backend raise_for_status raises on each.

is_succeeded property

is_succeeded: bool

True for :attr:SUCCEEDED — terminal with a full result.

is_rejected property

is_rejected: bool

True for :attr:REJECTED — refused at submission / accept.

is_canceled property

is_canceled: bool

True for :attr:CANCELED.

is_expired property

is_expired: bool

True for :attr:EXPIRED — TTL / day-rollover terminal.

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'State'

Coerce any Python value into a :class:State.

See module docstring for the accepted shapes. default swallows unknown / unparseable input; without it, unknown tokens raise :class:ValueError and unsupported types raise :class:TypeError.

is_valid classmethod

is_valid(value: Any) -> bool

Return True when :meth:from_ would succeed for value.

IOKind

Bases: IntEnum

What a backend reports a path/holder entry is.

EngineType

Bases: IntEnum

Compute engine for a Databricks :class:~yggdrasil.databricks.table.table.Table read / write.

  • :attr:YGGDRASIL — yggdrasil's native DeltaFolder (direct _delta_log
  • parquet over UC-vended credentials), no warehouse.
  • :attr:DATABRICKS_SQL_WAREHOUSE — the Databricks SQL warehouse.
  • :attr:SPARK — a Spark session (Databricks Connect / cluster / notebook).

from_str classmethod

from_str(value: Any, *, default: Any = ...) -> 'Optional[EngineType]'

Coerce an alias string into an :class:EngineType.

Matches a forgiving alias table ("warehouse" / "sql"DATABRICKS_SQL_WAREHOUSE; "ygg" / "native"YGGDRASIL; "spark" / "connect"SPARK). On an unknown string: raise :class:ValueError when default is ..., else return default.

from_numeric classmethod

from_numeric(value: Any, *, default: Any = ...) -> 'Optional[EngineType]'

Coerce an integer code into an :class:EngineType.

On an out-of-range / non-integer code: raise when default is ..., else return default. bool is rejected (it's an int subclass but never a meaningful engine code).

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Optional[EngineType]'

Coerce an :class:EngineType / alias string / int code / None.

None (or ...) is "unset" and returns None. A string routes to :meth:from_str, a number to :meth:from_numeric; both honour default (... → raise, else return the fallback).

AWSRegion

Bases: str, Enum

partition property

partition: str

The AWS partition this region belongs to (aws / aws-us-gov / aws-cn).

from_ classmethod

from_(value: Any, *, default: Any = ...) -> 'Optional[AWSRegion]'

Coerce value (an :class:AWSRegion, a region string, or None) to a member. Unknown / None returns default if given, else raises.

from_text classmethod

from_text(text: Any, *, default: Any = None) -> 'Optional[AWSRegion]'

Find an AWS region code embedded in text (e.g. a bucket name like acme-dls3-eu-central-1-p), or default if none is present.

Matches a region only when it stands as a whole, delimiter-bounded token (so eu-central-12 / ...-1foo don't false-positive), and prefers the longest code at a position (ap-southeast-1 over ap-south-1).

StructField dataclass

StructField(*args: Any, **kwargs: Any)

Bases: Field

A :class:Field whose dtype is a :class:StructType.

StructField([f1, f2, ...]) builds a struct field from its children directly — sugar for the equivalent Field(name=..., dtype=StructType(fields=(...,))) chain that :class:Field's constructor accepts. Every schema-shaped method (mapping surface, set operators, engine schema export, autotag, struct-aware equals) is inherited from :class:Field.

StructField(fields, name=..., metadata=..., ...).

fields is the children list — :class:Field instances, :class:pa.Field instances, or anything :meth:Field.from_any accepts. dtype=<StructType> is accepted as an alternative for callers (and the Field.__new__ redirect) that already have a built struct dtype in hand; pass one or the other, not both.

The wide *args / **kwargs shape is here because Field.__new__ redirects struct-shaped Field(...) calls (positional or keyword) to this class and Python then re-enters __init__ on the already-stamped instance with the original Field arguments. The re-init guard below absorbs that pass before any signature parsing runs.

position property

position: int | None

Optional 0-based index this field claims in a parent schema.

Stored in :attr:metadata under :data:POSITION_KEY. Used by :meth:select_in_field (and the engine-specific select_in_* helpers) as the last-resort fallback when :attr:name doesn't match a child name in the receiving schema — the receiver's children[position] (or column at position) is then resolved by name and used.

None (the default) leaves position-based lookup disabled, matching the historical name-only resolver.

default_value property

default_value

Field's default Python value (or the dtype-level default).

Reads :data:DEFAULT_VALUE_KEY from :attr:metadata first; falls back to self.dtype.default_pyobj when the metadata slot is unset. Renamed from default so the constructor classmethod :meth:Field.default can take that name — field.default would otherwise shadow it via descriptor lookup.

media_type property

media_type: 'Any | None'

:class:MediaType describing how this field's data is stored.

Decodes the b"media_type" metadata key — the mime-string canonical form ("application/vnd.apache.arrow.file", "application/vnd.apache.parquet", …) round-tripped through :meth:MediaType.from_. None when no media-type hint has been stamped.

Populated by :class:Folder._persist_schema so a schema loaded from a folder's .ygg/schema.arrow sidecar tells the reader which on-disk format the rows were last written in (Arrow IPC, Parquet, …) without walking the part files. Schema-level (top-level :class:StructField) is the canonical slot, but the accessor lives on :class:Field so per-column hints (e.g. the response-body field's HTTP Content-Type) can use the same property.

fields property

fields: list['Field']

Children excluding constraint-only fields.

inner_fields property

inner_fields: 'OrderedDict[str, Field]'

Compat view of the children as an ordered {name: field} map.

select_fields

select_fields(
    identifiers: "SelectType | Iterable[SelectType]" = (),
    *others: SelectType,
    raise_error: bool = True
) -> list["Field"]

Resolve one or more identifiers into the matching :class:Field objects.

Accepts strings (resolved by name), ints (resolved by index), and existing :class:Field instances (resolved by .name against this container — so callers can copy a field set between sibling schemas without first stringifying everything).

Calling shapes that all work the same way:

  • schema.select_fields("price") — single identifier.
  • schema.select_fields("price", "qty", 0) — multiple positionals.
  • schema.select_fields(["price", "qty"]) — single iterable.
  • schema.select_fields(other_schema.children) — copy a sibling's fields by name into this schema.
  • schema.select_fields("price", ["qty", "ts"], 0) — mixed; each positional is itself flattened so iterables and scalars can be interleaved.

:param identifiers: First identifier or iterable of identifiers. :param others: Additional identifiers. Each is flattened the same way as the first. :param raise_error: True (default) — missing identifiers raise via :meth:field_by with the same suggestion-rich error message used elsewhere. False — missing identifiers yield None in the returned list, preserving caller order.

:returns: A list of :class:Field (or Field | None when raise_error=False), one entry per resolved identifier in caller order. Duplicates in the input produce duplicates in the output — this is intentional, since select is the natural place to express a projection and projections sometimes repeat columns.

:raises KeyError: With suggestions, when raise_error is True and an identifier doesn't resolve. :raises TypeError: When an identifier is not a str / int / Field.

short

short() -> str

A compact name:dtype header tag — the dtype via :meth:~yggdrasil.data.types.base.DataType.short (recursive for nested types). Used for the column headers in :meth:yggdrasil.io.tabular.Tabular.display.

markers

markers() -> str

The main schema markers for a preview header, space-joined ("" when none): the key / layout flags (PK / FK / CK / partition / cluster / sorted / IK) and a * for a non-nullable (required) column. The compact cousin of :meth:_pretty_markers.

default classmethod

default(
    name: str = "",
    dtype: DataType = ObjectType(),
    nullable: bool = True,
    metadata: dict[bytes, bytes] | None = None,
    tags: dict[bytes, bytes] | None = None,
    default: Any = None,
)

Build a default-typed Field (ObjectType() unless overridden).

Convenience constructor for the "I just have a name" path — callers passing a plain string into APIs that expect a :class:Field (e.g. CastOptions(match_by=["id"])) land here. The instance-side default accessor was renamed to :attr:default_value so this name was free for the constructor.

pretty_format

pretty_format(indent: int = 2, level: int = 0) -> str

Pretty-print this field with the header on one line and the dtype below.

Layout is uniform across flat and nested dtypes — every field renders as a single field: 'name' <dtype>{markers} header line, with nested dtypes walking their inner fields inline at level + 1 so the tree reads as a flat list of consistent rows::

field: 'row' struct
  field: 'id' int64 not null [PK]
  field: 'name' string
  field: 'inner' struct
    field: 'age' int64
    field: 'email' string

indent is the per-level step in spaces; level is the current depth. The header carries the dtype kind (struct / list / map for nested, the primitive pretty-format for flat), the not null marker, the bracketed marker group (primary / foreign / constraint key, partition / cluster / sorted, any caller-defined tags, default value), and the comment.

Map dtypes flatten the synthetic entry struct into field: 'key' … / field: 'value' … lines so the key / value framing reads at the same level as a struct's own children rather than under an artificial wrapper.

Examples::

>>> print(field("id", "int64", nullable=False,
...             tags={"primary_key": True}).pretty_format())
field: 'id' int64 not null [PK]

>>> print(field("date", "date32",
...             tags={"partition_by": True}).pretty_format())
field: 'date' date32 [partition]

>>> print(field("user", StructType.from_fields([
...     field("id", "int64"),
...     field("email", "string"),
... ])).pretty_format())
field: 'user' struct
  field: 'id' int64
  field: 'email' string

invalidate_cache

invalidate_cache(*, cascade: bool = True) -> None

Drop cached engine projections, cascading to ancestors by default.

Public surface over :meth:_invalidate_cache. Callers that mutate the underlying state outside of the with_* mutators (custom DataType subclass that swaps children in place, external code that pokes dtype.fields directly) should call this once to make sure the next to_arrow_field / to_polars_field / to_pyspark_field / *_schema request rebuilds with the new state. With cascade=True (the default) every ancestor reachable via :attr:parent also drops its cache, so a struct's cached arrow schema gets rebuilt after one of its children mutates.

equals

equals(
    other: Any,
    check_names: bool = True,
    check_dtypes: bool = True,
    check_nullable: bool = True,
    check_metadata: bool = True,
) -> bool

Structural equality check with configurable scope.

Mirrors :meth:DataType.equals. Coerces other to a Field so that callers can pass a pa.Field / dict / etc. without manual conversion. Returns False on coercion failure instead of raising.

  • check_names: compare this field's name and recurse into child field names for nested types. For struct-shaped fields the comparison is order-independent (children matched by name) when check_names is True, mirroring how Arrow schemas are name-keyed.
  • check_dtypes: recurse into the dtype and compare nullable (both are structural, schema-defining attributes).
  • check_metadata: compare this field's metadata and recurse.

set_position

set_position(value: int | None) -> 'Field'

Set / clear :attr:position on self in place.

Negative values are rejected — positions are forward indices into the parent schema; if you need a last-element fallback, resolve it before calling.

with_position

with_position(position: int | None) -> 'Field'

Return a copy of this field with :attr:position set / cleared.

check_pandas_metadata

check_pandas_metadata(source: Any = None) -> 'Field'

Stamp pandas index tags onto child fields from a b"pandas" blob.

pandas carries its DataFrame index layout in the pyarrow b"pandas" schema metadata (index_columns). This reads that blob and marks each matching child as an index level via :meth:with_index_key, so a struct-shaped Field round-trips the index when it later rebuilds a DataFrame.

source is whatever carries the blob — a pa.Schema, a pa.Table, raw bytes / str JSON, or an already-parsed dict. When omitted, falls back to self.metadata[b"pandas"] (which :meth:from_arrow_schema preserves). Mutates and returns self for chaining; a no-op when there's no blob or no string index columns.

PARITY: Python/pandas-only. The TS port has no pandas
counterpart, so there is no mirror for this method.

with_field

with_field(
    field: "Field | pa.Field | str",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True,
    **kwargs: Any
) -> "Field"

Return self with field appended or merged in.

mode controls collision behavior when a child with the same name already exists. Accepts a :class:Mode member or any alias :meth:Mode.from_ understands.

  • :data:Mode.AUTO / :data:Mode.OVERWRITE — replace the existing child verbatim with field.
  • :data:Mode.APPEND — append a fresh child even if the name collides (struct semantics: last-write-wins for duplicate names; both entries survive in the children tuple).
  • :data:Mode.IGNORE — keep the existing child; drop the incoming.
  • :data:Mode.ERROR_IF_EXISTS — raise :class:ValueError on collision.
  • :data:Mode.UPSERT / :data:Mode.MERGE — :meth:merge_with the existing child against the incoming one (dtype, nullability, metadata), keeping the existing child's identity.

Auto-promotion to struct: when self isn't a struct (a primitive Field, a list/map, …) the call returns a fresh struct Field whose first child is the previous self (renamed to its current name so it's addressable) and whose second child is field. The promoted struct keeps self's name, nullability, and metadata — only the dtype changes.

Bare-string shorthand: self.with_field("price") reads as "make sure a child named 'price' exists." That call goes through :meth:Field.from_any which infers a sensible default dtype.

inplace=True (the default) mutates self and returns it. inplace=False returns a fresh copy.

with_fields

with_fields(
    fields: "Iterable[Field | pa.Field | str]",
    *,
    mode: "Mode | str | None" = None,
    inplace: bool = True
) -> "Field"

Apply :meth:with_field for every entry in fields.

Same mode semantics as :meth:with_field; the loop short- circuits :data:Mode.IGNORE once any one collision keeps the existing child (no global "first one wins, drop the rest" gymnastics — collisions are evaluated per name).

Auto-promotes self to a struct on the first call when needed; subsequent fields land on that struct.

autotag

autotag(tags: dict[AnyStr, AnyStr] | None = None) -> 'Field'

Stamp this field with tags derived from its dtype and name.

Writes Databricks-friendly auto-tags in place:

  • Everything from :meth:DataType.autotag (kind plus dtype detail like unit / tz / precision / scale / signed / iso / srid).
  • nullable for data-quality policies.
  • Name-based heuristics for governance: role=identifier for *_id / *_uuid, role=audit_timestamp for created_at patterns, plus pii / sensitive stamps for columns that obviously carry personal or credential data.

For struct-shaped fields (schemas) primary_key / partition_by / cluster_by entries on this field's metadata get consumed into per-child tags, and each child is autotagged in turn — so schema.autotag() propagates without the caller having to walk children manually.

Returns a new struct-shaped Field for schema-style autotagging, or self for primitive autotagging — both modes also stamp in place so existing f.autotag() chains keep working.

from_field classmethod

from_field(f: 'Field') -> 'Field'

Lift a :class:Field to cls.

For cls is Field this is identity. For subclasses (e.g. :class:Schema) it normalises the input to the subclass shape — for struct dtypes we keep the children, for non-struct we wrap the field as a single-child struct so the schema-shape contract holds.

from_fields classmethod

from_fields(
    fields: Iterable["Field | Any"],
    *,
    name: str = DEFAULT_FIELD_NAME,
    nullable: bool = False,
    metadata: dict[bytes | str, bytes | str | object] | None = None,
    tags: dict[bytes | str, bytes | str | object] | None = None
) -> "Field"

Build a struct-shaped instance from a list of fields.

from_spark_column classmethod

from_spark_column(column: 'ps.Column') -> 'Field'

Build a :class:Field from a pyspark.sql.Column.

Column objects don't expose a typed dtype on the public Python surface — we read the SQL-rendered expression instead and parse that:

  • id — bare reference. Name is id, dtype defers to the fallback (ObjectType) since neither the JVM nor the Spark Connect proxy exposes the underlying schema on a free-standing column.
  • CAST(<expr> AS <dtype>) / CAST(<expr> AS <dtype>) — name follows the inner <expr>'s leaf, dtype reads straight off <dtype> through :meth:DataType.from_str. Covers df["x"].cast("string"), df["x"].astype("decimal(10,2)"), F.col("x").cast(StringType()).
  • <expr> AS <alias> — name follows <alias>, dtype comes from the inner <expr> (recurses, so a cast inside an alias keeps its dtype).
  • Anything else falls back to the full SQL string as the name with :class:ObjectType as the dtype, since we can't infer the dtype of an arbitrary Catalyst expression without binding it through :meth:SparkSession.createDataFrame (which would be a live JVM round trip the caller didn't ask for).

Source of the SQL string, in order:

  1. Classic Spark: column._jc.toString() — the JVM Column.
  2. Spark Connect: column._expr.__repr__() — the proxy doesn't have _jc (accessing it raises PySparkAttributeError(JVM_ATTRIBUTE_NOT_SUPPORTED)) but _expr.__repr__ is exactly what Column.__repr__ wraps as "Column<'<sql>'>".
  3. repr(column) stripped of the Column<'…'> wrapper — last-resort for any future PySpark whose internal slots renamed.

Use :meth:Field.from_spark_field instead when the caller already has the resolved StructField (e.g. from df.schema.fields[i]) — that path keeps the precise dtype without going through the SQL string.

to_dict

to_dict(dump_parent: bool = False) -> dict[str, Any]

Serialize this field to a JSON-friendly dict.

dump_parent (default False) controls whether :attr:parent — the structural back-pointer to the field this one is nested under — is included. Children are still emitted via the dtype's to_dict (a struct field's dtype carries its members), so dropping parent prevents the recursion that would otherwise echo the whole ancestor chain into every nested field's payload.

to_arrow_field

to_arrow_field(dump_json: bool = False) -> pa.Field

Project to a :class:pa.Field.

Arrow preserves nested-type structure (struct, list, map) with per-field metadata recursively, so the dtype intent round-trips natively without us stuffing a type_json blob into the metadata. Only callers that need the exact :class:DataType subclass back (e.g. Decimal precision / Timestamp tz / extension types) should pass dump_json=True.

dump_json defaults to False; the cached path is the canonical (no-blob) shape, which is what every internal caller wants now that :meth:from_arrow_field falls back through :meth:DataType.from_arrow_type when the blob is missing.

to_arrow_schema

to_arrow_schema() -> pa.Schema

Project this field as a top-level :class:pa.Schema.

Struct-shaped fields (including :class:~yggdrasil.data.Schema) unfold their children into the schema's columns; non-struct fields produce a single-column schema with self as that column. The schema-level metadata mirrors self.metadata, plus the field's name / nullable flag re-embedded as b"name" / b"nullable" so :meth:Field.from_arrow_schema can recover them (pa.Schema has no native slot for either).

to_polars_schema

to_polars_schema() -> 'polars.Schema'

Project this field as a :class:polars.Schema.

Struct-shaped fields unfold into the schema's columns; non-struct fields produce a single-column schema.

to_pyspark_field

to_pyspark_field() -> 'pst.StructField'

Project to a Spark :class:StructField.

Spark's :class:StructType preserves struct children with their own metadata, so primitive and struct dtypes don't need a type_json round-trip blob. Spark's :class:MapType / :class:ArrayType only carry the element / key+value Spark types and lose any field-level metadata on the way through, so we dump the dtype JSON for those (and only those) to recover the original yggdrasil dtype on read.

to_spark_schema

to_spark_schema() -> 'pst.StructType'

Project this field as a top-level Spark :class:StructType.

Struct-shaped fields unfold their children into the StructType's fields; non-struct fields produce a single-field StructType.

cast

cast(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast obj to this field using its native engine.

Routing is by module prefix via :meth:ObjectSerde.module_and_name:

  • pyarrow.* → :meth:cast_arrow
  • polars.* → :meth:cast_polars
  • pandas.* → :meth:cast_pandas
  • pyspark.* → :meth:cast_spark
  • iterator / iterable → recurse per element (lazy generator)
  • everything else → :class:TypeError

self.dtype.type_id == OBJECT is handled by the narrow methods — they pass obj through unchanged because a variant column must never be cast. No redundant guard here.

cast_arrow

cast_arrow(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pyarrow object — dispatch by shape.

Table/RecordBatch → :meth:cast_arrow_tabular, Array/ChunkedArray → :meth:cast_arrow_array.

cast_polars

cast_polars(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any polars object — dispatch by shape.

DataFrame/LazyFrame → :meth:cast_polars_tabular, Series → :meth:cast_polars_series, Expr → :meth:cast_polars_expr.

cast_pandas

cast_pandas(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any pandas object — dispatch by shape.

DataFrame → :meth:cast_pandas_tabular + index check, Series → :meth:cast_pandas_series.

cast_spark

cast_spark(obj: Any, options: 'CastOptions | None' = None, **more: Any) -> Any

Cast any spark object — dispatch by shape.

DataFrame → :meth:cast_spark_tabular, Column → :meth:cast_spark_column.

cast_arrow_batch_iterator

cast_arrow_batch_iterator(
    batches: "Iterable[pa.RecordBatch]",
    options: "CastOptions | None" = None,
    **more
) -> "Iterator[pa.RecordBatch]"

Cast a stream of :class:pa.RecordBatch against this field.

Object targets passthrough (variant). Otherwise the dtype's struct view owns the per-batch tabular cast and byte_size rechunk — same shape contract as :meth:cast_arrow_tabular, just lazy.

fill_nulls

fill_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in obj using the native engine — engine + shape detection.

Routes the same way :meth:cast does. See :meth:fill_arrow / :meth:fill_polars / :meth:fill_pandas / :meth:fill_spark for the per-engine behaviour.

fill_arrow

fill_arrow(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pyarrow object.

Arrays go through :meth:fill_arrow_array_nulls directly. Tables / RecordBatches re-use the tabular cast path with self as the target — a no-op cast that still runs the per-column null-fill via the struct walk.

fill_polars

fill_polars(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any polars object.

Series / Expr go through :meth:fill_polars_array_nulls — which handles both shapes uniformly (Expr is the lazy counterpart of Series; the fill operator grafts onto each identically). DataFrame / LazyFrame route through :meth:cast_polars_tabular as a self-targeted cast.

fill_pandas

fill_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any pandas object.

fill_spark

fill_spark(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls in any spark object.

polars_alias

polars_alias(obj: Any) -> Any

Rename a polars Series / Expr to match this field's name.

No-op when the target name matches the current name, or when this field only has the sentinel name. Calling defensively is free — zero-cost on the no-rename path.

spark_alias

spark_alias(obj: Any) -> Any

Rename a Spark Column to match this field's name.

Spark DataFrames aren't handled — renaming a DataFrame requires a projection with named columns, which isn't a single-method operation. Column is the rename target here.

pandas_alias

pandas_alias(obj: Any) -> Any

Rename a pandas Series to match this field's name.

Pandas has no .alias() — rename is series.name = ..., which mutates. This helper returns the series so it chains like :meth:polars_alias / :meth:spark_alias. DataFrames aren't handled (column rename is a projection, not a single-method op).

finalize_arrow_array

finalize_arrow_array(
    array: Array | ChunkedArray, *, default_scalar: Scalar | None = None
)

Fill nulls on a pyarrow Array / ChunkedArray.

No alias step: pa.Array / ChunkedArray don't carry a name. Tabular naming lives in the pa.Field that wraps the array in a Table/RecordBatch, which :meth:cast_arrow_tabular handles through the struct walk.

finalize_arrow

finalize_arrow(obj: Any, *, default_scalar: Scalar | None = None) -> Any

Finalize any pyarrow object — dispatch by shape.

Array/ChunkedArray → fill. Table/RecordBatch → identity.

finalize_polars_series

finalize_polars_series(series: 'polars.Series', *, default_scalar: Any = None)

Fill nulls, alias a polars Series to the target name.

finalize_polars_expr

finalize_polars_expr(expr: 'polars.Expr', *, default_scalar: Any = None)

Fill nulls, alias a polars Expr to the target name.

Same as :meth:finalize_polars_series — polars Series and Expr share the fill + alias primitives, so the finalize shape is identical. Separate method for call-site clarity.

finalize_polars

finalize_polars(
    obj: "polars.Series | polars.Expr | polars.DataFrame | polars.LazyFrame",
    *,
    default_scalar: Any = None
)

Finalize any polars object — dispatch by shape.

Series/Expr → fill + alias. DataFrame/LazyFrame → identity (tabular cast already finalized per-column via the struct walk).

finalize_pandas_series

finalize_pandas_series(series: 'pd.Series', *, default_scalar: Any = None)

Fill nulls, rename a pandas Series to the target name.

finalize_pandas

finalize_pandas(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any pandas object — dispatch by shape.

Series → fill + rename. DataFrame → identity.

check_pandas_indexes

check_pandas_indexes(obj: Any) -> Any

Promote columns tagged index_key to the DataFrame index.

Collects children with :attr:index_key set, sorted by :attr:index_key_level, and calls set_index on the DataFrame. __index_level_N__ placeholder names are mapped back to None so the round-trip matches the source.

For a Series whose field is itself tagged index_key, the Series is returned as-is — the caller decides how to attach it as an index.

Passthrough when no children carry the tag or when the object is not a DataFrame.

finalize_spark_column

finalize_spark_column(
    column: "ps.Column", *, default_scalar: Any = None
) -> "ps.Column"

Fill nulls, alias a Spark Column to the target name.

finalize_spark

finalize_spark(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any spark object — dispatch by shape.

Column → fill + alias. DataFrame → identity (tabular cast already finalized).

finalize

finalize(obj: Any, *, default_scalar: Any = None) -> Any

Finalize obj using its native engine — module-prefix dispatch.

Mirrors :meth:cast / :meth:fill_nulls routing.

select

select(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field with only the selected children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).

drop

drop(
    *identifiers: "str | int | Field | Iterable[str | int | Field] | None",
) -> "Field"

Return a new struct-shaped Field without the specified children.

Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).

to_field

to_field() -> Field

Identity — StructField IS a Field. Kept for type-narrowing callers.

as_spark

as_spark() -> 'StructField'

Return a :class:StructField whose dtype is Spark-compatible.

as_polars

as_polars() -> 'StructField'

Return a :class:StructField whose dtype is Polars-compatible.

unit_family_for

unit_family_for(value: Any) -> type[Unit]

Find the unit family that recognises value (member or symbol).

Useful when a curated row carries a free-form unit token and the caller needs to dispatch to the right family without hard-coding a per-source switch. Returns the first family for which :meth:Unit.is_valid returns True; raises :class:ValueError when no family claims the token (the message lists every family that was consulted).