yggdrasil.data¶
data ¶
TimeUnit ¶
Bases: str, Enum
Canonical time-unit token for temporal DataType instances.
Members carry the lowercase short form so the enum is a drop-in string replacement::
TimeType(unit=TimeUnit.MICROSECOND)
TimeType(unit="us") # equivalent — both store ``"us"``
Use :meth:parse when accepting external input — it canonicalizes
aliases ("microseconds", "micros", "µs") to a member
and raises :class:ValueError for unknown tokens.
seconds
property
¶
Seconds per one of this unit (used for scalar epoch math).
Calendar-style interval units (year_month / day_time /
month_day_nano) have no fixed second-count and return
float('nan') so comparisons surface the mismatch instead of
silently truncating to zero.
order
property
¶
Precision rank — higher = finer.
Used by TemporalType._merge_with_same_id to pick the wider
of two units. Calendar interval units sit at rank -1 so
they don't outrank fixed-precision ones in normal merges.
from_
classmethod
¶
Coerce any Python value into a :class:TimeUnit.
Accepts:
- :class:
TimeUnit(returned as-is); - any string the alias table or canonical member values know —
s/ms/us/ns/d/ interval forms, plurals (microseconds), long forms (millisecond), mixed case,µs, hyphens / spaces; - objects exposing a
time_unit/unitattribute (PolarsDatetime/Duration, PyArrowTimestampType/DurationType/Time32Type/Time64Type); the attribute is re-funneled throughfrom_; None— returns default if supplied, else raises.
default swallows unknown / unparseable input. Without it,
unknown tokens raise :class:ValueError and unsupported types
raise :class:TypeError.
is_valid
classmethod
¶
Return True when :meth:from_ would succeed for value.
Timezone
dataclass
¶
An immutable wrapper around an IANA timezone identifier.
Instances are created via :meth:from_ (accepts strings,
ZoneInfo, datetime.tzinfo, other Timezone objects, or
None → UTC) or directly::
tz = Timezone("Europe/Paris")
tz = Timezone.from_("CET") # → Timezone("Europe/Paris")
tz = Timezone.from_("+01:00") # → Timezone("Etc/GMT-1")
tz = Timezone.from_(ZoneInfo("Europe/Paris"))
The naive case is represented by :attr:Timezone.NAIVE — a
sentinel instance whose iana is the empty string and whose
__bool__ is False. Use it instead of None so a
tz: Timezone field type can stay non-optional.
utc_seconds_offset
property
¶
Fixed UTC offset in seconds, or None for non-fixed zones.
Returns 0 for UTC and its equivalents (Etc/UTC,
GMT, Etc/GMT+0, …), the parsed offset for
Etc/GMT±N (note IANA's sign-flip — Etc/GMT-3 means
UTC+3, so the property reports +10800 not -10800),
and None for any zone whose offset depends on DST or for
:attr:Timezone.NAIVE. Use :meth:utc_offset instead when
you need a DST-aware offset for a specific instant.
Naming follows the "type-suffix unit" convention so
utc_seconds_offset makes the unit explicit at the call
site — utc_offset returns a :class:datetime.timedelta,
utc_offset_hours returns fractional hours, and this
property returns whole seconds (or None).
from_
classmethod
¶
Coerce any Python value into a :class:Timezone.
Accepts:
- :class:
Timezone(returned as-is — includingTimezone.NAIVE); - :class:
zoneinfo.ZoneInfo(extractskey); - a string — IANA name, alias (
CET/EST/Z/GMT/Etc/UTC/+00:00/ …), or fixed offset ("+01:00","UTC-05","-0530"); datetime.tzinfoinstance (ZoneInfo,datetime.timezone, third-party zones) — thekey/zoneattribute is preferred; otherwise falls back to the fixed UTC offset;- timezone-aware
datetime/time(extractstzinfo); - objects exposing a
tz/time_zone/timezone/ianaattribute (PyArrowTimestampType, PolarsDatetime, foreign Timezone classes); the attribute is re-funneled throughfrom_; None— returns :attr:UTCfor backward compatibility unlessdefaultis supplied.
default swallows unknown / unparseable input. Without it,
bad input raises :class:ValueError (string parse failure) or
:class:TypeError (unsupported value type).
to_zoneinfo
cached
¶
Return the zoneinfo.ZoneInfo for this timezone.
Raises :class:ValueError for :attr:Timezone.NAIVE since
there is no concrete zone to materialize.
utc_offset ¶
Return the UTC offset at instant at (default: now).
The result accounts for DST transitions.
Timezone("Europe/Paris").utc_offset(datetime(2024, 7, 1)) datetime.timedelta(seconds=7200) # +02:00 (CEST)
utc_offset_hours ¶
Return the UTC offset in fractional hours.
Timezone("Asia/Kolkata").utc_offset_hours() 5.5
is_fixed_offset ¶
Return True if this timezone has a fixed UTC offset (no DST).
is_dst ¶
Return True if DST is active at instant at (default: now).
Timezone("Europe/Paris").is_dst(datetime(2024, 7, 1)) True Timezone("Europe/Paris").is_dst(datetime(2024, 1, 1)) False
dst_offset ¶
Return the DST adjustment at instant at.
Returns timedelta(0) when DST is not active or for fixed-offset
timezones, and for :attr:Timezone.NAIVE.
abbreviation ¶
Return the timezone abbreviation (e.g. "CET", "CEST") at at.
Timezone("Europe/Paris").abbreviation(datetime(2024, 7, 1)) 'CEST' Timezone("Europe/Paris").abbreviation(datetime(2024, 1, 1)) 'CET'
distance_to ¶
Return the offset difference between self and other at at.
A positive result means other is ahead of self.
Timezone.UTC.distance_to(Timezone.CET, datetime(2024, 7, 1)) datetime.timedelta(seconds=7200)
now ¶
Return the current wall-clock time in this timezone.
The returned datetime is timezone-aware.
localize ¶
Stamp a naive datetime with this timezone (no conversion).
Raises:
| Type | Description |
|---|---|
ValueError
|
If naive is already timezone-aware, or if
this is :attr: |
convert ¶
Convert a timezone-aware datetime to this timezone.
Raises:
| Type | Description |
|---|---|
ValueError
|
If aware is naive, or if this is
:attr: |
midnight ¶
Return midnight (00:00) in this timezone for date (default: today).
from_polars_type
classmethod
¶
Extract the timezone from a Polars Datetime dtype.
Returns None if the dtype is not Datetime or has no timezone.
Timezone.from_polars_type(pl.Datetime("us", "Europe/Paris")) Timezone('Europe/Paris')
polars_normalize
classmethod
¶
polars_normalize(
col: "polars.Series | polars.Expr",
*,
lazy: bool = True,
return_value: Literal["iana"] = "iana"
) -> "polars.Series | polars.Expr"
Normalize timezone strings in a Polars column using alias replacement only.
from_arrow_type
classmethod
¶
Extract the timezone from a PyArrow TimestampType.
Returns None if the type has no tz attribute or it is empty.
import pyarrow as pa Timezone.from_arrow_type(pa.timestamp("us", tz="Europe/Paris")) Timezone('Europe/Paris')
arrow_timestamp_type ¶
Return a pa.timestamp(unit, tz=self.iana) type.
Timezone.CET.arrow_timestamp_type("ns") TimestampType(timestamp[ns, tz=Europe/Paris])
all_iana
classmethod
¶
Return all IANA timezone identifiers available on this system.
Mode ¶
Bases: IntEnum
readable
property
¶
True when the mode admits reads.
Every :class:Mode canonically resolves to a + POSIX form
(rb, rb+, wb+, ab+, xb+) — all of those
admit reads. Only the strict :data:READ_ONLY rb and
:data:IGNORE (which is no-op) deny writes; nothing here
denies reads.
appendable
property
¶
True when writes append at EOF rather than at the cursor.
Only :data:APPEND carries POSIX O_APPEND semantics; every
other write mode positions writes at the explicit cursor.
os_mode
property
¶
Stdlib :func:open mode string for this :class:Mode.
- :attr:
READ_ONLY→"rb" - :attr:
OVERWRITE/ :attr:TRUNCATE→"wb+" - :attr:
APPEND→"ab+" - :attr:
ERROR_IF_EXISTS→"xb+" - everything else (AUTO, IGNORE, UPSERT, MERGE) →
"rb+"(in-place edit; the disposition is enforced higher up).
from_
classmethod
¶
Normalize value into a :class:Mode.
Accepts:
- :class:
Mode(returned as-is, idempotent). - Aliases like
"overwrite","OVERWRITE","error-if-exists","replace","add"— see :data:STR_MAPPING. - POSIX-style mode strings —
"rb","wb","ab+","x","r+b"— parsed structurally; any combination of one primary character (r/w/a/x) plus optionalb/t/+flags is accepted. None→ returns default if supplied, else :data:Mode.AUTO.
Falls back to :class:ValueError for unrecognized strings.
Numeric / non-string non-Mode inputs raise
:class:TypeError — the input grammar is "string or enum,"
not "anything stringifiable."
MimeType
dataclass
¶
MimeType(
name: str,
value: str,
extensions: tuple[str, ...] = (),
magics: tuple[MagicMatcher, ...] = (),
is_codec: bool = False,
is_tabular: bool = False,
is_blob: bool = False,
)
Dataclass MIME descriptor + registries.
- extensions: dotless, lower-case keys
- magics: ordered matchers
- is_codec: compression / wrapper formats
- is_tabular: row/tabular-ish formats (read into a frame)
- is_blob: opaque single-file payload — straight byte IO, no row
structure (images, pdf, archives, pickle, …). Mutually exclusive
with
is_tabular; codecs and directory/connector mimes are neither.
get
classmethod
¶
Pure lookup — never raises. Returns None on miss.
Kept as the low-level "resolve by name/value" entry point:
:meth:from_ / :meth:from_str / :meth:from_magic layer
the default-handling contract on top.
parse_many
classmethod
¶
Resolve obj to a flat, deduped list of :class:MimeType.
Accepts anything :meth:from_ accepts, plus:
- iterables (list, tuple, set, generator) of any supported input
- Accept-header strings:
"application/json, text/csv;q=0.8" - composite
format+codecstrings:"application/csv+gzip","parquet+zstd","trades.parquet.zst" - wildcard strings:
"*/*","text/*","image/*" None→[]
Order: first-seen wins, deduped by identity. For composites, the base format is emitted first, then the codec (codec wraps format).
Never raises on an unresolvable element — unknowns are dropped
silently. For strict per-element resolution, call :meth:from_.
from_magic
classmethod
¶
from_magic(
magic: Union[bytes, bytearray, memoryview, IO[bytes], str, Path],
default: "MimeType | None" = _RAISE,
) -> "MimeType | None"
Resolve by sniffing magic bytes from magic.
Accepts raw bytes/memoryview, an IO, or anything the buffer class can wrap. Reads the first 64 bytes and walks the registered magic matchers in definition order.
:param default: see :class:MimeType class docstring for the
shared default-handling contract.
:raises ValueError: on a miss when default was not supplied.
from_str
classmethod
¶
Resolve a :class:str — path-like, bare extension, or mime value.
Tries, in order:
- Direct lookup against the lower-cased input (covers
"text/csv"/"json"/".csv"without paying a :class:pathlib.Pathallocation). - If the string looks path-like (contains
/or\), take its suffix as an extension key. - Fall back to :meth:
get(name / mime-value lookup with theapplication//text// … prefix stripping). - Structural sniff on leading
{/[.
:param default: see :class:MimeType class docstring for the
shared default-handling contract.
:raises ValueError: on a miss when default was not supplied.
MimeTypes ¶
Singleton MIME definitions.
MediaType
dataclass
¶
from_many
classmethod
¶
Compose a MediaType from an ordered mime list (e.g. URL extensions).
Two conventions land here:
trades.csv.zst→["csv", "zst"]: the codec is the outer wrapper (you have to unzst before you can parse csv) →MediaType(CSV, codec=ZSTD).part-xxx.zstd.parquet→["zstd", "parquet"]: the format is the outer wrapper and the codec is the parquet page-codec hint baked into the file. Parquet handles the decompression internally →MediaType(PARQUET)with no outer codec; setting one would route the read through a decompressor that doesn't belong on this byte stream.
Order matters: the last mime decides. Last-is-codec promotes the codec to the wrapper slot; last-is-format keeps the format alone and drops earlier codec hints.
from_magic
classmethod
¶
from_magic(
buf: Union[bytes, bytearray, memoryview, IO[bytes]],
*,
default: "MediaType" = ...
) -> "MediaType"
Sniff a :class:MediaType from raw bytes.
Two-stage: outer magic identifies the wrapper; if the wrapper is a codec, decompress a head-window and sniff the inner format. See module docstring for the full story.
Accepts bytes / bytearray / memoryview directly. For
convenience, also accepts an IO[bytes] — in that case it
delegates to :meth:from_io, which manages cursor save/restore.
:param default: Fallback when the outer sniff finds nothing.
... (default) raises :class:ValueError; any other
value (including None) is returned as-is.
from_io
classmethod
¶
Sniff a :class:MediaType from a file-like, seeking freely.
Captures the cursor on entry and restores it on exit, so the caller's stream position is unaffected. Inside the call we seek freely:
seek(0)and read the first :data:_OUTER_PEEKbytes for outer-magic resolution.- If the outer mime is a codec,
seek(0)again and drive the codec's streaming decompressor directly to read the first :data:_INNER_PEEKbytes of decoded payload. - Sniff the decoded head for the inner mime.
The codec resolution is one dict lookup
(:meth:Codec.from_mime) — the outer magic loop already
landed on the registered :class:MimeType singleton, so we
reuse it as the key without re-resolving. For streaming
codecs (gzip / zstd / lz4 / bz2 / xz / zlib / lzma) the
decompressor is opened directly on the IO via
:meth:Codec._open_decompress_reader — no IO wrap, no
full-buffer materialization. Non-streaming codecs (snappy,
brotli) fall back to reading the compressed body and calling
:meth:Codec.decompress_bytes.
Decompression errors during the inner sniff are swallowed —
the worst case is returning a less-specific MediaType (the
outer codec is preserved). A separate caller that wants to
validate decompressibility should call :meth:Codec.decompress
directly.
:param io_obj: Any file-like with read + seek (stdlib
io.BytesIO, an open file, our own IO, etc.).
:param default: Fallback when the outer sniff finds nothing.
... (default) raises :class:ValueError; any other
value (including None) is returned as-is.
Codec ¶
Bases: ABC
Abstract compression codec.
is_streaming
property
¶
True when both compress and decompress have streaming paths.
Callers with large (GiB-scale) inputs should check this before
passing them to :meth:compress / :meth:decompress. When
False, those methods fall back to materializing the full
payload in memory through :meth:compress_bytes /
:meth:decompress_bytes.
compress ¶
Compress src into a new :class:IO.
Streams chunk-by-chunk when :meth:_open_compress_writer is
available. Otherwise falls back to a full-in-memory bytes
roundtrip — callers with multi-GiB inputs should inspect
:attr:is_streaming first.
The source cursor is restored on exit.
decompress ¶
Decompress src into a new :class:IO.
Streams chunk-by-chunk when :meth:_open_decompress_reader is
available. Otherwise falls back to a full-in-memory bytes
roundtrip — callers with multi-GiB compressed inputs should
inspect :attr:is_streaming first.
The source cursor is restored on exit.
read_start_end ¶
read_start_end(
src: "IO | bytes | bytearray | memoryview",
*,
n_start: int = 64,
n_end: int = 64,
chunk_size: int = _CHUNK
) -> tuple[bytes, bytes]
Return the first n_start and last n_end bytes of the decoded stream.
Streams the decompression and keeps only a bounded amount of
state in memory (:math:n\_end + chunk\_size bytes) — safe
for very large compressed inputs when the codec supports
streaming decompression.
When the codec does NOT expose a streaming decoder, falls back
to a full :meth:decompress_bytes call, which materializes
the whole uncompressed payload in memory.
from_
classmethod
¶
Parse an arbitrary input into a Codec instance.
Accepts:
- :class:Codec instances (returned as-is).
- Short names like "gzip", "zstd" (case-insensitive).
- Anything :meth:MimeType.parse can resolve to a codec mime.
- None → returns default.
Codecs ¶
Singleton class for accessing all registered codecs.
ByteUnit ¶
Bases: IntEnum
Canonical IEC binary byte-unit token + scalar value.
Each member's value IS the byte count for one unit, so 128 *
ByteUnit.MIB reads naturally at the call site and slots into
any int field. Use :meth:parse_size when accepting external
config / API input — it canonicalizes "128 MB" /
"1.5 GiB" / raw integers / ByteUnit members all to a
plain integer byte count.
from_
classmethod
¶
Coerce any Python value into a :class:ByteUnit member.
Accepts:
- :class:
ByteUnit(returned as-is); - any string the alias table knows —
B/KB/MiB/gigabyte/ mixed case / trailings; - an integer matching a member's byte value (
1024→ :attr:KIB); None— returns default if supplied, else raises.
default swallows unknown / unparseable input. Without it,
unknown tokens raise :class:ValueError and unsupported types
raise :class:TypeError.
parse_size
classmethod
¶
Coerce a size-like value to an integer byte count.
The single entry point for "config gave me a size, give me bytes." Accepts:
- :class:
int— passed through (must be non-negative); - :class:
ByteUnit— its scalar value; - a quantity string
"128 MB"/"1.5 GiB"/"512"— parsed with this enum's IEC conventions; - a bare unit string
"MiB"— yields one unit (1024**2); None— returns default if supplied, else raises.
Floating-point quantities round to the nearest byte ("1.5 KiB"
→ 1536). Negative values raise :class:ValueError.
format
classmethod
¶
Format an integer byte count as a human-readable string.
Picks the largest unit at which n divides cleanly into a
scalar ≥ 1, defaulting to IEC tokens ("128 MiB"); pass
iec=False for the colloquial short form ("128 MB").
precision controls fractional digits.
pretty
classmethod
¶
pretty(
v: float,
unit: "ByteUnit | str | None" = None,
*,
iec: bool = True,
precision: int = 1
) -> str
Human-readable rendering of a quantity v expressed in unit.
v is a scalar count of unit (bytes by default); it's
scaled to a byte count and handed to :meth:format. The companion
to :meth:format for the common "I have N MiB, show it nicely"
case::
ByteUnit.pretty(1536) # "1.5 KiB"
ByteUnit.pretty(8, ByteUnit.MIB) # "8.0 MiB"
ByteUnit.pretty(1.5, "gb") # "1.5 GiB"
unit defaults to :attr:B (resolved at call time so the class
default is usable inside the method body); pass iec=False for
the colloquial short form.
Unit ¶
Mixin providing :meth:from_ / :meth:convert / scalar helpers.
Subclasses combine this with :class:Enum and declare members
as (symbol, factor_to_canonical, offset_to_canonical) tuples
(or (symbol, factor) — offset defaults to 0.0). The
canonical member of each family has factor=1.0 and
offset=0.0; conversion goes through that canonical pivot.
Extra spellings ("megawatthours" → MWH) live in a
<Family>._ALIASES dict assigned outside the enum body — the
enum metaclass would treat a class-body dict as a member.
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
EnergyUnit ¶
Bases: Unit, Enum
Energy unit (canonical = joule).
Members cover SI scale (J / kJ / MJ / GJ / TJ), the watt-hour family that dominates electricity-market ingestion (Wh / kWh / MWh / GWh / TWh), and the imperial / domestic units (BTU / cal / kcal / therm).
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
PowerUnit ¶
Bases: Unit, Enum
Power unit (canonical = watt).
SI scale (W / kW / MW / GW / TW) plus mechanical horsepower. Energy-market feeds publish capacity in MW, transmission limits in GW, household-scale appliances in W — having one enum cover all of them is what keeps schema-per-source curated views from re-stringing the unit token at every read.
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
MassUnit ¶
Bases: Unit, Enum
Mass unit (canonical = kilogram).
SI scale (mg / g / kg / t / Mt) plus the imperial pair (lb / oz) commonly used in commodity feeds.
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
LengthUnit ¶
Bases: Unit, Enum
Length unit (canonical = metre).
SI scale (mm / cm / m / km) plus the common imperial / aviation / maritime tokens (in / ft / yd / mi / nmi).
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
VolumeUnit ¶
Bases: Unit, Enum
Volume unit (canonical = cubic metre).
SI scale (mL / L / m³) plus the gallon variants and the oil barrel — commodity feeds split between US gallons, UK gallons, and the 42-gallon oil barrel, and a curated view that doesn't keep them straight ships wrong totals.
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
TemperatureUnit ¶
Bases: Unit, Enum
Temperature unit (canonical = kelvin).
The one affine family — Celsius and Fahrenheit have non-zero
offsets relative to kelvin, so the conversion is
canonical = value * factor + offset (not pure scaling).
Symbols use the standard degree glyphs (°C / °F) so the
curated tables render legibly in BI tools; aliases cover the
plain-ascii forms ("C" / "F" / "K").
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
PressureUnit ¶
Bases: Unit, Enum
Pressure unit (canonical = pascal).
SI scale (Pa / kPa / MPa / hPa) plus the bar family (bar / mbar), atmosphere, psi, and torr. Weather feeds publish in hPa or mbar, oil & gas in psi or bar, process control in kPa or MPa — one enum keeps them aligned.
from_
classmethod
¶
Coerce value to a member of this unit family.
Accepts: a member (passed through), a known symbol
("MWh" / "°C" / "psi"), the enum member name
("KWH"), or any alias the per-family _ALIASES dict
knows ("megawatthours" / "celsius" / "pound").
Lookup is case-insensitive.
Pass default to swallow unknown / unparseable input; without
it, unknown strings raise :class:ValueError with a hint
listing the valid symbols, and non-string non-member input
raises :class:TypeError.
convert
classmethod
¶
Convert value from source unit to target unit (scalar).
to_canonical ¶
Convert value in self's unit to the family's canonical unit.
from_canonical ¶
Convert value in the canonical unit to self's unit.
canonical
classmethod
¶
The family's canonical member (factor=1.0, offset=0.0).
factor_map
classmethod
¶
{symbol: factor} for every member + alias, for per-row maps.
offset_map
classmethod
¶
{symbol: offset} for every member + alias, for per-row maps.
JoinType ¶
Bases: IntEnum
Canonical join kind for tabular join surfaces.
Use :meth:from_ when accepting external input — it canonicalizes
aliases ("left", "LEFT JOIN", "anti", "outer",
integer codes) to a member and raises :class:ValueError for
unknown tokens.
Pass :attr:arrow / :attr:polars / :attr:sql to engine join
APIs — pyarrow's :meth:pa.Table.join and polars'
:meth:DataFrame.join accept different spellings of the same
concept, and storing the integer code keeps the enum a clean
discriminator without binding to either spelling.
polars
property
¶
polars' :meth:DataFrame.join how token.
Polars has no built-in right semi / right anti form —
those raise :class:NotImplementedError. Swap operands and use
the left-side equivalent.
from_
classmethod
¶
Coerce any Python value into a :class:JoinType.
Accepts:
- :class:
JoinType(returned as-is); - any string the alias table or canonical Arrow tokens know —
"inner","left","left outer","LEFT JOIN","anti","left_anti","outer","cross", …; mixed case, hyphens / underscores / spaces all normalize; - an integer code matching a member's value (round-trips with
int(JoinType.X)); None— returns default if supplied, else raises.
default swallows unknown / unparseable input. Without it,
unknown tokens raise :class:ValueError and unsupported types
raise :class:TypeError.
is_valid
classmethod
¶
Return True when :meth:from_ would succeed for value.
Scheme ¶
Bases: str, Enum
Canonical URL-scheme token for a Yggdrasil :class:URLBased
subclass.
Subclasses :class:str so a member is interchangeable with its
scheme token everywhere a string is expected (url.scheme ==
Scheme.DBFS works, f"{Scheme.S3}://bucket/key" reads
naturally). The lazy-import resolver lives on :meth:path_class.
from_
classmethod
¶
Coerce value to a :class:Scheme member.
Accepts:
- :class:
Scheme(returned as-is); - a scheme string — case-insensitive, trailing
://and whitespace tolerated, common aliases ("s3a"→ :attr:S3,"local"→ :attr:FILE,"memory"→ :attr:MEMORY); None— returns default if supplied, else raises.
default swallows unknown / unparseable input. Without it,
unknown tokens raise :class:ValueError and unsupported types
raise :class:TypeError.
path_class ¶
Return the concrete :class:URLBased subclass for this scheme.
Lazy: the backend module is imported on first use, which fires
the :meth:URLBased.__init_subclass__ side-effect that
registers the class. Subsequent calls hit the per-process cache.
Raises :class:ImportError when the backend's optional
dependencies aren't installed (databricks-sdk for the
Databricks schemes, boto3 for s3, …) — the message
names the missing extra so the caller can install it.
resolve
classmethod
¶
Shortcut: cls.from_(value).path_class().
Useful where the caller has a raw scheme string (or URL) and
just wants the concrete URLBased class. default is forwarded
to :meth:from_ for forgiving lookup; it does not catch
:class:ImportError from :meth:path_class.
NodeType ¶
Bases: str, Enum
Canonical Databricks node_type_id values.
Member values are the exact strings the Databricks SDK expects. Use
:attr:NodeType.DEFAULT for the codebase-wide default node type, the
semantic-size aliases (:attr:SMALL / :attr:MEDIUM / :attr:LARGE /
:attr:XLARGE) for intent-driven sizing, or the explicit cloud-specific
members when you need a precise SKU.
The enum is intentionally not exhaustive — Databricks publishes
hundreds of SKUs per cloud and most callers want a small, opinionated
set. Coerce caller-provided strings through :meth:to_id /
:meth:from_ so unknown SKUs round-trip as plain strings instead of
raising.
from_
classmethod
¶
Coerce value into a :class:NodeType member.
Accepts:
- :class:
NodeType(returned as-is); - any string in :data:
_NODETYPE_ALIASES, case-insensitive; - a bare member name (
"FLEET_XLARGE"); None— returns default if supplied, else :attr:NodeType.DEFAULT.
Raises :class:ValueError for unknown strings unless default is
supplied. To accept arbitrary SKU strings (without rejecting valid
cloud-specific identifiers), call :meth:to_id instead.
to_id
classmethod
¶
to_id(
value: Union[str, "NodeType", None],
*,
default: Optional[Union[str, "NodeType"]] = None
) -> str
Coerce value to a Databricks node_type_id string.
Unlike :meth:from_, this is forgiving: arbitrary SKU strings
round-trip unchanged so callers can pass cloud-specific identifiers
the enum does not enumerate ("r5d.metal", custom marketplace
images, …). Aliases are resolved when present.
Resolution order:
NodeTypemember → its string value.- Alias / member name / value match → the canonical string.
- Bare string → trimmed and returned as-is.
None→defaultif supplied, else :attr:NodeType.DEFAULT.
NodeSpec
dataclass
¶
NodeSpec(
cpu_cores: int,
ram_gib: float,
gpu_count: int = 0,
local_disk_gib: float = 0.0,
cloud: str = "",
)
Hardware specs for a :class:NodeType member.
The numbers reflect the vendor's published per-VM characteristics for
a single worker (or driver) — not the cluster-wide totals. ram_gib
is binary GiB (1024 ** 3 bytes), matching every cloud's own
documentation convention.
Attributes¶
cpu_cores
Virtual CPU count exposed to Spark.
ram_gib
Memory available to Spark, in IEC GiB.
gpu_count
Number of GPUs (0 for non-GPU SKUs).
local_disk_gib
Local SSD/NVMe storage attached to the instance. 0 when the
SKU has no local disk (relies on remote/elastic storage).
cloud
Cloud family identifier ("fleet" / "aws" / "azure" /
"gcp") used by :meth:NodeType.from_cpu_and_ram when
prefer is set.
State ¶
Bases: IntEnum
Unified execution state for async statement / job results.
Use :meth:from_ to normalize backend-specific tokens, and the
is_* predicates to derive done / failed / started
without re-implementing the membership sets per backend.
is_queued
property
¶
True for :attr:QUEUED — prepared, sitting in a submission queue.
is_accepted
property
¶
True for :attr:ACCEPTED — acknowledged but not yet running.
is_started
property
¶
True for anything from :attr:RUNNING onward.
Mirrors :attr:StatementResult.started: once the backend has
actually started executing, is_started flips and stays
True through every terminal state. :attr:ACCEPTED is
not started — the venue holds the order, no execution yet.
is_active
property
¶
True for non-terminal states with backend awareness.
Covers :attr:QUEUED through :attr:PARTIAL — anything the
caller can reasonably wait on. :attr:IDLE is excluded
(nothing has been submitted yet) and every terminal state is
excluded (no more transitions).
is_done
property
¶
True for terminal states (:attr:SUCCEEDED, :attr:REJECTED,
:attr:FAILED, :attr:CANCELED, :attr:EXPIRED) — no more
transitions expected.
is_failed
property
¶
True for :attr:REJECTED / :attr:FAILED / :attr:CANCELED /
:attr:EXPIRED.
Every non-success terminal state counts as failed because each
one means "the caller asked for a result and didn't get one":
cancellation, rejection, mid-run error, or TTL expiry all leave
the operation incomplete from the caller's view, and the
per-backend raise_for_status raises on each.
from_
classmethod
¶
Coerce any Python value into a :class:State.
See module docstring for the accepted shapes. default
swallows unknown / unparseable input; without it, unknown
tokens raise :class:ValueError and unsupported types raise
:class:TypeError.
is_valid
classmethod
¶
Return True when :meth:from_ would succeed for value.
IOKind ¶
Bases: IntEnum
What a backend reports a path/holder entry is.
EngineType ¶
Bases: IntEnum
Compute engine for a Databricks :class:~yggdrasil.databricks.table.table.Table
read / write.
- :attr:
YGGDRASIL— yggdrasil's native DeltaFolder (direct_delta_log - parquet over UC-vended credentials), no warehouse.
- :attr:
DATABRICKS_SQL_WAREHOUSE— the Databricks SQL warehouse. - :attr:
SPARK— a Spark session (Databricks Connect / cluster / notebook).
from_str
classmethod
¶
Coerce an alias string into an :class:EngineType.
Matches a forgiving alias table ("warehouse" / "sql" →
DATABRICKS_SQL_WAREHOUSE; "ygg" / "native" → YGGDRASIL;
"spark" / "connect" → SPARK). On an unknown string: raise
:class:ValueError when default is ..., else return default.
from_numeric
classmethod
¶
Coerce an integer code into an :class:EngineType.
On an out-of-range / non-integer code: raise when default is ...,
else return default. bool is rejected (it's an int subclass
but never a meaningful engine code).
from_
classmethod
¶
Coerce an :class:EngineType / alias string / int code / None.
None (or ...) is "unset" and returns None. A string routes
to :meth:from_str, a number to :meth:from_numeric; both honour
default (... → raise, else return the fallback).
AWSRegion ¶
Bases: str, Enum
partition
property
¶
The AWS partition this region belongs to (aws / aws-us-gov /
aws-cn).
from_
classmethod
¶
Coerce value (an :class:AWSRegion, a region string, or None)
to a member. Unknown / None returns default if given, else
raises.
from_text
classmethod
¶
Find an AWS region code embedded in text (e.g. a bucket name like
acme-dls3-eu-central-1-p), or default if none is present.
Matches a region only when it stands as a whole, delimiter-bounded
token (so eu-central-12 / ...-1foo don't false-positive), and
prefers the longest code at a position (ap-southeast-1 over
ap-south-1).
StructField
dataclass
¶
Bases: Field
A :class:Field whose dtype is a :class:StructType.
StructField([f1, f2, ...]) builds a struct field from its
children directly — sugar for the equivalent
Field(name=..., dtype=StructType(fields=(...,))) chain that
:class:Field's constructor accepts. Every schema-shaped method
(mapping surface, set operators, engine schema export, autotag,
struct-aware equals) is inherited from :class:Field.
StructField(fields, name=..., metadata=..., ...).
fields is the children list — :class:Field instances,
:class:pa.Field instances, or anything :meth:Field.from_any
accepts. dtype=<StructType> is accepted as an alternative
for callers (and the Field.__new__ redirect) that already
have a built struct dtype in hand; pass one or the other, not
both.
The wide *args / **kwargs shape is here because
Field.__new__ redirects struct-shaped Field(...) calls
(positional or keyword) to this class and Python then
re-enters __init__ on the already-stamped instance with
the original Field arguments. The re-init guard below
absorbs that pass before any signature parsing runs.
position
property
¶
Optional 0-based index this field claims in a parent schema.
Stored in :attr:metadata under :data:POSITION_KEY. Used
by :meth:select_in_field (and the engine-specific
select_in_* helpers) as the last-resort fallback when
:attr:name doesn't match a child name in the receiving
schema — the receiver's children[position] (or column
at position) is then resolved by name and used.
None (the default) leaves position-based lookup
disabled, matching the historical name-only resolver.
default_value
property
¶
Field's default Python value (or the dtype-level default).
Reads :data:DEFAULT_VALUE_KEY from :attr:metadata first;
falls back to self.dtype.default_pyobj when the metadata
slot is unset. Renamed from default so the constructor
classmethod :meth:Field.default can take that name —
field.default would otherwise shadow it via descriptor
lookup.
media_type
property
¶
:class:MediaType describing how this field's data is stored.
Decodes the b"media_type" metadata key — the mime-string
canonical form ("application/vnd.apache.arrow.file",
"application/vnd.apache.parquet", …) round-tripped through
:meth:MediaType.from_. None when no media-type hint has
been stamped.
Populated by :class:Folder._persist_schema so a schema
loaded from a folder's .ygg/schema.arrow sidecar tells the
reader which on-disk format the rows were last written in
(Arrow IPC, Parquet, …) without walking the part files.
Schema-level (top-level :class:StructField) is the canonical
slot, but the accessor lives on :class:Field so per-column
hints (e.g. the response-body field's HTTP Content-Type)
can use the same property.
inner_fields
property
¶
Compat view of the children as an ordered {name: field} map.
select_fields ¶
select_fields(
identifiers: "SelectType | Iterable[SelectType]" = (),
*others: SelectType,
raise_error: bool = True
) -> list["Field"]
Resolve one or more identifiers into the matching :class:Field objects.
Accepts strings (resolved by name), ints (resolved by index),
and existing :class:Field instances (resolved by .name
against this container — so callers can copy a field set
between sibling schemas without first stringifying everything).
Calling shapes that all work the same way:
schema.select_fields("price")— single identifier.schema.select_fields("price", "qty", 0)— multiple positionals.schema.select_fields(["price", "qty"])— single iterable.schema.select_fields(other_schema.children)— copy a sibling's fields by name into this schema.schema.select_fields("price", ["qty", "ts"], 0)— mixed; each positional is itself flattened so iterables and scalars can be interleaved.
:param identifiers:
First identifier or iterable of identifiers.
:param others:
Additional identifiers. Each is flattened the same way
as the first.
:param raise_error:
True (default) — missing identifiers raise via
:meth:field_by with the same suggestion-rich error
message used elsewhere. False — missing identifiers
yield None in the returned list, preserving caller
order.
:returns:
A list of :class:Field (or Field | None when
raise_error=False), one entry per resolved identifier
in caller order. Duplicates in the input produce
duplicates in the output — this is intentional, since
select is the natural place to express a projection
and projections sometimes repeat columns.
:raises KeyError:
With suggestions, when raise_error is True and an
identifier doesn't resolve.
:raises TypeError:
When an identifier is not a str / int / Field.
short ¶
A compact name:dtype header tag — the dtype via
:meth:~yggdrasil.data.types.base.DataType.short (recursive for nested
types). Used for the column headers in
:meth:yggdrasil.io.tabular.Tabular.display.
markers ¶
The main schema markers for a preview header, space-joined (""
when none): the key / layout flags (PK / FK / CK /
partition / cluster / sorted / IK) and a * for a
non-nullable (required) column. The compact cousin of
:meth:_pretty_markers.
default
classmethod
¶
default(
name: str = "",
dtype: DataType = ObjectType(),
nullable: bool = True,
metadata: dict[bytes, bytes] | None = None,
tags: dict[bytes, bytes] | None = None,
default: Any = None,
)
Build a default-typed Field (ObjectType() unless overridden).
Convenience constructor for the "I just have a name" path —
callers passing a plain string into APIs that expect a
:class:Field (e.g. CastOptions(match_by=["id"])) land
here. The instance-side default accessor was renamed to
:attr:default_value so this name was free for the
constructor.
pretty_format ¶
Pretty-print this field with the header on one line and the dtype below.
Layout is uniform across flat and nested dtypes — every field
renders as a single field: 'name' <dtype>{markers} header
line, with nested dtypes walking their inner fields inline at
level + 1 so the tree reads as a flat list of consistent
rows::
field: 'row' struct
field: 'id' int64 not null [PK]
field: 'name' string
field: 'inner' struct
field: 'age' int64
field: 'email' string
indent is the per-level step in spaces; level is the
current depth. The header carries the dtype kind (struct /
list / map for nested, the primitive pretty-format for
flat), the not null marker, the bracketed marker group
(primary / foreign / constraint key, partition / cluster /
sorted, any caller-defined tags, default value), and the
comment.
Map dtypes flatten the synthetic entry struct into
field: 'key' … / field: 'value' … lines so the
key / value framing reads at the same level as a struct's
own children rather than under an artificial wrapper.
Examples::
>>> print(field("id", "int64", nullable=False,
... tags={"primary_key": True}).pretty_format())
field: 'id' int64 not null [PK]
>>> print(field("date", "date32",
... tags={"partition_by": True}).pretty_format())
field: 'date' date32 [partition]
>>> print(field("user", StructType.from_fields([
... field("id", "int64"),
... field("email", "string"),
... ])).pretty_format())
field: 'user' struct
field: 'id' int64
field: 'email' string
invalidate_cache ¶
Drop cached engine projections, cascading to ancestors by default.
Public surface over :meth:_invalidate_cache. Callers that
mutate the underlying state outside of the with_* mutators
(custom DataType subclass that swaps children in place,
external code that pokes dtype.fields directly) should call
this once to make sure the next to_arrow_field /
to_polars_field / to_pyspark_field / *_schema
request rebuilds with the new state. With cascade=True
(the default) every ancestor reachable via :attr:parent
also drops its cache, so a struct's cached arrow schema gets
rebuilt after one of its children mutates.
equals ¶
equals(
other: Any,
check_names: bool = True,
check_dtypes: bool = True,
check_nullable: bool = True,
check_metadata: bool = True,
) -> bool
Structural equality check with configurable scope.
Mirrors :meth:DataType.equals. Coerces other to a Field so
that callers can pass a pa.Field / dict / etc. without manual
conversion. Returns False on coercion failure instead of raising.
check_names: compare this field's name and recurse into child field names for nested types. For struct-shaped fields the comparison is order-independent (children matched by name) whencheck_namesis True, mirroring how Arrow schemas are name-keyed.check_dtypes: recurse into the dtype and comparenullable(both are structural, schema-defining attributes).check_metadata: compare this field's metadata and recurse.
set_position ¶
Set / clear :attr:position on self in place.
Negative values are rejected — positions are forward indices into the parent schema; if you need a last-element fallback, resolve it before calling.
with_position ¶
Return a copy of this field with :attr:position set / cleared.
check_pandas_metadata ¶
Stamp pandas index tags onto child fields from a b"pandas" blob.
pandas carries its DataFrame index layout in the pyarrow
b"pandas" schema metadata (index_columns). This reads
that blob and marks each matching child as an index level via
:meth:with_index_key, so a struct-shaped Field round-trips the
index when it later rebuilds a DataFrame.
source is whatever carries the blob — a pa.Schema, a
pa.Table, raw bytes / str JSON, or an already-parsed
dict. When omitted, falls back to self.metadata[b"pandas"]
(which :meth:from_arrow_schema preserves). Mutates and returns
self for chaining; a no-op when there's no blob or no string
index columns.
PARITY: Python/pandas-only. The TS port has no pandas¶
counterpart, so there is no mirror for this method.¶
with_field ¶
with_field(
field: "Field | pa.Field | str",
*,
mode: "Mode | str | None" = None,
inplace: bool = True,
**kwargs: Any
) -> "Field"
Return self with field appended or merged in.
mode controls collision behavior when a child with the
same name already exists. Accepts a :class:Mode member or
any alias :meth:Mode.from_ understands.
- :data:
Mode.AUTO/ :data:Mode.OVERWRITE— replace the existing child verbatim with field. - :data:
Mode.APPEND— append a fresh child even if the name collides (struct semantics: last-write-wins for duplicate names; both entries survive in the children tuple). - :data:
Mode.IGNORE— keep the existing child; drop the incoming. - :data:
Mode.ERROR_IF_EXISTS— raise :class:ValueErroron collision. - :data:
Mode.UPSERT/ :data:Mode.MERGE— :meth:merge_withthe existing child against the incoming one (dtype, nullability, metadata), keeping the existing child's identity.
Auto-promotion to struct: when self isn't a struct
(a primitive Field, a list/map, …) the call returns a fresh
struct Field whose first child is the previous self
(renamed to its current name so it's addressable) and
whose second child is field. The promoted struct keeps
self's name, nullability, and metadata — only the dtype
changes.
Bare-string shorthand: self.with_field("price") reads as
"make sure a child named 'price' exists." That call goes
through :meth:Field.from_any which infers a sensible
default dtype.
inplace=True (the default) mutates self and returns
it. inplace=False returns a fresh copy.
with_fields ¶
with_fields(
fields: "Iterable[Field | pa.Field | str]",
*,
mode: "Mode | str | None" = None,
inplace: bool = True
) -> "Field"
Apply :meth:with_field for every entry in fields.
Same mode semantics as :meth:with_field; the loop short-
circuits :data:Mode.IGNORE once any one collision keeps the
existing child (no global "first one wins, drop the rest"
gymnastics — collisions are evaluated per name).
Auto-promotes self to a struct on the first call when
needed; subsequent fields land on that struct.
autotag ¶
Stamp this field with tags derived from its dtype and name.
Writes Databricks-friendly auto-tags in place:
- Everything from :meth:
DataType.autotag(kindplus dtype detail likeunit/tz/precision/scale/signed/iso/srid). nullablefor data-quality policies.- Name-based heuristics for governance:
role=identifierfor*_id/*_uuid,role=audit_timestampforcreated_atpatterns, pluspii/sensitivestamps for columns that obviously carry personal or credential data.
For struct-shaped fields (schemas) primary_key /
partition_by / cluster_by entries on this field's
metadata get consumed into per-child tags, and each child is
autotagged in turn — so schema.autotag() propagates without
the caller having to walk children manually.
Returns a new struct-shaped Field for schema-style autotagging,
or self for primitive autotagging — both modes also stamp
in place so existing f.autotag() chains keep working.
from_field
classmethod
¶
Lift a :class:Field to cls.
For cls is Field this is identity. For subclasses (e.g.
:class:Schema) it normalises the input to the subclass shape
— for struct dtypes we keep the children, for non-struct we
wrap the field as a single-child struct so the schema-shape
contract holds.
from_fields
classmethod
¶
from_fields(
fields: Iterable["Field | Any"],
*,
name: str = DEFAULT_FIELD_NAME,
nullable: bool = False,
metadata: dict[bytes | str, bytes | str | object] | None = None,
tags: dict[bytes | str, bytes | str | object] | None = None
) -> "Field"
Build a struct-shaped instance from a list of fields.
from_spark_column
classmethod
¶
Build a :class:Field from a pyspark.sql.Column.
Column objects don't expose a typed dtype on the public
Python surface — we read the SQL-rendered expression instead
and parse that:
id— bare reference. Name isid, dtype defers to the fallback (ObjectType) since neither the JVM nor the Spark Connect proxy exposes the underlying schema on a free-standing column.CAST(<expr> AS <dtype>)/CAST(<expr> AS <dtype>)— name follows the inner<expr>'s leaf, dtype reads straight off<dtype>through :meth:DataType.from_str. Coversdf["x"].cast("string"),df["x"].astype("decimal(10,2)"),F.col("x").cast(StringType()).<expr> AS <alias>— name follows<alias>, dtype comes from the inner<expr>(recurses, so a cast inside an alias keeps its dtype).- Anything else falls back to the full SQL string as the
name with :class:
ObjectTypeas the dtype, since we can't infer the dtype of an arbitrary Catalyst expression without binding it through :meth:SparkSession.createDataFrame(which would be a live JVM round trip the caller didn't ask for).
Source of the SQL string, in order:
- Classic Spark:
column._jc.toString()— the JVM Column. - Spark Connect:
column._expr.__repr__()— the proxy doesn't have_jc(accessing it raisesPySparkAttributeError(JVM_ATTRIBUTE_NOT_SUPPORTED)) but_expr.__repr__is exactly whatColumn.__repr__wraps as"Column<'<sql>'>". repr(column)stripped of theColumn<'…'>wrapper — last-resort for any future PySpark whose internal slots renamed.
Use :meth:Field.from_spark_field instead when the caller
already has the resolved StructField (e.g. from
df.schema.fields[i]) — that path keeps the precise dtype
without going through the SQL string.
to_dict ¶
Serialize this field to a JSON-friendly dict.
dump_parent (default False) controls whether
:attr:parent — the structural back-pointer to the field
this one is nested under — is included. Children are still
emitted via the dtype's to_dict (a struct field's
dtype carries its members), so dropping parent
prevents the recursion that would otherwise echo the whole
ancestor chain into every nested field's payload.
to_arrow_field ¶
Project to a :class:pa.Field.
Arrow preserves nested-type structure (struct, list, map)
with per-field metadata recursively, so the dtype intent
round-trips natively without us stuffing a type_json blob
into the metadata. Only callers that need the exact
:class:DataType subclass back (e.g. Decimal precision /
Timestamp tz / extension types) should pass
dump_json=True.
dump_json defaults to False; the cached path is the
canonical (no-blob) shape, which is what every internal caller
wants now that :meth:from_arrow_field falls back through
:meth:DataType.from_arrow_type when the blob is missing.
to_arrow_schema ¶
Project this field as a top-level :class:pa.Schema.
Struct-shaped fields (including :class:~yggdrasil.data.Schema)
unfold their children into the schema's columns; non-struct
fields produce a single-column schema with self as that
column. The schema-level metadata mirrors self.metadata,
plus the field's name / nullable flag re-embedded as
b"name" / b"nullable" so :meth:Field.from_arrow_schema
can recover them (pa.Schema has no native slot for either).
to_polars_schema ¶
Project this field as a :class:polars.Schema.
Struct-shaped fields unfold into the schema's columns; non-struct fields produce a single-column schema.
to_pyspark_field ¶
Project to a Spark :class:StructField.
Spark's :class:StructType preserves struct children with
their own metadata, so primitive and struct dtypes don't
need a type_json round-trip blob. Spark's :class:MapType
/ :class:ArrayType only carry the element / key+value Spark
types and lose any field-level metadata on the way through, so
we dump the dtype JSON for those (and only those) to recover
the original yggdrasil dtype on read.
to_spark_schema ¶
Project this field as a top-level Spark :class:StructType.
Struct-shaped fields unfold their children into the StructType's fields; non-struct fields produce a single-field StructType.
cast ¶
Cast obj to this field using its native engine.
Routing is by module prefix via :meth:ObjectSerde.module_and_name:
pyarrow.*→ :meth:cast_arrowpolars.*→ :meth:cast_polarspandas.*→ :meth:cast_pandaspyspark.*→ :meth:cast_spark- iterator / iterable → recurse per element (lazy generator)
- everything else → :class:
TypeError
self.dtype.type_id == OBJECT is handled by the narrow
methods — they pass obj through unchanged because a variant
column must never be cast. No redundant guard here.
cast_arrow ¶
Cast any pyarrow object — dispatch by shape.
Table/RecordBatch → :meth:cast_arrow_tabular,
Array/ChunkedArray → :meth:cast_arrow_array.
cast_polars ¶
Cast any polars object — dispatch by shape.
DataFrame/LazyFrame → :meth:cast_polars_tabular,
Series → :meth:cast_polars_series,
Expr → :meth:cast_polars_expr.
cast_pandas ¶
Cast any pandas object — dispatch by shape.
DataFrame → :meth:cast_pandas_tabular + index check,
Series → :meth:cast_pandas_series.
cast_spark ¶
Cast any spark object — dispatch by shape.
DataFrame → :meth:cast_spark_tabular,
Column → :meth:cast_spark_column.
cast_arrow_batch_iterator ¶
cast_arrow_batch_iterator(
batches: "Iterable[pa.RecordBatch]",
options: "CastOptions | None" = None,
**more
) -> "Iterator[pa.RecordBatch]"
Cast a stream of :class:pa.RecordBatch against this field.
Object targets passthrough (variant). Otherwise the dtype's
struct view owns the per-batch tabular cast and byte_size
rechunk — same shape contract as :meth:cast_arrow_tabular,
just lazy.
fill_nulls ¶
Fill nulls in obj using the native engine — engine + shape detection.
Routes the same way :meth:cast does. See
:meth:fill_arrow / :meth:fill_polars / :meth:fill_pandas
/ :meth:fill_spark for the per-engine behaviour.
fill_arrow ¶
Fill nulls in any pyarrow object.
Arrays go through :meth:fill_arrow_array_nulls directly.
Tables / RecordBatches re-use the tabular cast path with
self as the target — a no-op cast that still runs the
per-column null-fill via the struct walk.
fill_polars ¶
Fill nulls in any polars object.
Series / Expr go through :meth:fill_polars_array_nulls —
which handles both shapes uniformly (Expr is the lazy
counterpart of Series; the fill operator grafts onto each
identically). DataFrame / LazyFrame route through
:meth:cast_polars_tabular as a self-targeted cast.
fill_pandas ¶
Fill nulls in any pandas object.
fill_spark ¶
Fill nulls in any spark object.
polars_alias ¶
Rename a polars Series / Expr to match this field's name.
No-op when the target name matches the current name, or when this field only has the sentinel name. Calling defensively is free — zero-cost on the no-rename path.
spark_alias ¶
Rename a Spark Column to match this field's name.
Spark DataFrames aren't handled — renaming a DataFrame requires a projection with named columns, which isn't a single-method operation. Column is the rename target here.
pandas_alias ¶
Rename a pandas Series to match this field's name.
Pandas has no .alias() — rename is series.name = ...,
which mutates. This helper returns the series so it chains
like :meth:polars_alias / :meth:spark_alias. DataFrames
aren't handled (column rename is a projection, not a
single-method op).
finalize_arrow_array ¶
Fill nulls on a pyarrow Array / ChunkedArray.
No alias step: pa.Array / ChunkedArray don't carry a name.
Tabular naming lives in the pa.Field that wraps the array in
a Table/RecordBatch, which :meth:cast_arrow_tabular handles
through the struct walk.
finalize_arrow ¶
Finalize any pyarrow object — dispatch by shape.
Array/ChunkedArray → fill. Table/RecordBatch → identity.
finalize_polars_series ¶
Fill nulls, alias a polars Series to the target name.
finalize_polars_expr ¶
Fill nulls, alias a polars Expr to the target name.
Same as :meth:finalize_polars_series — polars Series and Expr
share the fill + alias primitives, so the finalize shape is
identical. Separate method for call-site clarity.
finalize_polars ¶
finalize_polars(
obj: "polars.Series | polars.Expr | polars.DataFrame | polars.LazyFrame",
*,
default_scalar: Any = None
)
Finalize any polars object — dispatch by shape.
Series/Expr → fill + alias. DataFrame/LazyFrame → identity (tabular cast already finalized per-column via the struct walk).
finalize_pandas_series ¶
Fill nulls, rename a pandas Series to the target name.
finalize_pandas ¶
Finalize any pandas object — dispatch by shape.
Series → fill + rename. DataFrame → identity.
check_pandas_indexes ¶
Promote columns tagged index_key to the DataFrame index.
Collects children with :attr:index_key set, sorted by
:attr:index_key_level, and calls set_index on the
DataFrame. __index_level_N__ placeholder names are mapped
back to None so the round-trip matches the source.
For a Series whose field is itself tagged index_key, the
Series is returned as-is — the caller decides how to attach it
as an index.
Passthrough when no children carry the tag or when the object is not a DataFrame.
finalize_spark_column ¶
Fill nulls, alias a Spark Column to the target name.
finalize_spark ¶
Finalize any spark object — dispatch by shape.
Column → fill + alias. DataFrame → identity (tabular cast already finalized).
finalize ¶
Finalize obj using its native engine — module-prefix dispatch.
Mirrors :meth:cast / :meth:fill_nulls routing.
select ¶
Return a new struct-shaped Field with only the selected children.
Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).
drop ¶
Return a new struct-shaped Field without the specified children.
Accepts strings (by name), ints (by index), Field instances (by name), iterables thereof, or None (skipped).
as_polars ¶
Return a :class:StructField whose dtype is Polars-compatible.
unit_family_for ¶
Find the unit family that recognises value (member or symbol).
Useful when a curated row carries a free-form unit token and the
caller needs to dispatch to the right family without hard-coding
a per-source switch. Returns the first family for which
:meth:Unit.is_valid returns True; raises :class:ValueError
when no family claims the token (the message lists every family
that was consulted).