yggdrasil.plan.execution_plan¶
execution_plan ¶
Autonomous, mutable execution plans for lazy Tabular transformations.
The plan tree is two-tier:
- :class:
ExecutionPlan— abstract base, also a :class:Tabular. Subclasses carry every object they need to run standalone (target / source Tabulars, predicates, transformations) so :meth:executetakes onlywait/raise_error. - :class:
SelectPlan— builder for SELECT-like reads. Returns the read result as a :class:Tabular. - :class:
InsertPlan— builder forINSERT INTO target (...) .... Returns an :class:OperationResultwith row counts + post-write target. - :class:
MergePlan— builder forMERGE INTO target USING source ON ... WHEN MATCHED ... WHEN NOT MATCHED .... Returns :class:OperationResult.
Because each plan IS a :class:Tabular, callers can pass a plan
anywhere a Tabular is expected; reading the plan triggers
:meth:execute and exposes the result (for SELECT) or the operation
metadata as a one-row table (for INSERT / MERGE).
ExecutionPlan ¶
Bases: Tabular[O], Generic[O]
Abstract base — a plan IS a Tabular.
Subclasses carry every object they need to execute standalone
(source/target Tabulars, predicates, transformations). The
:meth:execute entry point takes only wait and
raise_error — same calling convention every async-aware
backend in yggdrasil uses.
Reading the plan (plan.read_arrow_table() /
plan.read_arrow_batches() / …) executes the plan once and
streams the result. Writing into the plan is a type error —
plans are read-only as Tabulars.
Initialize the tree-parent slot.
Subclasses chain super().__init__(**kwargs) and don't
otherwise touch :attr:tabular_parent — aggregators pass
tabular_parent=self at child construction.
The slot is named tabular_parent (not parent) so it
doesn't collide with :attr:yggdrasil.io.path.path.Path.parent,
which is the path-shaped parent directory and therefore
property-backed and read-only.
from_sql
classmethod
¶
from_sql(
sql: str, dialect: "Dialect | str | None" = None, default: Any = ...
) -> "ExecutionPlan | Any"
Parse sql into a concrete plan (SelectPlan / InsertPlan / MergePlan).
Returns default on parse failure when default is not
...; raises otherwise. The returned plan is not yet
bound to concrete Tabulars — bind via the constructor
keywords (source=, target=) before calling
:meth:execute.
execute
abstractmethod
¶
Run the plan; returns a :class:Tabular (SELECT) or
:class:OperationResult (INSERT / MERGE / …).
open ¶
Acquire the resource and cascade into owned children.
Order:
- Run our own :meth:
_acquire(subclass body). - Flip :attr:
openedto True and mark_self_opened. -
For each owned child, in registration order:
-
If the child is already opened, just :meth:
_claimit. It stays self-opened — the existing self-open is what keeps it alive after we let go. - Otherwise, call :meth:
openon the child (which recursively cascades into ITS owned children), then clear the child's_self_openedflag so the child knows its open is parent-driven, then :meth:_claimit. Without that flag clear, the eventual :meth:_unclaimwould refuse to close — it would see "I'm self-opened, someone explicitly opened me, leave me alone."
Both branches record the child in our per-frame scratch
list so :meth:_release knows what to unclaim.
Transactional rollback: if any child's open or claim raises,
we walk back through the children we already touched (in
reverse), unclaim each, then call our own :meth:_release
with committed=False and re-raise the original exception.
From the caller's view, the open atomically either succeeded
with the whole graph live, or failed with nothing changed.
Not reentrant: raises :class:RuntimeError if already opened.
Nesting is expressed via with self: blocks, not via paired
:meth:open calls.
close ¶
Drop the schema cache and forward to any cooperative close.
Tabular itself has no resources to release — the schema cache
is the only state it owns. Subclasses that mix Tabular with a
lifecycle (Disposable-derived IO, holders, …) inherit
this hook through cooperative super().close(); pure
Tabular subclasses without a lifecycle peer get a harmless
no-op forward.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
from_(
obj: Any,
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
as_folder: bool = False,
**kwargs: Any
) -> "Tabular | None"
Coerce obj into a :class:Tabular.
Routes:
None— returns default (Nonewhendefault=None).- :class:
Tabular— returned as-is. When as_folder isTrueand obj is a local :class:Path, wraps it in a :class:Folder. str/ :class:os.PathLike— coerced via :class:Path.from_. When as_folder isTrue, wraps in :class:Folder.- File-like objects — drained into :class:
Memory; media_type required.
Falls back to default on unrecognised shapes when supplied;
otherwise raises :class:TypeError.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
SelectPlan ¶
Bases: ExecutionPlan[O]
Mutable SELECT-like plan: projection, filter, join, union, dedup, resample, cast, limit.
Bind a source at construction (SelectPlan(source=tabular)) or
later via :meth:bind to make the plan autonomous —
:meth:execute then runs without any extra arguments.
is_schema_preserving
property
¶
True iff the plan can't reshape the source's column set/types.
Filter, limit, offset and order-by only drop or reorder rows; the
output schema is identical to the source. Anything that projects,
aggregates, joins, unions, resamples or casts can change it. Callers
(e.g. :meth:LazyTabular._collect_schema) use this to skip a full
plan execution when they only need column names/types.
bind ¶
Attach source to this plan so :meth:execute runs standalone.
execute ¶
execute(
source: "Tabular | WaitingConfigArg | None" = None,
*,
wait: WaitingConfigArg = True,
raise_error: bool = True
) -> "Tabular"
Run the SELECT plan against the bound or supplied source.
Accepts a positional :class:Tabular as the source for
ergonomics (and backward compat with the previous
execute(source) shape). Otherwise uses the source bound
at construction / via :meth:bind.
wait and raise_error are routed through to async
sources (warehouse statements, remote runs); fully-local
Arrow sources ignore them — there's no work to wait for.
open ¶
Acquire the resource and cascade into owned children.
Order:
- Run our own :meth:
_acquire(subclass body). - Flip :attr:
openedto True and mark_self_opened. -
For each owned child, in registration order:
-
If the child is already opened, just :meth:
_claimit. It stays self-opened — the existing self-open is what keeps it alive after we let go. - Otherwise, call :meth:
openon the child (which recursively cascades into ITS owned children), then clear the child's_self_openedflag so the child knows its open is parent-driven, then :meth:_claimit. Without that flag clear, the eventual :meth:_unclaimwould refuse to close — it would see "I'm self-opened, someone explicitly opened me, leave me alone."
Both branches record the child in our per-frame scratch
list so :meth:_release knows what to unclaim.
Transactional rollback: if any child's open or claim raises,
we walk back through the children we already touched (in
reverse), unclaim each, then call our own :meth:_release
with committed=False and re-raise the original exception.
From the caller's view, the open atomically either succeeded
with the whole graph live, or failed with nothing changed.
Not reentrant: raises :class:RuntimeError if already opened.
Nesting is expressed via with self: blocks, not via paired
:meth:open calls.
close ¶
Drop the schema cache and forward to any cooperative close.
Tabular itself has no resources to release — the schema cache
is the only state it owns. Subclasses that mix Tabular with a
lifecycle (Disposable-derived IO, holders, …) inherit
this hook through cooperative super().close(); pure
Tabular subclasses without a lifecycle peer get a harmless
no-op forward.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
from_(
obj: Any,
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
as_folder: bool = False,
**kwargs: Any
) -> "Tabular | None"
Coerce obj into a :class:Tabular.
Routes:
None— returns default (Nonewhendefault=None).- :class:
Tabular— returned as-is. When as_folder isTrueand obj is a local :class:Path, wraps it in a :class:Folder. str/ :class:os.PathLike— coerced via :class:Path.from_. When as_folder isTrue, wraps in :class:Folder.- File-like objects — drained into :class:
Memory; media_type required.
Falls back to default on unrecognised shapes when supplied;
otherwise raises :class:TypeError.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
from_sql
classmethod
¶
from_sql(
sql: str, dialect: "Dialect | str | None" = None, default: Any = ...
) -> "ExecutionPlan | Any"
Parse sql into a concrete plan (SelectPlan / InsertPlan / MergePlan).
Returns default on parse failure when default is not
...; raises otherwise. The returned plan is not yet
bound to concrete Tabulars — bind via the constructor
keywords (source=, target=) before calling
:meth:execute.
InsertPlan ¶
InsertPlan(
*,
target: "Tabular | None" = None,
source: "Tabular | ExecutionPlan | None" = None,
columns: "list[str] | None" = None,
values: "list[list[Any]] | None" = None,
mode: "Mode | str | None" = None,
**kwargs: Any
)
Bases: ExecutionPlan[O]
Autonomous INSERT INTO target ... plan.
Carries the concrete target Tabular plus the source — either
another :class:Tabular, an :class:ExecutionPlan (a SELECT plan
feeding the insert), or an inline VALUES list.
open ¶
Acquire the resource and cascade into owned children.
Order:
- Run our own :meth:
_acquire(subclass body). - Flip :attr:
openedto True and mark_self_opened. -
For each owned child, in registration order:
-
If the child is already opened, just :meth:
_claimit. It stays self-opened — the existing self-open is what keeps it alive after we let go. - Otherwise, call :meth:
openon the child (which recursively cascades into ITS owned children), then clear the child's_self_openedflag so the child knows its open is parent-driven, then :meth:_claimit. Without that flag clear, the eventual :meth:_unclaimwould refuse to close — it would see "I'm self-opened, someone explicitly opened me, leave me alone."
Both branches record the child in our per-frame scratch
list so :meth:_release knows what to unclaim.
Transactional rollback: if any child's open or claim raises,
we walk back through the children we already touched (in
reverse), unclaim each, then call our own :meth:_release
with committed=False and re-raise the original exception.
From the caller's view, the open atomically either succeeded
with the whole graph live, or failed with nothing changed.
Not reentrant: raises :class:RuntimeError if already opened.
Nesting is expressed via with self: blocks, not via paired
:meth:open calls.
close ¶
Drop the schema cache and forward to any cooperative close.
Tabular itself has no resources to release — the schema cache
is the only state it owns. Subclasses that mix Tabular with a
lifecycle (Disposable-derived IO, holders, …) inherit
this hook through cooperative super().close(); pure
Tabular subclasses without a lifecycle peer get a harmless
no-op forward.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
from_(
obj: Any,
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
as_folder: bool = False,
**kwargs: Any
) -> "Tabular | None"
Coerce obj into a :class:Tabular.
Routes:
None— returns default (Nonewhendefault=None).- :class:
Tabular— returned as-is. When as_folder isTrueand obj is a local :class:Path, wraps it in a :class:Folder. str/ :class:os.PathLike— coerced via :class:Path.from_. When as_folder isTrue, wraps in :class:Folder.- File-like objects — drained into :class:
Memory; media_type required.
Falls back to default on unrecognised shapes when supplied;
otherwise raises :class:TypeError.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
from_sql
classmethod
¶
from_sql(
sql: str, dialect: "Dialect | str | None" = None, default: Any = ...
) -> "ExecutionPlan | Any"
Parse sql into a concrete plan (SelectPlan / InsertPlan / MergePlan).
Returns default on parse failure when default is not
...; raises otherwise. The returned plan is not yet
bound to concrete Tabulars — bind via the constructor
keywords (source=, target=) before calling
:meth:execute.
MergePlan ¶
MergePlan(
*,
target: "Tabular | None" = None,
source: "Tabular | ExecutionPlan | None" = None,
on: "Predicate | None" = None,
on_keys: "list[str] | None" = None,
when_matched: "list[dict] | None" = None,
when_not_matched: "list[dict] | None" = None,
**kwargs: Any
)
Bases: ExecutionPlan[O]
Autonomous MERGE INTO target USING source ON ... plan.
Carries the target Tabular, the source (Tabular or
:class:ExecutionPlan), the ON predicate (or simple key list),
and the WHEN MATCHED / WHEN NOT MATCHED action dicts.
The Arrow execution path handles the common in-memory patterns:
ON t.<key> = s.<key>(equality on one or more columns).WHEN MATCHED THEN UPDATE SET <col> = <s.col | literal>WHEN MATCHED THEN DELETEWHEN NOT MATCHED THEN INSERT (cols) VALUES (s.cols | literals)
open ¶
Acquire the resource and cascade into owned children.
Order:
- Run our own :meth:
_acquire(subclass body). - Flip :attr:
openedto True and mark_self_opened. -
For each owned child, in registration order:
-
If the child is already opened, just :meth:
_claimit. It stays self-opened — the existing self-open is what keeps it alive after we let go. - Otherwise, call :meth:
openon the child (which recursively cascades into ITS owned children), then clear the child's_self_openedflag so the child knows its open is parent-driven, then :meth:_claimit. Without that flag clear, the eventual :meth:_unclaimwould refuse to close — it would see "I'm self-opened, someone explicitly opened me, leave me alone."
Both branches record the child in our per-frame scratch
list so :meth:_release knows what to unclaim.
Transactional rollback: if any child's open or claim raises,
we walk back through the children we already touched (in
reverse), unclaim each, then call our own :meth:_release
with committed=False and re-raise the original exception.
From the caller's view, the open atomically either succeeded
with the whole graph live, or failed with nothing changed.
Not reentrant: raises :class:RuntimeError if already opened.
Nesting is expressed via with self: blocks, not via paired
:meth:open calls.
close ¶
Drop the schema cache and forward to any cooperative close.
Tabular itself has no resources to release — the schema cache
is the only state it owns. Subclasses that mix Tabular with a
lifecycle (Disposable-derived IO, holders, …) inherit
this hook through cooperative super().close(); pure
Tabular subclasses without a lifecycle peer get a harmless
no-op forward.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
from_(
obj: Any,
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
as_folder: bool = False,
**kwargs: Any
) -> "Tabular | None"
Coerce obj into a :class:Tabular.
Routes:
None— returns default (Nonewhendefault=None).- :class:
Tabular— returned as-is. When as_folder isTrueand obj is a local :class:Path, wraps it in a :class:Folder. str/ :class:os.PathLike— coerced via :class:Path.from_. When as_folder isTrue, wraps in :class:Folder.- File-like objects — drained into :class:
Memory; media_type required.
Falls back to default on unrecognised shapes when supplied;
otherwise raises :class:TypeError.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
from_sql
classmethod
¶
from_sql(
sql: str, dialect: "Dialect | str | None" = None, default: Any = ...
) -> "ExecutionPlan | Any"
Parse sql into a concrete plan (SelectPlan / InsertPlan / MergePlan).
Returns default on parse failure when default is not
...; raises otherwise. The returned plan is not yet
bound to concrete Tabulars — bind via the constructor
keywords (source=, target=) before calling
:meth:execute.