yggdrasil.plan.lazy¶
lazy ¶
:class:LazyTabular — defers execution until a read_* call.
Wraps a concrete source :class:Tabular and a :class:SelectPlan.
Transformation methods mutate the plan; every read_* materialises.
LazyTabular ¶
Bases: Tabular[O], Generic[O]
Lazy wrapper: accumulates a plan, executes on every read.
into ¶
Build an :class:InsertPlan that feeds this lazy SELECT into target.
Returns the unexecuted :class:InsertPlan so callers can
chain .execute() or hand the plan around as a Tabular.
merge_into ¶
Open a :class:MergePlan with this lazy SELECT as the source.
open ¶
Acquire the resource and cascade into owned children.
Order:
- Run our own :meth:
_acquire(subclass body). - Flip :attr:
openedto True and mark_self_opened. -
For each owned child, in registration order:
-
If the child is already opened, just :meth:
_claimit. It stays self-opened — the existing self-open is what keeps it alive after we let go. - Otherwise, call :meth:
openon the child (which recursively cascades into ITS owned children), then clear the child's_self_openedflag so the child knows its open is parent-driven, then :meth:_claimit. Without that flag clear, the eventual :meth:_unclaimwould refuse to close — it would see "I'm self-opened, someone explicitly opened me, leave me alone."
Both branches record the child in our per-frame scratch
list so :meth:_release knows what to unclaim.
Transactional rollback: if any child's open or claim raises,
we walk back through the children we already touched (in
reverse), unclaim each, then call our own :meth:_release
with committed=False and re-raise the original exception.
From the caller's view, the open atomically either succeeded
with the whole graph live, or failed with nothing changed.
Not reentrant: raises :class:RuntimeError if already opened.
Nesting is expressed via with self: blocks, not via paired
:meth:open calls.
close ¶
Drop the schema cache and forward to any cooperative close.
Tabular itself has no resources to release — the schema cache
is the only state it owns. Subclasses that mix Tabular with a
lifecycle (Disposable-derived IO, holders, …) inherit
this hook through cooperative super().close(); pure
Tabular subclasses without a lifecycle peer get a harmless
no-op forward.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
from_(
obj: Any,
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
as_folder: bool = False,
**kwargs: Any
) -> "Tabular | None"
Coerce obj into a :class:Tabular.
Routes:
None— returns default (Nonewhendefault=None).- :class:
Tabular— returned as-is. When as_folder isTrueand obj is a local :class:Path, wraps it in a :class:Folder. str/ :class:os.PathLike— coerced via :class:Path.from_. When as_folder isTrue, wraps in :class:Folder.- File-like objects — drained into :class:
Memory; media_type required.
Falls back to default on unrecognised shapes when supplied;
otherwise raises :class:TypeError.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))