`yggdrasil.execution`¶

execution ¶

Expression DSL for predicates and SQL generation.

Alias ¶

Alias(expr: Expression, name: str)

Bases: Expression

expr AS name — wraps any expression with a user-visible alias.

Used in SELECT lists, sub-expressions, and CTE column naming. The inner expr carries the computation; name is the output label.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Arithmetic ¶

Arithmetic(op: ArithmeticOp, left: Expression, right: Expression)

Bases: Expression

Two-operand arithmetic. Result type is the widened operand type.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Between ¶

Between(
    target: Expression, low: Expression, high: Expression, negated: bool = False
)

Bases: Predicate

column BETWEEN low AND high — inclusive on both bounds.

negated=True flips to NOT BETWEEN. The Python and pyarrow emitters honour the inclusive contract; SQL renders natively.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

filter_arrow_batch ¶

filter_arrow_batch(batch: 'Any') -> 'Any'

Filter batch — hashset shortcut for InList / AND(InList), :meth:pa.RecordBatch.filter otherwise.

The hashset shortcut probes each row's column value against a :class:frozenset and skips pyarrow's per-call compile + scan — ~30x faster than the kernel on the 1-row-per-leaf cache shape. Empty input is returned unchanged; an all-drop result is a zero-row slice with the source schema. Null semantics match :func:pyarrow.compute.is_in (includes_null=True keeps null rows, False drops them).

filter_arrow_table ¶

filter_arrow_table(table: 'Any') -> 'Any'

Filter table — same dispatch as :meth:filter_arrow_batch.

filter_arrow_batches ¶

filter_arrow_batches(batches: 'Iterable[Any]') -> 'Iterator[Any]'

Streaming filter — yield surviving batches one at a time.

Decomposition happens once outside the loop. InList / AND(InList) shapes route through :func:_apply_inlist per-batch (one pc.is_in kernel per clause); everything else compiles a single pa.Expression on the first non-empty batch (so a stream of empty batches doesn't pay the compile cost) and reuses it across the stream. Empty / fully-dropped batches are skipped so consumers see only "non-empty rows that match".

filter_polars_frame ¶

filter_polars_frame(frame: 'Any') -> 'Any'

Filter a :class:polars.DataFrame / :class:polars.LazyFrame.

Both shapes expose .filter(expr) accepting a :class:`polars.Expr, so one method covers both. The target schema feeds the same temporal-tz rewrite as the Arrow filter — a Polars frame whosets`` column is UTC and a predicate built with a Paris-tz Cast lands with the literal already in UTC by the time polars sees the expression.

filter_pandas_frame ¶

filter_pandas_frame(frame: 'Any') -> 'Any'

Filter a :class:pandas.DataFrame via the pyarrow kernel.

Pandas has no first-class predicate kernel and df.apply row-wise costs ~20 ms / 5 k rows (the slowest path in bench_predicate.py). Round-trip through Arrow instead: pa.Table.from_pandas → :meth:filter_arrow_table (which already auto-tunes between the hashset shortcut and the pyarrow filter kernel) → Table.to_pandas — the zero-copy legs land at sub-millisecond range on the same fixture, and downstream behaves identically (same row order, same dtype per column) to the original frame.

filter_spark_frame ¶

filter_spark_frame(frame: 'Any') -> 'Any'

Filter a :class:pyspark.sql.DataFrame.

filter_pylist ¶

filter_pylist(rows: 'Iterable[Any]') -> 'list[Any]'

Filter a list / iterable of Mapping rows (the :meth:Tabular.read_pylist shape).

Uses the same InList / AND(InList, ...) hashset shortcut as the arrow filter: when the predicate decomposes into (col, value_set, includes_null) clauses, each row becomes a dict-lookup + set-probe instead of compiling the full AST to a callable. Speeds up the cache-key lookup shape by ~2× on bench-sized inputs.

filter_pydict ¶

filter_pydict(data: 'Mapping[str, Any]') -> 'dict[str, list]'

Filter a column-oriented {col: [values, ...]} dict.

Mirrors :meth:Tabular.read_pydict: builds a temporary Arrow table, runs the same hashset / pyarrow filter as :meth:filter_arrow_table, and unzips the survivors back into a fresh {col: list} dict. Empty input or no surviving rows return a dict with the same key set and empty lists.

filter_iterable ¶

filter_iterable(
    items: "Iterable[Any]", *, key: "Any | None" = None
) -> "Iterator[Any]"

Filter an arbitrary iterable. Items default to dict-shaped rows (Mapping[str, Any]); pass key= to project the comparison target (e.g. key=attrgetter('payload')) when the predicate columns address a sub-shape of each item. Yields each item the predicate accepts.

Uses the same hashset shortcut as :meth:filter_pylist when the predicate decomposes into InList clauses — same speedup, just lazy (yields one at a time) instead of materialising into a list.

filter ¶

filter(target: 'Any') -> 'Any'

Generic dispatch — pick the right filter_* for target's type.

Recognised shapes:

:class:pyarrow.Table → :meth:filter_arrow_table.
:class:pyarrow.RecordBatch → :meth:filter_arrow_batch.
:class:pandas.DataFrame → :meth:filter_pandas_frame.
:class:polars.DataFrame / :class:polars.LazyFrame → :meth:filter_polars_frame.
:class:pyspark.sql.DataFrame → :meth:filter_spark_frame.
:class:dict of columns → :meth:filter_pydict.
:class:list / :class:tuple → :meth:filter_pylist.
Anything else iterable → :meth:filter_iterable.

Optional dependencies (polars / pandas / pyspark) are detected by module-name sniffing on type(target) so this method never imports them just to dispatch — the relevant filter_* does the import when called.

CaseWhen ¶

CaseWhen(
    branches: "tuple[tuple[Expression, Expression], ...]",
    else_expr: "Expression | None" = None,
    operand: "Expression | None" = None,
)

Bases: Expression

CASE [operand] WHEN cond THEN val … [ELSE val] END.

Two forms:

Searched (operand=None): each branch condition is a full boolean expression.
Simple (operand=expr): each branch condition is compared against operand with implicit equality.

branches is a tuple of (condition, result) pairs. else_expr is the fallback value (None ⇒ implicit NULL).

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Cast ¶

Cast(operand: Expression, dtype: 'DataType')

Bases: Expression

Explicit type cast. Returns a typed scalar expression.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Column ¶

Column(
    name: str,
    field: "Field | None" = None,
    alias: "str | None" = None,
    qualifier: "str | None" = None,
)

Bases: Expression

A reference to a column inside an expression tree.

Owns the expression-side lookup state: the column name the backend resolves on the frame, an optional SQL alias rename (so col("foo").with_alias("bar") renders as foo AS bar), and the table-level qualifier for T.col style addressing inside a MERGE / aliased SQL query.

field is the origin metadata — the typed :class:Field the column was sourced from (dtype, nullability, source-schema children). Backends consult it for typed-literal casts and engine-flavoured dtypes, but it deliberately does not carry expression-side knobs like the table qualifier — those would bleed back into Field consumers (schema diffs, cast registry, pickle round-trips) that have no business with them.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

CompareOp ¶

Bases: str, Enum

Binary comparison operator, target-engine-agnostic.

Backends translate to their own dialect: EQ becomes = in SQL, __eq__ in Python, pa.compute.equal in pyarrow, etc.

Expression ¶

Abstract base for every node in the AST.

Expressions are immutable by convention — combinators don't mutate operands but wrap them in a new node, so an expression tree can be safely cached, reused, or shared across threads.

Concrete subclasses opt out of :class:dataclasses.dataclass: a manual __slots__ + explicit __init__ is roughly 3× the construction throughput of @dataclass(frozen=True, slots=True) on the per-node hot path, and the AST builds enough nodes per predicate (one Comparison per c == v, one Logical per | / &) that the savings show up in benchmarks/io/tabular/bench_predicate.py.

The __eq__ / < / > / >= / <= / != operator overloads on this base produce :class:Comparison nodes instead of returning a bool — that's what makes col("price") >= 100 work. Structural equality uses :meth:equals; identity-based hashing keeps nodes usable as dict keys.

Subclasses override nothing structural; this class is a marker plus the operator surface and to_* dispatchers. Backend- specific compilation lives in the matching yggdrasil.execution.expr.backends.* module — kept off the node so a build that excludes (say) pyspark doesn't import the optional dependency.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

FunctionCall ¶

FunctionCall(
    name: str, args: "tuple[Expression, ...]" = (), distinct: bool = False
)

Bases: Expression

Arbitrary function call — UPPER(col), COUNT(DISTINCT col), etc.

name is stored upper-cased so comparisons are case-insensitive. args are the positional arguments as Expression nodes. distinct renders DISTINCT inside the parentheses (aggregate functions like COUNT(DISTINCT …)).

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

InList ¶

InList(
    target: Expression,
    values: "Iterable[Any]" = (),
    negated: bool = False,
    includes_null: bool = False,
)

Bases: Predicate

column IN (...) against a finite literal value list.

Values are stored as tuples of literal Python objects so the node remains hashable. The :attr:includes_null flag carries forward through round-trips — backends that don't natively handle NULL inside IN (most SQL dialects) expand to ... OR col IS NULL.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

filter_arrow_batch ¶

filter_arrow_batch(batch: 'Any') -> 'Any'

Filter batch — hashset shortcut for InList / AND(InList), :meth:pa.RecordBatch.filter otherwise.

The hashset shortcut probes each row's column value against a :class:frozenset and skips pyarrow's per-call compile + scan — ~30x faster than the kernel on the 1-row-per-leaf cache shape. Empty input is returned unchanged; an all-drop result is a zero-row slice with the source schema. Null semantics match :func:pyarrow.compute.is_in (includes_null=True keeps null rows, False drops them).

filter_arrow_table ¶

filter_arrow_table(table: 'Any') -> 'Any'

Filter table — same dispatch as :meth:filter_arrow_batch.

filter_arrow_batches ¶

filter_arrow_batches(batches: 'Iterable[Any]') -> 'Iterator[Any]'

Streaming filter — yield surviving batches one at a time.

Decomposition happens once outside the loop. InList / AND(InList) shapes route through :func:_apply_inlist per-batch (one pc.is_in kernel per clause); everything else compiles a single pa.Expression on the first non-empty batch (so a stream of empty batches doesn't pay the compile cost) and reuses it across the stream. Empty / fully-dropped batches are skipped so consumers see only "non-empty rows that match".

filter_polars_frame ¶

filter_polars_frame(frame: 'Any') -> 'Any'

Filter a :class:polars.DataFrame / :class:polars.LazyFrame.

Both shapes expose .filter(expr) accepting a :class:`polars.Expr, so one method covers both. The target schema feeds the same temporal-tz rewrite as the Arrow filter — a Polars frame whosets`` column is UTC and a predicate built with a Paris-tz Cast lands with the literal already in UTC by the time polars sees the expression.

filter_pandas_frame ¶

filter_pandas_frame(frame: 'Any') -> 'Any'

Filter a :class:pandas.DataFrame via the pyarrow kernel.

Pandas has no first-class predicate kernel and df.apply row-wise costs ~20 ms / 5 k rows (the slowest path in bench_predicate.py). Round-trip through Arrow instead: pa.Table.from_pandas → :meth:filter_arrow_table (which already auto-tunes between the hashset shortcut and the pyarrow filter kernel) → Table.to_pandas — the zero-copy legs land at sub-millisecond range on the same fixture, and downstream behaves identically (same row order, same dtype per column) to the original frame.

filter_spark_frame ¶

filter_spark_frame(frame: 'Any') -> 'Any'

Filter a :class:pyspark.sql.DataFrame.

filter_pylist ¶

filter_pylist(rows: 'Iterable[Any]') -> 'list[Any]'

Filter a list / iterable of Mapping rows (the :meth:Tabular.read_pylist shape).

Uses the same InList / AND(InList, ...) hashset shortcut as the arrow filter: when the predicate decomposes into (col, value_set, includes_null) clauses, each row becomes a dict-lookup + set-probe instead of compiling the full AST to a callable. Speeds up the cache-key lookup shape by ~2× on bench-sized inputs.

filter_pydict ¶

filter_pydict(data: 'Mapping[str, Any]') -> 'dict[str, list]'

Filter a column-oriented {col: [values, ...]} dict.

Mirrors :meth:Tabular.read_pydict: builds a temporary Arrow table, runs the same hashset / pyarrow filter as :meth:filter_arrow_table, and unzips the survivors back into a fresh {col: list} dict. Empty input or no surviving rows return a dict with the same key set and empty lists.

filter_iterable ¶

filter_iterable(
    items: "Iterable[Any]", *, key: "Any | None" = None
) -> "Iterator[Any]"

Filter an arbitrary iterable. Items default to dict-shaped rows (Mapping[str, Any]); pass key= to project the comparison target (e.g. key=attrgetter('payload')) when the predicate columns address a sub-shape of each item. Yields each item the predicate accepts.

Uses the same hashset shortcut as :meth:filter_pylist when the predicate decomposes into InList clauses — same speedup, just lazy (yields one at a time) instead of materialising into a list.

filter ¶

filter(target: 'Any') -> 'Any'

Generic dispatch — pick the right filter_* for target's type.

Recognised shapes:

:class:pyarrow.Table → :meth:filter_arrow_table.
:class:pyarrow.RecordBatch → :meth:filter_arrow_batch.
:class:pandas.DataFrame → :meth:filter_pandas_frame.
:class:polars.DataFrame / :class:polars.LazyFrame → :meth:filter_polars_frame.
:class:pyspark.sql.DataFrame → :meth:filter_spark_frame.
:class:dict of columns → :meth:filter_pydict.
:class:list / :class:tuple → :meth:filter_pylist.
Anything else iterable → :meth:filter_iterable.

Optional dependencies (polars / pandas / pyspark) are detected by module-name sniffing on type(target) so this method never imports them just to dispatch — the relevant filter_* does the import when called.

Like ¶

Like(
    target: Expression,
    pattern: str,
    case_insensitive: bool = False,
    negated: bool = False,
)

Bases: Predicate

SQL-style LIKE / ILIKE.

Pattern uses % and _ as wildcards. Set case_insensitive for ILIKE semantics; negated for NOT LIKE.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

filter_arrow_batch ¶

filter_arrow_batch(batch: 'Any') -> 'Any'

Filter batch — hashset shortcut for InList / AND(InList), :meth:pa.RecordBatch.filter otherwise.

The hashset shortcut probes each row's column value against a :class:frozenset and skips pyarrow's per-call compile + scan — ~30x faster than the kernel on the 1-row-per-leaf cache shape. Empty input is returned unchanged; an all-drop result is a zero-row slice with the source schema. Null semantics match :func:pyarrow.compute.is_in (includes_null=True keeps null rows, False drops them).

filter_arrow_table ¶

filter_arrow_table(table: 'Any') -> 'Any'

Filter table — same dispatch as :meth:filter_arrow_batch.

filter_arrow_batches ¶

filter_arrow_batches(batches: 'Iterable[Any]') -> 'Iterator[Any]'

Streaming filter — yield surviving batches one at a time.

Decomposition happens once outside the loop. InList / AND(InList) shapes route through :func:_apply_inlist per-batch (one pc.is_in kernel per clause); everything else compiles a single pa.Expression on the first non-empty batch (so a stream of empty batches doesn't pay the compile cost) and reuses it across the stream. Empty / fully-dropped batches are skipped so consumers see only "non-empty rows that match".

filter_polars_frame ¶

filter_polars_frame(frame: 'Any') -> 'Any'

Filter a :class:polars.DataFrame / :class:polars.LazyFrame.

Both shapes expose .filter(expr) accepting a :class:`polars.Expr, so one method covers both. The target schema feeds the same temporal-tz rewrite as the Arrow filter — a Polars frame whosets`` column is UTC and a predicate built with a Paris-tz Cast lands with the literal already in UTC by the time polars sees the expression.

filter_pandas_frame ¶

filter_pandas_frame(frame: 'Any') -> 'Any'

Filter a :class:pandas.DataFrame via the pyarrow kernel.

Pandas has no first-class predicate kernel and df.apply row-wise costs ~20 ms / 5 k rows (the slowest path in bench_predicate.py). Round-trip through Arrow instead: pa.Table.from_pandas → :meth:filter_arrow_table (which already auto-tunes between the hashset shortcut and the pyarrow filter kernel) → Table.to_pandas — the zero-copy legs land at sub-millisecond range on the same fixture, and downstream behaves identically (same row order, same dtype per column) to the original frame.

filter_spark_frame ¶

filter_spark_frame(frame: 'Any') -> 'Any'

Filter a :class:pyspark.sql.DataFrame.

filter_pylist ¶

filter_pylist(rows: 'Iterable[Any]') -> 'list[Any]'

Filter a list / iterable of Mapping rows (the :meth:Tabular.read_pylist shape).

Uses the same InList / AND(InList, ...) hashset shortcut as the arrow filter: when the predicate decomposes into (col, value_set, includes_null) clauses, each row becomes a dict-lookup + set-probe instead of compiling the full AST to a callable. Speeds up the cache-key lookup shape by ~2× on bench-sized inputs.

filter_pydict ¶

filter_pydict(data: 'Mapping[str, Any]') -> 'dict[str, list]'

Filter a column-oriented {col: [values, ...]} dict.

Mirrors :meth:Tabular.read_pydict: builds a temporary Arrow table, runs the same hashset / pyarrow filter as :meth:filter_arrow_table, and unzips the survivors back into a fresh {col: list} dict. Empty input or no surviving rows return a dict with the same key set and empty lists.

filter_iterable ¶

filter_iterable(
    items: "Iterable[Any]", *, key: "Any | None" = None
) -> "Iterator[Any]"

Filter an arbitrary iterable. Items default to dict-shaped rows (Mapping[str, Any]); pass key= to project the comparison target (e.g. key=attrgetter('payload')) when the predicate columns address a sub-shape of each item. Yields each item the predicate accepts.

Uses the same hashset shortcut as :meth:filter_pylist when the predicate decomposes into InList clauses — same speedup, just lazy (yields one at a time) instead of materialising into a list.

filter ¶

filter(target: 'Any') -> 'Any'

Generic dispatch — pick the right filter_* for target's type.

Recognised shapes:

:class:pyarrow.Table → :meth:filter_arrow_table.
:class:pyarrow.RecordBatch → :meth:filter_arrow_batch.
:class:pandas.DataFrame → :meth:filter_pandas_frame.
:class:polars.DataFrame / :class:polars.LazyFrame → :meth:filter_polars_frame.
:class:pyspark.sql.DataFrame → :meth:filter_spark_frame.
:class:dict of columns → :meth:filter_pydict.
:class:list / :class:tuple → :meth:filter_pylist.
Anything else iterable → :meth:filter_iterable.

Optional dependencies (polars / pandas / pyspark) are detected by module-name sniffing on type(target) so this method never imports them just to dispatch — the relevant filter_* does the import when called.

Literal ¶

Literal(value: Any, dtype: 'DataType | None' = None)

Bases: Expression

A scalar literal value.

dtype is optional — when left None the backend infers from the Python value. Pinning a dtype is useful when the inferred type would be wrong (e.g. naive datetime you want rendered as DATE instead of TIMESTAMP).

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Predicate ¶

Bases: Expression

Marker mix-in for boolean-valued expressions.

Every comparison / logical / membership / null-check node inherits this so callers can guard a where= argument with isinstance(x, Predicate). Adds two convenience filters (:meth:filter_arrow_batch, :meth:filter_arrow_table) that compile the predicate to a :class:pyarrow.compute.Expression and run the row-level filter natively in C++ — no Python row iteration. Used by :meth:yggdrasil.io.tabular.base.Tabular.delete on every leaf rewrite.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

filter_arrow_batch ¶

filter_arrow_batch(batch: 'Any') -> 'Any'

Filter batch — hashset shortcut for InList / AND(InList), :meth:pa.RecordBatch.filter otherwise.

The hashset shortcut probes each row's column value against a :class:frozenset and skips pyarrow's per-call compile + scan — ~30x faster than the kernel on the 1-row-per-leaf cache shape. Empty input is returned unchanged; an all-drop result is a zero-row slice with the source schema. Null semantics match :func:pyarrow.compute.is_in (includes_null=True keeps null rows, False drops them).

filter_arrow_table ¶

filter_arrow_table(table: 'Any') -> 'Any'

Filter table — same dispatch as :meth:filter_arrow_batch.

filter_arrow_batches ¶

filter_arrow_batches(batches: 'Iterable[Any]') -> 'Iterator[Any]'

Streaming filter — yield surviving batches one at a time.

Decomposition happens once outside the loop. InList / AND(InList) shapes route through :func:_apply_inlist per-batch (one pc.is_in kernel per clause); everything else compiles a single pa.Expression on the first non-empty batch (so a stream of empty batches doesn't pay the compile cost) and reuses it across the stream. Empty / fully-dropped batches are skipped so consumers see only "non-empty rows that match".

filter_polars_frame ¶

filter_polars_frame(frame: 'Any') -> 'Any'

Filter a :class:polars.DataFrame / :class:polars.LazyFrame.

Both shapes expose .filter(expr) accepting a :class:`polars.Expr, so one method covers both. The target schema feeds the same temporal-tz rewrite as the Arrow filter — a Polars frame whosets`` column is UTC and a predicate built with a Paris-tz Cast lands with the literal already in UTC by the time polars sees the expression.

filter_pandas_frame ¶

filter_pandas_frame(frame: 'Any') -> 'Any'

Filter a :class:pandas.DataFrame via the pyarrow kernel.

Pandas has no first-class predicate kernel and df.apply row-wise costs ~20 ms / 5 k rows (the slowest path in bench_predicate.py). Round-trip through Arrow instead: pa.Table.from_pandas → :meth:filter_arrow_table (which already auto-tunes between the hashset shortcut and the pyarrow filter kernel) → Table.to_pandas — the zero-copy legs land at sub-millisecond range on the same fixture, and downstream behaves identically (same row order, same dtype per column) to the original frame.

filter_spark_frame ¶

filter_spark_frame(frame: 'Any') -> 'Any'

Filter a :class:pyspark.sql.DataFrame.

filter_pylist ¶

filter_pylist(rows: 'Iterable[Any]') -> 'list[Any]'

Filter a list / iterable of Mapping rows (the :meth:Tabular.read_pylist shape).

Uses the same InList / AND(InList, ...) hashset shortcut as the arrow filter: when the predicate decomposes into (col, value_set, includes_null) clauses, each row becomes a dict-lookup + set-probe instead of compiling the full AST to a callable. Speeds up the cache-key lookup shape by ~2× on bench-sized inputs.

filter_pydict ¶

filter_pydict(data: 'Mapping[str, Any]') -> 'dict[str, list]'

Filter a column-oriented {col: [values, ...]} dict.

Mirrors :meth:Tabular.read_pydict: builds a temporary Arrow table, runs the same hashset / pyarrow filter as :meth:filter_arrow_table, and unzips the survivors back into a fresh {col: list} dict. Empty input or no surviving rows return a dict with the same key set and empty lists.

filter_iterable ¶

filter_iterable(
    items: "Iterable[Any]", *, key: "Any | None" = None
) -> "Iterator[Any]"

Filter an arbitrary iterable. Items default to dict-shaped rows (Mapping[str, Any]); pass key= to project the comparison target (e.g. key=attrgetter('payload')) when the predicate columns address a sub-shape of each item. Yields each item the predicate accepts.

Uses the same hashset shortcut as :meth:filter_pylist when the predicate decomposes into InList clauses — same speedup, just lazy (yields one at a time) instead of materialising into a list.

filter ¶

filter(target: 'Any') -> 'Any'

Generic dispatch — pick the right filter_* for target's type.

Recognised shapes:

:class:pyarrow.Table → :meth:filter_arrow_table.
:class:pyarrow.RecordBatch → :meth:filter_arrow_batch.
:class:pandas.DataFrame → :meth:filter_pandas_frame.
:class:polars.DataFrame / :class:polars.LazyFrame → :meth:filter_polars_frame.
:class:pyspark.sql.DataFrame → :meth:filter_spark_frame.
:class:dict of columns → :meth:filter_pydict.
:class:list / :class:tuple → :meth:filter_pylist.
Anything else iterable → :meth:filter_iterable.

Optional dependencies (polars / pandas / pyspark) are detected by module-name sniffing on type(target) so this method never imports them just to dispatch — the relevant filter_* does the import when called.

SortOrder ¶

SortOrder(
    expr: Expression, ascending: bool = True, nulls_first: "bool | None" = None
)

Bases: Expression

ORDER BY expr [ASC|DESC] [NULLS FIRST|LAST].

ascending=True for ASC, False for DESC. nulls_first=None leaves the null ordering to the engine default; explicit True / False renders NULLS FIRST / NULLS LAST.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Star ¶

Star(qualifier: 'str | None' = None)

Bases: Expression

SELECT * or COUNT(*) — a bare wildcard reference.

qualifier handles table.* style addressing: Star(qualifier="t") renders as t.*.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

Subscript ¶

Subscript(expr: Expression, index: Expression)

Bases: Expression

expr[index] — array element access or map key lookup.

expr is the collection expression; index is the key / offset expression. SQL renders as expr[index].

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

WindowFunction ¶

WindowFunction(function: Expression, window: WindowSpec)

Bases: Expression

function OVER window_spec — a windowed aggregate or ranking call.

function is typically a :class:FunctionCall (e.g. ROW_NUMBER(), SUM(col)). window carries the OVER (…) specification.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

WindowSpec ¶

WindowSpec(
    partition_by: "tuple[Expression, ...]" = (),
    order_by: "tuple[SortOrder, ...]" = (),
    frame_start: "str | None" = None,
    frame_end: "str | None" = None,
)

Bases: Expression

OVER (PARTITION BY … ORDER BY … ROWS BETWEEN … AND …).

partition_by and order_by are tuples of Expression / SortOrder nodes respectively. frame_start / frame_end are raw SQL frame-boundary strings ("UNBOUNDED PRECEDING", "CURRENT ROW", "3 PRECEDING", etc.) — kept as strings because the combinatorial space of frame specs is huge and rarely manipulated programmatically.

equals ¶

equals(other: Any) -> bool

Structural equality.

a.equals(b) is True when a and b are the same concrete node type with field-by-field equality. The plain == operator on Expressions builds a :class:Comparison node instead, so use :meth:equals whenever the test target is an Expression you'd otherwise be comparing for identity.

cast ¶

cast(dtype: 'DataType') -> 'Expression'

Coerce self to dtype.

Smart factory — not all calls return a fresh :class:Cast node:

When self already advertises dtype (a :class:Column whose field.dtype matches, or a :class:Cast already targeting dtype), :meth:cast is a no-op and returns self unchanged.
When self is an Object- or Null-typed :class:Column, the column carries no real type information yet — its field.dtype is replaced with dtype in a new :class:Column (no :class:Cast wrapper needed: the downstream filter / SQL emitter reads the field's dtype directly).
When self is a :class:Cast already, the inner cast collapses: Cast(x, A).cast(B) becomes Cast(x, B) so backends emit one cast, not two.
Otherwise, wrap self in :class:Cast.

Construct :class:Cast directly to opt out of these rewrites (e.g. when a test wants the raw wrapper).

to_python ¶

to_python(*, strict: bool = False)

Compile to a Callable[[Mapping[str, Any]], Any].

With strict=True, missing columns raise KeyError on evaluation; the default treats them as None (matching SQL three-valued logic).

to_sql ¶

to_sql(flavor: 'str | None' = None, *, dialect: 'str | None' = None) -> str

Render to a SQL string for the named flavor / dialect.

flavor is the canonical parameter name and accepts "databricks" / "postgres" / "sqlite" / "mysql" / "ansi". dialect is a deprecated alias kept for callers that already use the older keyword. Default mirrors yggdrasil's primary target (Databricks).

to_arrow ¶

to_arrow()

Lift to a :class:pyarrow.compute.Expression.

Canonical name; :meth:to_pyarrow is kept as an alias so callers don't have to update on rename.

to_polars ¶

to_polars()

Lift to a :class:polars.Expr.

to_pyspark ¶

to_pyspark()

Lift to a :class:pyspark.sql.Column.

to_engine ¶

to_engine(engine: str, **kwargs: Any) -> Any

Dispatch by backend name.

engine ∈ {"python", "sql", "arrow", "polars", "spark"} — the same set :meth:from_ accepts. **kwargs are forwarded to the matching to_* method (e.g. SQL takes flavor / strict for Python). Useful for code that picks the target at runtime (configuration-driven emitters, dispatch tables, …).

merge_with ¶

merge_with(other: 'Expression') -> 'Expression'

Combine self with other into a single expression.

Both sides predicates → conjunction (self AND other). Both sides structurally equal → return self (idempotent). Anything else raises :class:TypeError — a "merge" between a scalar and a different scalar isn't a well-defined operation; callers needing arithmetic combination should spell out the operator (self + other, etc.).

from_ `classmethod` ¶

from_(source: Any, **kwargs: Any) -> 'Expression'

Auto-detect lifter.

Routes to the matching from_* based on the source's runtime type:

str → :meth:from_sql
pyarrow.compute.Expression → :meth:from_arrow
polars.Expr → :meth:from_polars
pyspark.sql.Column → :meth:from_spark
already an :class:Expression → returned unchanged

**kwargs are forwarded to the chosen lifter (e.g. SQL takes flavor= / dialect=).

from_sql `classmethod` ¶

from_sql(
    sql: str, flavor: "str | None" = None, *, dialect: "str | None" = None
) -> "Expression"

Parse a SQL predicate string into our AST.

Uses the in-tree tokenizer + recursive-descent parser in backends.sql — no third-party SQL parser dependency. See :func:backends.sql.from_sql for the supported grammar.

from_arrow `classmethod` ¶

from_arrow(expr: Any) -> 'Expression'

Lift a :class:pyarrow.compute.Expression.

from_polars `classmethod` ¶

from_polars(expr: Any) -> 'Expression'

Lift a :class:polars.Expr.

from_spark `classmethod` ¶

from_spark(expr: Any) -> 'Expression'

Lift a :class:pyspark.sql.Column.

all_of ¶

all_of(*operands: Any) -> Logical

Conjunction over an arbitrary number of operands.

Equivalent to chaining a & b & c but produces a flat :class:Logical instead of a left-leaning tree — useful when callers know the conjunction is commutative and want the cheaper round-trip.

any_of ¶

any_of(*operands: Any) -> Logical

Disjunction over an arbitrary number of operands.

col ¶

col(
    name: "str | Field",
    *,
    field: "Field | None" = None,
    alias: "str | None" = None,
    qualifier: "str | None" = None,
    dtype: "Any | None" = None
) -> Column

Build a :class:Column reference.

name is the column identifier; pass a pre-built :class:Field here to reuse the typed metadata (the bound dtype, nullability, children, …). dtype is the convenience knob for the common "I know the column type" case — when omitted the (synthesised) field's dtype defaults to :class:ObjectType so backends fall back to engine-side inference.

alias adds a column-level rename so emitters render foo AS bar. qualifier adds the table-level T.col addressing used by aliased SQL / MERGE rewrites. Both live on the :class:Column itself — :class:Field stays as origin metadata only.

field= is the explicit entry: when the caller already has a :class:Field instance, hand it in directly and the builder skips the construction. field and dtype are mutually exclusive — the field's dtype wins if both are supplied.

lit ¶

lit(value: Any, dtype: 'DataType | None' = None) -> Literal

Build a :class:Literal. Convenience for the builder API.

neg ¶

neg(expr: Expression) -> Not

Return NOT expr. Same as ~expr but explicit.

yggdrasil.execution¶

execution ¶

Alias ¶

equals ¶

cast ¶

to_python ¶

to_sql ¶

to_arrow ¶

to_polars ¶

to_pyspark ¶

to_engine ¶

merge_with ¶

from_ classmethod ¶

from_sql classmethod ¶

from_arrow classmethod ¶

from_polars classmethod ¶

from_spark classmethod ¶

Arithmetic ¶

equals ¶

cast ¶

to_python ¶

to_sql ¶

to_arrow ¶

to_polars ¶

to_pyspark ¶

to_engine ¶

merge_with ¶

from_ classmethod ¶

from_sql classmethod ¶

from_arrow classmethod ¶

from_polars classmethod ¶

from_spark classmethod ¶

Between ¶

equals ¶

cast ¶

to_python ¶

to_sql ¶

to_arrow ¶

to_polars ¶

to_pyspark ¶

to_engine ¶

merge_with ¶

from_ classmethod ¶

from_sql classmethod ¶

from_arrow classmethod ¶

from_polars classmethod ¶

from_spark classmethod ¶

filter_arrow_batch ¶

filter_arrow_table ¶

filter_arrow_batches ¶

filter_polars_frame ¶

filter_pandas_frame ¶

filter_spark_frame ¶

filter_pylist ¶

filter_pydict ¶

filter_iterable ¶

filter ¶

CaseWhen ¶

equals ¶

cast ¶

to_python ¶

to_sql ¶

to_arrow ¶

to_polars ¶

to_pyspark ¶

to_engine ¶

merge_with ¶

from_ classmethod ¶

from_sql classmethod ¶

from_arrow classmethod ¶

from_polars classmethod ¶

from_spark classmethod ¶

Cast ¶

equals ¶

cast ¶

to_python ¶

to_sql ¶

to_arrow ¶

to_polars ¶

to_pyspark ¶

`yggdrasil.execution`¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶

from_ `classmethod` ¶

from_sql `classmethod` ¶

from_arrow `classmethod` ¶

from_polars `classmethod` ¶

from_spark `classmethod` ¶