yggdrasil.execution.expr.backends.pyspark_backend

pyspark_backend

pyspark.sql.Column emitter + best-effort lifter.

:func:to_pyspark translates the AST to a :class:pyspark.sql.Column. Output is suitable as the predicate of :meth:pyspark.sql.DataFrame.filter / :meth:pyspark.sql.DataFrame.where. Type-aware literals route through :func:pyspark.sql.functions.lit plus a cast when the :class:Literal carries a pinned :class:DataType — keeps Spark from inferring the wrong dtype on ambiguous Python values (date becoming timestamp, etc.).

:func:from_pyspark falls back to the column's underlying Expression-string representation. Spark only exposes Column.expr.json() / Column.expr.toString() reliably; the lifter parses the SQL produced by Column.expr.sql() via the SQL backend's :func:from_sql. That reuses one parser and means this lifter is best-effort but consistent with what the SQL backend handles.

to_pyspark

to_pyspark(expr: Expression)

Emit expr as a :class:pyspark.sql.Column.

from_pyspark

from_pyspark(spark_col) -> Expression

Lift a :class:pyspark.sql.Column back into our AST.

Strategy: render the column to SQL via Spark's own Column.expr.sql() (or str(col)-style fallback), then parse it through the SQL backend's :func:from_sql. That keeps one parser instead of two and means anything :func:from_sql handles is supported here too.

Raises :class:NotImplementedError when Spark won't surface the SQL representation (e.g. UDFs, Column.expr private APIs that aren't stable across Spark versions).