Skip to content

yggdrasil.data.types.nested.struct_spark

struct_spark

Spark cast helpers for :class:StructType targets.

Spark casts lower to :class:pyspark.sql.Column expressions — projection-time, no row-by-row work. The cast becomes part of the physical plan, fused with any pushdown filters and the final select(*cols).

  • :func:cast_spark_struct_column — struct → struct via F.struct(...), with a null-preservation guard (F.when(col.isNull(), F.lit(None)).otherwise(...)) because F.struct always returns a non-null row.
  • :func:cast_spark_map_column — map → struct via F.element_at(col, F.lit(name)) per child.
  • :func:cast_spark_list_column — list → struct via F.get(col, F.lit(i)) (null-on-OOB; plain col[i] raises ArrayIndexOutOfBoundsException in Spark 4.x).
  • :func:cast_spark_tabular — DataFrame select with one cast Column per target field.