Skip to content

yggdrasil.spark.cast

cast

Spark <-> Arrow casting helpers and converters.

This module provides bidirectional type-mapping and data-casting between Apache Spark (PySpark) and Apache Arrow (PyArrow). Every public function accepts an optional CastOptions argument that carries the target schema and a safe flag:

  • safe=True (default) – best-effort: invalid casts become null / empty values; missing columns are filled with type-appropriate defaults.
  • safe=False – strict: any type mismatch or missing column raises immediately.

The module is structured in three layers:

  1. Type converters – stateless, pure functions that map Arrow ↔ Spark type objects without touching any data.
  2. Field / schema converters – wrap type converters to also carry name, nullability, and metadata.
  3. Data converters – operate on live Spark DataFrames / Columns using the type information from layers 1-2.

JSON decoding for compound targets

When the source column is StringType or BinaryType and the target type is a compound type (struct, array, or map), the casters automatically attempt to parse the column as JSON using from_json before applying field-level / element-level casting. This handles the common commodity-data pattern where nested structures are stored as JSON strings in a flat column (e.g. a STRING column containing '{"bid":99.5,"ask":100.0}').

  • BinaryType sources are decoded to UTF-8 string first via CAST(... AS STRING).
  • Spark's from_json silently returns null for rows whose JSON is malformed or structurally incompatible — consistent with safe=True semantics. In safe=False mode callers can enforce non-nullability on the result via the standard null-fill machinery.

spark_dataframe_to_arrow 

spark_dataframe_to_arrow(df: 'pyspark_sql.DataFrame') -> pa.Table

Materialize df as a pa.Table, working on PySpark 3 and 4.

PySpark 4.x added the native :meth:pyspark.sql.DataFrame.toArrow helper that we prefer when available — single driver-side collect that round-trips through Arrow IPC. Older releases (3.x) don't expose it; we fall back to the streaming :meth:toArrowBatchIterator API when present, and finally to a pandas round-trip.

Both Arrow paths can fail at the JVM level on Java 17+ when the cluster JVM was started without the sun.misc.Unsafe / java.nio.DirectByteBuffer --add-opens flags (IllegalAccessException deep inside the Spark allocator). We catch that and fall back to toPandas so callers see a working table instead of a JVM stack trace. The pandas path itself benefits from Spark's Arrow optimization when the session has spark.sql.execution.arrow.pyspark.enabled=true.

Centralizing the polyfill here keeps callers from sprinkling hasattr(df, "toArrow") and exception-handling boilerplate throughout the codebase.

spark_dataframe_to_pandas 

spark_dataframe_to_pandas(df: 'pyspark_sql.DataFrame')

Public alias for :func:_to_pandas_no_arrow.

Same JVM-Arrow-bypass behavior; exposed so tests and callers that just want a pandas DataFrame don't have to round-trip through Arrow or hand-roll the same conf-toggle dance.