yggdrasil.pickle.ser.pyspark

pyspark

PySpark serialization support.

Wire-tag range: 700–799 (PYSPARK_BASE = 700)

Objects covered

pyspark.sql.DataFrame -> PYSPARK_DATAFRAME (700) via PyArrow IPC file pyspark.sql.Row -> PYSPARK_ROW (701) via Arrow record-batch pyspark.sql.types.StructType -> PYSPARK_SCHEMA (702) via Arrow IPC schema pyspark.sql.types.DataType -> PYSPARK_DATATYPE (703) via JSON repr pyspark.sql.Column -> PYSPARK_COLUMN (704) via pickle/cloudpickle pyspark.RDD -> PYSPARK_RDD (705) collect() → Arrow IPC pyspark.sql.SparkSession -> PYSPARK_SESSION (706) not reconstructed; serialised as a metadata-only stub

DataFrame / RDD payloads delegate to pyarrow. All other PySpark objects that cannot be naturally expressed with Arrow fall back to cloudpickle (if available) or standard pickle.

Decoding note

SparkSession references are not reconstructed on load. value returns None for PYSPARK_SESSION because a live Spark context cannot be round- tripped through bytes. The serialized form serves as documentation / provenance only.

PySparkSerialized dataclass

PySparkSerialized(head: Header, data: IO)

Bases: Serialized[TPySpark], Generic[TPySpark]

Abstract base for all PySpark-flavoured Serialized subclasses.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

PySparkDataFrameSerialized dataclass

PySparkDataFrameSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.sql.DataFrame by collecting it to the driver and storing the result as an Arrow IPC file.

On deserialization a pyarrow.Table is returned (not a DataFrame) because there is no live SparkSession available at load time. Call .to_pyspark(spark) to re-wrap if a session is available.

value property

value: Any

Return the collected data as a pyarrow.Table.

to_pyspark

to_pyspark(spark: Any = None) -> Any

Re-create a pyspark.sql.DataFrame from the stored Arrow data.

Uses :func:~yggdrasil.spark.cast.arrow_table_to_spark_dataframe which requires an active SparkSession (obtained via SparkSession.getActiveSession() when spark is None).

Parameters

spark: An active pyspark.sql.SparkSession, or None to use the currently-active session.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkRowSerialized dataclass

PySparkRowSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.sql.Row as a one-row Arrow RecordBatch.

value property

value: Any

Return the data as a pyspark.sql.Row (requires PySpark on load).

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkSchemaSerialized dataclass

PySparkSchemaSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.sql.types.StructType as an Arrow IPC schema.

On load, if PySpark is available the Arrow schema is converted back to a StructType; otherwise a pyarrow.Schema is returned.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkDataTypeSerialized dataclass

PySparkDataTypeSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.sql.types.DataType via its JSON representation.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkColumnSerialized dataclass

PySparkColumnSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.sql.Column expression.

Column objects are opaque JVM-backed objects. We use cloudpickle (preferred) or standard pickle as the storage strategy. Note that the deserialized Column requires the same PySpark / JVM version to be available.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkRDDSerialized dataclass

PySparkRDDSerialized(head: Header, data: IO)

Bases: PySparkSerialized[Any]

Serialize a pyspark.RDD by collecting all elements to the driver.

Elements must be homogeneous and Arrow-convertible (e.g. Rows, dicts, tuples, or primitives). The collected data is stored as an Arrow IPC file.

On deserialization, a pyarrow.Table is returned.

value property

value: Any

Return collected RDD data as a pyarrow.Table.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.

PySparkSessionSerialized dataclass

PySparkSessionSerialized(head: Header, data: IO)

Bases: PySparkSerialized[None]

Provenance stub for a pyspark.sql.SparkSession.

A live SparkSession cannot be serialized in a meaningful way – it is a client handle to a running JVM. We store app-name, master URL, and Spark version as metadata, and return None on deserialization.

module_and_name staticmethod

module_and_name(obj: Any, *, fallback: str = '') -> tuple[str, str]

Return (module, qualname) for obj.

Robust across Python objects, including many C-extension / PyArrow objects where module may be missing or misleading.

decode_arrow_buffer

decode_arrow_buffer() -> Any

Return payload as a pyarrow.Buffer for IPC reads.