yggdrasil.spark.setup

setup

Spark local environment setup utilities for Windows (and other platforms).

This module provides helpers to bootstrap a working local Spark environment from scratch — particularly on Windows, where Hadoop native binaries (winutils.exe) are not shipped with PySpark and Java compatibility can be tricky.

Usage::

from yggdrasil.spark.setup import ensure_spark_env, install_spark

# Full one-shot setup: installs pyspark, winutils, sets env vars,
# and returns a ready SparkSession.
spark = ensure_spark_env()

# Or step by step:
install_spark()                # pip-install pyspark if missing
ensure_hadoop_home()           # download winutils and set HADOOP_HOME
configure_java_compat()        # patch JVM args for Java 17+
spark = create_local_session() # build a local SparkSession

quiet_spark_loggers

quiet_spark_loggers() -> None

Mute the chattiest Spark / py4j loggers at the Python level.

spark_home_dir

spark_home_dir() -> Path

Return the default Yggdrasil-managed Spark home directory.

On Windows this is %LOCALAPPDATA%\.yggdrasil_spark. On other platforms it's ~/.yggdrasil_spark.

get_java_version

get_java_version() -> int | None

Return the major Java version (e.g. 17, 21, 24), or None if Java isn't found.

get_java_version_from_bin

get_java_version_from_bin(java_bin: str | Path) -> int | None

Return the major Java version for a specific java binary, or None on failure.

ensure_java

ensure_java(auto_download: bool = True, force: bool = False) -> Path

Make sure a compatible Java is available, downloading Zulu JDK 21 if needed.

Resolution order: 1. Check the system java on PATH — if it's a compatible version, use it. 2. Check the managed Yggdrasil Java at ~/.yggdrasil/java — if present and valid, prepend it to PATH/JAVA_HOME and use it. 3. If auto_download is True, download Azul Zulu JDK 21 into ~/.yggdrasil/java and set it up. 4. Otherwise, raise with guidance on what to install.

Parameters:

Name	Type	Description	Default
auto_download	bool	Whether to download Zulu JDK 21 when no compatible Java is found. Defaults to True.	True
force	bool	Re-download even if a managed JDK already exists.	False

Returns:

Type	Description
Path	The resolved JAVA_HOME path.

Raises:

Type	Description
RuntimeError	If no compatible Java is available and auto_download is False, or the download fails.

install_spark

install_spark(
    version: str | None = None, extras: list[str] | None = None
) -> None

Install PySpark via pip if it's not already importable.

Parameters:

Name	Type	Description	Default
version	str | None	PySpark version pin, e.g. "4.1.1". If None, installs the latest version compatible with the current Python.	None
extras	list[str] | None	Additional pip packages to install alongside PySpark, e.g. ["delta-spark"].	None

ensure_hadoop_home

ensure_hadoop_home(
    hadoop_home: str | Path | None = None, force: bool = False
) -> Path

Ensure HADOOP_HOME is set and contains winutils.exe (Windows only).

On non-Windows platforms this is a no-op — Hadoop native libs aren't needed for local-mode Spark.

Parameters:

Name	Type	Description	Default
hadoop_home	str | Path | None	Explicit directory to use. If None, uses the Yggdrasil-managed default under spark_home_dir().	None
force	bool	Re-download even if the binaries already exist.	False

Returns:

Type	Description
Path	The resolved HADOOP_HOME path.

configure_java_compat

configure_java_compat(java_version: int | None = None) -> list[str]

Return (and set) JVM options needed for modern Java + Spark.

Java 17+ restricts reflective access that Spark/Hadoop need. PySpark 4.x handles most of this internally, but some edge cases (especially on Windows with older Hadoop jars) still need extra --add-opens flags.

This function detects the Java version and sets PYSPARK_SUBMIT_ARGS so the Spark driver JVM picks up the right flags.

Parameters:

Name	Type	Description	Default
java_version	int | None	Override auto-detected Java major version.	None

Returns:

Type	Description
list[str]	The list of extra JVM flags that were applied.

create_local_session

create_local_session(
    app_name: str = "yggdrasil-local",
    cores: str = "local[*]",
    extra_config: dict[str, str] | None = None,
    **kwargs: Any
) -> "SparkSession"

Create a local-mode SparkSession with sensible defaults.

Parameters:

Name	Type	Description	Default
app_name	str	Application name shown in the Spark UI.	'yggdrasil-local'
cores	str	Master URL, default "local[*]" uses all cores.	'local[*]'
extra_config	dict[str, str] | None	Additional Spark config key/value pairs.	None
**kwargs	Any	Passed through to SparkSession.builder.config.	{}

Returns:

Type	Description
'SparkSession'	A ready-to-use SparkSession.

ensure_spark_env

ensure_spark_env(
    app_name: str = "yggdrasil-local",
    cores: str = "local[*]",
    install: bool = True,
    pyspark_version: str | None = None,
    hadoop_home: str | Path | None = None,
    extra_config: dict[str, str] | None = None,
    **kwargs: Any
) -> "SparkSession"

One-shot bootstrap: install, configure, and return a local SparkSession.

This is the easiest way to get Spark running on a fresh Windows machine. Call it once and you're good::

from yggdrasil.spark.setup import ensure_spark_env
spark = ensure_spark_env()

It handles: 1. Installing PySpark if missing 2. Downloading winutils.exe for Hadoop on Windows 3. Configuring JVM flags for Java 17+ / 24+ compatibility 4. Creating a local SparkSession with Arrow optimizations

Parameters:

Name	Type	Description	Default
app_name	str	Spark application name.	'yggdrasil-local'
cores	str	Master URL (default: all local cores).	'local[*]'
install	bool	Whether to pip-install PySpark if missing.	True
pyspark_version	str | None	Pin a specific PySpark version.	None
hadoop_home	str | Path | None	Custom HADOOP_HOME path (Windows only).	None
extra_config	dict[str, str] | None	Extra Spark config entries.	None
**kwargs	Any	Passed to create_local_session.	{}

Returns:

Type	Description
'SparkSession'	A ready SparkSession.

Raises:

Type	Description
ImportError	If PySpark is not installed and install=False.
RuntimeError	If Java is not found on PATH.