Skip to content

Yggdrasil

Schema-aware data interchange for Python. One conversion registry that moves values cleanly between Python types, dataclasses, Arrow, Polars, pandas, Spark, Databricks, and the wire — without losing schema, nullability, or metadata.

pip install ygg

Why people pick this up

  • Stop writing brittle hand-rolled casts between dicts, dataframes, and warehouse schemas.
  • Treat Arrow schema as the contract: names, order, nullability, metadata, nested structure are preserved across boundaries.
  • Use one converter registry instead of separate utilities per engine.
  • Optional dependencies — pull pandas / polars / spark / databricks only when you need them.

60-second tour

from yggdrasil.data.cast.registry import convert

convert("42", int)              # 42
convert("yes", bool)            # True
convert("2024-06-01", "date")   # datetime.date(2024, 6, 1)

from dataclasses import dataclass
from yggdrasil.data.cast.registry import convert

@dataclass
class Order:
    id: int
    amount: float
    paid: bool = False

convert({"id": "7", "amount": "99.50", "paid": "yes"}, Order)
# Order(id=7, amount=99.5, paid=True)

import pyarrow as pa
from yggdrasil.arrow.cast import cast_arrow_tabular
from yggdrasil.data.options import CastOptions

raw = pa.table({"id": ["1", "2"], "score": ["9.1", "8.7"]})
target = pa.schema([
    pa.field("id",    pa.int64(),   nullable=False),
    pa.field("score", pa.float64(), nullable=False),
])
out = cast_arrow_tabular(raw, CastOptions(target=target))

from yggdrasil.databricks import DatabricksClient

stmt = DatabricksClient().sql.execute("SELECT * FROM main.default.orders LIMIT 100")
stmt.to_arrow_table()
stmt.to_pandas()
stmt.to_polars()
stmt.to_spark()

Where to go next

  • Getting Started Install, first conversions, a working example for every layer.

  • Architecture Cast registry, dispatch order, CastOptions, optional-dep guards.

  • Casting guide Scalar conversion, schema-aware tabular cast, engine bridges.

  • IO & HTTP URL, HTTPSession, batch dispatch, response conversions.

  • Databricks SQL, Unity Catalog, Compute, DBFS/Volumes, Secrets, IAM.

  • Databricks CLI ygg databricks — clusters, warehouses, sql, jobs, fs, wheel, deploy, seed.

  • Development Tests, lint, docs, optional dependencies.

  • Module walkthrough Curated index of focused module pages.

  • API Reference Auto-generated from the yggdrasil source tree.

Install patterns

pip install ygg                   # core: pyarrow + polars + xxhash + orjson
pip install "ygg[bigdata]"        # pyspark
pip install "ygg[databricks]"     # databricks-sdk
pip install "ygg[api]"            # fastapi, uvicorn, pydantic
pip install "ygg[http]"           # xxhash
pip install "ygg[pickle]"         # cloudpickle, dill, zstandard, blake3
pip install "ygg[mongo]"          # mongoengine
pip install "ygg[postgres]"       # psycopg, adbc-driver-postgresql
pip install "ygg[kafka]"          # confluent-kafka
pip install "ygg[delta]"          # deltalake

The only hard runtime deps are pyarrow>=20, polars>=1.3, xxhash, and orjson>=3.10.