yggdrasil.databricks.registry

registry

:class:WorkspacePyPIRegistry — workspace-backed shared wheel cache.

A lightweight PEP 503-shaped layout under a single :class:WorkspacePath so colleagues on the same workspace can share locally-built wheels without round-tripping through a real PyPI server. The path is <base>/<pkg>/<pkg>-<version>-py3-none-any.whl; the registry is lazy (wheels only build when a consumer asks for the dep) and idempotent (existing entries are reused unless the caller explicitly forces a rebuild, or the dep is an editable install — those re-publish every load with the local hostname folded into the version so each developer's working copy lands at a stable but unique slot).

Used by :meth:DatabricksClient.spark to wire serverless deps into :class:DatabricksEnv.withDependencies:

• Public PyPI distributions go through verbatim ("ygg==0.7.85"); the cluster's pip handles them.
• Editable / private distributions get a wheel built locally (pip wheel <name> --no-deps), uploaded to the workspace registry once, then downloaded back to a local cache and declared via the local: prefix Databricks Connect understands.

The "is this a public PyPI package?" check is opt-in via check_public=True — an HTTPS HEAD against pypi.org/pypi/<name>/json — so an offline registry doesn't pay the latency by default. The default classification reads direct_url.json straight off the installed distribution (PEP 610) to spot editables, and otherwise treats every dep as private (publish locally, share via the workspace cache).

DependencyKind

Enum-shaped marker for how a dep ships to the cluster.

PUBLIC class-attribute

PUBLIC: str = 'public'

Pip-installable from a public index — pass the spec straight to :meth:DatabricksEnv.withDependencies.

LOCAL class-attribute

LOCAL: str = 'local'

Installed locally from a private source — build a wheel and share through the workspace registry.

EDITABLE class-attribute

EDITABLE: str = 'editable'

pip install -e . install — version-stamp with the local hostname so each developer's working copy gets its own slot in the registry; always re-upload on resolve.

DependencyInfo dataclass

DependencyInfo(
    name: str,
    version: Optional[str] = None,
    kind: str = DependencyKind.LOCAL,
    source: Optional[Path] = None,
    spec: Optional[str] = None,
)

Resolution of a single dep into a Databricks Connect spec.

Attributes:

Name	Type	Description
name	str	PyPI project name (case-preserving). Used as both the install spec for public deps and the directory key under the registry for everything else.
version	Optional[str]	Resolved version string. Editable deps get a PEP 440 local-version suffix (+host-<hostname>) so the same project from different machines lands at distinct registry entries.
kind	str	One of :class:DependencyKind constants.
source	Optional[Path]	Filesystem location of the source tree (editable / local) or None for public deps.
spec	Optional[str]	The pip install spec for public deps ("ygg==0.7.85"); None for everything else.

wheel_basename property

wheel_basename: str

Conventional wheel filename under the registry layout.

Always emits a pure-python wheel name — <name>-<version>-py3-none-any.whl — so reads from the registry don't have to guess at the tag. The build step is the authoritative source; this method is for cache lookup only.

WorkspacePyPIRegistry dataclass

WorkspacePyPIRegistry(
    client: "DatabricksClient",
    base_path: "WorkspacePath" = None,
    local_cache: Path = None,
)

PEP 503-shaped wheel cache living under a :class:WorkspacePath.

Layout::

<base_path>/
    <package>/
        <package>-<version>-py3-none-any.whl
        <package>-<other-version>-py3-none-any.whl
        ...

base_path defaults to the shared /Workspace/Shared/pypi — the same workspace-wide index the serverless job image deploys to — so the Spark Connect path and the job image reuse one registry. Pass an explicit workspace path for a private / per-user root.

:meth:publish is the entry point. It classifies the dep, builds a wheel locally when needed, uploads it to the workspace under the layout above, and downloads it back to local_cache (default: a temp dir) so the caller can pass the result to :meth:DatabricksEnv.withDependencies with the local: prefix Databricks Connect understands. Public PyPI specs short-circuit to "<name>==<version>" with no upload.

The registry is lazy: existing entries are reused as-is, and only editable deps are re-uploaded on every call (their hostname-stamped version slot is the same on the same machine, so this is a single overwriting upload, not a leaky history).

publish_many

publish_many(
    deps: Sequence[Any], *, check_public: bool = False
) -> Tuple[list[str], list["WorkspacePath"]]

Resolve a batch of deps. Returns (specs, remote_paths).

specs is the list of strings to feed straight into :meth:DatabricksEnv.withDependencies — a mix of "name==version" (public) and "local:<path>" (private). remote_paths collects the workspace entries (None-free, for the public deps that didn't upload anywhere) so callers can mirror / inspect the cache.

publish

publish(
    obj: Any, *, check_public: bool = False
) -> Tuple[str, Optional["WorkspacePath"]]

Resolve a single dep into a withDependencies-ready spec.

• PUBLIC → returns ("<name>==<version>", None).
• EDITABLE → builds a wheel with the hostname-stamped version, uploads under <base>/<name>/<wheel>, downloads back to local_cache, returns ("local:<local-path>", <workspace-path>).
• LOCAL → same as editable, but only uploads when the workspace entry is missing (the registry is lazy).

resolve_local

resolve_local(remote: 'WorkspacePath') -> _LocalPath

Pull remote into :attr:local_cache and return the path.

Useful as a separate entry for consumers that already know the registry path (built earlier, or pre-populated by a teammate) and just want it on disk to feed withDependencies(local:...).

classify_dependency

classify_dependency(obj: Any, *, check_public: bool = False) -> DependencyInfo

Resolve obj into a :class:DependencyInfo.

Accepts the same shapes as the rest of the path API:

• "ygg" / "ygg==0.7.85" — already a pip spec. Spec with an operator goes straight to PUBLIC. A bare name consults importlib.metadata (and, when check_public=True, pypi.org) to pick between PUBLIC / EDITABLE / LOCAL.
• :class:os.PathLike — always LOCAL; the path is the source tree.
• Any object exposing __module__ — resolves via its top package name.