`yggdrasil.databricks.aws`¶

aws ¶

Databricks-vended AWS credentials providers.

Two :class:AwsCredentialsProvider subclasses backed by Unity Catalog's temporary-credentials APIs:

:class:AWSDatabricksVolumeCredentials — vended through temporary_volume_credentials.generate_temporary_volume_credentials, scoped to a volume id.
:class:AWSDatabricksTableCredentials — vended through temporary_table_credentials.generate_temporary_table_credentials, scoped to a table id.
:class:AWSDatabricksPathCredentials — vended through temporary_path_credentials.generate_temporary_path_credentials, scoped to a storage URL. This is the only endpoint that vends creds for an external location's storage credential (the service-credential endpoint rejects those), so it backs :class:~yggdrasil.databricks.external.location.resource.ExternalLocation.

Each provider is a process-wide singleton per resource id and handles both read and write modes internally — :meth:get_credentials(mode=...) resolves the requested :class:Mode into the right UC operation and returns the matching credentials. The per-region :class:AWSClient cache is keyed by (mode, region) so reads and writes mint distinct boto sessions while still sharing one provider, one bound :class:DatabricksClient, and the singleton-by-resource_id guarantee.

The bound :class:DatabricksClient is mutable: every constructor call updates the live binding so refreshes that follow a client rotation pick up the fresh workspace auth.

Vended credentials can also be cached in a Databricks secret scope, one per resource — aws.<kind>.<resource> (e.g. aws.volume.cat.sch.vol / aws.table.cat.sch.tbl), with the credentials stored under a single credentials key (a read/write map). A later resolution — in this process, on a Spark executor, or in a fresh run — reuses a still-valid cached credential instead of re-vending it from Unity Catalog; an in-process memo short-circuits repeat calls without even reading the secret. This is off by default: a caller opts in with secret_cache=True (on the provider, or Volume.aws / Table.aws / credentials_refresher). The aws scope prefix is overridable via YGG_DATABRICKS_CREDS_SECRET_PREFIX (empty also disables it). The whole layer is best-effort: any Secrets-API failure falls back to a fresh UC vend, so it never blocks credential resolution.

AWSDatabricksVolumeCredentials ¶

AWSDatabricksVolumeCredentials(
    volume_id: str,
    *,
    client: Any = None,
    resource_url: "str | None" = None,
    secret_cache: bool = False
)

Bases: _DatabricksCredentialsBase

Refreshable AWS creds for a Unity Catalog volume.

Singleton-cached per volume_id. One provider serves both read (READ_VOLUME) and write (WRITE_VOLUME) modes — the requested mode is resolved at :meth:get_credentials / :meth:aws_client time.

client `property` ¶

client: 'DatabricksClient'

Bound :class:DatabricksClient. Lazily resolves to :meth:DatabricksClient.current when no client was passed.

workspace `property` ¶

workspace: Any

Shortcut for self.client.workspace_client().

get_credentials ¶

get_credentials(mode: ModeLike = None) -> AwsCredentials

Return fresh credentials scoped to mode.

mode accepts a :class:Mode enum, a mode-like string ("read", "overwrite", …), or None (uses :attr:DEFAULT_MODE). Read-only modes vend the UC "read" operation; everything else vends the writable operation.

If the SDK rejects the call with PermissionDenied: ... EXTERNAL USE SCHEMA on Schema 'cat.sch', we make exactly one attempt to self-grant EXTERNAL_USE_SCHEMA on the offending schema and retry. Owners of UC schemas commonly forget this grant — when they own the schema they have permission to fix it, and silently succeeding is dramatically less surprising than asking them to read the error and run a follow-up SQL. If the recovery itself fails (non-owner, network error, …) the original PermissionDenied propagates so the failure mode stays obvious.

aws_client ¶

aws_client(
    *, mode: ModeLike = None, region: Optional[str] = None
) -> "AWSClient"

Return the cached :class:AWSClient for this provider / mode / region.

First call per (mode, region) seeds a botocore :class:RefreshableCredentials-backed session whose refresher is bound to mode; subsequent calls with the same key return the same client and reuse the connection pool, boto-client cache, and in-flight refresh state.

with_client ¶

with_client(client: Any) -> '_DatabricksCredentialsBase'

Replace the bound client. Returns self.

AWSDatabricksTableCredentials ¶

AWSDatabricksTableCredentials(
    table_id: str,
    *,
    client: Any = None,
    resource_url: "str | None" = None,
    secret_cache: bool = False
)

Bases: _DatabricksCredentialsBase

Refreshable AWS creds for a Unity Catalog table.

Singleton-cached per table_id. One provider serves both read (READ) and write (READ_WRITE) modes — the requested mode is resolved at :meth:get_credentials / :meth:aws_client time.

client `property` ¶

client: 'DatabricksClient'

Bound :class:DatabricksClient. Lazily resolves to :meth:DatabricksClient.current when no client was passed.

workspace `property` ¶

workspace: Any

Shortcut for self.client.workspace_client().

get_credentials ¶

get_credentials(mode: ModeLike = None) -> AwsCredentials

Return fresh credentials scoped to mode.

mode accepts a :class:Mode enum, a mode-like string ("read", "overwrite", …), or None (uses :attr:DEFAULT_MODE). Read-only modes vend the UC "read" operation; everything else vends the writable operation.

If the SDK rejects the call with PermissionDenied: ... EXTERNAL USE SCHEMA on Schema 'cat.sch', we make exactly one attempt to self-grant EXTERNAL_USE_SCHEMA on the offending schema and retry. Owners of UC schemas commonly forget this grant — when they own the schema they have permission to fix it, and silently succeeding is dramatically less surprising than asking them to read the error and run a follow-up SQL. If the recovery itself fails (non-owner, network error, …) the original PermissionDenied propagates so the failure mode stays obvious.

aws_client ¶

aws_client(
    *, mode: ModeLike = None, region: Optional[str] = None
) -> "AWSClient"

Return the cached :class:AWSClient for this provider / mode / region.

First call per (mode, region) seeds a botocore :class:RefreshableCredentials-backed session whose refresher is bound to mode; subsequent calls with the same key return the same client and reuse the connection pool, boto-client cache, and in-flight refresh state.

with_client ¶

with_client(client: Any) -> '_DatabricksCredentialsBase'

Replace the bound client. Returns self.

AWSDatabricksPathCredentials ¶

AWSDatabricksPathCredentials(
    url: str, *, client: Any = None, secret_cache: bool = False
)

Bases: _DatabricksCredentialsBase

Refreshable AWS creds for a Unity Catalog storage URL.

Singleton-cached per url. Vends through temporary_path_credentials.generate_temporary_path_credentials, which — unlike the service-credential endpoint — works for the storage credential backing an external location. One provider serves read (PATH_READ) and write (PATH_READ_WRITE) modes; the requested mode resolves at :meth:get_credentials / :meth:aws_client time.

The URL is normalised to a directory prefix (trailing /) so every key under the same external-location prefix collapses to one provider, one refresh cycle, and one boto session.

client `property` ¶

client: 'DatabricksClient'

Bound :class:DatabricksClient. Lazily resolves to :meth:DatabricksClient.current when no client was passed.

workspace `property` ¶

workspace: Any

Shortcut for self.client.workspace_client().

get_credentials ¶

get_credentials(mode: ModeLike = None) -> AwsCredentials

Return fresh credentials scoped to mode.

mode accepts a :class:Mode enum, a mode-like string ("read", "overwrite", …), or None (uses :attr:DEFAULT_MODE). Read-only modes vend the UC "read" operation; everything else vends the writable operation.

If the SDK rejects the call with PermissionDenied: ... EXTERNAL USE SCHEMA on Schema 'cat.sch', we make exactly one attempt to self-grant EXTERNAL_USE_SCHEMA on the offending schema and retry. Owners of UC schemas commonly forget this grant — when they own the schema they have permission to fix it, and silently succeeding is dramatically less surprising than asking them to read the error and run a follow-up SQL. If the recovery itself fails (non-owner, network error, …) the original PermissionDenied propagates so the failure mode stays obvious.

aws_client ¶

aws_client(
    *, mode: ModeLike = None, region: Optional[str] = None
) -> "AWSClient"

Return the cached :class:AWSClient for this provider / mode / region.

First call per (mode, region) seeds a botocore :class:RefreshableCredentials-backed session whose refresher is bound to mode; subsequent calls with the same key return the same client and reuse the connection pool, boto-client cache, and in-flight refresh state.

with_client ¶

with_client(client: Any) -> '_DatabricksCredentialsBase'

Replace the bound client. Returns self.

yggdrasil.databricks.aws¶

aws ¶

AWSDatabricksVolumeCredentials ¶

client property ¶

workspace property ¶

get_credentials ¶

aws_client ¶

with_client ¶

AWSDatabricksTableCredentials ¶

client property ¶

workspace property ¶

get_credentials ¶

aws_client ¶

with_client ¶

AWSDatabricksPathCredentials ¶

client property ¶

workspace property ¶

get_credentials ¶

aws_client ¶

with_client ¶

`yggdrasil.databricks.aws`¶

client `property` ¶

workspace `property` ¶

client `property` ¶

workspace `property` ¶

client `property` ¶

workspace `property` ¶