`yggdrasil.databricks.schema`¶

schema ¶

Unity Catalog schema resource + service.

UCSchema ¶

UCSchema(
    data: Any = None,
    service: "Schemas | None" = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    *,
    infos_ttl: float | None = None,
    infos: SchemaInfo | None = None,
    infos_fetched_at: float | None = None,
    url: URL | None = None,
    path_prefix: str | None = None,
    singleton_ttl: "int | None" = ...
)

Bases: DatabricksPath

A single Unity Catalog schema — lifecycle, table navigation, tags.

Identity is (client, catalog_name, schema_name, path_prefix): two callers asking for the same schema on the same navigation surface under the same client collapse onto one instance via the :class:Singleton cache, so the cached :class:SchemaInfo and tag state are shared. :attr:path_prefix (inherited from the parent catalog) fixes what a / path-join descends into — /Volumes/ → :class:Volume (then :class:VolumePath), /Tables/ → :class:Table — so the child type is known up front rather than guessed.

URL-addressable through :class:DatabricksPath under :attr:Scheme.DATABRICKS_SCHEMA (dbfs+schema://); the Path / Holder byte primitives raise — a schema is a logical UC resource, not a positional byte buffer. Mirrors the same (DatabricksPath, Singleton) shape that :class:Catalog uses.

sql `property` ¶

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened `property` ¶

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed `property` ¶

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url `property` `writable` ¶

url: 'URL'

Canonical URL identifying this holder.

size_known `property` ¶

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite `property` ¶

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type `property` `writable` ¶

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming `property` ¶

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest `property` ¶

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder `property` ¶

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder `property` ¶

owns_holder: bool

Whether closing self also closes the bound parent.

mode `property` ¶

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client `property` ¶

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

explore_url `property` ¶

explore_url: URL

Workspace UI URL pointing at this schema's Catalog Explorer page.

catalog `property` ¶

catalog: 'UCCatalog'

Navigate up to the parent :class:UCCatalog.

The parent inherits this schema's :attr:path_prefix so a round-trip up-then-down (schema.catalog / schema_name) lands back on the same surface — and the same singleton.

tags `property` ¶

tags: tuple[Any, ...]

Schema-level entity-tag assignments — served from client.entity_tags.

open ¶

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit ¶

commit()

Commit current state

rollback ¶

rollback()

Rollback current state

close ¶

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty ¶

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme `classmethod` ¶

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch `classmethod` ¶

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_url ¶

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton ¶

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton ¶

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type `classmethod` ¶

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static ¶

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ `classmethod` ¶

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

options_class `classmethod` ¶

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options `classmethod` ¶

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup ¶

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize ¶

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

collect_schema ¶

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count ¶

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches ¶

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table ¶

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader ¶

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table ¶

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table ¶

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union ¶

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset ¶

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator ¶

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records ¶

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique ¶

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters¶

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns¶

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample ¶

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters¶

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns¶

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select ¶

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop ¶

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter ¶

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast ¶

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display ¶

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy ¶

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath ¶

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes `classmethod` ¶

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder `classmethod` ¶

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder `classmethod` ¶

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

The explicit media_type kwarg, when supplied.
holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes `classmethod` ¶

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv ¶

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv ¶

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve ¶

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

resize ¶

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate ¶

truncate(n: int) -> int

Resize the object to exactly n bytes.

Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

stat ¶

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime ¶

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire ¶

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush ¶

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread ¶

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite ¶

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview ¶

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv ¶

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes ¶

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes ¶

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text ¶

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text ¶

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head ¶

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail ¶

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto ¶

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline ¶

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines ¶

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell ¶

tell() -> int

Current cursor position.

seek ¶

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path ¶

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream ¶

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder ¶

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload ¶

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

:class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
:class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download ¶

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes ¶

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue ¶

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode ¶

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64 ¶

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64 ¶

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64 ¶

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes ¶

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream ¶

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream ¶

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable ¶

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type ¶

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media ¶

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno ¶

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read ¶

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall ¶

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write ¶

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32 ¶

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32 ¶

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load ¶

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

Explicit media_type kwarg.
Cached :attr:media_type on the IO.
Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress ¶

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls ¶

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink ¶

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove ¶

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone ¶

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch ¶

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module ¶

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module ¶

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file ¶

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range ¶

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io ¶

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

from_url `classmethod` ¶

from_url(url: 'URL | str', **kwargs: Any) -> 'UCSchema'

Build a :class:Schema from a dbfs+volume:///cat/sch URL.

Used by the :class:DatabricksPath dispatcher when a caller passes a POSIX volume path that resolves to schema depth (DatabricksPath("/Volumes/main/sales") → Schema("main", "sales")).

full_name ¶

full_name(safe: str | bool | None = None) -> str

Return the two-part schema name, optionally backtick-quoted.

clear ¶

clear() -> 'UCSchema'

Public alias for :meth:_reset_cache; returns self.

exists ¶

exists() -> bool

True if this schema is reachable via the Unity Catalog API.

table ¶

table(name: str) -> 'Table'

Return a :class:Table within this schema.

Parameters:

Name	Type	Description	Default
`name`	`str`	Table name (unqualified).	required

tables ¶

tables(name: str | None = None) -> Iterator['Table']

Iterate over tables in this schema, optionally filtered by name.

create ¶

create(
    *,
    comment: str | None = None,
    properties: Optional[Mapping[str, str]] = None,
    storage_root: str | None = None,
    missing_ok: bool = True
) -> "UCSchema"

Create this schema in Unity Catalog.

Parameters:

Name	Type	Description	Default
`comment`	`str \| None`	Human-readable description.	`None`
`properties`	`Optional[Mapping[str, str]]`	Extra key/value properties.	`None`
`storage_root`	`str \| None`	External storage root URI.	`None`
`missing_ok`	`bool`	Silently succeed if the schema already exists.	`True`

get_or_create ¶

get_or_create(
    *,
    comment: str | None = None,
    properties: Optional[Mapping[str, str]] = None,
    storage_root: str | None = None
) -> "UCSchema"

Create this schema (and any missing parent catalog) if it doesn't exist, then return self. :meth:create is itself idempotent and ensures the parent on a not-found, so this is just a named alias.

delete ¶

delete(
    *,
    force: bool = False,
    wait: WaitingConfigArg = True,
    raise_error: bool = True
) -> "UCSchema"

Delete this schema from Unity Catalog.

Parameters:

Name	Type	Description	Default
`force`	`bool`	Cascade-delete all child tables.	`False`
`wait`	`WaitingConfigArg`	Block until the API call returns.	`True`
`raise_error`	`bool`	Re-raise :exc:`DatabricksError` on failure.	`True`

clone ¶

clone(
    target: "str | UCSchema | None" = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    deep: bool = True,
    mode: ModeLike = Mode.IGNORE,
    include_views: bool = True,
    name: str | None = None,
    max_workers: int | None = None
) -> dict[str, str]

Clone every child of this schema into target, in parallel.

The target schema (and any missing parent catalog) is created first, then each child table / view is cloned concurrently. mode is the single existence policy, applied uniformly to every sub-clone:

IGNORE (default) — CREATE … IF NOT EXISTS: a child already present is skipped (left untouched), a missing one is created.
OVERWRITE / TRUNCATE — CREATE OR REPLACE: overwrite same-named targets.
ERROR_IF_EXISTS — plain CREATE: record a failure per clash.

A target that exists but changed kind (table ⇄ view) can't be cross-replaced by Delta, so it is dropped first and recreated as the source's current kind (except under ERROR_IF_EXISTS, which lets the clash surface as a failure). One child's failure is recorded and never aborts the rest of the batch.

Parameters:

Name	Type	Description	Default
`target`	`'str \| UCSchema \| None'`	Destination schema — a :class:`UCSchema`, a `"catalog.schema"` / `"schema"` dotted name, or `None` when catalog_name / schema_name are passed explicitly. A bare name reuses this schema's catalog.	`None`
`catalog_name`	`str \| None`	Target catalog override (defaults to this schema's).	`None`
`schema_name`	`str \| None`	Target schema override.	`None`
`deep`	`bool`	DEEP clone (independent copy) vs SHALLOW (metadata only, shares the source's files).	`True`
`mode`	`ModeLike`	Existence policy (`Mode` or mode-like string) forwarded to every sub-clone — see above. Defaults to `IGNORE` (skip what's already there).	`IGNORE`
`include_views`	`bool`	Also clone view-shaped children (re-emitting their definition); `False` clones only tables.	`True`
`name`	`str \| None`	Optional child-name filter (exact or glob) — clone a subset.	`None`
`max_workers`	`int \| None`	Thread-pool size for the fan-out (defaults to the child count, capped at 16).	`None`

Returns:

Type	Description
`dict[str, str]`	`{table_name: status}` where status is `"created"`,
`dict[str, str]`	`"skipped"` (already present), or `"failed: <error>"`.

set_tags_ddl ¶

set_tags_ddl(
    tags: Mapping[str, str] | None,
    *,
    tag_collation: str | None = DEFAULT_TAG_COLLATION
) -> str

Build an ALTER SCHEMA … SET TAGS DDL statement.

Retained for dry-run / logging contexts; :meth:set_tags no longer executes this DDL — it goes through the entity_tag_assignments REST API instead.

set_tags ¶

set_tags(
    tags: Mapping[str, str] | None,
    *,
    tag_collation: str | None = DEFAULT_TAG_COLLATION,
    mode: ModeLike | None = None
) -> "UCSchema"

Apply schema-level tags via the UC entity_tag_assignments API.

tag_collation is accepted for API compatibility and ignored — collations only matter for the legacy DDL literal form.

mode selects the write strategy ("upsert" default, "overwrite" for strict replace, "append" / "ignore" / "error_if_exists").

unset_tags ¶

unset_tags(tag_keys: Iterable[str], *, if_exists: bool = True) -> 'UCSchema'

Delete schema-level tag assignments by key.

permissions ¶

permissions(*, principal: str | None = None) -> tuple[PrivilegeAssignment, ...]

Direct grants on this schema (no inherited privileges).

Calls the Unity Catalog grants.get endpoint.

Parameters:

Name	Type	Description	Default
`principal`	`str \| None`	Optional filter — return only grants for this user / group / service principal.	`None`

Returns:

Type	Description
`PrivilegeAssignment`	Tuple of :class:`PrivilegeAssignment` (one per principal
`...`	with at least one direct grant).

effective_permissions ¶

effective_permissions(*, principal: str | None = None) -> tuple[Any, ...]

Effective grants on this schema, including privileges inherited from the parent catalog / metastore.

Calls the Unity Catalog grants.get_effective endpoint.

can_use_external ¶

can_use_external(*, refresh: bool = False) -> bool

True when the current user holds EXTERNAL USE SCHEMA on this schema (directly or inherited from the catalog / metastore).

That privilege is Unity Catalog's prerequisite for touching an external securable's backing cloud storage directly — so callers gate a direct storage-path read / write on it and otherwise fall back to the Files API. The result is cached on this (singleton) schema, so a single grants.get_effective decides for the whole process; refresh forces a re-check. Any lookup failure (no current-user, denied grants read, …) resolves to False without raising — the safe default is "go through the Files API".

mark_external_unusable ¶

mark_external_unusable() -> None

Record that direct external-storage access didn't actually work for this schema (e.g. the grant is present but the bucket policy denied the I/O) so callers stop re-trying it and route through the Files API.

grant ¶

grant(
    principal: str, privileges: "str | Privilege | Iterable[str | Privilege]"
) -> "UCSchema"

Add one or more privileges for principal on this schema.

Privileges may be passed as :class:Privilege enums or as strings (case-insensitive, - / spaces accepted in place of _). Example::

schema.grant("alice@example.com", "EXTERNAL USE SCHEMA")
schema.grant("data-engs", [Privilege.USE_SCHEMA, "SELECT"])

revoke ¶

revoke(
    principal: str, privileges: "str | Privilege | Iterable[str | Privilege]"
) -> "UCSchema"

Remove one or more privileges for principal on this schema.

set_permissions ¶

set_permissions(
    principal: str, privileges: "str | Privilege | Iterable[str | Privilege]"
) -> "UCSchema"

Replace principal's direct grants on this schema with exactly privileges.

Computes the diff against the current direct grants and emits a single grants.update call that adds the missing privileges and removes the extras. Inherited grants are not touched (they belong to the parent securable).

update_permissions ¶

update_permissions(
    changes: "Iterable[PermissionsChange | Mapping[str, Any]]",
) -> "UCSchema"

Apply a batch of PermissionsChange to this schema.

Accepts :class:PermissionsChange instances or plain mappings ({"principal": ..., "add": [...], "remove": [...]}). Empty / no-op changes are filtered out before the API call.

update ¶

update(
    *,
    comment: str | None = None,
    owner: str | None = None,
    properties: Optional[Mapping[str, str]] = None
) -> "UCSchema"

Update schema metadata in-place and refresh the local cache.

rename ¶

rename(new_name: str) -> 'UCSchema'

Rename this schema in-place (ALTER SCHEMA … RENAME TO …).

The catalog parent is unchanged; new_name is the unqualified schema name.

Schemas ¶

Schemas(
    client=None, catalog_name: str | None = None, schema_name: str | None = None
)

Bases: DatabricksService

Collection-level service for Unity Catalog schemas.

Attach a default catalog_name (and optional schema_name) so callers don't have to repeat them on every call::

schemas = client.schemas(catalog_name="main")

# navigate by subscript
schema = schemas["sales"]                    # Schema ("main.sales")
schema = schemas["main.sales"]               # Schema  (fully qualified)
table  = schemas["main.sales.orders"]        # Table

# explicit factory methods
schema  = schemas.schema("main.sales")       # Schema
table   = schemas.table("main.sales.orders") # Table
catalog = schemas.catalog("main")            # Catalog

# list
for sch in schemas.list():
    for tbl in sch.tables():
        ...

wheels `property` ¶

wheels: 'Wheels'

Wheel registry service (shorthand for client.wheels).

environments `property` ¶

environments: 'Environments'

Base-environment service (shorthand for client.environments).

tables `property` ¶

tables: 'Tables'

Collection-level Unity Catalog table service (shorthand for client.tables).

views `property` ¶

views: 'Tables'

Alias for :attr:tables — :class:Table covers both managed/external tables and view-shaped securables.

catalogs `property` ¶

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service (shorthand for client.catalogs).

schemas `property` ¶

schemas: 'Schemas'

Collection-level Unity Catalog schema service (shorthand for client.schemas).

volumes `property` ¶

volumes: 'Volumes'

Collection-level Unity Catalog volume service (shorthand for client.volumes).

genie `property` ¶

genie: 'Genie'

Genie service (shorthand for client.genie).

ai `property` ¶

ai: 'DatabricksAI'

Databricks AI umbrella service (shorthand for client.ai).

default_tags ¶

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

Returns:

Type	Description
`dict[str, str]`	A dict of default tags.

schema ¶

schema(
    location: "UCSchema | str | None" = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> UCSchema

Return a :class:UCSchema bound to this service.

Uses the module-level cache when available.

Parameters:

Name	Type	Description	Default
`location`	`'UCSchema \| str \| None'`	Two-part dotted `"catalog.schema"` name (optional if catalog_name / schema_name are provided).	`None`
`catalog_name`	`str \| None`	Override catalog (falls back to `self.catalog_name`).	`None`
`schema_name`	`str \| None`	Override schema (falls back to `self.schema_name`).	`None`

table ¶

table(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    table_name: str | None = None
) -> Table

Return a :class:Table via the client-level tables service.

Parameters:

Name	Type	Description	Default
`location`	`str \| None`	Three-part dotted name `"catalog.schema.table"` (or fewer parts, filled by service defaults).	`None`
`catalog_name`	`str \| None`	Override catalog.	`None`
`schema_name`	`str \| None`	Override schema.	`None`
`table_name`	`str \| None`	Override table.	`None`

catalog ¶

catalog(name: str | None = None) -> UCCatalog

Return a :class:Catalog using this service's client.

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Catalog name (falls back to `self.catalog_name`).	`None`

list ¶

list(
    name: str | None = None,
    *,
    catalog_name: str | None = None,
    use_cache: bool = True
) -> Iterator[UCSchema]

Iterate over visible schemas, optionally scoped to a catalog.

Parameters:

Name	Type	Description	Default
`name`	`str \| None`	Optional schema-name filter. When it contains ``, matching uses a case-insensitive glob (e.g. `"sales_"`, `"_raw"`, `""`).	`None`
`catalog_name`	`str \| None`	Catalog to list (falls back to `self.catalog_name`). When `None` the iteration fans out across every visible catalog. When it contains `*` the iteration fans out across catalogs matching the glob.	`None`
`use_cache`	`bool`	Populate `_SCHEMA_INFO_CACHE` from results.	`True`

find_remote ¶

find_remote(
    catalog_name: str, schema_name: str, *, raise_error: bool = True
) -> Optional[SchemaInfo]

Raw API lookup — GET by fully-qualified name, no cache.

Returns None on miss when raise_error=False.

find ¶

find(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    raise_error: bool = True,
    cache_ttl: float | None = 300.0
) -> Optional[UCSchema]

Resolve a schema by name.

Caching is controlled by cache_ttl. Set cache_ttl=None to bypass the cache for this lookup.

Parameters:

Name	Type	Description	Default
`location`	`str \| None`	Two-part dotted `"catalog.schema"` name.	`None`
`catalog_name`	`str \| None`	Override catalog (falls back to service default).	`None`
`schema_name`	`str \| None`	Override schema (falls back to service default).	`None`
`raise_error`	`bool`	Raise :exc:`ResourceDoesNotExist` when not found.	`True`
`cache_ttl`	`float \| None`	Entry TTL in seconds (`None` → bypass cache).	`300.0`

parse_location ¶

parse_location(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> tuple[Optional[str], Optional[str]]

Parse a 1- or 2-part dotted name into (catalog, schema).

Keyword overrides take precedence over parts extracted from location. Service defaults fill any remaining blanks.

invalidate ¶

invalidate(
    schema: UCSchema | str | None = None,
    *,
    catalog_name: Optional[str] = None,
    schema_name: Optional[str] = None
) -> None

Evict one entry from the module-level schema-info cache.

invalidate_all `classmethod` ¶

invalidate_all() -> None

Clear the entire module-level schema-info cache.

yggdrasil.databricks.schema¶

schema ¶

UCSchema ¶

sql property ¶

opened property ¶

closed property ¶

url property writable ¶

size_known property ¶

holder_is_overwrite property ¶

media_type property writable ¶

is_streaming property ¶

xxh3_64_digest property ¶

holder property ¶

owns_holder property ¶

mode property ¶

workspace_client property ¶

explore_url property ¶

catalog property ¶

tags property ¶

open ¶

commit ¶

rollback ¶

close ¶

mark_dirty ¶

for_scheme classmethod ¶

dispatch classmethod ¶

to_url ¶

to_singleton ¶

invalidate_singleton ¶

class_for_media_type classmethod ¶

matches_static ¶

from_ classmethod ¶

options_class classmethod ¶

check_options classmethod ¶

cleanup ¶

optimize ¶

collect_schema ¶

count ¶

scan_arrow_batches ¶

scan_arrow_table ¶

scan_arrow_batch_reader ¶

read_table ¶

write_table ¶

union ¶

read_spark_dataset ¶

read_record_iterator ¶

read_records ¶

unique ¶

Parameters¶

Returns¶

resample ¶

Parameters¶

Returns¶

select ¶

drop ¶

filter ¶

cast ¶

display ¶

lazy ¶

joinpath ¶

from_bytes classmethod ¶

from_holder classmethod ¶

for_holder classmethod ¶

registered_classes classmethod ¶

read_mv ¶

write_mv ¶

reserve ¶

resize ¶

truncate ¶

stat ¶

touch_mtime ¶

acquire ¶

flush ¶

pread ¶

pwrite ¶

memoryview ¶

iter_mv ¶

read_bytes ¶

write_bytes ¶

read_text ¶

`yggdrasil.databricks.schema`¶

sql `property` ¶

opened `property` ¶

closed `property` ¶

url `property` `writable` ¶

size_known `property` ¶

holder_is_overwrite `property` ¶

media_type `property` `writable` ¶

is_streaming `property` ¶

xxh3_64_digest `property` ¶

holder `property` ¶

owns_holder `property` ¶

mode `property` ¶

workspace_client `property` ¶

explore_url `property` ¶

catalog `property` ¶

tags `property` ¶

for_scheme `classmethod` ¶

dispatch `classmethod` ¶

class_for_media_type `classmethod` ¶

from_ `classmethod` ¶

options_class `classmethod` ¶

check_options `classmethod` ¶

from_bytes `classmethod` ¶

from_holder `classmethod` ¶

for_holder `classmethod` ¶

registered_classes `classmethod` ¶

from_url `classmethod` ¶

wheels `property` ¶

environments `property` ¶

tables `property` ¶