yggdrasil.databricks.table

table

Unity Catalog table resource + service.

DatabricksTableInsert dataclass

DatabricksTableInsert(
    target: "Table | str",
    mode: Mode,
    data: "Tabular | Path | str",
    client: "DatabricksClient | None" = None,
    schema: "Field | None" = None,
    predicate: "Predicate | None" = None,
    match_by: "list[str] | None" = None,
    update_column_names: "list[str] | None" = None,
    schema_mode: "Mode | str | None" = None,
    zorder_by: "list[str] | None" = None,
    optimize_after_merge: bool = False,
    vacuum_hours: "int | None" = None,
    safe_merge: bool = False,
)

Bases: _InsertExecution

One insert operation — the full arrow_insert surface in one object.

Carries the target table, the save mode, the staged data location (a :class:Path / :class:VolumePath, or a uniform-URL string), and the keyed-write surface (schema, predicate, match_by, update_column_names, schema_mode, zorder_by, optimize_after_merge, vacuum_hours, safe_merge).

target may be a :class:Table or its full name; data is the staged Parquet source — a :class:Path / :class:VolumePath, or its uniform URL as a string (reconstructed through the bound client at execute time).

result property

result: Any

The inner :class:StatementBatch driving the load (None until :meth:start / :meth:execute).

target_name property

target_name: str

catalog.schema.table — the resolved target name.

progress

progress() -> 'float | None'

A 0..1 completion fraction for a progress bar, or None if unknown.

A UI hook: a generic awaitable can't know its fraction, so the base returns None (drive a spinner, not a bar). Subclasses that do know — a batch's children done, a statement's rows fetched — override this. Consumed by :func:yggdrasil.cli.style.track.

watch

watch(
    on_tick: "Any" = None, *, interval: float = 0.1, raise_error: bool = True
) -> "Awaitable"

Drive to completion, calling on_tick(self) each poll.

The hook a UI (spinner / progress bar) connects to without this trait importing any UI — keeping the layering clean. Starts the awaitable if it hasn't been, polls until done, then surfaces a failure (unless raise_error is False). Pairs with :func:yggdrasil.cli.style.track.

execute

execute(
    *,
    target: Any = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    engine: Any = None,
    retry: WaitingConfigArg | None = None
) -> "_InsertExecution"

Build and run this insert. target rebinds the destination :class:Table; engine forces the "api" / "spark" backend, retry a per-statement retry policy. With wait (default) blocks until the statements finish; wait=False fires them and returns immediately (poll via :meth:wait / await).

data_path

data_path(client: Any = None) -> 'Path'

Resolve the staged file data to a concrete :class:Path.

Already a :class:Path → returned as-is; a uniform-URL string → reconstructed through the bound (or supplied) client so the warehouse can read it wherever it landed.

staged_source

staged_source(client: Any = None) -> Any

Rebuild the staged data into the concrete :class:Path the warehouse reads. A live :class:Path is returned unchanged; a serialized URL is rebuilt through the bound (or supplied) client.

select_sql

select_sql(client: Any = None) -> str

Back-compat alias for :func:make_sql_select over this op.

TableOptions dataclass

TableOptions(
    source: "Field | None" = None,
    target: "Field | None" = None,
    safe: bool = False,
    checked_cast: bool = False,
    mode: Mode = Mode.AUTO,
    schema_mode: Mode = Mode.AUTO,
    row_size: int | None = None,
    byte_size: int | None = None,
    row_limit: int | None = None,
    use_threads: bool = True,
    match_by: list["Field"] | None = None,
    unique_by: list["Field"] | None = None,
    time_sample_by: list["Field"] | None = None,
    fill_strategy: str = "ffill",
    predicate: Predicate | None = None,
    wait: WaitingConfig = WaitingConfig.default(),
    spark_session: "SparkSession | None" = None,
    arrow_memory_pool: MemoryPool | None = None,
    update_column_names: list[str] | None = None,
    zorder_by: list[str] | None = None,
    optimize_after_merge: bool = False,
    vacuum_hours: int | None = None,
    retry: WaitingConfigArg | None = None,
    return_data: bool = False,
    safe_merge: bool = False,
    sync_metadata: bool = True,
    engine: "EngineType | None" = None,
)

Bases: CastOptions

:class:CastOptions for a Unity Catalog :class:Table.

Inherits the full cast / projection / predicate / merge-maintenance surface (target, predicate, row_limit, mode, match_by, zorder_by, vacuum_hours, …) and adds the table-only routing knob:

• :attr:engine — pick the read/write compute (an :class:EngineType):

• :attr:~EngineType.YGGDRASIL — yggdrasil's native DeltaFolder (a direct _delta_log + parquet path over UC-vended credentials) when the table is Delta-backed. Native writes need an external Delta table (UC vends read-only credentials for managed tables); a non-Delta or managed-Delta write falls back to the warehouse.

• :attr:~EngineType.DATABRICKS_SQL_WAREHOUSE — the SQL warehouse.
• :attr:~EngineType.SPARK — a Spark session.
• None (default) — guess best per call: an active Spark session → SPARK; otherwise DATABRICKS_SQL_WAREHOUSE. The native DeltaFolder path is never auto-selected — it bypasses the warehouse (and its governance / staging), so it is taken only when requested explicitly with engine=YGGDRASIL.

In every case, if the native path can't get UC credentials for the table's storage, the read/write transparently falls back to the warehouse.

merged property

merged: Field | None

Target reconciled with source under :attr:schema_mode.

With the default :attr:Mode.AUTO, a target field that matches a source field by name is merged — so variant (ObjectType / NullType) target slots adopt the source dtype — while the target's field set is preserved (source-only columns are not pulled in). This is the schema casts coerce to: see the cast_* dispatchers, which run against merged rather than the raw target so a bare columns= projection autotypes against the bound source.

column_names property

column_names: list[str] | None

The target field's column names, if a target field is bound.

match_by_keys property

match_by_keys: list[str] | None

Resolved key column names to dedup on.

Pulls the :attr:Field.name of each entry in :attr:match_by. Returns None when no keys are set so callers can branch on "keys vs no-keys" with a single truthiness check.

select_source_column_names

select_source_column_names() -> list[str] | None

The source field's column names, if a source field is bound.

read_columns

read_columns() -> list[str] | None

Columns a source reader must keep — the projection plus the predicate's columns.

:attr:column_names is what the read should end up with, but the predicate row-filter runs before the cast projects down to it, so any column the predicate touches has to survive the read even when the caller didn't ask for it (columns=["a"] + predicate on b). None means no projection — read everything.

check classmethod

check(options: CastOptionsArg = None, /, **overrides: Any) -> T

Canonical entry point — coerce anything into a :class:CastOptions.

Dispatch by what options is:

• None — construct fresh :class:CastOptions(**overrides).
• :class:CastOptions — if no overrides given, return it unchanged; if overrides given, .copy(**overrides).
• :class:Mapping (including dict) — merge into overrides and construct fresh (override args win on key collision).
• :class:pa.DataType / :class:pa.Field / :class:pa.Schema / :class:Field / :class:Schema — treat as a target hint. Equivalent to check(target=options, **overrides).

source= / target= go straight into the dataclass slots (after :meth:Field.from_ normalization in __post_init__). Callers that want the peek-and-bind "only set if not already bound" semantic should chain :meth:check_source / :meth:check_target after the call.

columns= shortcut: a sequence of column names describing the desired output projection. When a target is already bound the names narrow it (target.select(columns)); otherwise they are promoted to a struct-shaped target field whose children default to :class:ObjectType — a "I want these columns, leave their types alone" placeholder that drives projection without casting. It lands on target (not source) so it never shadows the real source schema inferred at read time.

:raises TypeError: if options is a type the dispatch table doesn't cover.

field_names cached classmethod

field_names() -> frozenset[str]

Frozenset of this class's constructor-accepting field names.

Used by :meth:_build to filter **overrides down to keys the constructor will accept — callers funnel mixed kwargs through .check() (DataIO public methods often pass user kwargs straight through), and we don't want a stray filter= or columns= to crash construction.

Excludes init=False fields (the private memoization slots for merged / merged_schema); those are not valid __init__ keywords and a copy via dataclasses.replace would crash if it tried to forward them.

Cached per-class via :func:functools.cache so subclasses with extra fields get their own expanded set on first access.

copy

copy(**overrides: Any) -> T

Return a copy with overrides applied.

... values in overrides are ignored (keep existing). Pass source=/target= to swap either slot — :class:Field normalization runs in __post_init__ so any :class:Field-shaped input (pa.Schema, pa.DataType, dict, …) is accepted.

Implementation note: bypasses :func:dataclasses.replace, which rebuilds via cls(**all_fields) and pays a full __init__ + __post_init__ traversal even when the caller only tweaked a single bool. Cast pipelines call copy repeatedly per batched write (with_source / check_source / with_target all funnel through here), so the fast clone below — :func:object.__new__ + slot copy + targeted __post_init__ normalization for the overridden keys — is meaningfully cheaper.

check_source

check_source(obj: Any = None, *, copy: bool = True) -> T

Bind a :attr:source if one isn't already set.

Two ways to supply one:

1. source= on :meth:check / :meth:copy — explicit Field / Schema / pa type. Wins even if self.source is already set (explicit override).
2. obj= here — a peekable object. Only runs the peek when self.source is currently None — an already- bound field is never clobbered by a peek.

Returns self unchanged when neither is given. Used from :class:DataIO methods (collect_schema, read_arrow_dataset) that want to pin a source schema before running a batch walk.

checked_cast=True short-circuits — the caller guarantees the batch shape matches the target, so the peek (which would rebuild a yggdrasil :class:Field from the batch's :class:pa.Schema) is wasted work. Combined with the :meth:cast_arrow_tabular short-circuit, this collapses every per-batch cast pass to a single attribute read on the leaf write path — ~150 us / batch saved on a RESPONSE_SCHEMA-shaped write.

check_target

check_target(obj: Any = ..., *, copy: bool = True) -> T

Bind a :attr:target if one isn't already set.

Symmetry partner for :meth:check_source. See that method for the argument semantics — source/target behave identically.

with_source

with_source(source: 'Field', copy: bool = False) -> T

Return a copy with source as the new source field.

Accepts the same shapes :meth:Field.from_ does (pa schema, yggdrasil Field, dict, etc.) — normalized in __post_init__ via :func:dataclasses.replace. The frozen slot is updated through :func:object.__setattr__ in the post-init hook; we don't bypass it here because going through replace gets the normalization for free.

with_target

with_target(target: 'Field', copy: bool = True) -> T

Return a copy with target as the new target field.

with_checked_cast

with_checked_cast(value: bool = True, copy: bool = False) -> T

Return a copy (or in-place) with :attr:checked_cast set.

Mirror of :meth:with_source / :meth:with_target — keeps the per-call mutation behind a named method instead of having every writer-side caller reach for :func:dataclasses.replace / :func:object.__setattr__. Set when the caller knows every batch already matches the target (came from a :class:pa.Table, a :class:pa.RecordBatchReader, a polars / pandas frame, or another writer that just emitted the same schema); the leaf's :meth:check_source / :meth:cast_arrow_tabular then short-circuit straight to the write path.

need_cast

need_cast(
    source: Any | None = None,
    target: Any | None = None,
    check_names=False,
    check_dtypes=True,
    check_metadata=False,
    check_nullable: bool = False,
) -> bool

Return True if source and target fields differ enough to need casting.

When either field is unbound, returns False — there's nothing to compare against, so assume caller already sorted it.

Field equality semantics are the :meth:Field.equals rules: names, dtypes, metadata — each independently gateable. Metadata is off by default because it's commonly decorative (pandas preserves indices through metadata, arrow carries codec hints in field metadata) and comparing on it would demand a cast for cosmetic differences.

check_nullable is off by default because nullability rarely warrants a real value-level cast — primitives and lists pass through unchanged when only the flag differs. Tabular / struct casts pass check_nullable=True so the rebuild fires when child fields differ on nullability: Spark / Delta refuse to implicitly cast nullable→NOT NULL inside a struct (even when the data is in fact non-null), so the cast has to emit the target's field types verbatim to keep MERGE happy.

finalize

finalize(obj: Any, *, default_scalar: Any = None) -> Any

Finalize any object — delegates to :meth:Field.finalize.

finalize_spark_cast

finalize_spark_cast(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls and alias a Spark Column to the target name.

Direct parallel of :meth:finalize_polars_cast — Spark Columns, like polars Series/Expr, carry a name that can diverge from the target after a cast chain, so the alias step belongs in finalize rather than in each cast site.

finalize_arrow_cast

finalize_arrow_cast(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls on a pyarrow object to finish a cast chain.

No alias step: :class:pa.Array / :class:pa.ChunkedArray don't carry a name, and tabular rename (Table/RecordBatch) is a schema-level rebuild that :meth:cast_arrow_tabular already handles inline via the target schema. Finalize here just means "apply the default-scalar null fill."

finalize_pandas_cast

finalize_pandas_cast(obj: Any, *, default_scalar: Any = None) -> Any

Fill nulls on a pandas object to finish a cast chain.

No alias step exposed on :class:CastOptions for pandas — Series .name and DataFrame column labels get set by the cast methods directly. Finalize is fill-only, matching :meth:finalize_arrow_cast.

cast

cast(obj: Any) -> Any

Cast obj to :attr:target using its native engine.

Dispatches arrow types through :meth:cast_arrow, Tabular through :meth:cast_tabular. Everything else delegates to :meth:Field.cast.

cast_pyarrow

cast_pyarrow(obj: Any) -> Any

Cast any pyarrow object — delegates to :meth:Field.cast_arrow.

cast_arrow_array

cast_arrow_array(array: Any) -> Any

Cast a :class:pa.Array or :class:pa.ChunkedArray.

cast_arrow_batch

cast_arrow_batch(batch: 'pa.RecordBatch') -> 'pa.RecordBatch'

Filter + cast a :class:pa.RecordBatch.

cast_arrow_table

cast_arrow_table(table: 'pa.Table') -> 'pa.Table'

Filter + cast a :class:pa.Table.

cast_arrow_tabular

cast_arrow_tabular(data: 'ArrowTabular') -> 'ArrowTabular'

Filter + cast an :class:ArrowTabular (batch by batch).

cast_arrow

cast_arrow(data: Any) -> Any

Dispatch arrow types to the specific cast method.

cast_tabular

cast_tabular(data: Any) -> Any

Cast any Tabular-like object.

dedup_columns_on_read

dedup_columns_on_read() -> 'list[str]'

Return the column names that need client-side dedup at read time.

Sourced from :attr:unique_by — each Field's :attr:Field.name is the column the read pass must deduplicate on. Returns an empty list when :attr:unique_by is unset / empty.

dedup_arrow_batches

dedup_arrow_batches(
    batches: "Iterator[pa.RecordBatch]",
) -> "Iterator[pa.RecordBatch]"

Collapse duplicate rows on the columns flagged unique.

Resolves the dedup column set via :meth:dedup_columns_on_read, then delegates to :func:yggdrasil.arrow.ops.dedup_arrow_batches for the pure-Arrow group-by + take pass. Identity short-circuit when no column needs collapsing keeps the read path zero-cost on the common case (no target / no unique column / source already unique).

resample_on_read

resample_on_read() -> 'tuple[str, int, list[str], str] | None'

Return (time_column, sampling_seconds, partition_by, fill_strategy) to resample.

Picks the first entry of :attr:time_sample_by whose time_sampling metadata carries a positive ISO-8601 duration. The result drives :func:yggdrasil.arrow.ops.resample_arrow_table — a single (column, interval) is all that op consumes (you can only have one time axis per table to resample on at a time).

Each Field's sampling lives under its :attr:Field.metadata's non-prefixed b"time_sampling" key as an ISO-8601 duration string ("PT1H" / "P1D"). The non-prefixed key keeps the value off the schema-level tag registry (it's a per-call option, not a contract that rides with the data on disk).

partition_by is derived from the target schema's :attr:Field.primary_key set, minus the resample column itself if it's also primary. The rationale: on a per-entity time series (one symbol per row, partitioned by symbol), each entity's timeline should bucket independently — without partition_by the resample would collapse rows across instruments. Schemas with no primary keys (or where the only primary is the timestamp) fall back to a flat resample.

Returns None when :attr:time_sample_by is unset / empty or every listed Field's metadata fails to parse.

resample_arrow_batches

resample_arrow_batches(
    batches: "Iterator[pa.RecordBatch]",
) -> "Iterator[pa.RecordBatch]"

Snap rows to the target's time_sampling grid.

Resolves the resample column / interval / partition keys via :meth:resample_on_read, then delegates to :func:yggdrasil.arrow.ops.resample_arrow_batches. Identity short-circuit when no field is flagged keeps the read path zero-cost on the common case.

apply_post_read_table

apply_post_read_table(table: 'pa.Table') -> 'pa.Table'

Run column projection + resample + dedup on a materialised :class:pa.Table.

Same operations and same order as the streaming wraps — column projection first (trim I/O cost before any compute), resample second (its bucket collapse trims rows before the unique-tag walk), then dedup. Identity short-circuit when no pass is configured so the common case stays zero-cost.

Pyarrow / polars / pandas read paths that already produce a Table funnel through this method instead of the iterator wraps; the result is one Table.from_batches + one Table.take (per pass) instead of two Table.from_batches + a Table.to_batches rebatch sandwich.

apply_post_read_spark_frame

apply_post_read_spark_frame(df: Any) -> Any

Run resample + dedup directly on a Spark DataFrame.

Spark-side mirror of :meth:apply_post_read_table — same op order (resample first, dedup second), same identity short-circuit when neither is configured. Routes through :mod:yggdrasil.spark.ops so the heavy lifting stays on the executors (groupBy + applyInArrow for the partitioned resample, SQL window functions otherwise) instead of collecting the frame to the driver as Arrow.

Used by :meth:yggdrasil.spark.tabular.Dataset._read_spark_frame to apply the read-time passes before handing the frame back — saving a full df.toArrow → arrow.ops → createDataFrame round trip per configured op.

cast_arrow_batch_iterator

cast_arrow_batch_iterator(batches: Any) -> Any

Cast a stream of :class:pa.RecordBatch and rechunk by byte_size / row_size.

With a bound target: per-batch tabular cast + streamed rechunking via :meth:Field.cast_arrow_batch_iterator (which routes through the struct-side helper).

Without a target: rechunk-only when byte_size / row_size is set, otherwise passthrough. Lets callers that did an in-engine cast upstream still pick up the optimized rechunker.

fill_arrow_nulls

fill_arrow_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Engine-level null fill — delegates to :meth:Field.fill_arrow.

fill_arrow_array_nulls

fill_arrow_array_nulls(array: Any, *, default_scalar: Any = None) -> Any

Narrow null fill for a :class:pa.Array / :class:pa.ChunkedArray.

cast_polars

cast_polars(obj: Any) -> Any

Cast any polars object — delegates to :meth:Field.cast_polars.

cast_polars_series

cast_polars_series(series: Any, *, default_scalar: Any = None) -> Any

Cast a :class:pl.Series.

cast_polars_expr

cast_polars_expr(expr: Any, *, default_scalar: Any = None) -> Any

Cast a :class:pl.Expr.

Wraps the expression tree with a cast operator — actual work fires when the containing LazyFrame is collected.

cast_polars_tabular

cast_polars_tabular(data: Any) -> Any

Cast a :class:pl.DataFrame or :class:pl.LazyFrame.

fill_polars_nulls

fill_polars_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Engine-level polars null fill — delegates to :meth:Field.fill_polars.

polars_alias

polars_alias(obj: Any) -> Any

Rename a polars Series/Expr to the target name — no-op if matching.

Delegates to :meth:Field.polars_alias. When target is unbound there's no name to rename to, so we pass through.

cast_pandas

cast_pandas(obj: Any) -> Any

Cast any pandas object — delegates to :meth:Field.cast_pandas.

fill_pandas_nulls

fill_pandas_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Engine-level pandas null fill — delegates to :meth:Field.fill_pandas.

cast_spark_frame

cast_spark_frame(df: Any) -> Any

Filter + cast a Spark DataFrame.

Applies :attr:predicate (when set) then delegates to :meth:Field.cast_spark_tabular for schema coercion.

cast_spark_tabular

cast_spark_tabular(data: Any) -> Any

Filter + cast a :class:Dataset (Spark Tabular wrapper).

cast_spark_column

cast_spark_column(obj: Any) -> Any

Cast a Spark Column.

cast_spark

cast_spark(obj: Any) -> Any

Dispatch spark types to the specific cast method.

fill_spark_nulls

fill_spark_nulls(obj: Any, *, default_scalar: Any = None) -> Any

Engine-level spark null fill — delegates to :meth:Field.fill_spark.

spark_alias

spark_alias(obj: Any) -> Any

Rename a Spark Column to the target name — delegates to :meth:Field.spark_alias.

Table

Table(
    service: "Tables | None" = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    table_name: str | None = None,
    *,
    infos: TableInfo | None = None,
    infos_fetched_at: float | None = None,
    columns: list[Column] | None = None,
    url: URL | None = None,
    temporary: bool = False,
    singleton_ttl: "int | None" = ...
)

Bases: DatabricksPath

A single Unity Catalog table — DDL, DML, schema, storage helpers.

Registers under :attr:Scheme.DATABRICKS_TABLE (dbfs+table://) so a URL of the shape dbfs+table://[creds@]host/<catalog>/<schema>/<table>?… round-trips a Table through :meth:from_url / :meth:to_url. Reads and writes flow through the active :class:SQLEngine via the existing :class:Tabular hooks (_read_arrow_batches / _write_arrow_batches); the byte-level :class:Holder primitives are intentionally not implemented because a SQL table is not a positional byte buffer — callers should use the Tabular surface (read_arrow_table / write_arrow_table / …).

Identity is (client, catalog_name, schema_name, table_name): two callers asking for the same fully-qualified table under the same client collapse onto one instance via the :class:Singleton cache, so the cached :class:TableInfo / columns / staging volume slot are shared across views into the same UC resource.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

explore_url property

explore_url: URL

Workspace UI deep-link for this table (/explore/data/...).

Mirrors :attr:Catalog.explore_url / :attr:Schema.explore_url. The canonical addressable URL for this table lives on :attr:url (inherited from :class:Holder); explore_url is the human-friendly Catalog Explorer link.

catalog property

catalog: 'UCCatalog'

Navigate up to the parent :class:UCCatalog.

Returns the singleton-cached :class:UCCatalog for this client + catalog name — repeated calls hand back the same instance with shared :class:CatalogInfo cache.

schema property

schema: 'UCSchema'

Navigate up to the parent :class:UCSchema.

Returns the singleton-cached :class:UCSchema for this client + (catalog, schema) — repeated calls hand back the same instance with shared :class:SchemaInfo cache.

table_id property

table_id: str | None

The table's Unity Catalog id, or None when the table doesn't exist (resolved with default=None so a missing table reads as None instead of raising).

table_type property

table_type: Optional[TableType]

:class:TableType from the cached infos.

Returns None when the table hasn't been resolved against Unity Catalog yet — the property never triggers a network round trip on its own. Callers that need a guaranteed-fresh answer should access self.infos.table_type directly.

is_view property

is_view: bool

True for VIEW / MATERIALIZED_VIEW / METRIC_VIEW securables.

Reads the cached :attr:table_type; returns False until the table's infos has been resolved at least once.

is_delta property

is_delta: bool

True for a Delta-backed table (USING DELTA), from cached infos.

Reads the cached infos only — never a network round trip; returns False until the table has been resolved at least once. Views are never Delta.

view_definition property

view_definition: Optional[str]

The SQL SELECT text for a view; None for non-views.

Reads the cached infos; does not trigger a remote fetch.

view_dependencies property

view_dependencies

Upstream dependencies declared by a view (cached only).

owner property writable

owner: Optional[str]

The table's Unity Catalog owner principal (user / group / SP).

Resolves infos (a remote read if not cached), mirroring :attr:Catalog.owner / :attr:Schema.owner. Assigning re-owners the securable via ALTER TABLE|VIEW … OWNER TO.

properties property

properties: TableProperties

Live, mutable view of the table's Unity Catalog TBLPROPERTIES.

Returns a :class:TableProperties (a MutableMapping): reads resolve cached :attr:infos, while item assignment / deletion / :meth:dict.update transparently issue ALTER … SET/UNSET TBLPROPERTIES — skipping the remote call whenever the value is already what's requested::

t.properties["delta.appendOnly"] = "true"   # one ALTER
t.properties["delta.appendOnly"] = "true"   # no-op, no network
del t.properties["stale.key"]               # UNSET … IF EXISTS

view_name property writable

view_name: str

Alias for :attr:table_name so view-style call sites keep working.

tags property

tags: tuple[EntityTagAssignment, ...]

Table-level entity-tag assignments — served from client.entity_tags.

column_tags property

column_tags: Mapping[str, tuple[EntityTagAssignment, ...]]

Per-column entity-tag assignments.

Fan-out is parallelised so wide tables pay one aggregate wall-clock round trip rather than N sequential ones; cache hits inside client.entity_tags short-circuit each leg.

storage_location property

storage_location: str | None

Return the raw storage-location URL string for this table, or None when the table has no resolvable metadata.

For a Delta table this is the cloud-object root that contains the parquet data files plus the _delta_log directory. :meth:storage_path wraps the same URL in an :class:AWSClient-backed Path so callers can iterdir() / read_bytes() it directly.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

safe_name classmethod

safe_name(raw: str | None) -> str

Build a Unity-Catalog-safe table name from any raw string.

Centralized "raw string → table name" builder so every caller (URL paths, free-text in user code, composed names from upstream metadata) lands on the same identifier without duplicating the sanitization logic.

Pipeline:

1. Lowercase the input, collapse every run of non-alphanumeric characters to a single _ (/, ., query-string punctuation, whitespace, non-ASCII all fold to the same separator).
2. Strip surrounding _; substitute "root" for the empty result so "/" / "" / None still yield a legal identifier.
3. Hand off to :func:safe_table_name for the 255-char UC ceiling — overflow tokens are joined and BLAKE2b-hashed into a 32-char suffix so distinct overflows stay distinct.

When the returned name differs from raw (sanitization or truncation kicked in), a :class:logging.WARNING is emitted on this module's logger so the rewrite is visible in the wall of logs that any pipeline already collects. An identifier that's already safe round-trips silently — no warning churn for the steady-state case.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'Table'

Build a :class:Table from a dbfs+table://... URL.

Reads the catalog / schema / table from the URL path (/catalog/schema/table) and, when service is not passed in kwargs, infers the underlying :class:DatabricksClient from the URL via :meth:DatabricksClient.from_url — userinfo carries the PAT / OAuth secret, the URL host is the workspace, and remaining query items are forwarded as DatabricksClient init kwargs. Then a :class:Tables service is built on top of that client.

Caller-supplied service / catalog_name / schema_name / table_name overrides anything the URL provided.

to_url

to_url() -> URL

Render this Table as a dbfs+table://... URL.

Layers the table's /catalog/schema/table path on top of :meth:DatabricksClient.to_url so credentials / profile / account_id ride along the same URL — symmetric with :meth:from_url.

lazy

lazy(sql: str | PreparedStatement | None = None, **kwargs) -> 'Tabular'

Return a deferred :class:Tabular for sql against this table.

When sql is provided, submits the query via :attr:sql.execute and returns the resulting :class:StatementResult — itself a :class:Tabular. The warehouse executes the query eagerly so the result handle is ready, but the rows aren't materialised until the caller invokes a Tabular hook (read_arrow_table / read_arrow_batches / read_pandas_frame …)::

handle = tbl.lazy(sql="SELECT id, val FROM {self} WHERE id > 5")
arrow = handle.read_arrow_table()

{self} in the query string is substituted with the backtick-quoted full name of this table — saves the caller from concatenating tbl.full_name(safe=True) into every query. When no {self} placeholder is present, the SQL flows through verbatim.

Calling lazy() with sql=None returns the table itself (already a :class:Tabular) so callers that want to chain on the table's own data hand back the same object.

column_full_name

column_full_name(column_name: str) -> str

Fully-qualified column name suitable for entity_tag_assignments.

read_infos

read_infos(default: Any = ...)

Basic :class:TableInfo — TTL-cached.

set_tags

set_tags(tags: Mapping[str, str] | None) -> 'Table'

Apply table-level tags via the UC entity_tag_assignments API.

tag_collation is accepted for API compatibility and ignored — collations only matter for the legacy DDL literal form.

unset_tags

unset_tags(tag_keys: Iterable[str], *, if_exists: bool = True) -> 'Table'

Delete table-level tag assignments by key.

update_columns_tags

update_columns_tags(
    tags_by_column: (
        Mapping[str, Mapping[str, str] | list[EntityTagAssignment]] | None
    ),
    *,
    mode: ModeLike | None = None,
    parallel_columns: int | bool | None = None,
    parallel_per_column: int | bool | None = None,
    cache_ttl: float | None = 300.0,
    continue_on_error: bool = True,
    validate: bool = True
) -> dict[str, BaseException | None]

Apply tag batches to many columns of this table in parallel.

Per-column counterpart of :meth:set_tags. Each column's batch is routed through :meth:EntityTags.update_entities_tags with the same mode and cache_ttl; columns are processed concurrently up to parallel_columns.

Parameters:

Name	Type	Description	Default
tags_by_column	Mapping[str, Mapping[str, str] | list[EntityTagAssignment]] | None	Mapping of column name to its tag batch. Each batch may be a {tag_key: tag_value} dict or a list of :class:EntityTagAssignment (entity addressing on the assignments is filled in here — callers don't need to set it).	required
mode	ModeLike | None	Batch mode applied per column. See :meth:EntityTags.update_entity_tags for semantics.	None
parallel_columns	int | bool | None	Outer concurrency — columns processed at once. Defaults to 4.	None
parallel_per_column	int | bool | None	Inner concurrency — writes within a single column's batch. Defaults to 1; bump only when the workspace can absorb the extra load (rate limits are workspace-wide).	None
cache_ttl	float | None	TTL for the per-column tag-list cache reads used to diff before writing. None bypasses the cache.	300.0
continue_on_error	bool	When True (default), per-column failures are returned in the result rather than aborting the whole call. With False, the first exception propagates.	True
validate	bool	When True (default), unknown column names raise :class:ValueError before any write goes out. Turn off when applying tags speculatively against a partially-known schema.	True

Returns:

Type	Description
dict[str, BaseException | None]	{column_name: None | BaseException}. None denotes success.

api_create

api_create(
    definition: Union[Schema, Any],
    *,
    storage_location: str | None = None,
    comment: str | None = None,
    properties: Optional[dict[str, str]] = None,
    table_type: TableType | None = None,
    data_source_format: DataSourceFormat = DataSourceFormat.DELTA,
    missing_ok: bool = False,
    record_ygg_properties: bool = True
) -> "Table"

Create the table via the Unity Catalog tables.create REST API.

Targets EXTERNAL tables — the SDK tables.create endpoint requires an explicit storage_location. For MANAGED tables, prefer :meth:sql_create, which is also the only path that exposes Delta-specific knobs (CLUSTER BY, OPTIMIZE, TBLPROPERTIES, column mapping mode, …).

comment and constraints (PK / FK / CHECK) carried by the schema are applied post-create — the SDK call itself only takes columns + storage + properties — so the behaviour ends up symmetric with :meth:sql_create.

create_view_ddl

create_view_ddl(
    query: str,
    *,
    or_replace: bool = False,
    missing_ok: bool = False,
    columns: Iterable[str] | None = None,
    comment: str | None = None,
    properties: Optional[Mapping[str, Any]] = None
) -> str

Render a CREATE [OR REPLACE] VIEW [IF NOT EXISTS] DDL statement.

Mirrors the legacy :meth:View.create_ddl shape; or_replace and missing_ok are mutually exclusive, and the SELECT text is required.

create_view

create_view(
    query: str,
    *,
    mode: ModeLike = None,
    or_replace: bool | None = None,
    missing_ok: bool | None = None,
    columns: Iterable[str] | None = None,
    comment: str | None = None,
    properties: Optional[Mapping[str, Any]] = None,
    tags: Mapping[str, str] | None = None,
    wait: WaitingConfigArg = True
) -> "Table"

Create (or replace) this Table as a Unity Catalog view.

When neither or_replace nor missing_ok is provided the keywords are derived from mode:

• :data:Mode.OVERWRITE → or_replace=True
• :data:Mode.AUTO / :data:Mode.APPEND / :data:Mode.UPSERT / :data:Mode.IGNORE → missing_ok=True
• :data:Mode.ERROR_IF_EXISTS → plain CREATE VIEW

concat_tables

concat_tables(
    tables: Iterable["Table"],
    *,
    by_name: bool = True,
    cast: bool = True,
    comment: str | None = None,
    mode: ModeLike = Mode.OVERWRITE
) -> "Table"

Create or replace this Table as the UNION ALL of tables.

When cast is True (default), the union is "smart": column names are aligned across inputs, types are promoted to the widest compatible :class:DataType via merge_with(upcast=True), each input projects the unified column list in order, and any column missing from a given input is emitted as CAST(NULL AS <ddl>) so the unified schema is preserved.

When cast is False the method falls back to a plain SELECT * FROM <table> UNION ALL [BY NAME] ... and lets Databricks reconcile the schemas at query time.

rename

rename(
    new_name: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    table_name: str | None = None
) -> "Table"

Rename this table in-place (ALTER TABLE … RENAME TO …).

Accepts an unqualified name ("new_orders"), a two-part name ("sales.new_orders" → cross-schema move within the same catalog), or a three-part name ("main.sales.new_orders"). Catalog/schema keyword overrides win over parts parsed from new_name.

Unity Catalog allows cross-schema renames within the same catalog; moves across catalogs are rejected here with a clear error rather than letting the server return a generic failure.

clone

clone(
    target: "str | Table | None" = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    table_name: str | None = None,
    deep: bool = True,
    replace: bool = False,
    missing_ok: bool = False,
    mode: "ModeLike | None" = None,
    properties: Mapping[str, Any] | None = None,
    location: str | None = None,
    version: int | None = None,
    timestamp: "str | _dt.datetime | _dt.date | None" = None
) -> "Table"

Clone this table to target via Delta CREATE TABLE … CLONE.

Emits one of::

CREATE TABLE [IF NOT EXISTS] <target> [SHALLOW|DEEP] CLONE <source>
    [TBLPROPERTIES (...)] [LOCATION '...']
CREATE OR REPLACE TABLE <target> [SHALLOW|DEEP] CLONE <source> ...

Parameters:

Name	Type	Description	Default
target	'str | Table | None'	Target location — :class:Table, a ½/3-part dotted name, or None when catalog_name / schema_name / table_name are passed explicitly.	None
deep	bool	True (default) → DEEP CLONE (independent copy); False → SHALLOW CLONE (metadata only, shares files).	True
replace	bool	Emit CREATE OR REPLACE TABLE.	False
missing_ok	bool	Emit CREATE TABLE IF NOT EXISTS. Mutually exclusive with replace.	False
mode	'ModeLike | None'	Existence policy as a :class:Mode (overrides replace / missing_ok when set): OVERWRITE / TRUNCATE → CREATE OR REPLACE, IGNORE → CREATE … IF NOT EXISTS, ERROR_IF_EXISTS → plain CREATE (fails if the target exists).	None
properties	Mapping[str, Any] | None	Optional TBLPROPERTIES overrides.	None
location	str | None	External storage path for the target.	None
version	int | None	Delta source version (VERSION AS OF).	None
timestamp	'str | _dt.datetime | _dt.date | None'	Delta source timestamp (TIMESTAMP AS OF).	None

Returns:

Name	Type	Description
A	'Table'	class:Table bound to this service pointing at the target.

insert

insert(
    data: Any,
    *,
    mode: ModeLike = None,
    match_by: Optional[list[str]] = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    spark_session: Optional["SparkSession"] = None,
    return_data: bool = False,
    **kwargs
) -> "Tabular | None"

Insert data into this table — thin wrapper over :meth:insert_into.

auto_loader

auto_loader(
    source: "str | None" = None,
    *,
    name: "str | None" = None,
    file_format: str = "parquet",
    checkpoint: "str | None" = None,
    available_now: bool = True,
    file_arrival: bool = True,
    file_arrival_min_seconds: int = 60,
    trigger: "Any" = None,
    clean_source: bool = False,
    clean_source_retention: str = "8 days",
    bundle_dependencies: bool = True,
    environment: "str | None" = ...,
    deploy: bool = True
) -> "Any"

Get-or-create a Databricks Auto Loader ingestion job for this table.

Builds a serverless job — leveraging the ygg wheel + environment via the :class:~yggdrasil.databricks.job.skeleton.Flow machinery — whose single task runs :func:yggdrasil.databricks.table.auto_loader.auto_load on the cluster: Spark Structured Streaming + cloudFiles incrementally ingests files dropped under source into this table (exactly-once, schema-evolving). The job is named [YGG][AUTOLOADER] <full_name> and upserted by name (:meth:Jobs.create_or_update), so repeated calls reconfigure the same job rather than piling up duplicates.

Parameters:

Name	Type	Description	Default
source	'str | None'	Cloud path Auto Loader watches (s3://… / /Volumes/…). None (default) uses this table's dedicated staging volume as the governed /Volumes/<cat>/<sch>/<vol>/STAGE_SUBPATH path — on-cluster cloudFiles reads it through Unity Catalog's own optimized access, so it works for managed and external staging volumes. (Uploads via :meth:stage_insert still take the direct cloud-storage fast path when the volume is external.) Files staged there are ingested with no explicit wiring.	None
name	'str | None'	Job name override (default [YGG][AUTOLOADER] <full_name>).	None
file_format	str	cloudFiles.format (parquet / json / csv / avro / …).	'parquet'
checkpoint	'str | None'	Streaming checkpoint + schema location. None (default) on the zero-config path (when source also defaults) co-locates it with the staging area on the same staging volume — the governed /Volumes/…/CHECKPOINT_SUBPATH path — kept off a MANAGED table's own governed __unitystorage storage, which Unity Catalog forbids Auto Loader from writing into. With an explicit source, None instead lets the on-cluster step derive <table-location>/_ygg_autoloader.	None
available_now	bool	True → a one-shot Trigger.AvailableNow sweep (the shape a scheduled / file-arrival run wants); False → continuous micro-batch.	True
file_arrival	bool	True (default) → attach a file-arrival trigger on source so the job fires automatically when new files land — the natural shape for an ingestion job watching a drop path. The trigger establishes a baseline when it's created and only fires for files arriving after that, so stage rows after deploying (a file already present at creation won't trigger it). False deploys the job with no trigger (run it on a schedule, manually, or via .run()). Ignored when an explicit trigger is given.	True
file_arrival_min_seconds	int	Databricks polling floor for the file-arrival trigger (min_time_between_triggers_seconds) — it won't evaluate source more often than this. Databricks' minimum is 60s; smaller values are clamped up to it. Default 60.	60
trigger	'Any'	An explicit Databricks TriggerSettings (schedule / file-arrival), passed through as-is. Takes precedence over file_arrival.	None
clean_source	bool	True makes Auto Loader delete each staged file once it's been ingested and is older than clean_source_retention (cloudFiles.cleanSource = DELETE) so the staging area is self-cleaning. A rolling janitor — it does not delete files within the same one-shot sweep that ingests them. Default False.	False
clean_source_retention	str	Retention window for clean_source; Databricks requires an interval greater than 7 days (default "8 days").	'8 days'
bundle_dependencies	bool	True (default) ships the whole transitive dependency closure as wheels so the serverless environment installs with zero PyPI access ("0 pip install"); False ships only the ygg wheel and resolves deps from the workspace index at install.	True
environment	'str | None'	Name of a reusable serverless base environment to create-or-update and reference. Default (unset) resolves to the canonical, version-pinned ygg image (:func:~yggdrasil.databricks.environments.service.environment_stem, ygg-<version>-py3XX) — the same <name>.yml file ygg databricks deploy writes under /Workspace/Shared/ environments, so the job reuses the seeded wheel-built image (or self-provisions the identical one). Pass an explicit name to point at a different shared env; None inlines the dependency list on the job instead.	...
deploy	bool	True (default) get-or-creates the job now and returns the :class:~yggdrasil.databricks.job.job.Job; False returns the configured (un-deployed) :class:Flow for inspection / a manual .deploy(client).	True

Returns the deployed :class:Job (deploy=True) or the :class:Flow.

stage_insert

stage_insert(data: Any, *, options: Optional[CastOptions] = None) -> 'Path'

Stage data as Parquet under this table's Auto Loader staging area and return the path it landed at — no warehouse statement runs.

Writes a fresh, uniquely-named Parquet file under the staging volume's STAGE_SUBPATH prefix — the same path a deployed :meth:auto_loader job watches — so staged rows are ingested with no extra wiring: stage_insert → Auto Loader → table. staging_volume / STAGE_SUBPATH resolves to the volume's direct cloud storage (s3://…, no Files-API hop) when the volume is EXTERNAL and reachable, and to the governed Files-API /Volumes/… path otherwise (e.g. a managed staging volume) — both land the file where the watcher expects it.

insert_into

insert_into(
    data: Union[
        Table,
        RecordBatch,
        RecordBatchReader,
        dict,
        list,
        str,
        PreparedStatement,
        StatementResult,
        "pandas.DataFrame",
        "polars.DataFrame",
        "pyspark.sql.DataFrame",
    ],
    *,
    mode: Mode | str | None = None,
    schema_mode: Mode | str | None = None,
    options: Optional[CastOptions] = None,
    overwrite_schema: bool | None = None,
    match_by: Optional[list[str]] = None,
    update_column_names: Optional[list[str]] = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    zorder_by: Optional[list[str]] = None,
    optimize_after_merge: bool = False,
    vacuum_hours: int | None = None,
    spark_session: Optional["pyspark.sql.SparkSession"] = None,
    spark_options: Optional[Dict[str, Any]] = None,
    predicate: Predicate | None = None,
    retry: Optional[WaitingConfigArg] = None,
    return_data: bool = False,
    safe_merge: bool = False
) -> "StatementBatch | Tabular | None"

Insert data into this table using the most appropriate backend.

Routing:

• Spark DataFrame (or anything when a SparkSession is reachable) → :meth:spark_insert
• Otherwise → :meth:arrow_insert (the specialized warehouse path with Volume staging, which delegates to :class:~yggdrasil.databricks.table.insert.DatabricksTableInsert).

Returns the submitted :class:StatementBatch by default. With return_data=True the backend that ran the write hands back its source payload as a :class:Tabular — :class:ArrowTabular from :meth:arrow_insert, :class:Dataset from :meth:spark_insert — for downstream chaining without re-querying the target.

insert_volume_path

insert_volume_path(
    target: "Table | None" = None,
    *,
    volume: "Volume | None" = None,
    temporary: bool = True
) -> VolumePath

Mint a fresh Parquet staging path under the target table's :attr:staging_volume.

Roots the file at <staging_volume>/.sql/tmp/tmp-<epoch_ms>-<seed>.parquet (same shape as :meth:staging_folder but with a unique leaf per call). target defaults to self; pass another :class:Table when the staging hierarchy needs to live next to a different table (e.g. dispatch fan-out). Lifted out of :meth:arrow_insert so callers — and tests — can pre-mint or swap the staging location without driving the full insert.

arrow_insert

arrow_insert(
    data,
    *,
    engine: Literal["api", "spark"] | None = None,
    mode: Mode | str | None = None,
    schema_mode: Mode | str | None = None,
    options: Optional[CastOptions] = None,
    overwrite_schema: bool | None = None,
    match_by: Optional[list[str]] = None,
    update_column_names: Optional[list[str]] = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    zorder_by: Optional[list[str]] = None,
    optimize_after_merge: bool = False,
    vacuum_hours: int | None = None,
    predicate: Predicate | None = None,
    retry: Optional[WaitingConfigArg] = None,
    return_data: bool = False,
    safe_merge: bool = False,
    staging_volume: "Volume | None" = None
) -> "StatementBatch | Tabular | None"

Insert through the warehouse SQL path with staged Parquet.

safe_merge controls keyed-write strategy:

• safe_merge=False (default) — emits a single MERGE INTO statement. Databricks / Delta plans the keyed dedup once.
• safe_merge=True — sidesteps MERGE: keyed APPEND becomes INSERT ... WHERE NOT EXISTS (...), keyed UPSERT becomes DELETE matching keys then INSERT. Useful for backends without native MERGE or callers that want explicit dedup semantics.

Returns the submitted :class:StatementBatch by default. With return_data=True, returns an :class:ArrowTabular wrapping the staged source rows so callers can chain on the payload without re-reading from the target.

spark_insert

spark_insert(
    data: Any,
    *,
    mode: Mode | str | None = None,
    schema_mode: Mode | str | None = None,
    options: Optional[CastOptions] = None,
    overwrite_schema: bool | None = None,
    match_by: Optional[list[str]] = None,
    update_column_names: Optional[list[str]] = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    zorder_by: Optional[list[str]] = None,
    optimize_after_merge: bool = False,
    vacuum_hours: int | None = None,
    spark_options: Optional[Dict[str, Any]] = None,
    predicate: Predicate | None = None,
    spark_session: Optional["pyspark.sql.SparkSession"] = None,
    retry: Optional[WaitingConfigArg] = None,
    return_data: bool = False,
    safe_merge: bool = False
) -> "StatementBatch | Tabular | None"

Insert into this table using Spark.

retry is applied to DML statements (INSERT/MERGE/DELETE/UPDATE) only — TRUNCATE/OPTIMIZE/VACUUM stay non-retryable. :class:SparkStatementResult already auto-promotes transient Delta failures (ConcurrentAppendException, …) to retryable; passing retry=True (or any :class:WaitingConfig arg) makes the policy explicit instead of relying on auto-promote.

Returns the submitted :class:StatementBatch by default. With return_data=True, returns a :class:Dataset wrapping the materialised source DataFrame — handy for chaining downstream transforms without re-querying the target.

external_location

external_location(*, refresh: bool = False) -> ExternalLocation | None

The Unity Catalog external location governing this table's backing storage — the most specific one whose URL the table's storage_location sits under (longest-prefix match over the cached external-location list, :meth:ExternalLocations.find_url) — or None when the table has no resolvable storage or no accessible location covers it (e.g. a MANAGED table on governed __unitystorage). Never raises.

Memoised on the table: resolved once and reused (a resolved None is cached too) so repeated precheck calls don't re-walk the location list. refresh=True re-resolves; the memo is dropped on any info refresh (:meth:_store_infos).

can_read

can_read(*, refresh: bool = False) -> bool

Global precheck — can this table's storage be read directly at the cloud layer (bypassing the warehouse)? True when an accessible external location covers it. Cheap and cached (no per-object probe) — gate :meth:delta / :meth:storage_path bulk work on it.

can_write

can_write(*, refresh: bool = False) -> bool

Global precheck — can this table's storage be written directly at the cloud layer? True when a covering external location exists and is not read-only.

storage_path

storage_path(*, write: 'bool | None' = None) -> 'Path | None'

Return the table's backing storage as an addressable :class:Path.

For a Delta table, tbl.storage_path() yields a Path that contains the parquet data files plus the _delta_log transaction directory — list(tbl.storage_path().iterdir()) is the natural way to inspect the on-disk layout.

write picks the UC temporary-credential scope the Path's :class:AWSClient vends — important because a principal can hold read but not write on a table:

• None (default) — the operation default for the table type (READ for managed, READ_WRITE for external);
• False — READ (least-privilege; reads a table you can't write);
• True — READ_WRITE (collapses to READ for managed, which UC never vends write creds for).

delta

delta(*, write: 'bool | None' = None) -> 'DeltaFolder'

Return a :class:~yggdrasil.io.delta.DeltaFolder over this table's backing storage — the native Delta read/write surface.

Built from :meth:storage_path so the folder (and every parquet / _delta_log child it resolves) inherits the table's temporary-credential :class:AWSClient; constructing a DeltaFolder from the bare URI string would drop those creds. Lets callers read (tbl.delta().read_arrow_table()) or commit (tbl.delta().write_arrow_table(t, mode=Mode.APPEND)) straight against the transaction log, bypassing the warehouse.

write flows to :meth:storage_path to scope the vended credentials — write=False for a read-only handle (works even when the caller can't write the table), write=True for a commit.

aws

aws(
    operation: "TableOperation | ModeLike | None" = None,
    *,
    region: Optional[str] = None,
    secret_cache: bool = False
) -> "AWSClient"

Return an :class:AWSClient whose credentials self-refresh from Unity Catalog's temporary_table_credentials API.

Routes through :meth:credentials_refresher — every :class:Table instance pointing at the same UC table id collapses to one provider that handles both read and write modes internally. The provider caches its :class:AWSClient per (mode, region) so the boto session, :class:RefreshableCredentials, connection pool, and STS vending are shared across every caller on the same scope.

operation accepts a :class:TableOperation, a :class:Mode / mode-like string, or None (defaults to the right operation for this table's type). secret_cache=True backs the vended credentials with a per-table Databricks secret scope (off by default).

credentials_refresher

credentials_refresher(
    *, secret_cache: bool = False
) -> "AWSDatabricksTableCredentials"

Return the process-wide singleton credentials provider for this table.

Keyed by table_id; handles both read and write modes internally via :meth:AWSDatabricksTableCredentials.get_credentials.

secret_cache=True opts the provider into persisting its vended AWS credentials in a per-table Databricks secret scope (off by default); the opt-in is sticky across the shared singleton.

Tables

Tables(
    client=None, catalog_name: str | None = None, schema_name: str | None = None
)

Bases: DatabricksService

Collection-level service for Unity Catalog tables.

Attach default catalog / schema context so callers don't have to repeat them on every call::

tables = client.tables(catalog_name="main", schema_name="sales")
table  = tables.find_table("orders")
for t in tables.list_tables():
    ...

wheels property

wheels: 'Wheels'

Wheel registry service (shorthand for client.wheels).

environments property

environments: 'Environments'

Base-environment service (shorthand for client.environments).

tables property

tables: 'Tables'

Collection-level Unity Catalog table service (shorthand for client.tables).

views property

views: 'Tables'

Alias for :attr:tables — :class:Table covers both managed/external tables and view-shaped securables.

catalogs property

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service (shorthand for client.catalogs).

schemas property

schemas: 'Schemas'

Collection-level Unity Catalog schema service (shorthand for client.schemas).

volumes property

volumes: 'Volumes'

Collection-level Unity Catalog volume service (shorthand for client.volumes).

genie property

genie: 'Genie'

Genie service (shorthand for client.genie).

ai property

ai: 'DatabricksAI'

Databricks AI umbrella service (shorthand for client.ai).

default_tags

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

Returns:

Type	Description
dict[str, str]	A dict of default tags.

table

table(
    location: Table | str | None = None,
    *,
    table_name: str | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> "Table"

Return a :class:~yggdrasil.databricks.table.table.Table bound to this service.

view

view(
    location: Table | str | None = None,
    *,
    table_name: str | None = None,
    view_name: str | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> "Table"

Return a :class:~yggdrasil.databricks.table.table.Table bound to this service.

Alias for :meth:table — Unity Catalog stores views in the same tables API as managed/external tables, so the returned :class:Table covers both. view_name is accepted as a convenience alias for table_name.

catalog

catalog(name: str | None = None) -> 'UCCatalog'

Return a :class:UCCatalog using this service's client.

Parameters:

Name	Type	Description	Default
name	str | None	Catalog name (falls back to self.catalog_name).	None

schema

schema(
    name: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> "UCSchema"

Return a :class:UCSchema using this service's client.

Parameters:

Name	Type	Description	Default
name	str | None	Two-part "catalog.schema" name (optional if catalog_name / schema_name are provided).	None
catalog_name	str | None	Override catalog (falls back to self.catalog_name).	None
schema_name	str | None	Override schema (falls back to self.schema_name).	None

find_table_remote

find_table_remote(
    catalog_name: str,
    schema_name: str,
    table_name: str | None = None,
    *,
    table_id: str | None = None,
    default: Any = ...
) -> Optional[TableInfo]

Raw API lookup — three strategies in order, no cache.

1. Search by table_id (full list scan, preferred when id is known).
2. GET by fully qualified name (fast path for normal lookups).
3. Case-insensitive list scan (handles edge cases in naming / quoting).

Returns None on miss when raise_error=False.

find_table

find_table(
    location: str | Table | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    table_name: str | None = None,
    table_id: str | None = None,
    default: Any = ...,
    cache_ttl: float | None = 300.0
) -> Optional["Table"]

Resolve a table by name or Unity Catalog ID.

Caching is controlled only by cache_ttl. Set cache_ttl=None to bypass the cache for this lookup.

Parameters:

Name	Type	Description	Default
location	str | Table | None	Full string location	None
table_name	str | None	Name.	None
catalog_name	str | None	Override catalog (falls back to service default).	None
schema_name	str | None	Override schema (falls back to service default).	None
table_id	str | None	Unity Catalog table UUID — triggers id-based search.	None
default	Any	Raise :exc:ResourceDoesNotExist when not found.	...
cache_ttl	float | None	Entry TTL in seconds (None → 5 min default).	300.0

list_tables

list_tables(
    name: str | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    *,
    table_types: Iterable[TableType] | None = None,
    cache_ttl: float | None = 300.0
) -> Iterator["Table"]

Iterate over tables in the resolved catalog/schema scope.

Any of name, catalog_name, or schema_name may be a case-insensitive glob ("sales_*", "*_raw", "prefix_*_table", "*"). Globbed catalog/schema names fan out across the matching resources; None still means "all" at that level.

Parameters:

Name	Type	Description	Default
name	str | None	Optional table-name filter (exact or glob).	None
catalog_name	str | None	Override catalog (falls back to service default). Accepts a glob to fan out across catalogs.	None
schema_name	str | None	Override schema (falls back to service default). Accepts a glob to fan out across schemas. When None, iterates every schema in the resolved catalog scope. When both catalog and schema are None, iterates every visible catalog and schema.	None
table_types	Iterable[TableType] | None	Restrict the yielded :attr:Table.table_type set. None (default) yields every securable UC reports — managed / external tables and view-shaped securables alike. Pass :data:_VIEW_TABLE_TYPES to filter to views, or any subset to narrow further.	None
cache_ttl	float | None	Entry TTL in seconds (None → 5 min default).	300.0

list_views

list_views(
    name: str | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    *,
    table_types: Iterable[TableType] | None = None,
    cache_ttl: float | None = 300.0
) -> Iterator["Table"]

Iterate over view-shaped securables only.

Convenience wrapper around :meth:list_tables with table_types defaulted to {VIEW, MATERIALIZED_VIEW, METRIC_VIEW}. Pass an explicit table_types to narrow further (e.g. only :data:TableType.MATERIALIZED_VIEW).

concat_tables

concat_tables(
    tables: Iterable["Table"],
    *,
    view_name: str | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    by_name: bool = True,
    cast: bool = True,
    comment: str | None = None,
    mode: ModeLike = Mode.OVERWRITE
) -> "Table"

Create or update a view that concatenates tables with UNION ALL.

Resolves the view name + parent (deriving from the inputs' shared prefix and the first input's catalog/schema when not given) and delegates the actual DDL to :meth:View.concat_tables — which does the smart by-name + type-promotion projection when cast is True.

Parameters:

Name	Type	Description	Default
tables	Iterable['Table']	Iterable of :class:Table or :class:View instances to union. At least one input is required.	required
view_name	str | None	Unqualified view name. When omitted, the longest shared prefix of the input table names (trimmed of trailing _ - .) is used. Raises ValueError when the inputs share no common prefix.	None
catalog_name	str | None	Override the view catalog. Falls back to the service default, then to the first input table's catalog.	None
schema_name	str | None	Override the view schema. Falls back to the service default, then to the first input table's schema.	None
by_name	bool	Forwarded to :meth:View.concat_tables. Only consulted when cast is False.	True
cast	bool	Forwarded to :meth:View.concat_tables — enables smart column-name alignment with CAST(NULL AS <ddl>) fills for columns missing from a given input. Default True.	True
comment	str | None	Optional COMMENT on the view.	None
mode	ModeLike	Passed through to :meth:View.create. Defaults to :attr:Mode.OVERWRITE so the view is created or replaced atomically.	OVERWRITE

Returns:

Name	Type	Description
The	'Table'	class:View that was created or updated.

make_sql_insert

make_sql_insert(
    op: "DatabricksTableInsert",
    *,
    target_location: "str | None" = None,
    source_sql: "str | None" = None,
    columns: "list[str] | None" = None,
    client: Any = None
) -> list[str]

Render the full statement list for one insert.

Yields the INSERT / MERGE / DELETE+INSERT / TRUNCATE / OPTIMIZE / VACUUM statement list for the op.

target_location / source_sql / columns let the synchronous paths supply their own source reference (the {__tmpsrc__} placeholder, a Spark temp-view name, or a wrapped user query) and pre-resolved target location; when omitted they're derived from the op's target and staged data.

make_sql_select

make_sql_select(
    op: "DatabricksTableInsert",
    *,
    client: Any = None,
    source: "str | None" = None
) -> str

The atomic per-op SELECT over the op's staged source.

Two source shapes:

• default — render `SELECT * FROM parquet.`` over the op's staged Parquet (resolved from its :class:Path` / uniform URL).
• explicit source — when the caller already has a source reference (the {__tmpsrc__} placeholder, which is substituted for the external-data VolumePath at prepare time), project the op's schema columns from it: SELECT <projection> FROM <source>.