yggdrasil.databricks

databricks

Databricks integrations and helpers for Yggdrasil.

DatabricksClient

DatabricksClient(
    *,
    host: Any = ...,
    account_id: Any = ...,
    workspace_id: Any = ...,
    token: Any = ...,
    client_id: Any = ...,
    client_secret: Any = ...,
    token_audience: Any = ...,
    cluster_id: Any = ...,
    serverless_compute_id: Any = ...,
    azure_workspace_resource_id: Any = ...,
    azure_use_msi: Any = ...,
    azure_client_secret: Any = ...,
    azure_client_id: Any = ...,
    azure_tenant_id: Any = ...,
    azure_environment: Any = ...,
    google_credentials: Any = ...,
    google_service_account: Any = ...,
    profile: Any = ...,
    config_file: Any = ...,
    auth_type: Any = ...,
    http_timeout_seconds: Any = ...,
    retry_timeout_seconds: Any = ...,
    debug_truncate_bytes: Any = ...,
    debug_headers: Any = ...,
    rate_limit: Any = ...,
    max_connection_pools: Any = ...,
    max_connections_per_pool: Any = ...,
    product: Any = ...,
    product_version: Any = ...,
    skip_verify: Any = ...,
    catalog_name: Any = ...,
    schema_name: Any = ...,
    singleton_ttl: "int | None" = ...
)

Bases: Singleton, URLBased

Thin wrapper around databricks.sdk.config.Config.

URL-addressable through the :class:URLBased base: cls.scheme is :attr:Scheme.DATABRICKS (dbks), so a single dbks://[client_id[:secret]@]<host>[?profile=...&account_id=...] URL round-trips a client through :meth:from_url / :meth:to_url. The userinfo carries the credential — a bare :<token>@ for a PAT, <client_id>:<client_secret>@ for an OAuth service principal — and the query string carries every other DatabricksClient field that __init__ accepts. Sensitive fields (host, token, client_id, client_secret) are kept out of the query so they don't get duplicated when the URL is logged or persisted.

Public state is intentionally minimal

• config: Config

Extra wrapper-only metadata is kept separately.

Cross-process / cross-host serialization is supported via :meth:__getstate__ / :meth:__setstate__. SDK clients, Configs, and lazy sub-service caches are dropped (all rebuild on demand from the dataclass fields). A best-effort session token snapshot is carried alongside so the receiving side can warm-start without re-running the auth dance (browser flow, MSI probe, gcloud, etc).

If the deserializing host is itself a Databricks runtime (driver node), the carried credentials are discarded and auth_type is forced to "runtime" — DBR's notebook-scoped auth is short-lived, identity-correct, and the right default in that environment.

explore_url property

explore_url: URL

Workspace UI root for the Catalog Explorer (/explore/data).

Mirrors :attr:Catalog.explore_url / :attr:Schema.explore_url so the whole resource hierarchy advertises a deep-link in one place.

project property writable

project: Optional[str]

The client project — an alias of :attr:product, always lowercased (the canonical project identifier). yggdrasil and ygg are the same project: yggdrasil is the name (and the import package), ygg is only its PyPI distribution alias. That distribution mapping lives in the wheel / environment layer (:func:~yggdrasil.databricks.wheels.service.distribution_for), not here — so client.project stays the human project name while the deployed wheels / base environment land under the ygg distribution folder. Defaults to "yggdrasil"; set it (or product) to name a project and it persists with the client (product is one of :data:_INIT_NAMES, so it rides config, session snapshots, and clones).

product_name property

product_name: Optional[str]

A nice, capitalized display name for the client :attr:project — the real project name (yggdrasil → Yggdrasil, my-app → My App), or None when unset. The project's default warehouse and cluster are named for this. The ygg PyPI alias is not applied here — it belongs to the wheel / distribution layer, not the project's identity.

wheels property

wheels: 'Wheels'

Wheel registry service — build/upload/deploy/browse wheels.

environments property

environments: 'Environments'

Base-environment service — assemble/deploy serverless + cluster images, deploy projects.

compute property

compute: 'Compute'

Default cluster helper for this client.

secrets property

secrets: 'Secrets'

Default secrets helper for this client.

tables property

tables: 'Tables'

Collection-level Unity Catalog table service for this client.

views property

views: 'Tables'

Alias for :attr:tables — Unity Catalog stores views in the same tables API, and :class:Table handles both shapes.

columns property

columns: 'Columns'

Collection-level Unity Catalog column service for this client.

catalogs property

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service for this client.

Provides dict-like access to catalogs, schemas, and tables::

client.catalogs["main"]                   # Catalog
client.catalogs["main"]["sales"]          # Schema
client.catalogs["main"]["sales"]["orders"]  # Table

external property

external: 'DatabricksExternal'

Unity Catalog external data umbrella service for this client.

Groups the securables that bind UC to outside storage — external locations and storage credentials::

client.external.locations["raw_zone"]          # ExternalLocation
client.external.credentials.create_aws("prod_s3", "arn:aws:iam::123:role/R")
client.external.credentials["prod_s3"].aws_client(region="us-east-1")

external_locations property

external_locations: 'ExternalLocations'

Unity Catalog external-locations service for this client.

Flat alias onto :attr:external — client.external.locations::

client.external_locations["raw_zone"]          # ExternalLocation
client.external_locations.list()               # Iterator[ExternalLocation]
client.external_locations.create(name, url, credential_name)

credentials property

credentials: 'Credentials'

Unity Catalog storage-credentials service for this client.

Flat alias onto :attr:external — client.external.credentials::

client.credentials.create_aws("prod_s3", "arn:aws:iam::123:role/R")
client.credentials["prod_s3"].aws_client(region="us-east-1")  # refreshable

schemas property

schemas: 'Schemas'

Collection-level Unity Catalog schema service for this client.

Provides dict-like access to schemas and tables::

client.schemas["main.sales"]             # Schema
client.schemas["main.sales.orders"]      # Table
client.schemas(catalog_name="main")      # Schemas scoped to "main"

volumes property

volumes: 'Volumes'

Collection-level Unity Catalog volume service for this client.

Provides dict-like access to volumes::

client.volumes["main.sales.uploads"]                # Volume
client.volumes(catalog_name="main", schema_name="sales")["uploads"]
client.volumes.list(catalog_name="main")            # Iterator[Volume]

genie property

genie: 'Genie'

Databricks AI/BI Genie service — manage spaces + ask questions by code.

dbc.genie.spaces() dbc.genie["01ef…"].ask("revenue by region last month").to_polars()

ai property

ai: 'DatabricksAI'

Databricks AI umbrella service (vector search today, serving/registry next).

Reach the concrete services through it::

client.ai.vector_search.endpoint("rag").ensure_created()
client.ai.vector_search.index("main.rag.docs").query(
    query_text="…", columns=["id", "text"],
)

is_serverless_compute property

is_serverless_compute: bool

True when this client explicitly targets serverless compute.

Only returns True when serverless_compute_id was set by the caller. A bare client with no cluster_id and no serverless_compute_id is NOT serverless — it simply has no compute target and will resolve one lazily when needed.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Pop self from the per-class _INSTANCES cache.

Mutating ops on a Singleton-cached object (writes, deletes, schema invalidations on a Databricks table, put_object on an :class:S3Path) want to make sure the next caller asking for the same key gets a fresh build rather than collapsing onto this stale handle — that's what remove_global=True (the default) does. The pop is :meth:identity-guarded: only an entry that still points at self is removed, so a concurrent re-construction that already raced past this thread is left alone.

remove_global=False is a no-op. The keyword exists so subclass invalidators (invalidate_singleton, _invalidate_entity_tag_cache, …) can offer the same switch without branching at the call site.

to_url

to_url(scheme: str | None = None) -> URL

Render this client as a dbks://... URL.

Pack everything __init__ would need to rebuild the client into the URL: the workspace host as the URL host, the credential (PAT or OAuth client_id/secret) as userinfo, and every other non-default field as query items. Sensitive fields (host, token, client_id, client_secret) are intentionally kept out of the query so they don't get duplicated alongside the userinfo.

scheme overrides :attr:scheme for callers that want a different URL scheme (e.g. "https" for the bare workspace URL); defaults to :attr:Scheme.DATABRICKS.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> TC

Build a client from a dbks://... URL.

Reads:

• the workspace host from url.host (preferred) or a host= query param;
• credentials from url.userinfo — <client_id>:<client_secret>@ for OAuth, :<token>@ (or anything-as-password) for a PAT;
• every other init field of :class:DatabricksClient from the query string (profile, auth_type, account_id, workspace_id, http_timeout_seconds, …).

kwargs overrides anything the URL provides so callers can layer programmatic overrides on top of a parsed URL without an extra replace call.

files_session

files_session() -> 'HTTPSession'

Authenticated :class:HTTPSession bound to this workspace host.

Volume / Files-API traffic routes through yggdrasil's own HTTP transport instead of the SDK's requests-based client: the :class:HTTPSession owns a per-host keep-alive connection pool, status-aware tiered retry (429 / 5xx with backoff), and — via the :class:HTTPStream response body — transparent resume-on-disconnect for SSL UNEXPECTED_EOF / connection-reset mid-download, the failure modes the SDK's Files client handles poorly.

:class:HTTPSession is itself a process-wide singleton keyed by (base_url, verify, …), so repeated calls collapse onto one shared pool. skip_verify flows through to verify=False.

files_authorization

files_authorization() -> str

Fresh Authorization header value for Files-API requests.

Delegates to the SDK Config's auth flow (:meth:databricks.sdk.config.Config.authenticate) so every supported credential type — PAT, OAuth M2M, Azure SP, GCP — and the SDK's own token-refresh caching apply unchanged; only the wire transport is ours. Raises when the resolved auth produces no bearer header (e.g. a misconfigured profile).

files_headers

files_headers() -> dict[str, str]

Base header set for Files-API requests, matching the SDK transport.

The SDK's ApiClient stamps User-Agent (product + SDK version + platform + auth type) on every call and X-Databricks-Workspace-Id when the config carries a workspace id — Databricks' edge uses both to attribute and rate-limit traffic, so requests without them are classified as anonymous and throttled (429) far more aggressively. Mirror them here since Files traffic bypasses the SDK transport (:meth:files_session).

default_tags

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

On create (update=False) the tag set is enriched with environment-derived owner metadata pulled from :class:~yggdrasil.environ.UserInfo:

• Product / ProductVersion from the client config.
• Owner — UserInfo email when available.
• Hostname — local hostname so per-user pools / clusters are distinguishable in shared workspaces.
• User — whoami key, useful when no email is reachable.

Returns:

Type	Description
dict[str, str]	A dict of default tags.

user_scoped_name

user_scoped_name(
    base: str, *, separator: str = "-", max_length: int = 80
) -> str

Return base suffixed with a stable per-user slug.

Resolution order for the slug:

1. UserInfo.email local part (alice@example.com → alice).
2. UserInfo.key (whoami).
3. UserInfo.hostname.

Each candidate is sanitized with :meth:safe_tag_value so the result is a legal Databricks resource name. Falls back to base unchanged when no candidate is available — useful in test harnesses where the environment carries no identity. The result is truncated to max_length characters (default 80, well under the Databricks cluster / pool name cap of 100).

safe_tag_value staticmethod

safe_tag_value(value: str, *, repl: str = '-') -> str

Sanitize a tag string to match: ^[\d \w+-=:.:/@]*$ Replaces any illegal characters with repl and collapses repeats.

dbfs_path

dbfs_path(
    parts: Union[list[str], str], temporary: bool = False
) -> DatabricksPath

Create a DatabricksPath in this workspace.

.. deprecated:: 0.8.31 Use :meth:open for byte IO (client.open("/Volumes/cat/sch/vol/x", "rb")), or :meth:path when you need the (non-opened) resource itself (client.path("/Volumes/cat/sch/vol/x")).

Parameters:

Name	Type	Description	Default
parts	Union[list[str], str]	Path parts or string to parse.	required
temporary	bool	Temporary path	False

Returns:

Type	Description
DatabricksPath	A DatabricksPath instance.

open

open(path: Any, mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Open path against this workspace and return an :class:IO cursor.

Defaults to a :class:DatabricksPath bound to this client — strings like /Volumes/cat/sch/vol/x or dbfs+volume:///cat/sch/vol/x dispatch to the right concrete subclass (DBFS / Volumes / Workspace). A pre-built :class:~yggdrasil.path.Path is opened verbatim so callers can mix in S3/HTTP/local paths without losing the workspace binding.

mode and **kwargs ride straight through to :meth:Path.open (which forwards to :meth:IO.open).

path

path(path: Any, *, temporary: bool = False, **kwargs: Any) -> 'Path'

Build a path bound to this workspace — without opening it.

The non-opening companion to :meth:open: strings like /Volumes/cat/sch/vol/x or dbfs+volume:///cat/sch/vol/x dispatch to the right concrete :class:DatabricksPath subclass (DBFS / Volumes / Workspace), while a pre-built :class:~yggdrasil.path.Path is returned verbatim so callers can mix in S3 / HTTP / local paths without losing the workspace binding.

Use this when you want the path resource itself — ls / stat / write_table / child navigation — and :meth:open when you want an :class:IO byte cursor. temporary and any extra **kwargs ride straight through to :meth:DatabricksPath.from_.

tmp_path

tmp_path(
    suffix: str | None = None,
    extension: str | None = None,
    max_lifetime: float | None = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
    base_path: str | None = None,
) -> "DatabricksPath"

Shared cache base under Volumes for the current user.

Parameters:

Name	Type	Description	Default
suffix	str | None	Optional suffix	None
extension	str | None	Optional extension suffix to append.	None
max_lifetime	float | None	Max lifetime of temporary path	None
catalog_name	str | None	Unity catalog name for volume path	None
schema_name	str | None	Unity schema name for volume path	None
volume_name	str | None	Unity volume name for volume path	None
base_path	str | None	Base temporary path	None

Returns:

Type	Description
'DatabricksPath'	A DatabricksPath pointing at the shared cache location.

lazy_property staticmethod

lazy_property(
    *, cache_attr: str, factory: Callable[[], T], use_cache: bool
) -> T

Public helper kept for backwards compatibility.

New properties on :class:DatabricksClient inline the lookup directly through self.__dict__ (one dict get, one dict set on miss) — the function-call + lambda overhead of routing every sub-service through this helper used to cost ~1.3 us per access on the hot path. Callers outside this class keep the same surface.

dataset

dataset(sql_or_table: str, *, schema: Any = None)

Return a :class:~yggdrasil.spark.tabular.Dataset from SQL or a table name.

Resolves the Spark session via :meth:spark (Databricks Connect) and builds a :class:Dataset directly — no intermediate executor hop::

dbc = DatabricksClient()
ds = dbc.dataset("SELECT * FROM main.sales.orders")
ds = dbc.dataset("main.sales.orders")

The result is a full :class:Dataset — call .map, .filter, .to_table, .toArrow, etc. on it.

parallelize

parallelize(
    inputs: "Any",
    function: "Callable | None" = None,
    *,
    schema: Any = None,
    byte_size: int = 128 * 1024 * 1024
)

Distribute function over inputs via Spark executors, or create a :class:~yggdrasil.spark.tabular.Dataset directly from inputs when no function is given::

dbc = DatabricksClient()
# With function
results = dbc.parallelize(urls, fetch, schema=output_schema)
# Without function — just wrap inputs as a Dataset
ds = dbc.parallelize(rows, schema=output_schema)

spark

spark(
    *dependencies: "Any",
    registry: "Optional[Any]" = None,
    check_public: bool = False,
    cache_dir: "Optional[Union[str, os.PathLike]]" = None,
    cluster: "str | Cluster | None" = None
)

Open a Databricks Connect session with auto-resolved deps.

Returns a live :class:pyspark.sql.SparkSession (Spark Connect variant) configured against this client's workspace host and credentials. The bound :class:DatabricksClient is stashed on the session as session.ygg_client so downstream helpers (UDFs, :class:Dataset extensions, ad-hoc resource lookups) can reach the same auth without an extra DatabricksClient.current() call.

Each dependency is classified once via :func:classify_dependency:

• Public PyPI specs ("ygg[data,databricks]==0.7.85", "numpy>=1.0", …) ride straight to the cluster via :meth:DatabricksEnv.withDependencies. ygg is always declared via :meth:default_ygg_spec — pinned to the driver's installed version with the [data, databricks] extras so the cluster sees the exact same runtime + pandas / numpy / databricks-sdk surface the driver is using. Override by passing an explicit ygg spec (e.g. client.spark("ygg==0.7.0") or client.spark("ygg") for an editable-mode rebuild).
• Editable installs (pip install -e .) get their local working copy built into a wheel whose version carries the local hostname (0.7.85+host.<host>). The wheel lives at /Workspace/Users/<me>/.ygg/pypi/simple/<pkg>/<wheel> (overridable via registry) and is re-uploaded on every load so the registry slot tracks the developer's working code.
• Private / non-PyPI installs get the same wheel-build + workspace-upload treatment, but lazily — the upload is skipped when the workspace slot already exists, so a team sharing one registry path only re-uploads on version bumps.

Both editable and private wheels are then handed to :meth:DatabricksEnv.withDependencies via the local:<path> prefix Databricks Connect understands; the wheel itself is read back from the workspace into a local cache so the spec is reachable from the driver process.

Serverless compute (the default — no cluster_id) wires deps through DatabricksEnv + withEnvironment; classic compute falls back to :meth:SparkSession.addArtifacts with pyfile=True since classic clusters don't honour the declarative environment.

Arguments:

• dependencies — variadic. Each entry is anything :func:classify_dependency accepts (PyPI spec string, bare module name, :class:os.PathLike, or any object with __module__). client.spark("polars", "my_internal", Path("/some/pkg")) is the headline shape; ygg[data,databricks] is appended automatically unless the caller already provides their own ygg spec.
• registry — a :class:WorkspacePyPIRegistry (or any shape its constructor accepts) to use as the shared wheel cache. Defaults to a registry rooted at /Workspace/Users/<me>/.ygg/pypi/simple so a single-user setup needs no configuration.
• check_public — when True, an HTTPS probe to pypi.org decides whether an installed dist counts as public. Off by default so an offline registry stays fast; turn on when shipping mixed public / private deps.
• cache_dir — local scratch dir used by the classic compute fallback (and for wheel materialization when no explicit registry is passed).

When a :class:pyspark.sql.SparkSession is already active in the process (notebook driver, an outer client.spark() call, a Databricks Job task), that session is returned as-is — dependency classification and wheel publishing are skipped, since the active session's environment is already fixed. The client is still stashed on it as session.ygg_client so downstream helpers find the same auth.

DatabricksPath

DatabricksPath(
    data: Any = None,
    *,
    url: URL | None = None,
    service: Optional[DatabricksService] = None,
    client: Optional["DatabricksClient"] = None,
    temporary: bool = False,
    retry_sleep: Optional[Callable[[float], None]] = None,
    singleton_ttl: Any = ...,
    **kwargs: Any
)

Bases: RemotePath, DatabricksResource

Abstract :class:RemotePath for Databricks namespaces.

Mutualizes the :class:DatabricksResource surface — self.service, the inherited client / sql properties, generic resource pickling — with :class:RemotePath's URL-keyed singleton machinery. A path is a resource: it has a service, the service has a client, the client routes SDK calls. Subclasses (:class:DBFSPath, :class:VolumePath, :class:WorkspacePath) get the client/sql accessors for free.

Registers under :attr:Scheme.DBFS (the dbfs:// family root) and acts as the dispatcher: :meth:from_url inspects the URL and forwards to the right concrete subclass (DBFS, Volumes, Workspace) based on the compound dbfs+<surface>:// scheme, or — for the legacy un-prefixed dbfs:// form — on the URL path's leading namespace (/Volumes/... → :class:VolumePath, /Workspace/... → :class:WorkspacePath, everything else → :class:DBFSPath).

Like every leaf :class:RemotePath, Databricks file paths are not singleton-cached (_SINGLETON_TTL = ... inherited): the workspace client lives on the shared :class:DatabricksClient singleton, so a path is a cheap redirector and two callers asking for the same URL get independent stat caches — a delete or external change seen through one never hides behind a sibling's stale snapshot. The per-class _INSTANCES dict is retained only so an explicit singleton_ttl= / :meth:to_singleton opt-in partitions away from a hot S3Path / HTTPPath construction. The Unity Catalog resources (UCCatalog / UCSchema / Volume) keep their own process-lifetime singleton identity.

explore_url property

explore_url: Optional[URL]

Workspace UI deep-link for this resource, or None.

Concrete resources (:class:Catalog, :class:Schema, :class:Volume, :class:Table, :class:SQLWarehouse, :class:Job, :class:VolumePath, …) override this to return the /explore/data/... / /sql/warehouses/... / /jobs/... URL that opens the resource in the workspace UI. The inherited :class:ExploreUrlRepr keys off the override — anything that returns a non-None URL gets a ClassName(<url>) repr (and a clickable _repr_html_) for free without restating it on every subclass.

sql property

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

full_path abstractmethod

full_path() -> str

Backend-native string form of this path's URL.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'DatabricksPath'

Construct the right concrete subclass from a Databricks URL.

Four URL shapes are supported on :class:DatabricksPath itself:

• dbfs+dbfs://, dbfs+volume://, dbfs+workspace:// — the compound :class:Scheme form, dispatched by URL scheme alone.
• dbfs+table://[creds@]host/<cat>/<sch>/<tbl>?… — Unity Catalog logical table; dispatches to :class:yggdrasil.databricks.table.table.Table (not a :class:DatabricksPath, but on the same scheme family).
• dbfs:// — un-qualified family URL. Dispatched by the URL path's leading namespace: /Volumes/... → :class:VolumePath, /Workspace/... → :class:WorkspacePath, anything else → :class:DBFSPath.

Concrete subclasses (DBFSPath / VolumePath / WorkspacePath) bypass the dispatcher and forward straight to cls(url=url). Catalog Explorer deep-links (https://<host>/explore/data/..., including a volume's ?volumePath= file selection) are recognized too, so a :class:VolumePath can be built straight from a URL copied out of the workspace UI.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

reserve

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

DBFSPath

DBFSPath(
    data: Any = None,
    *,
    url: URL | None = None,
    service: Optional[DatabricksService] = None,
    client: Optional["DatabricksClient"] = None,
    temporary: bool = False,
    retry_sleep: Optional[Callable[[float], None]] = None,
    singleton_ttl: Any = ...,
    **kwargs: Any
)

Bases: DatabricksPath

Path under /dbfs/... via the DBFS SDK API.

explore_url property

explore_url: Optional[URL]

Workspace UI deep-link for this resource, or None.

Concrete resources (:class:Catalog, :class:Schema, :class:Volume, :class:Table, :class:SQLWarehouse, :class:Job, :class:VolumePath, …) override this to return the /explore/data/... / /sql/warehouses/... / /jobs/... URL that opens the resource in the workspace UI. The inherited :class:ExploreUrlRepr keys off the override — anything that returns a non-None URL gets a ClassName(<url>) repr (and a clickable _repr_html_) for free without restating it on every subclass.

sql property

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

api_path property

api_path: str

Path as the DBFS SDK expects it — leading slash, no /dbfs/ prefix.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'DatabricksPath'

Construct the right concrete subclass from a Databricks URL.

Four URL shapes are supported on :class:DatabricksPath itself:

• dbfs+dbfs://, dbfs+volume://, dbfs+workspace:// — the compound :class:Scheme form, dispatched by URL scheme alone.
• dbfs+table://[creds@]host/<cat>/<sch>/<tbl>?… — Unity Catalog logical table; dispatches to :class:yggdrasil.databricks.table.table.Table (not a :class:DatabricksPath, but on the same scheme family).
• dbfs:// — un-qualified family URL. Dispatched by the URL path's leading namespace: /Volumes/... → :class:VolumePath, /Workspace/... → :class:WorkspacePath, anything else → :class:DBFSPath.

Concrete subclasses (DBFSPath / VolumePath / WorkspacePath) bypass the dispatcher and forward straight to cls(url=url). Catalog Explorer deep-links (https://<host>/explore/data/..., including a volume's ?volumePath= file selection) are recognized too, so a :class:VolumePath can be built straight from a URL copied out of the workspace UI.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

VolumePath

VolumePath(
    data: Any = None,
    *,
    url: "URL | None" = None,
    volume: "Volume | None" = None,
    service: Any = None,
    **kwargs: Any
)

Bases: DatabricksPath

Path under /Volumes/<cat>/<sch>/<vol>/... via the Files API.

Per-volume metadata (VolumeInfo, storage location, temporary credentials, AWS client) lives on the :class:Volume resource accessible via :attr:volume. Every :class:VolumePath pointing at the same UC volume collapses to the same :class:Volume singleton, so the SDK round trip and the auto-refreshing :class:AWSClient are shared process-wide.

sql property

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

catalog_name property

catalog_name: Optional[str]

The Unity Catalog catalog this volume lives under, or None.

schema_name property

schema_name: Optional[str]

The Unity Catalog schema this volume lives under, or None.

volume_name property

volume_name: Optional[str]

The Unity Catalog volume name, or None when the URL path doesn't address a volume.

volume property

volume: 'Volume'

Return the :class:Volume resource backing this path.

Lazily resolved on first access and cached on the instance. Because :class:Volume instances are singletons per (host, catalog, schema, name), every :class:VolumePath on the same UC volume shares the same live metadata cache, the same :class:VolumeInfo snapshot, and the same credentials refresher.

Raises :class:ValueError when the URL path doesn't address a volume (no /cat/sch/vol prefix).

catalog property

catalog: 'UCCatalog'

Return a :class:Catalog instance for this volume's parent catalog.

Delegates to :attr:volume.catalog so the underlying :class:Catalog instance is reused across every path on this UC volume.

Raises :class:ValueError when the URL path doesn't address a volume (no /cat/sch/vol prefix).

schema property

schema: 'UCSchema'

Return a :class:Schema instance for this volume's parent schema.

Delegates to :attr:volume.schema so the underlying :class:Schema instance is reused across every path on this UC volume.

Raises :class:ValueError when the URL path doesn't address a volume (no /cat/sch/vol prefix).

storage_location property

storage_location: str

This path's backing cloud-storage URL string — the volume's :attr:Volume.storage_location root with this path's suffix appended (s3://bkt/root/sub/file.bin), not the bare volume root. Raises :class:ValueError when the volume has no resolvable storage location.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'DatabricksPath'

Construct the right concrete subclass from a Databricks URL.

Four URL shapes are supported on :class:DatabricksPath itself:

• dbfs+dbfs://, dbfs+volume://, dbfs+workspace:// — the compound :class:Scheme form, dispatched by URL scheme alone.
• dbfs+table://[creds@]host/<cat>/<sch>/<tbl>?… — Unity Catalog logical table; dispatches to :class:yggdrasil.databricks.table.table.Table (not a :class:DatabricksPath, but on the same scheme family).
• dbfs:// — un-qualified family URL. Dispatched by the URL path's leading namespace: /Volumes/... → :class:VolumePath, /Workspace/... → :class:WorkspacePath, anything else → :class:DBFSPath.

Concrete subclasses (DBFSPath / VolumePath / WorkspacePath) bypass the dispatcher and forward straight to cls(url=url). Catalog Explorer deep-links (https://<host>/explore/data/..., including a volume's ?volumePath= file selection) are recognized too, so a :class:VolumePath can be built straight from a URL copied out of the workspace UI.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

volume_info

volume_info(refresh: bool = False) -> Any

Return the SDK's :class:VolumeInfo for this volume.

Delegates to :meth:Volume.read_info. The result is shared across every :class:VolumePath on this UC volume (via the :class:Volume singleton) and refreshed when the cached snapshot is past the 5-minute TTL.

Raises :class:ValueError when the URL path doesn't address a volume.

storage_path

storage_path(
    *, write: bool = False, region: Optional[str] = None, refresh: bool = False
) -> Path | None

This path's cloud-storage :class:Path (an :class:S3Path today) — the single entry point for direct external-storage access.

Resolves the volume's directly-reachable storage root (:meth:Volume.external_storage_root — gated on EXTERNAL + EXTERNAL USE SCHEMA + s3:// + (for a write) not a __unitystorage layout, the verdict cached per mode) and appends this path's suffix, so the byte / stat / list / remove primitives hit the object store directly — no Files-API hop, no Unity Catalog quota burn. Returns None when the volume isn't directly reachable for the requested write / read mode (managed volume, no grant, non-s3): the caller then uses the Files API. Never raises into the I/O flow.

external_location

external_location(*, refresh: bool = False) -> ExternalLocation | None

The Unity Catalog external location governing this path's backing storage (delegates to :meth:Volume.external_location), or None when the path doesn't address a volume, no accessible location covers it, or listing isn't permitted. Never raises.

can_read

can_read(*, refresh: bool = False) -> bool

Global precheck — can this path's storage be read directly at the cloud layer? Delegates to :meth:Volume.can_read; False when the path doesn't address a volume. Cheap + cached, never raises.

can_write

can_write(*, refresh: bool = False) -> bool

Global precheck — can this path's storage be written directly at the cloud layer? Delegates to :meth:Volume.can_write; False when the path doesn't address a volume. Cheap + cached, never raises.

temporary_credentials

temporary_credentials(*, mode: ModeLike = Mode.AUTO) -> Any

Vend temporary cloud credentials for this volume. Delegates to :meth:Volume.temporary_credentials.

credentials_refresher

credentials_refresher() -> 'AWSDatabricksVolumeCredentials'

Return the process-wide singleton credentials provider for this volume. Delegates to :meth:Volume.credentials_refresher.

aws

aws(*, mode: ModeLike = None, region: Optional[str] = None) -> 'AWSClient'

Return an :class:AWSClient whose credentials self-refresh from :meth:temporary_credentials. Delegates to :meth:Volume.aws.

arrow_filesystem

arrow_filesystem(*, operation: Any = None, region: Optional[str] = None) -> Any

Build a :class:pyarrow.fs.S3FileSystem for this volume.

Delegates to :meth:Volume.arrow_filesystem. operation is passed through as the credential mode.

WorkspacePath

WorkspacePath(
    data: Any = None,
    *,
    url: URL | None = None,
    service: Optional[DatabricksService] = None,
    client: Optional["DatabricksClient"] = None,
    temporary: bool = False,
    retry_sleep: Optional[Callable[[float], None]] = None,
    singleton_ttl: Any = ...,
    **kwargs: Any
)

Bases: DatabricksPath

Path under /Workspace/... via the Workspace API.

explore_url property

explore_url: Optional[URL]

Workspace UI deep-link for this resource, or None.

Concrete resources (:class:Catalog, :class:Schema, :class:Volume, :class:Table, :class:SQLWarehouse, :class:Job, :class:VolumePath, …) override this to return the /explore/data/... / /sql/warehouses/... / /jobs/... URL that opens the resource in the workspace UI. The inherited :class:ExploreUrlRepr keys off the override — anything that returns a non-None URL gets a ClassName(<url>) repr (and a clickable _repr_html_) for free without restating it on every subclass.

sql property

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'DatabricksPath'

Construct the right concrete subclass from a Databricks URL.

Four URL shapes are supported on :class:DatabricksPath itself:

• dbfs+dbfs://, dbfs+volume://, dbfs+workspace:// — the compound :class:Scheme form, dispatched by URL scheme alone.
• dbfs+table://[creds@]host/<cat>/<sch>/<tbl>?… — Unity Catalog logical table; dispatches to :class:yggdrasil.databricks.table.table.Table (not a :class:DatabricksPath, but on the same scheme family).
• dbfs:// — un-qualified family URL. Dispatched by the URL path's leading namespace: /Volumes/... → :class:VolumePath, /Workspace/... → :class:WorkspacePath, anything else → :class:DBFSPath.

Concrete subclasses (DBFSPath / VolumePath / WorkspacePath) bypass the dispatcher and forward straight to cls(url=url). Catalog Explorer deep-links (https://<host>/explore/data/..., including a volume's ?volumePath= file selection) are recognized too, so a :class:VolumePath can be built straight from a URL copied out of the workspace UI.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

create_notebook

create_notebook(
    language: str = "PYTHON",
    *,
    content: str | bytes | None = None,
    overwrite: bool = False
) -> "WorkspacePath"

Create a notebook at this Workspace path via the import API.

Imports content (or an empty body) as a notebook in language (PYTHON / SQL / SCALA / R) through workspace.upload(format=SOURCE, language=…) — the import endpoint that stores the object as a real notebook rather than a plain workspace file. The language-specific … Databricks notebook source magic header that Databricks stamps on exported notebooks is prepended when the body doesn't already carry it, so a hand-written .py / .sql body round-trips as a clean notebook. Parent directories are created as needed; overwrite replaces an existing object (otherwise a pre-existing path raises :class:FileExistsError).

run_notebook

run_notebook(
    parameters: Mapping[str, Any] | None = None,
    *,
    cluster: "Cluster | str | None" = None,
    environment: Any | str | None = None,
    run_name: str | None = None,
    timeout_seconds: int | None = None,
    wait: WaitingConfigArg = True,
    raise_error: bool = True,
    **submit_kwargs: Any
) -> "JobRun"

Submit this notebook as a one-time job run.

Wraps the notebook in a single SubmitTask / NotebookTask and fires it through :meth:Jobs.submit — a one-shot run that has a run_id but no persisted job_id. parameters are passed as the notebook task's base_parameters, so inside the run they land on the notebook's widget bindings: catchable by :class:yggdrasil.environ.SystemParameters (which reads the union of widgets + {{job.parameters.*}} via dbutils.notebook.entry_point.getCurrentBindings()) or directly with dbutils.widgets.get("<name>"). Values are stringified — the Databricks parameter channel is string-typed, and :class:SystemParameters casts them back to the declared field types on the way out.

Compute is defaulted for you:

• pass a cluster (:class:Cluster or cluster-id string) to pin existing compute;
• otherwise the run goes serverless, and the environment is resolved automatically — an explicit :class:JobEnvironment or a seeded base-environment name / .yml path is used as given, while the default (None) picks up the running client project's deployed environment when present, else the seeded ygg base environment under the shared project folder (/Workspace/Shared/environment/ygg/ygg-<version>-py3XX.yml), falling back to the workspace's default serverless compute when none is seeded.

This collapses the verbose dbc.jobs.submit(tasks=[SubmitTask( environment_key=…, notebook_task=NotebookTask(…))], environments=[…]) boilerplate into one call.

wait defaults to True (block until terminal) — a notebook run is usually awaited for its dbutils.notebook.exit result; pass wait=False to fire-and-forget, or a number for a timeout in seconds. raise_error raises on terminal failure when waiting. Returns the :class:JobRun.

Example::

nb = WorkspacePath("/Workspace/Shared/etl")
run = nb.run_notebook({"date": "2024-01-01"})
out = run.task_output("etl").notebook_output.result

# pin a named, seeded serverless environment
nb.run_notebook({"category": "wind"}, environment="meteologica")

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "WorkspacePath"

Pack a local module / package and import it into the workspace.

Mirrors the :meth:Path.upload_module contract but bypasses the generic read_bytes() round-trip: the produced .zip is streamed straight into workspace.upload from an open file handle, so a large archive doesn't get materialized into a Python bytes object before the upload.

Destination resolution stays consistent with the base: self with a .zip / .whl suffix is taken verbatim; anything else gets self / <name or "<module>.zip">. The format hint is :class:ImportFormat.AUTO — the Workspace API detects the binary content type and stores the object as a workspace file (not a notebook).

Genie

Genie(client: Optional[DatabricksClient] = None)

Bases: DatabricksService

Collection-level Databricks Genie service (dbc.genie).

wheels property

wheels: 'Wheels'

Wheel registry service (shorthand for client.wheels).

environments property

environments: 'Environments'

Base-environment service (shorthand for client.environments).

tables property

tables: 'Tables'

Collection-level Unity Catalog table service (shorthand for client.tables).

views property

views: 'Tables'

Alias for :attr:tables — :class:Table covers both managed/external tables and view-shaped securables.

catalogs property

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service (shorthand for client.catalogs).

schemas property

schemas: 'Schemas'

Collection-level Unity Catalog schema service (shorthand for client.schemas).

volumes property

volumes: 'Volumes'

Collection-level Unity Catalog volume service (shorthand for client.volumes).

genie property

genie: 'Genie'

Genie service (shorthand for client.genie).

ai property

ai: 'DatabricksAI'

Databricks AI umbrella service (shorthand for client.ai).

api property

api

The underlying SDK GenieAPI bound to this client's workspace.

default_tags

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

Returns:

Type	Description
dict[str, str]	A dict of default tags.

list_spaces

list_spaces(*, page_size: Optional[int] = None) -> Iterator[GenieSpace]

Iterate every Genie space, transparently paging the API.

spaces

spaces() -> list[GenieSpace]

All Genie spaces as a list.

space

space(space_id: str) -> GenieSpace

A :class:GenieSpace handle for space_id (metadata is lazy).

find

find(title: str) -> Optional[GenieSpace]

Find a space by exact title, or None.

create_space

create_space(
    warehouse_id: str,
    serialized_space: str,
    *,
    title: Optional[str] = None,
    description: Optional[str] = None,
    parent_path: Optional[str] = None
) -> GenieSpace

Create a Genie space from a serialized definition.

ask

ask(
    space_id: str, question: str, *, timeout: timedelta = _DEFAULT_TIMEOUT
) -> GenieAnswer

Ask question of space_id in a fresh conversation and wait.

GenieAnswer

GenieAnswer(
    message: "GenieMessage",
    *,
    client: "DatabricksClient",
    space_id: str,
    warehouse_id: Optional[str] = None
)

The completed result of a Genie turn — text + generated SQL + rows.

text property

text: str

Genie's natural-language reply (all text attachments joined).

query property

query: Optional[str]

The SQL Genie generated for this turn, if it answered with a query.

result

result()

Re-attach to the generated query's result as a StatementResult.

Reuses the SQL engine's statement re-attach path, so the rows are read on the space's warehouse without re-running the query. Raises when the turn produced no query.

GenieSpace

GenieSpace(
    space_id: str,
    *,
    client: "DatabricksClient",
    info: "Optional[SDKGenieSpace]" = None
)

A single Genie space — resource-style lifecycle + Q&A.

info property

info: 'SDKGenieSpace'

The SDK GenieSpace metadata (fetched once, then cached).

refresh

refresh() -> 'GenieSpace'

Re-fetch the space metadata from the workspace.

update

update(
    *,
    title: Optional[str] = None,
    description: Optional[str] = None,
    warehouse_id: Optional[str] = None,
    serialized_space: Optional[str] = None,
    parent_path: Optional[str] = None
) -> "GenieSpace"

Update the space's settings; refreshes the cached info.

trash

trash() -> None

Move the space to trash (delete).

ask

ask(question: str, *, timeout: timedelta = _DEFAULT_TIMEOUT) -> GenieAnswer

Start a new conversation, ask question, and wait for the answer.

follow_up

follow_up(
    conversation_id: str,
    question: str,
    *,
    timeout: timedelta = _DEFAULT_TIMEOUT
) -> GenieAnswer

Ask a follow-up question in an existing conversation.

conversations

conversations() -> 'Iterator[GenieConversation]'

Iterate the space's conversations (most recent first).

messages

messages(conversation_id: str) -> 'Iterator[GenieMessage]'

Iterate the messages of one conversation.

Volume

Volume(
    data: Any = None,
    service: "Volumes | None" = None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
    *,
    client: DatabricksClient | None = None,
    infos: VolumeInfo | None = None,
    infos_fetched_at: float | None = None,
    infos_ttl: float | None = None,
    singleton_ttl: "int | None" = ...,
    **kwargs
)

Bases: DatabricksPath

A single Unity Catalog volume — metadata, credentials, storage path.

Construct via :meth:Volumes.volume / :meth:Volumes.__getitem__ for the cache-aware path, or directly when callers already have the coordinates. Either way the instance is interned per (client, catalog, schema, name) so subsequent calls collapse to the same live cache — same convention :class:Catalog / :class:Schema / :class:Table use.

sql property

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

workspace_client property

workspace_client: Any

Shortcut for self.client.workspace_client() — the live Databricks SDK workspace handle every SDK call routes through.

infos property

infos: VolumeInfo

Alias for :attr:info — matches the Schema / Table shape.

info property

info: VolumeInfo

Cached :class:VolumeInfo (15-minute TTL by default).

storage_location property writable

storage_location: str | None

Volume's backing storage URL string (e.g. s3://bucket/...).

Pure read from :meth:read_info — no AWS auth resolution, no Path construction. Use this when you only need the URL rendering for logging / config plumbing; reach for :meth:storage_path when you'll do actual I/O against the location.

catalog property

catalog: 'UCCatalog'

Navigate up to the parent :class:Catalog.

Returns the singleton-cached :class:Catalog for this client + catalog name — repeated calls hand back the same instance with shared :class:CatalogInfo cache. The per-instance _catalog slot is kept as a one-attribute shortcut so the navigation path skips the client.catalogs.catalog(...) round trip on hot loops.

schema property writable

schema: 'UCSchema'

Navigate up to the parent :class:Schema.

Returns the singleton-cached :class:Schema for this client + (catalog, schema) — repeated calls hand back the same instance with shared :class:SchemaInfo cache. The per-instance _schema slot is kept as a one-attribute shortcut so the navigation path skips the client.schemas.schema(...) round trip on hot loops.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'DatabricksPath'

Coerce obj (string / URL / :class:Path / dict) into the right concrete subclass.

On the abstract :class:DatabricksPath, this is the friendly entry point: POSIX strings like /Volumes/cat/sch/vol/x are coerced through :func:_coerce_to_url_str and routed by scheme to :class:VolumePath / :class:WorkspacePath / :class:DBFSPath, while compound dbfs+...:// URLs dispatch by scheme alone (including dbfs+table:// → :class:Table). On a concrete subclass, the call returns an instance of that subclass without redispatching — the standard :meth:Path.from_ contract.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'DatabricksPath'

Join segments onto this path, always extending it.

The bare :class:Holder join follows pathlib semantics, where a segment with a leading / resets to an absolute path and a trailing / duplicate slash leaves an empty component. A Databricks path is anchored in a namespace it must not escape, and the logical handles (:class:UCCatalog / :class:UCSchema / :class:Volume) pick the child type from the path's segment count — so every join goes through :func:_relative_join_parts first. A single multi-part string ("a/b/c"), several segments, embedded / trailing / duplicate slashes, and . components all flatten into clean relative components, so cat / "sales/raw" reliably reaches the volume and cat / "sales/raw/" doesn't over-count into a VolumePath.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Range read with an aggressive whole-file fast path.

The base :meth:Holder.read_mv runs self.size (an :meth:_stat probe) to convert n < 0 into a concrete byte count and to bounds-check the requested window. On Databricks backends that probe costs a Unity Catalog / Workspace round trip every read — wasted for read_bytes() / read_arrow_table() and other "give me everything" calls, because each backend's :meth:_read_mv already handles EOF natively (chunked-until-short-page on DBFS, full-object download on Volumes / Workspace).

Whole-file shape (n < 0 and pos == 0) skips the size probe entirely. Partial / positional reads keep the base bounds check so out-of-range windows still raise.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve

reserve(n: int) -> None

No-op — Databricks backends have no capacity layer.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

external_access

external_access(*, write: bool) -> Optional[bool]

Cached per-mode direct-storage accessibility: True (usable), False (not usable / denied), or None (not yet determined — the caller should resolve it and record the result).

mark_external_ok

mark_external_ok(*, write: bool) -> None

Cache that this volume's storage is directly reachable for the mode — subsequent ops skip the eligibility check.

mark_external_denied

mark_external_denied(*, write: bool) -> None

Cache that this volume's storage is not directly reachable for the mode (ineligible, or a permission error), so the fast path isn't retried.

external_storage_root

external_storage_root(
    *, write: bool, region: Optional[str] = None, refresh: bool = False
) -> Path | None

The volume's backing cloud-storage root :class:Path (an :class:S3Path today) when this volume is directly reachable for write / read — else None (the caller stays on the Files-API :class:VolumePath).

Eligibility — volume is EXTERNAL, the current identity holds EXTERNAL USE SCHEMA on its schema, the location is s3://, and — for a write — it isn't a UC-managed __unitystorage / __unitycatalog layout (where direct PutObject is denied) — is decided once per mode and cached on this singleton via :meth:external_access, so the check doesn't re-run on every path build / I/O. The root itself is resolved through :meth:storage_path (the single cloud-storage resolution point). Never raises into the caller — any metadata / vend / credential failure resolves to None and the Files-API path takes over.

external_location

external_location(*, refresh: bool = False) -> ExternalLocation | None

The Unity Catalog external location governing this volume's backing storage — the most specific one whose URL the volume's storage_location sits under (longest-prefix match over the cached external-location list, :meth:ExternalLocations.find_url) — or None when the volume has no resolvable storage or no accessible location covers it. Never raises.

Memoised on the volume: resolved once and reused (a resolved None is cached too), so repeated external_location / can_read / can_write calls don't re-walk the location list. refresh=True re-resolves (and re-lists); the memo is dropped on any info refresh / rebind (:meth:_store_infos / :meth:_reset_cache).

can_read

can_read(*, refresh: bool = False) -> bool

Global precheck — can this volume's storage be read at the cloud layer? True when an accessible external location covers it. Cheap and cached (no per-object probe); use it to gate bulk work before paying for the first I/O.

can_write

can_write(*, refresh: bool = False) -> bool

Global precheck — can this volume's storage be written at the cloud layer? True when a covering external location exists and is not read-only.

full_name

full_name(safe: str | bool | None = None) -> str

Return the three-part volume name, optionally backtick-quoted.

from_url classmethod

from_url(url: 'URL | str', **kwargs: Any) -> 'Volume'

Build a :class:Volume from a dbfs+volume:///cat/sch/vol URL.

Used by the :class:DatabricksPath dispatcher when a caller passes a POSIX volume path that resolves exactly to volume depth (DatabricksPath("/Volumes/main/sales/staging") → Volume("main", "sales", "staging")); deeper paths resolve to :class:VolumePath instead.

clear

clear() -> 'Volume'

Public alias for :meth:_reset_cache; returns self.

read_info

read_info(*, refresh: bool = False, default: Any = ...) -> VolumeInfo

Return the SDK's :class:VolumeInfo for this volume.

Refreshes whenever the cached entry is past :attr:DEFAULT_INFO_TTL (15 minutes by default), or when refresh=True forces it. If the underlying volumes.read raises :class:NotFound, auto-creates the volume (and any missing catalog / schema parents) and retries the read once.

exists

exists() -> bool

True if this volume is reachable via the Unity Catalog API.

Always hits the API (refresh=True): exists is a liveness probe, and a cached :class:VolumeInfo (5-minute TTL) would keep reporting True for minutes after the volume was dropped — e.g. right after a delete / a peer's cascade teardown. On a clean not-found the stale cache is dropped so a following :attr:info doesn't resurrect the deleted volume's metadata.

storage_path

storage_path(
    *,
    mode: ModeLike = Mode.AUTO,
    region: Optional[str] = None,
    refresh: bool = False
) -> Path

Return the volume's root storage :class:Path.

Dispatches to the right :class:URLBased subclass — :class:S3Path for s3://, generic Path registry for anything else. The returned Path is cached on the instance per credential scope (read-only vs write) and (for S3) carries the auto-refreshing :class:AWSClient session minted via :meth:credentials_refresher.

temporary_credentials

temporary_credentials(*, mode: ModeLike = Mode.AUTO) -> Any

Vend temporary cloud credentials for this volume.

Wraps temporary_volume_credentials.generate_temporary_volume_credentials. Read-only modes map to READ_VOLUME; everything else to WRITE_VOLUME.

credentials_refresher

credentials_refresher(
    *, secret_cache: bool = False
) -> AWSDatabricksVolumeCredentials

Return the process-wide singleton credentials provider for this volume.

Keyed by volume_id — every :class:Volume / :class:VolumePath pointing at the same UC volume collapses to one provider.

secret_cache=True opts the provider into persisting its vended AWS credentials in a per-volume Databricks secret scope (off by default); the opt-in is sticky across the shared singleton.

aws

aws(
    *,
    mode: ModeLike = None,
    region: Optional[str] = None,
    secret_cache: bool = False
) -> "AWSClient"

Return an :class:AWSClient whose credentials self-refresh from :meth:temporary_credentials.

secret_cache=True backs the vended credentials with a per-volume Databricks secret scope (off by default).

arrow_filesystem

arrow_filesystem(*, mode: ModeLike = None, region: Optional[str] = None) -> Any

Build a :class:pyarrow.fs.S3FileSystem for this volume.

path

path(
    sub: str = "",
    *,
    url: URL | None = None,
    volume: "Volume | None" = None,
    service: Any = None,
    temporary: bool = False,
    **kwargs
) -> "VolumePath"

Return a :class:VolumePath rooted at this volume.

sub is appended under the volume root ("" → the volume itself, "sub/x.parquet" → /Volumes/<cat>/<sch>/<vol>/sub/x.parquet).

create

create(
    *,
    refresh: bool = False,
    comment: str | None = None,
    storage_location: str | None = None,
    volume_type: "VolumeType | str" = None,
    missing_ok: bool = True
) -> "Volume"

Create this volume in Unity Catalog.

Idempotent — a successful read means it already exists (reads never auto-create). On a not-found create error the missing parent schema (and, through it, the catalog) is created and the create retried once.

Defaults to a managed volume. Pass storage_location + volume_type="EXTERNAL" for an external volume.

get_or_create

get_or_create(
    *,
    comment: str | None = None,
    storage_location: str | None = None,
    volume_type: Any = None
) -> "Volume"

Create this volume (and any missing parents) if it doesn't exist, then return self. :meth:create is itself idempotent and ensures the parents on a not-found, so this is just a named alias.

permissions

permissions(*, principal: str | None = None) -> tuple[PrivilegeAssignment, ...]

Direct grants on this volume (no inherited privileges).

grant

grant(
    principal: str, privileges: "str | Privilege | list[str | Privilege]"
) -> "Volume"

Add one or more privileges for principal on this volume.

revoke

revoke(
    principal: str, privileges: "str | Privilege | list[str | Privilege]"
) -> "Volume"

Remove one or more privileges for principal on this volume.

update_permissions

update_permissions(
    changes: "list[PermissionsChange | Mapping[str, Any]]",
) -> "Volume"

Apply a batch of PermissionsChange to this volume.

Volumes

Volumes(
    client=None,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
)

Bases: DatabricksService

Collection-level service for Unity Catalog volumes.

wheels property

wheels: 'Wheels'

Wheel registry service (shorthand for client.wheels).

environments property

environments: 'Environments'

Base-environment service (shorthand for client.environments).

tables property

tables: 'Tables'

Collection-level Unity Catalog table service (shorthand for client.tables).

views property

views: 'Tables'

Alias for :attr:tables — :class:Table covers both managed/external tables and view-shaped securables.

catalogs property

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service (shorthand for client.catalogs).

schemas property

schemas: 'Schemas'

Collection-level Unity Catalog schema service (shorthand for client.schemas).

volumes property

volumes: 'Volumes'

Collection-level Unity Catalog volume service (shorthand for client.volumes).

genie property

genie: 'Genie'

Genie service (shorthand for client.genie).

ai property

ai: 'DatabricksAI'

Databricks AI umbrella service (shorthand for client.ai).

default_tags

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

Returns:

Type	Description
dict[str, str]	A dict of default tags.

volume

volume(
    location: "Volume | str | None" = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None
) -> Volume

Return a :class:Volume bound to this service.

Routes through the singleton cache so repeated calls collapse to the same instance.

create

create(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
    comment: str | None = None,
    storage_location: str | None = None,
    volume_type=None,
    missing_ok: bool = True
) -> Volume

Create a volume by name, auto-creating missing schema / catalog.

Service-level entry point matching :meth:Warehouses.create — resolves the catalog / schema / volume parts (from location or keyword overrides, falling back to the service defaults), materialises the :class:Volume singleton, then delegates to :meth:Volume.create so the managed-volume-type default and the post-create _store_infos cache warm-up live in one place.

get_or_create

get_or_create(
    location: "str | Volume | None" = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
    storage_location: str | None = None,
    comment: str | None = None,
    volume_type=None
) -> Volume

Get-or-create a volume — external by storage URI, else managed by dotted name.

When the first argument (or storage_location) is a cloud object-store URI (s3://… / abfss://… / gs://… — see :data:EXTERNAL_LOCATION_SCHEMES), returns or creates an external volume backed by that location. volume_name is required and names the volume explicitly; catalog_name / schema_name default to the service scope. (An external location + storage credential covering the URI must already exist in Unity Catalog.)

Otherwise location is a 1-/2-/3-part dotted name (cat.sch.vol) and a managed volume is returned, created if absent. Mirrors the find-then-create shape of :meth:Jobs.create_or_update.

find_remote

find_remote(
    catalog_name: str,
    schema_name: str,
    volume_name: str,
    *,
    raise_error: bool = True
) -> Optional[VolumeInfo]

Raw API lookup — GET by fully-qualified name, no cache.

Returns None on miss when raise_error=False.

find

find(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None,
    raise_error: bool = True
) -> Optional[Volume]

Resolve a volume by name.

Caching lives on the :class:Volume singleton itself; this method drives a remote read when the cached info is stale (or missing) and seeds the resulting info onto the instance.

list

list(
    name: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None
) -> Iterator[Volume]

Iterate over visible volumes in the resolved catalog / schema scope.

catalog_name and schema_name default to this service's attached scope. When both are set the underlying SDK call is a single volumes.list against cat.sch; otherwise the iteration fans out across the matching catalogs / schemas.

parse_location

parse_location(
    location: str | None = None,
    *,
    catalog_name: str | None = None,
    schema_name: str | None = None,
    volume_name: str | None = None
) -> tuple[Optional[str], Optional[str], Optional[str]]

Parse a 1- / 2- / 3-part dotted name into (catalog, schema, volume).

Keyword overrides take precedence over parts extracted from location. Service defaults fill any remaining blanks.

invalidate

invalidate(
    volume: "Volume | str | None" = None,
    *,
    catalog_name: Optional[str] = None,
    schema_name: Optional[str] = None,
    volume_name: Optional[str] = None
) -> None

Drop one volume's singleton instance.

The next client.volumes[...] (or Volume(...)) call will mint a fresh instance with an empty info cache — i.e. forces a re-read of :class:VolumeInfo on first access.

invalidate_all classmethod

invalidate_all() -> None

Clear the entire singleton cache.