yggdrasil.http_

http_

HTTP transport for yggdrasil — single entry point :class:HTTPSession.

Construct an :class:HTTPSession (singleton-cached per config) and drive HTTP through its inherited verb methods; the returned :class:HTTPResponse carries the body, headers, status, and request bound on a single object. The supporting types (:class:HTTPRequest, :class:HTTPPath, :class:Cookies) round out the surface for paths and cookie jars.

The transport-shape side modules (:mod:yggdrasil.http_.retry, :mod:yggdrasil.http_.timeout, :mod:yggdrasil.http_.exceptions, :mod:yggdrasil.http_.headers) hold the stdlib-only, urllib3-shaped primitives. :class:HTTPSession IS the connection pool — no separate PoolManager indirection.

Submodule imports are deferred via PEP 562 __getattr__ so importing :mod:yggdrasil.http_.exceptions (used by :mod:yggdrasil.exceptions.http) does not pull the full session / response / path chain — that chain transitively reaches back into :mod:yggdrasil.exceptions and would otherwise close the loop.

HTTPSession

HTTPSession(
    base_url: Optional[URL | str] = None,
    verify: "bool | str | pathlib.Path" = True,
    pool_maxsize: int = 10,
    headers: "HTTPHeaders | Mapping[str, str] | None" = None,
    waiting: WaitingConfig = DEFAULT_WAITING_CONFIG,
    *,
    auth: Optional[Authorization] = None,
    proxy: Optional[URL | str] = None,
    no_proxy: Optional[str] = None
)

Bases: Session

HTTP/HTTPS session — singleton-keyed by (base_url, verify, pool_maxsize, headers, waiting, auth).

Inherits the singleton + pickle + job_pool plumbing from :class:~yggdrasil.io.session.Session and adds every HTTP-specific concern: a stdlib-backed connection pool (HTTPSession owns the per-host socket cache directly — no separate PoolManager indirection) connection pool, the prepare → cache-lookup → wire-send → cache-writeback pipeline (both single-request :meth:send and bulk :meth:send_many), Spark fan-out via mapInArrow, the verb sugar (:meth:get / :meth:post / :meth:put / :meth:patch / :meth:delete / :meth:head / :meth:options / :meth:request), cookie-header coercion, and the 403 auth-refresh retry loop.

No User-Agent generator, cookie jar, or browser-emulation layering is built in — per-vendor integrations subclass this for their own auth / pagination / rate-limit policy (see :class:yggdrasil.fxrate.FxRate for a worked example).

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Pop self from the per-class _INSTANCES cache.

Mutating ops on a Singleton-cached object (writes, deletes, schema invalidations on a Databricks table, put_object on an :class:S3Path) want to make sure the next caller asking for the same key gets a fresh build rather than collapsing onto this stale handle — that's what remove_global=True (the default) does. The pop is :meth:identity-guarded: only an entry that still points at self is removed, so a concurrent re-construction that already raced past this thread is left alone.

remove_global=False is a no-op. The keyword exists so subclass invalidators (invalidate_singleton, _invalidate_entity_tag_cache, …) can offer the same switch without branching at the call site.

local_cache

local_cache() -> 'Folder'

Return the session-scoped local cache folder, creating the directory on first access.

The folder lives under ~/.cache/http/<host>/<path> when :attr:base_url is set (so different APIs on the same machine don't collide), or ~/.cache/http/default otherwise.

Thread-safe: the directory is created under the session lock and the resulting :class:Folder is cached for the lifetime of the singleton.

clear_connections

clear_connections() -> None

Close every cached idle connection.

Lifecycle convenience — closes the per-host sockets the session accumulated. Not called automatically; explicit cleanup is the caller's responsibility (or rely on process exit).

fetch

fetch(
    method: str,
    url: URL | str,
    *,
    headers: Optional[Mapping[str, str]] = None,
    body: Any = None,
    timeout: Any = None,
    preload_content: bool = False,
    decode_content: bool = True,
    redirect: bool = True,
    tags: Optional[Mapping[str, str]] = None
) -> HTTPResponse

Low-level wire fetch — bypasses the cache / auth-refresh pipeline.

Build a synthetic :class:HTTPRequest from method / url / headers / body and run it through the same retry + redirect machinery :meth:_send_http does for the regular :meth:send path. Useful when the caller just wants a raw byte stream off a URL — Databricks external-link readers feeding :func:pa.input_stream are the canonical case.

insecure

insecure() -> 'HTTPSession'

Return a session with SSL verification disabled.

If self already has verify=False, returns self. Otherwise returns a new :class:HTTPSession with the same base_url / headers / waiting / auth / proxy / no_proxy but verify=False.

Emits :class:InsecureRequestWarning on first construction.

send

send(
    request: HTTPRequest,
    config: SendConfig | Mapping[str, Any] | None = None,
    *,
    wait: WaitingConfigArg = ...,
    raise_error: bool = ...,
    remote_cache: CacheConfig | Mapping[str, Any] | None = ...,
    local_cache: CacheConfig | Mapping[str, Any] | None = ...,
    cache_only: bool = ...,
    spark_session: Optional["SparkSession"] = ...,
    start: bool = True,
    **options
) -> HTTPResponse

Prepare, dispatch, and (optionally) await the response.

Single-request entry point — always returns a :class:Response. adds no value.

When spark_session is bound (or True / ... resolved via :meth:PyEnv.spark_session), the wire send is fanned out to an executor through the same mapInArrow path :meth:send_many uses — the driver does not cross the wire. Otherwise the call runs synchronously on the local pool.

start=True (default) fires the wire call. start=False builds the prepared request + response shell without crossing the wire — the :class:StatementExecutor override uses the same knob to return an idled :class:StatementResult whose backend submission is deferred until :meth:StatementResult.start fires. Plain HTTP sessions don't need an idle :class:Response (the network call is synchronous), so the base raises a clean NotImplementedError via :meth:_build_idle_response.

Per-request :attr:HTTPRequest.send_config is used as the base when no explicit config is passed — explicit kwargs still override individual fields.

refresh_auth

refresh_auth(request: HTTPRequest, force: bool = False) -> bool

Resolve the auth handler and stamp the Authorization header.

Called automatically by _local_send on 401/403 responses. Per-request request.auth wins over session-wide self.auth.

Returns True when a handler ran and the header was stamped, False when no handler is bound and force=False.

Override this method in your session subclass to implement custom auth flows (query-param tokens, API-key rotation, challenge- response, etc.)::

class MySession(HTTPSession):
    def refresh_auth(self, request, force=True):
        request.headers["X-API-Key"] = self._rotate_key()
        return True

prepare_request_before_send

prepare_request_before_send(request: HTTPRequest) -> HTTPRequest

Session-wide request hook fired once per outbound request.

Stamps the session reference, sent_at timestamp, merged headers, and auth. When the request's local_cache has received_from / received_to but no tabular, fills in the session's default local-cache folder. A bare session.get(url) with no cache config does no disk I/O.

prepare_response_after_received

prepare_response_after_received(response: HTTPResponse) -> HTTPResponse

Session-wide response hook fired once per completed network send.

Default returns response unchanged. Subclasses override to log, redact, enrich, or wrap responses returned from the wire. Runs in :meth:_send after :meth:_local_send and before cache writeback, so the persisted response reflects any post-processing. Cache hits bypass it. Travels with the session into Spark workers via __getstate__ / __setstate__.

send_many

send_many(
    requests: Iterator[HTTPRequest],
    config: SendConfig | Mapping[str, Any] | None = None,
    *,
    wait: WaitingConfigArg = ...,
    raise_error: bool = ...,
    remote_cache: CacheConfig | Mapping[str, Any] | None = ...,
    local_cache: CacheConfig | Mapping[str, Any] | None = ...,
    cache_only: bool = ...,
    spark_session: Optional["SparkSession"] = ...,
    batch_size: int | None = None,
    ordered: bool = False,
    max_in_flight: int | None = None,
    max_batch_ttl: float | None = None,
    **options
) -> Iterator[HTTPResponse]

Stream responses for a batch of requests.

Batch orchestration kwargs (batch_size, ordered, max_in_flight, max_batch_ttl) control chunking and concurrency; everything else is folded into a :class:SendConfig that gets stamped on each request.

Send-config kwargs default to ... (Ellipsis). When left unset the per-request :attr:HTTPRequest.send_config is preserved; an explicit value overrides that field on every request in the batch.

send_many_batches

send_many_batches(
    requests: Iterator[HTTPRequest],
    *,
    wait: WaitingConfigArg = ...,
    raise_error: bool = ...,
    remote_cache: CacheConfig | Mapping[str, Any] | None = ...,
    local_cache: CacheConfig | Mapping[str, Any] | None = ...,
    cache_only: bool = ...,
    spark_session: Optional["SparkSession"] = ...,
    batch_size: int | None = None,
    ordered: bool = False,
    max_in_flight: int | None = None,
    max_batch_ttl: float | None = None
) -> Iterator[HTTPResponseBatch]

Yield one :class:HTTPResponseBatch per processed chunk.

Send-config kwargs default to ... (Ellipsis). When left unset the per-request :attr:HTTPRequest.send_config is preserved; an explicit value overrides that field on every request in the batch.

HTTPResponse

HTTPResponse(
    request: HTTPRequest,
    status_code: int,
    headers: MutableMapping[str, str],
    tags: MutableMapping[str, str],
    buffer: Any,
    received_at: datetime,
    receiver: "Optional[UserInfo]" = None,
)

Bases: IO

HTTP response model — paired with the originating :class:HTTPRequest.

Implements :class:Tabular over the deterministic metadata projection: :meth:read_arrow_batches (and the engine fan-out derived from it) yields a single Arrow row built from :attr:arrow_values, matching :data:RESPONSE_SCHEMA.to_arrow_schema(). To parse the body as a tabular payload (Parquet, CSV, JSON …), open a cursor with :meth:open and use that cursor's Tabular surface — :class:IO dispatches to the right format leaf via the holder's media type — or call the existing :meth:to_arrow_batches with parse=True.

The Tabular dispatch flips the same parse knob through :class:ResponseOptions — pass parse=False to force the metadata projection even when the body would otherwise auto-parse.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url property writable

url: 'URL'

Canonical URL identifying this holder.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size property

size: int

Current visible size in bytes.

Cursor / format-leaf IOs read the bound parent's size; storage subclasses override directly.

size_known property

size_known: bool

True when reading :attr:size won't trigger a backend probe.

Always true for in-memory IOs (size is a slot). Path IOs override to True only when their stat cache is warm — callers that want to short-circuit on an empty buffer (parquet / arrow IPC / CSV readers checking size == 0) can guard the check on this predicate so a cold remote path doesn't pay a HeadObject / get_status / get_metadata round trip just to discover the file is non-empty. Cursor / format-leaf IOs delegate to the parent.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

mtime property

mtime: float

Last-modified time stamp.

is_memory property

is_memory: bool

True when the IO lives entirely in process memory.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (:class:Memory) override directly.

is_local_path property

is_local_path: bool

True when the IO is a path on the local filesystem.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (:class:LocalPath) override directly.

is_remote_path property

is_remote_path: bool

True when the IO is a path on a non-local backend.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (remote paths) override directly.

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

holder property

holder: Holder

The :class:Holder backing the response body — alias for :attr:buffer.

receiver property

receiver: UserInfo | None

:class:UserInfo snapshot for the side that received this response.

Defaults to UserInfo.current() at construction time. Use :meth:with_receiver to swap it.

partition_key property

partition_key: int

xxh3_64 of the joined :meth:partition_values — int64 partition column.

body_hash property

body_hash: int

xxh3_64 digest of the body bytes; 0 when body is absent.

Non-nullable in the schema — callers that need a "missing" signal should branch on :attr:buffer or :attr:body_size.

hash property

hash: int

Overall identity over (request.hash, status_code, headers, body).

public_hash property

public_hash: int

Cross-system identity stable across anonymize='remove'.

status property

status: int

urllib3-shaped alias for :attr:status_code.

data property

data: bytes

All body bytes from offset 0 (urllib3-shaped .data).

path property

path: 'HTTPPath'

:class:HTTPPath view of the request URL.

Bound to the response's attached :class:HTTPSession (when present) so the HTTPPath reuses the same connection pool. Useful for re-fetching the resource, issuing a HEAD probe via :meth:HTTPPath.stat, or doing a follow-up PUT / DELETE.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url classmethod

from_url(url: URL, **kwargs) -> 'IO'

Create a new IO from a URL.

When cls is abstract (has subclasses but isn't itself constructible — e.g. :class:Path), the URL scheme is resolved through the :class:URLBased registry to a concrete subclass; an unknown scheme raises :class:ValueError instead of producing the obscure "Can't instantiate abstract class" :class:TypeError.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Pop self from the per-class _INSTANCES cache.

Mutating ops on a Singleton-cached object (writes, deletes, schema invalidations on a Databricks table, put_object on an :class:S3Path) want to make sure the next caller asking for the same key gets a fresh build rather than collapsing onto this stale handle — that's what remove_global=True (the default) does. The pop is :meth:identity-guarded: only an entry that still points at self is removed, so a concurrent re-construction that already raced past this thread is left alone.

remove_global=False is a no-op. The keyword exists so subclass invalidators (invalidate_singleton, _invalidate_entity_tag_cache, …) can offer the same switch without branching at the call site.

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'IO'

Build a sibling IO at self.url joined with segments.

URL-shaped IOs (:class:LocalPath, remote paths) use this to mint a child path; :class:Memory and other non-URL leaves raise :class:ValueError.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Slice size bytes from offset as a :class:memoryview.

cursor=True ignores the explicit offset and reads from the holder's internal cursor (:attr:tell), advancing it past the bytes returned. cursor=False (default) keeps the cursor-less positional contract — the cursor is untouched.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's bounds-check uses its own size — avoids a redundant stat probe on remote backings when the cursor has no local size cache, and routes through any subclass _active override (lazy materialization on :class:ZipEntryFile, …).

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Splice data at offset, pre-growing the holder as needed.

size caps the byte count written — size=-1 (default) writes all of data; size>=0 writes min(len(data), size) bytes. Caps via a slice of data (zero-copy on memoryview / bytes), so downstream pipelines that only need the first N bytes of a larger buffer skip the trailing tail.

overwrite declares that this write replaces the holder's tail past offset + size — after the splice, :attr:size is set to offset + size. Callers that currently do truncate(0) followed by write_bytes(...) collapse to a single write_bytes(..., overwrite=True), which on whole-blob remote backends saves a SDK round trip (the atomic upload at offset == 0 already replaces the object — no preceding truncate needed).

Pipeline:

1. Slice data to size if capped.
2. Normalize offset (-1 → append, -N → self.size - N).
3. Pre-grow visible :attr:size to cover the splice via :meth:resize.
4. Hand the normalized (data, offset) to :meth:_write_mv.
5. Truncate tail past offset + n when overwrite.
6. Mark dirty + bump cached mtime if anything was written.

update_stat=False skips the post-write :meth:_touch_stat and :meth:mark_dirty calls. Use it for bulk loops that want a single stat refresh at the end (one :func:time.time call instead of one per write); the caller is then responsible for calling :meth:_touch_stat (or re-statting via the path-side _stat for filesystem backends) once the loop finishes.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's resize / bounds-check / dirty-marking fires once, on the backing storage — the cursor only advances its own _pos.

reserve

reserve(n: int) -> None

Pre-grow capacity to at least n bytes.

Capacity-only — does NOT change :attr:size. Idempotent when capacity ≥ n already. Subclasses with no growable capacity layer may treat this as a no-op. Cursor / format-leaf IOs delegate to the bound parent.

resize

resize(n: int) -> int

Grow visible :attr:size to at least n bytes (one-way).

Sister of :meth:truncate, but never shrinks. Used by :meth:write_mv to pre-allocate a known target before the splice so :meth:_write_mv doesn't have to manage size.

• n <= size → no-op, returns current :attr:size.
• n > size → extends with zero-padding via :meth:truncate, returns n.

Subclasses with a native grow-only primitive (capacity hint to a remote upload session, posix_fallocate on local fd) override for the cheaper path; the default works on every backend.

truncate

truncate(size: 'int | None' = None) -> int

Set the visible :attr:size to exactly size bytes.

Shrinks drop the tail; extends zero-pad. Returns the new size.

On a cursor (self._parent is not None), size=None truncates at the current cursor position and the cursor is clamped if it would exceed the post-truncate size. On a storage IO size=None is invalid — pass an explicit byte count.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Push buffered writes to the durable backing.

Cursor IOs forward the flush to their bound parent; storage IOs go through :meth:Disposable.commit (default no-op unless a subclass overrides).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: Any = None) -> 'IO'

Return a typed Tabular leaf bound to this buffer's holder.

.. deprecated:: Use :meth:open with a media_type instead — holder.open(media_type=...) dispatches to the same format leaf and returns an acquired cursor. as_media is retained for callers that haven't migrated.

Resolution: explicit media_type wins; otherwise the buffer's stamped media type is used. The leaf borrows the same backing storage so durable bytes are shared without a copy. When self is already an instance of the resolved leaf class, returns self unchanged.

Raises :class:KeyError when no media type can be resolved or the resolved type has no registered Tabular leaf.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

open

open(mode: str = 'rb+') -> IO

Open a fresh :class:IO cursor over the response's holder.

Dispatches to the format-specific leaf via the holder's media type (Parquet → :class:ParquetFile, CSV → :class:CSVFile, …). The returned cursor is non-owning: closing it does not close the underlying holder.

with_receiver

with_receiver(receiver: UserInfo | Mapping[str, Any] | None) -> 'Response'

Mutate :attr:receiver in place and return self.

Mirrors :meth:update_headers / :meth:update_tags — the response is mutable in this codebase, so this is a fluent in-place setter rather than a clone.

partition_values

partition_values() -> dict[str, Any]

Hook for subclasses to define what feeds partition_key.

Default: delegates to the embedded request so request and response rows share the same partition_key and co-locate. Override to give responses an independent partition strategy (e.g. partition by status class for an audit table).

values_to_arrow_batch classmethod

values_to_arrow_batch(responses: 'Iterable[Response]') -> pa.RecordBatch

Build one :class:pa.RecordBatch from N responses in a single C++ pass.

The per-row :meth:to_arrow_batch path does one pa.array([arrow_values], type=struct) plus RecordBatch.from_struct_array per call — N rows means N independent C++ struct walks plus an outer Table.from_batches(...).combine_chunks() concat. This classmethod folds the whole bulk into one struct walk: collect arrow_values once per response, hand the list of dicts to pyarrow, get back a single RecordBatch with N rows.

Used by every bulk-writeback path in the cache pipeline (Session._persist_remote, Session._responses_to_spark, :func:responses_to_tabular) — at 64 rows the bulk shape is ~30x faster than the per-row build it replaces.

from_record classmethod

from_record(
    record: "Mapping[str, Any]", *, normalize: bool = False
) -> "Response"

Build a :class:Response from a row-shaped mapping.

release_conn

release_conn() -> None

Return the underlying connection to the pool, once.

No-op on responses built from records / cache hits / test fixtures (_connection is None). On a wire-fetched response, defers to session._release_connection so the socket lands back in the per-host idle cache.

When the body stream reconnected mid-flight (HTTPStream issued a Range request on a transient error), the original connection is broken — we close it instead of pooling it — and the live replacement connection stored on the stream is returned to the pool instead.

drain_conn

drain_conn() -> None

Pull any remaining bytes from the source so the socket is reusable.

stream

stream(amt: int = 65536) -> Iterator[bytes]

Iterate body bytes in amt-sized chunks (urllib3-shaped).

Drives the underlying :class:MemoryStream through its internal cursor so the iteration is forward-only and releases the connection once EOF lands.

from_wire classmethod

from_wire(
    request: "HTTPRequest",
    raw: HTTPResponse,
    *,
    session: "Optional[HTTPSession]" = None,
    connection: Optional[HTTPConnection] = None,
    pool_key: Optional[Tuple[str, str, int]] = None,
    decode_content: bool = True,
    preload_content: bool = False,
    tags: Optional[Mapping[str, str]] = None,
    received_at: Optional[datetime] = None
) -> "HTTPResponse"

Build a response straight off a live http.client.HTTPResponse.

Replaces the urllib3-shim pool HTTPResponse that used to live in yggdrasil.http_._pool intermediary — :class:HTTPSession builds the wire response directly through this factory. The body is wrapped in a :class:MemoryStream whose source is the the raw socket; the connection / pool-key pair lets :meth:release_conn return the socket to the session's idle cache after drain.

from_pool classmethod

from_pool(
    request: "HTTPRequest",
    response: Any,
    tags: Optional[Mapping[str, str]],
    received_at: datetime,
    *,
    stream: bool = True,
    amt: int = 512 * 1024,
    release_conn: bool = True
) -> "HTTPResponse"

Compat shim — promote a duck-typed pool response to high-level.

Used by test fixtures and any legacy caller that still hands a urllib3-shaped object (with .status / .headers / .read / .stream / .release_conn). When the input is already a high-level :class:HTTPResponse it round-trips through itself; otherwise the body is drained into a fresh :class:MemoryStream and the new high-level response wraps that buffer.

HTTPRequest

HTTPRequest(
    method: str,
    url: URL,
    headers: MutableMapping[str, str],
    tags: MutableMapping[str, str],
    buffer: Optional[Holder],
    sent_at: Optional[datetime],
    sender: Optional[UserInfo] = None,
    auth: Optional[Authorization] = None,
    send_config: Optional["SendConfig"] = None,
    session: "Session | None" = None,
)

Immutable-ish request descriptor — fields are normalized in init.

_session is a transient back-reference used by :meth:attach_session; it's deliberately excluded from :meth:__getstate__ so a request stays portable when pickled to Spark executors and re-binds via attach_session once it lands on the worker. Per-request request/response hooks live on the owning :class:Session (_prepare_request / _prepare_response).

sender property

sender: UserInfo | None

:class:UserInfo snapshot for the side issuing this request.

Defaults to UserInfo.current() at construction time. Use :meth:with_sender to swap it for a different snapshot (returns a new request — :class:PreparedRequest is immutable-ish, so the underlying field is read-only).

send_config_or_default property

send_config_or_default: 'SendConfig'

Return :attr:send_config or the shared default singleton.

local_cache_config property

local_cache_config: 'CacheConfig | None'

Per-request local :class:CacheConfig, delegated to :attr:send_config.

remote_cache_config property

remote_cache_config: 'CacheConfig | None'

Per-request remote :class:CacheConfig, delegated to :attr:send_config.

holder property

holder: Optional[Holder]

The :class:Holder backing the request body — alias for :attr:buffer.

auth property writable

auth: Optional[Authorization]

The :class:Authorization handler bound to this request, if any.

When set, :meth:prepare_authorization resolves it lazily into the Authorization header just before the request is sent — every send picks up a freshly-minted header value.

partition_key property

partition_key: int

xxh3_64 of the joined :meth:partition_values — int64 partition column.

body_hash property

body_hash: int

xxh3_64 digest of the body bytes; 0 when body is absent.

Non-nullable in the schema — callers that need a "missing" signal should branch on :attr:buffer or :attr:body_size.

private_url_hash property

private_url_hash: int

xxh3_64 of (method, URL) exactly as captured (userinfo + full query).

public_url_hash property

public_url_hash: int

xxh3_64 of (method, URL) after anonymize('remove') — userinfo and sensitive query params dropped, so this hash is stable across the cache's anonymize step. Method is mixed in so verbs don't collide.

hash property

hash: int

xxh3_64 over (method, url, headers, body) — overall identity, including sensitive bits (userinfo, Authorization, …).

Use :attr:public_hash for matches that should survive cache anonymization (drops userinfo / sensitive query params / sensitive headers).

public_hash property

public_hash: int

xxh3_64 over the anonymize='remove' projection of (method, url, headers, body) — the cross-system identity that survives cache anonymization.

with_sender

with_sender(sender: UserInfo | Mapping[str, Any] | None) -> 'HTTPRequest'

Return a copy of this request with :attr:sender replaced.

Accepts a :class:UserInfo, a struct-shaped mapping (matching :data:USERINFO_STRUCT), or None to clear the snapshot.

open

open(mode: str = 'rb+') -> Optional[IO]

Open a fresh :class:IO cursor over the request's holder.

Returns None when the request has no body. Dispatches to the format-specific leaf via the holder's media type. The returned cursor is non-owning.

prepare_authorization

prepare_authorization() -> 'HTTPRequest'

Resolve the bound :class:Authorization handler into the header.

No-op when no handler is bound. Calls handler.authorization each time, so handlers that refresh internally (e.g. MSAL) emit the current token on every send.

partition_values

partition_values() -> dict[str, Any]

Hook for subclasses to define what feeds partition_key.

Returns the ordered mapping that gets fed to :attr:partition_key's xxh3_64 digest. Override to bucket requests differently — e.g. by tenant id, by url.host only, by a fixed shard. Order is stable: the helper concatenates values in iteration order so two configs that disagree on key ordering hash to different partitions.

Default: {"host": url.host, "path": url.path} — every call to the same endpoint shares one partition leaf.

values_to_arrow_batch classmethod

values_to_arrow_batch(requests: 'Iterable[PreparedRequest]') -> pa.RecordBatch

Build one :class:pa.RecordBatch from N requests in a single C++ pass.

Counterpart to :meth:Response.values_to_arrow_batch — same rationale: collect arrow_values once per request, hand the list of dicts to pyarrow, get back one RecordBatch with N rows. Replaces the pa.Table.from_batches([r.to_arrow_batch(...) for r in N]) shape with a single C++ struct walk; at 64 rows this is ~30x faster than per-row builds plus a downstream concat.

from_record classmethod

from_record(
    record: "Mapping[str, Any]", *, normalize: bool = True
) -> "HTTPRequest"

Build a :class:PreparedRequest from a row-shaped mapping.

HTTPPath

HTTPPath(
    data: Any = None,
    *,
    url: URL | None = None,
    session: "HTTPSession | None" = None,
    **kwargs: Any
)

Bases: RemotePath

:class:Path over an http:// / https:// URL.

Construction shapes::

HTTPPath("https://example.com/data.csv")
HTTPPath(url=URL("http://api.example/v1"), session=my_session)

The session kwarg accepts any :class:HTTPSession; when omitted, a default one is lazy-built on first access (mirroring :meth:PreparedRequest._send for orphan requests). Sharing a single session across paths reuses the connection pool.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known property

size_known: bool

True only when the stat cache carries a fresh entry.

Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe round trip just to short-circuit on size == 0: when the cache is cold the format reader will trip its own EOF / empty-file error which the caller catches and translates to an empty schema. When the cache is warm the cheap size read fires unchanged.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

session property

session: 'HTTPSession'

Attached :class:HTTPSession, building a default one on miss.

open

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url classmethod

from_url(url: URL, **kwargs) -> 'IO'

Create a new IO from a URL.

When cls is abstract (has subclasses but isn't itself constructible — e.g. :class:Path), the URL scheme is resolved through the :class:URLBased registry to a concrete subclass; an unknown scheme raises :class:ValueError instead of producing the obscure "Can't instantiate abstract class" :class:TypeError.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Drop this path's cached :class:IOStats, schema, and _INSTANCES entry — see :meth:Path.invalidate_singleton.

A mutation just ran, so the cached metadata is no longer authoritative; the next read re-probes the backend. Discards any un-flushed write scratch (callers must :meth:flush first to keep pending writes).

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ classmethod

from_(obj: Any, **kwargs: Any) -> 'Path'

Coerce obj (str / URL / pathlib / Path) into a :class:Path.

When called on the abstract :class:Path, dispatches via the :class:Holder scheme registry to the subclass registered for the URL's scheme; defaults to :class:LocalPath for path-shaped URLs without an explicit scheme. When called on a concrete subclass, returns an instance of that subclass.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'IO'

Build a sibling IO at self.url joined with segments.

URL-shaped IOs (:class:LocalPath, remote paths) use this to mint a child path; :class:Memory and other non-URL leaves raise :class:ValueError.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Read — served from the acquired write scratch when one is in flight, otherwise straight through to the backend.

Inside an acquired window with un-flushed writes the scratch file is the authoritative view (read-after-write within the handle); without a scratch this is the base :class:Holder read over the subclass :meth:_read_mv (ranged where the backend supports it).

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Whole-blob write — direct upload when closed, disk-paged when open.

• Closed (un-acquired). A whole-object overwrite from the start (offset == 0, overwrite, no cursor; what write_bytes(...) resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holder splice (download, splice, re-upload via :meth:_write_mv).
• Open (acquired — with path: / path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush / release streams the scratch to the backend in one upload.

reserve

reserve(n: int) -> None

No-op by default — files have no separate capacity layer.

resize

resize(n: int) -> int

No-op for remote-backend paths.

:class:Holder.resize would call :meth:truncate to pre-grow a holder before a positional write. On remote backends every truncate is a full-object upload, so the pre-grow would double the network traffic for every write. The upload that :meth:write_mv runs next will materialize the right size on its own.

truncate

truncate(n: int) -> int

Resize the object to exactly n bytes.

• Acquired (the open("wb") truncate-on-acquire, and explicit truncates inside a with): resize the disk scratch; the commit happens once on :meth:flush. An empty truncate(0) therefore costs no PUT until release — and open("wb") immediately followed by a write coalesces to a single upload.
• Closed: a whole-object upload — truncate(0) PUTs an empty object; truncate(n > 0) downloads, slices / zero-extends, re-uploads.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush

flush() -> None

Commit the acquired write scratch to the backend in one upload.

The single (streamed) PUT that an open("wb") window produces — every write() since acquire spliced into the disk scratch, and this drains it. The scratch streams off disk (bounded memory) on backends that support it; others read it back for the SDK's whole-object upload. A no-op when nothing was buffered. with path.open("wb"): pass still materialises an empty object (the acquire-time truncate(0) seeded an empty scratch).

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

• self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
• self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

arrow_random_access_file

arrow_random_access_file()

Yield a pyarrow random-access file backed by ranged _read_mv.

Lets pyarrow readers seek and pull only the bytes they touch — a Parquet column / row-group projection fetches the footer plus the projected chunks, instead of snapshotting the whole object the way :meth:arrow_input_stream does. :class:ParquetFile reaches for this when a projection is bound and the backend advertises :attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read still snapshots. Generic over any holder via _read_mv + size.

read_byte_range

read_byte_range(offset: int, length: int = -1) -> memoryview

Read exactly length bytes from offset — a ranged backend fetch.

The explicit byte-range surface for tabular / format readers that want a specific window (a Parquet footer, an Arrow IPC block) without snapshotting the whole object. Works whether the holder is opened or not: an in-flight write scratch is served from disk, otherwise the subclass :meth:_read_mv issues a ranged GET on backends that support it. length < 0 reads to EOF.

An explicit non-negative window goes straight to :meth:_read_mv — no self.size (HEAD) bounds probe, so a footer fetch is a single ranged GET. A short read near EOF is the caller's to interpret.

write_arrow_io

write_arrow_io(payload: 'Any') -> int

Commit an Arrow-encoded payload directly to the backend.

Accepts a pa.Buffer, bytes, bytearray, or memoryview and uploads it in one backend call — no truncate, no stat probe. Tabular IO files (ParquetFile, ArrowIPCFile, etc.) route through this after the format encoder finishes so the encoded bytes go straight to the remote object without intermediate copies. Whole-object replace: any in-flight write scratch is superseded.

Cookies

Cookies(initial: Optional[Mapping[str, str]] = None)

Mapping-like cookie jar with Cookie / Set-Cookie plumbing.

Stored as name -> value pairs without attribute parsing (Path, Expires, HttpOnly, Secure, Domain are ignored). That matches the rest of the browser-mode helpers, which intentionally stay close to a "browser-as-a-string-bag" model rather than reimplementing :mod:http.cookiejar semantics.

to_header

to_header() -> str

Serialize the jar into a Cookie request-header value.

Returns an empty string when the jar is empty so callers can if header: skip emitting the header entirely.

parse_set_cookie staticmethod

parse_set_cookie(header: str) -> tuple[str, str]

Parse a single Set-Cookie header value into (name, value).

Attribute pairs after the first ; are dropped on purpose — see the class docstring.

update_from_set_cookie

update_from_set_cookie(raw: str) -> None

Merge a raw Set-Cookie header value into the jar.

urllib3 collapses multiple Set-Cookie values with a comma, so we split and best-effort parse each piece.

update_from_response

update_from_response(response: 'HTTPResponse') -> None

Pull cookies out of response's Set-Cookie header(s).