`yggdrasil.http_.response_cache`¶

response_cache ¶

Specialized, content-addressed local cache for HTTP responses.

The generic cache stores responses as a Hive-partitioned Arrow dataset and scans it with a :class:Predicate on every lookup — the whole Folder / Tabular / parquet pipeline for what is really a key→blob store. :class:HttpResponseCache is the lean replacement for the local backend: it's a :class:Folder (so it drops in wherever the cache expects a Folder/Tabular holder — cache_tabular, local_cache_folder, the cleanup daemon under ~/.cache/http/response), but each response is stored as one tiny file named by the producing request's public_hash (sharded one byte deep): a 1-byte version, a JSON meta header (status, received-at, headers, tags) and the raw body — no Arrow on the hot path. A lookup is then a single O(1) file read + Response rebuild (the looked-up request, same identity, is reattached), a write a single atomic file replace (upsert). No dataset scan, partitioning, predicate, schema, or Arrow round-trip.

Over the on-disk store sits a small byte-bounded RAM hot tier (default 32 MB, process-wide) keyed by public_hash: the most-recent / most-reused responses are served straight from memory, everything else from the per-key file — so the cache's RAM can never balloon (an oversized response is kept disk-only rather than evicting the hot set), and day-old files are pruned on construction.

A cache hit measures several times faster than even a localhost HTTP call, with bounded memory vs the generic tabular cache — see benchmarks/http_/bench_response_cache.py. The remote (Databricks Table) backend is untouched and keeps the tabular path.

HttpResponseCache ¶

HttpResponseCache(
    data: Any = None,
    *,
    path: Any = None,
    tabular_parent: Any = None,
    **kwargs: Any
)

Bases: Folder

Content-addressed local HTTP-response cache (one lightweight file per key).

opened `property` ¶

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

closed `property` ¶

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

url `property` `writable` ¶

url: 'URL'

Canonical URL identifying this holder.

static_values `property` ¶

static_values: 'Mapping[str, Any]'

Partition KV parsed from the folder's bound :attr:path URL.

Walks the URL's segments (self.path.url.parts) and picks every Hive-shaped <col>=<val> chunk — a folder at <base>/year=2024/month=05/ reports {"year": 2024, "month": 5} without any constructor seed. Values are cast to their declared dtype when a parent :class:Folder (or this folder itself) already has the schema cached, otherwise the decoded string passes through; the downstream :meth:matches_static prune is conservative and just degrades to a row-level filter on undecidable compares.

Schema lookup walks tabular_parent first so partition children minted during a read reuse the root folder's cached schema instead of paying a per-child sidecar load. Only when no ancestor has the schema cached does the read fall through to :meth:collect_schema's sidecar / first- batch path.

The parsed mapping is memoised on the instance — URL parts are immutable for the bound path and the schema only flips when :meth:_persist_schema writes a new layout, which invalidates the cache. Calling once per matches_static per child (the prune hot path) costs one dict read after the first invocation.

parent `property` ¶

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents `property` ¶

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

size_known `property` ¶

size_known: bool

True when reading :attr:size won't trigger a backend probe.

Always true for in-memory IOs (size is a slot). Path IOs override to True only when their stat cache is warm — callers that want to short-circuit on an empty buffer (parquet / arrow IPC / CSV readers checking size == 0) can guard the check on this predicate so a cold remote path doesn't pay a HeadObject / get_status / get_metadata round trip just to discover the file is non-empty. Cursor / format-leaf IOs delegate to the parent.

holder_is_overwrite `property` ¶

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

media_type `property` `writable` ¶

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_streaming `property` ¶

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest `property` ¶

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder `property` ¶

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder `property` ¶

owns_holder: bool

Whether closing self also closes the bound parent.

mode `property` ¶

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

probe_hashes ¶

probe_hashes(requests: 'Iterable[PreparedRequest]') -> 'set[int]'

The subset of request public_hash keys that have a cached file — the cheap presence check the send pipeline uses to split hits/misses.

read_responses ¶

read_responses(
    requests: "Iterable[PreparedRequest]", *, config: "Any"
) -> "tuple[list, list]"

(hits, misses) — each request's per-key file decoded to a :class:Response and accepted by config.filter_response.

write_arrow ¶

write_arrow(data: 'Any') -> None

Persist a response :class:pa.RecordBatch/Table (or Spark frame), one file per row keyed by request_public_hash — overwriting the prior entry for that key (upsert).

open ¶

open(mode: 'Mode | str | None' = None, **kwargs: Any) -> 'IO'

Acquire the path and return an :class:IO cursor bound to it.

mode accepts a :class:Mode member, an alias string, or a stdlib open() mode string. None falls through to :meth:Holder.open which uses "rb+". Other keyword arguments (owns_holder, media_type, auto_open, …) ride through to :meth:Holder.open.

commit ¶

commit()

Commit current state

rollback ¶

rollback()

Rollback current state

close ¶

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

mark_dirty ¶

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme `classmethod` ¶

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch `classmethod` ¶

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

from_url `classmethod` ¶

from_url(url: URL, **kwargs) -> 'IO'

Create a new IO from a URL.

When cls is abstract (has subclasses but isn't itself constructible — e.g. :class:Path), the URL scheme is resolved through the :class:URLBased registry to a concrete subclass; an unknown scheme raises :class:ValueError instead of producing the obscure "Can't instantiate abstract class" :class:TypeError.

to_url ¶

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

to_singleton ¶

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton ¶

invalidate_singleton(remove_global: bool = True) -> None

Drop the cached :class:IOStats and pop self from _INSTANCES. Single canonical invalidator for paths.

Mutating ops (writes, deletes) call this so the next read on the same logical path picks up fresh state. Subclasses with their own caches (schema, table info, column lists) override and chain super().invalidate_singleton(...).

class_for_media_type `classmethod` ¶

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

matches_static ¶

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

from_ `classmethod` ¶

from_(obj: Any, **kwargs: Any) -> 'Path'

Coerce obj (str / URL / pathlib / Path) into a :class:Path.

When called on the abstract :class:Path, dispatches via the :class:Holder scheme registry to the subclass registered for the URL's scheme; defaults to :class:LocalPath for path-shaped URLs without an explicit scheme. When called on a concrete subclass, returns an instance of that subclass.

check_options `classmethod` ¶

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup ¶

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize ¶

optimize(
    byte_size: "int | None" = None,
    *,
    target_media_type: "MediaType | str | Any" = MediaTypes.ARROW_IPC,
    tolerance: float = OPTIMIZE_TOLERANCE,
    **kwargs: Any
) -> int

Compact small part files into byte_size-shaped bundles.

Walks the tree under :attr:path and at every directory that holds part files, groups them by combined size and rewrites each group as a single fresh part. Two flavors of the pass:

byte_size=None (the default and the shape the local-cache compaction loop in :class:Session calls with) — collapses every directory with more than one part into a single file.
byte_size=N — first-fit-decreasing bin pack into bins of capacity N bytes. Parts whose size is within ±tolerance of N (or already larger) are skipped: they are already "close enough" and rewriting them would just burn IO.

target_media_type (a :class:MediaType or anything :meth:MediaType.from_ accepts) selects the format the rewritten parts are encoded in. Defaults to Arrow IPC.

Returns the number of new part files created. Idempotent: a second call on a tree that's already at target leaves nothing to do and returns 0.

delete ¶

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema ¶

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count ¶

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches ¶

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table ¶

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader ¶

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table ¶

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table ¶

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union ¶

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset ¶

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator ¶

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records ¶

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique ¶

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters¶

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns¶

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample ¶

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters¶

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns¶

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select ¶

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop ¶

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter ¶

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast ¶

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display ¶

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy ¶

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath ¶

joinpath(*segments: Any) -> 'IO'

Build a sibling IO at self.url joined with segments.

URL-shaped IOs (:class:LocalPath, remote paths) use this to mint a child path; :class:Memory and other non-URL leaves raise :class:ValueError.

from_bytes `classmethod` ¶

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder `classmethod` ¶

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

for_holder `classmethod` ¶

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

The explicit media_type kwarg, when supplied.
holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes `classmethod` ¶

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv ¶

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Slice size bytes from offset as a :class:memoryview.

cursor=True ignores the explicit offset and reads from the holder's internal cursor (:attr:tell), advancing it past the bytes returned. cursor=False (default) keeps the cursor-less positional contract — the cursor is untouched.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's bounds-check uses its own size — avoids a redundant stat probe on remote backings when the cursor has no local size cache, and routes through any subclass _active override (lazy materialization on :class:ZipEntryFile, …).

write_mv ¶

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Splice data at offset, pre-growing the holder as needed.

size caps the byte count written — size=-1 (default) writes all of data; size>=0 writes min(len(data), size) bytes. Caps via a slice of data (zero-copy on memoryview / bytes), so downstream pipelines that only need the first N bytes of a larger buffer skip the trailing tail.

overwrite declares that this write replaces the holder's tail past offset + size — after the splice, :attr:size is set to offset + size. Callers that currently do truncate(0) followed by write_bytes(...) collapse to a single write_bytes(..., overwrite=True), which on whole-blob remote backends saves a SDK round trip (the atomic upload at offset == 0 already replaces the object — no preceding truncate needed).

Pipeline:

Slice data to size if capped.
Normalize offset (-1 → append, -N → self.size - N).
Pre-grow visible :attr:size to cover the splice via :meth:resize.
Hand the normalized (data, offset) to :meth:_write_mv.
Truncate tail past offset + n when overwrite.
Mark dirty + bump cached mtime if anything was written.

update_stat=False skips the post-write :meth:_touch_stat and :meth:mark_dirty calls. Use it for bulk loops that want a single stat refresh at the end (one :func:time.time call instead of one per write); the caller is then responsible for calling :meth:_touch_stat (or re-statting via the path-side _stat for filesystem backends) once the loop finishes.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's resize / bounds-check / dirty-marking fires once, on the backing storage — the cursor only advances its own _pos.

resize ¶

resize(n: int) -> int

Grow visible :attr:size to at least n bytes (one-way).

Sister of :meth:truncate, but never shrinks. Used by :meth:write_mv to pre-allocate a known target before the splice so :meth:_write_mv doesn't have to manage size.

n <= size → no-op, returns current :attr:size.
n > size → extends with zero-padding via :meth:truncate, returns n.

Subclasses with a native grow-only primitive (capacity hint to a remote upload session, posix_fallocate on local fd) override for the cheaper path; the default works on every backend.

clear ¶

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat ¶

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime ¶

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire ¶

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

flush ¶

flush() -> None

Push buffered writes to the durable backing.

Cursor IOs forward the flush to their bound parent; storage IOs go through :meth:Disposable.commit (default no-op unless a subclass overrides).

pread ¶

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite ¶

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview ¶

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv ¶

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes ¶

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes ¶

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text ¶

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text ¶

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head ¶

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail ¶

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto ¶

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline ¶

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines ¶

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell ¶

tell() -> int

Current cursor position.

seek ¶

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path ¶

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream ¶

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder ¶

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload ¶

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

:class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
:class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download ¶

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes ¶

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue ¶

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode ¶

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64 ¶

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64 ¶

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64 ¶

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes ¶

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream ¶

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream ¶

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable ¶

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type ¶

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media ¶

as_media(media_type: 'Any' = None) -> 'Any'

Wrap this path in the format leaf for its media type.

.. deprecated:: Use :meth:open with a media_type instead — path.open(media_type=...) already dispatches to the right format leaf and gives a properly acquired cursor with lifecycle handling. as_media returns an un-acquired leaf and is kept only for callers that haven't migrated.

Resolution: explicit media_type first, else the holder's :class:MediaType (path extension, magic-byte sniff, or content-type header). The resolved class is looked up in the :class:Holder format registry and instantiated bound to this path.

Raises :class:KeyError when the path's media type isn't registered as a tabular format.

fileno ¶

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read ¶

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall ¶

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write ¶

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32 ¶

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32 ¶

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load ¶

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

Explicit media_type kwarg.
Cached :attr:media_type on the IO.
Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress ¶

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.

ls ¶

ls(
    *,
    recursive: bool = False,
    limit: "int | None" = None,
    singleton_ttl: Any = False
) -> Iterator["Path"]

Yield children lazily. limit caps how many are produced — the underlying listing stays incremental, so a bounded ls over a huge prefix never materialises (or fetches) more than it needs.

unlink ¶

unlink(missing_ok: bool = True, wait: WaitingConfigArg = True) -> None

Remove the leaf — pathlib-compatible: refuses directories.

Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises :class:IsADirectoryError for directories so callers don't accidentally recursive-delete via unlink. Use :meth:remove for the directory case. Thin wrapper over :meth:_delete's path-removal mode.

remove ¶

remove(
    recursive: bool = True,
    missing_ok: bool = True,
    wait: WaitingConfigArg = True,
    fresher_than: Optional[TimeLike] = None,
    older_than: Optional[TimeLike] = None,
) -> "Path"

Remove this path — the file, or the whole subtree when recursive.

Thin wrapper over :meth:_delete's path-removal mode (the single deletion primitive). fresher_than / older_than scope the removal to children inside that mtime window.

wait_until_gone ¶

wait_until_gone(wait: WaitingConfigArg = True) -> 'Path'

Block until :meth:exists reports False or wait expires.

Polls the backend with a fresh probe each iteration — the stat cache is invalidated between checks so a TTL'd hit can't mask a deletion that landed after the cache was filled. Useful when a fire-and-forget unlink (e.g. WarehouseStatementBatch.clear_temporary_resources) means the caller can't observe completion through the original operation's return value.

Raises :class:TimeoutError when wait's deadline elapses and the path is still present.

touch ¶

touch() -> 'Path'

Create the path as an empty file if it doesn't exist.

write_bytes(b"") short-circuits in the holder fast path (zero bytes, no flush), which would leave a missing file behind — open + close around the empty write so the holder actually materialises the entry on the backing store.

upload_module ¶

upload_module(
    module: Any, *, name: str | None = None, overwrite: bool = True
) -> "Path"

Zip a local module / package and write it under this path.

module is anything :func:resolve_module_root accepts — an importable module name ("yggdrasil.io"), a :class:os.PathLike pointing at a package directory or an existing .zip / .whl archive, or a callable carrying a __module__ attribute. The module is packed into a deflated zip whose top-level entry is the package directory itself, so the archive can be added to sys.path directly (or fed to :meth:SparkSession.addArtifacts with pyfile=True).

Destination shape on self:

self names a file with a .zip / .whl suffix — archive bytes land at that exact path.
self is anything else — archive lands at self / <name or "<module>.zip">.

Returns the concrete :class:Path that now holds the archive. overwrite=False raises :class:FileExistsError when the destination already exists.

import_module ¶

import_module(
    module_name: str | None = None,
    *,
    install: bool = True,
    cache_dir: "Any" = None
) -> Any

Download a module archive at this path and import it.

Inverse of :meth:upload_module: fetch the archive bytes at self, drop them on local disk, prepend the archive (or its extracted parent) to :data:sys.path, and return the live module via :func:importlib.import_module.

module_name defaults to the archive's stem (filename minus suffix). cache_dir picks where the archive lands locally (default: a fresh :meth:LocalPath.staging_path-style directory).

install=True (the default) preserves the archive on disk so subsequent imports in the same process hit the cache. install=False makes the cache-dir lifetime the caller's problem.

iter_children ¶

iter_children(options: 'FolderOptions | None' = None) -> 'Iterator[Tabular]'

Yield every non-private direct entry of :attr:path (one level).

Sub-directories come back as a fresh :class:Folder; file entries resolve through :meth:_leaf_for to a registered :class:Tabular leaf (non-tabular files are skipped). A missing folder yields nothing.

Plain flat listing — the partition-aware, recursive walk lives in :meth:iter_leaves (which the read path uses). options is accepted for signature symmetry but isn't consulted here.

partition_columns ¶

partition_columns(
    options: "FolderOptions | None" = None, *, schema: Any = None
) -> "tuple[str, ...]"

Partition column names declared on the folder's schema.

schema overrides the lookup when supplied (write path passes the just-peeked batch schema). Otherwise the resolved schema is :meth:collect_schema — which itself reads the .ygg/schema.arrow sidecar when present and falls back to the first-batch shape — with options.target as a final fallback when even the sidecar / first batch is empty (a fresh folder asked for its layout before any write).

Columns already pinned by :attr:static_values get filtered out: a partition-key=v leaf would otherwise re-infer partition_key from its own data and recurse forever into partition_key=v/partition_key=v/…. :class:Field parses every tag off the Arrow schema, so the helper just walks schema.children and picks the partition_by=True ones.

make_child ¶

make_child(*, options: FolderOptions | None = None) -> 'Tabular'

Mint a fresh tabular leaf bound to a fresh path under :attr:path.

Filename shape: part-{epoch_ms}-{seed}.{ext} where ext is options.child_media_type.full_extension. The millisecond timestamp gives lexical-time ordering; a 2-byte seed (~65k-value space) breaks ties between writes that land in the same millisecond.

The :class:Tabular leaf is dispatched directly from the media type via :meth:IO.class_for_media_type, so the write path doesn't go through the path-extension reverse- lookup. A media type with no registered leaf falls back to a raw :class:IO so non-tabular extensions still get a working write.

Returns a closed leaf. Caller opens it inside a with block to write bytes.

iter_leaves ¶

iter_leaves(options: 'FolderOptions | None' = None) -> 'Iterator[Tabular]'

Recursively yield the surviving leaf data files under this folder.

Walks the tree itself — a non-recursive :meth:Path.ls at each level — skipping private entries (dot-prefixed) and empty (0-byte) files, and recursing into sub-directories the predicate doesn't prune on their path-level Hive :attr:static_values. Opens no data file: a pure listing + path-prune, reusable as a file-enumeration pass. The residual row-level filter stays in :meth:_read_arrow_batches.

Fast get: when the predicate pins this level's partition column to a finite value set (col == v / col in [...]), the matching <col>=<val>/ sub-paths are probed directly — a handful of stats instead of listing (and pruning) every sibling partition.

yggdrasil.http_.response_cache¶

response_cache ¶

HttpResponseCache ¶

opened property ¶

closed property ¶

url property writable ¶

static_values property ¶

parent property ¶

parents property ¶

size_known property ¶

holder_is_overwrite property ¶

media_type property writable ¶

is_streaming property ¶

xxh3_64_digest property ¶

holder property ¶

owns_holder property ¶

mode property ¶

probe_hashes ¶

read_responses ¶

write_arrow ¶

open ¶

commit ¶

rollback ¶

close ¶

mark_dirty ¶

for_scheme classmethod ¶

dispatch classmethod ¶

from_url classmethod ¶

to_url ¶

to_singleton ¶

invalidate_singleton ¶

class_for_media_type classmethod ¶

matches_static ¶

from_ classmethod ¶

check_options classmethod ¶

cleanup ¶

optimize ¶

delete ¶

collect_schema ¶

count ¶

scan_arrow_batches ¶

scan_arrow_table ¶

scan_arrow_batch_reader ¶

read_table ¶

write_table ¶

union ¶

read_spark_dataset ¶

read_record_iterator ¶

read_records ¶

unique ¶

Parameters¶

Returns¶

resample ¶

Parameters¶

Returns¶

select ¶

drop ¶

filter ¶

cast ¶

display ¶

lazy ¶

joinpath ¶

from_bytes classmethod ¶

from_holder classmethod ¶

for_holder classmethod ¶

registered_classes classmethod ¶

read_mv ¶

write_mv ¶

resize ¶

clear ¶

stat ¶

touch_mtime ¶

acquire ¶

flush ¶

pread ¶

pwrite ¶

memoryview ¶

iter_mv ¶

read_bytes ¶

write_bytes ¶

`yggdrasil.http_.response_cache`¶

opened `property` ¶

closed `property` ¶

url `property` `writable` ¶

static_values `property` ¶

parent `property` ¶

parents `property` ¶

size_known `property` ¶

holder_is_overwrite `property` ¶

media_type `property` `writable` ¶

is_streaming `property` ¶

xxh3_64_digest `property` ¶

holder `property` ¶

owns_holder `property` ¶

mode `property` ¶

for_scheme `classmethod` ¶

dispatch `classmethod` ¶

from_url `classmethod` ¶

class_for_media_type `classmethod` ¶

from_ `classmethod` ¶

check_options `classmethod` ¶

from_bytes `classmethod` ¶

from_holder `classmethod` ¶

for_holder `classmethod` ¶

registered_classes `classmethod` ¶