yggdrasil.io.base

base

Format-buffer / arrow-stream context managers and IO helpers.

After the Holder ↔ IO merge the canonical :class:IO class lives in :mod:yggdrasil.io.holder (storage + cursor + tabular handle in one class). This module survives as the home for two things:

1. Module-level helpers used by the IO class and by callers poking at the temp-file spill convention: :func:_mint_spill_path, :func:_as_byte_mv, :func:_local_path_for_handle.
2. Codec / pyarrow streaming context managers returned by :meth:IO._format_buffer / :meth:IO._format_input / :meth:IO.arrow_input_stream / :meth:IO.arrow_output_stream. They live here rather than on the IO class so the codec / memory-map machinery can grow without bloating the class body.

:class:IO is re-exported from this module for backwards compatibility — every prior from yggdrasil.io.base import IO call site keeps working unchanged.

IO

IO(
    data: Any = None,
    *,
    stat: IOStats | None = None,
    url: URL | None = None,
    binary: bytes | bytearray | memoryview | None = None,
    path: PathLike | None = None,
    holder: "IO | None" = None,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    temporary: bool = False,
    singleton_ttl: Any = ...,
    **kwargs
)

Bases: Tabular[O], BinaryIO, Generic[T, O]

Position-addressable byte holder + seekable cursor + tabular handle.

Three layered shapes share the class:

• Storage IOs — :class:yggdrasil.io.memory.Memory and :class:yggdrasil.io.path.Path subclasses. Own their bytes directly; implement the storage primitives (_read_mv / _write_mv / reserve / truncate / _clear / :attr:size / :meth:_stat).
• Cursor IOs — borrow a parent storage IO via _parent; every byte primitive delegates through :meth:_active to that parent. Built by :meth:open and by format-leaf construction with parent= / holder=.
• Format-leaf IOs — :class:ParquetFile, :class:CSVFile, :class:ArrowIPCFile, … — register a :class:MimeType to claim that format in :data:_HOLDER_FORMAT_REGISTRY. They inherit the cursor delegation and override the two :class:Tabular hooks against the bound parent's bytes.

An IO IS a :class:Disposable: it can be opened, closed, used in a with block, marked dirty / clean. It is also a :class:Tabular — the default :meth:_read_arrow_batches / :meth:_write_arrow_batches contextually open the IO (with self.open() as bio:) and delegate to whichever format-leaf the stamped :class:MediaType resolves to. That means LocalPath("data.xlsx").read_pandas_frame() works the same way LocalPath("data.xlsx").open() does — the open / dispatch / close cycle is hidden behind the Tabular surface.

Also subclasses :class:typing.BinaryIO so external libraries that type-check against the stdlib file-like interface (pandas, pyarrow, zipfile, …) accept Yggdrasil byte buffers without a separate facade. :attr:mode returns the typed :class:Mode enum; pandas/zipfile's "b" in handle.mode sniffs still work because :class:Mode delegates __contains__ to its :attr:~Mode.os_mode string.

Storage subclasses implement five primitives:

• :meth:_read_mv(n, pos) — slice n bytes from pos as a :class:memoryview. Receives normalized (n, pos).
• :meth:_write_mv(data, pos) — splice data at pos, growing the IO if needed. Returns bytes written.
• :meth:reserve(n) — pre-grow the underlying capacity to at least n bytes without changing the visible :attr:size.
• :meth:truncate(n) — set the visible :attr:size to n. Shrinks drop the tail; extends zero-pad.
• :meth:_clear — drop the payload entirely.

Plus the :attr:size property and :meth:resize (concrete, built on :meth:truncate). Cursor / format-leaf subclasses inherit the delegating defaults — they hand the call to self._active() (= self._parent).

Initialize the IO.

Exactly one of url / binary / path / data / holder determines the seed; the rest are mutually exclusive (validated in :meth:__new__).

holder= (alias: parent=) borrows an existing IO as backing storage — every byte primitive then delegates through :meth:_active. owns_holder=True transfers close-ownership so closing this IO also closes the parent.

temporary=True marks the IO for self-cleanup on release: :meth:_release calls :meth:clear so the payload is dropped when the IO closes. Default False — clears only happen when the caller asks.

mode follows stdlib :func:open semantics, normalized to a :class:Mode enum. Side effects fire on :meth:_acquire, not here: cursor stays at byte 0 until then.

stat lets callers seed the metadata cache (size / mtime / media_type) when they already know it — saves a backend probe on the first :meth:stat call.

opened property

opened: bool

True iff :meth:_acquire has run and :meth:_release hasn't.

parent property

parent: 'IO | None'

The IO one level up — cursor parent first, else URL parent.

Resolution order:

1. The cursor parent (self._parent, set by :meth:IO.open and by format-leaf construction with parent= / holder=). When set, this IO is a cursor and the parent is its backing storage.
2. The URL parent — a sibling IO of the same concrete class at self.url.parent. Used by URL-shaped storage leaves (:class:Path / :class:LocalPath / remote paths) to walk up the filesystem.

Returns None when neither applies (top-level storage with no URL hierarchy — e.g., :class:Memory, which overrides :meth:_url_parent to skip the URL branch).

parents property

parents: 'Iterator[IO]'

Walk the parent chain outward, yielding one IO per step.

Each step follows :attr:parent — cursor parent first, then URL parent (when applicable), terminating when .parent returns None. Empty on top-level non-URL storage (:class:Memory).

url property writable

url: 'URL'

Canonical URL identifying this holder.

size property

size: int

Current visible size in bytes.

Cursor / format-leaf IOs read the bound parent's size; storage subclasses override directly.

size_known property

size_known: bool

True when reading :attr:size won't trigger a backend probe.

Always true for in-memory IOs (size is a slot). Path IOs override to True only when their stat cache is warm — callers that want to short-circuit on an empty buffer (parquet / arrow IPC / CSV readers checking size == 0) can guard the check on this predicate so a cold remote path doesn't pay a HeadObject / get_status / get_metadata round trip just to discover the file is non-empty. Cursor / format-leaf IOs delegate to the parent.

holder_is_overwrite property

holder_is_overwrite: bool

True when the backing holder was opened in OVERWRITE mode.

Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.

mtime property

mtime: float

Last-modified time stamp.

media_type property writable

media_type

The holder's :class:MediaType, or None if unset.

Resolves lazily on first read: a fresh holder bound only by URL carries the sentinel ... in :attr:_media_type and runs :meth:URL.infer_media_type here once, caching the result back onto the slot. Subsequent reads (and pickling, IOStats snapshots, codec dispatch, …) hit the cached value.

Cursor IOs (those wrapping a :attr:parent storage) defer to the parent's stamped media type when their own slot is unset — the codec / format dispatch on a :class:JSONFile bound to a gzip-stamped :class:Memory parent needs to see the parent's media type, not its own (the cursor was constructed bare).

is_memory property

is_memory: bool

True when the IO lives entirely in process memory.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (:class:Memory) override directly.

is_local_path property

is_local_path: bool

True when the IO is a path on the local filesystem.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (:class:LocalPath) override directly.

is_remote_path property

is_remote_path: bool

True when the IO is a path on a non-local backend.

Cursor / format-leaf IOs delegate to the bound parent. Storage subclasses (remote paths) override directly.

is_streaming property

is_streaming: bool

True when :attr:size reflects only the bytes pulled so far.

Streaming holders (:class:MemoryStream over a live source) lazily pull bytes on read; their :attr:size grows as the cursor advances and may underreport the eventual total. Static holders (:class:Memory, :class:Path) know their full size up front so the default is False.

:class:IO.read checks this flag to decide whether to cap the requested byte count at :attr:size (static case — out-of-range reads would raise) or pass the request through unclamped (streaming case — the holder pulls until it has enough or EOF).

xxh3_64_digest property

xxh3_64_digest: bytes

8-byte big-endian payload digest — equivalent to xxh3_64().digest() but served from the cached :meth:xxh3_int64 so callers mixing the digest into a parent hash don't re-walk the payload.

holder property

holder: 'IO'

The bound parent IO (cursor case) or self (storage case).

Backwards-compatible alias preserved from the pre-merge IO.holder property — call sites that drilled through a cursor to reach its backing storage keep working.

owns_holder property

owns_holder: bool

Whether closing self also closes the bound parent.

mode property

mode: Mode

The typed :class:Mode enum this buffer was opened with.

pandas / pyarrow / zipfile inspect .mode for substrings like "b" to dispatch binary vs text reads; those sniffs still work because :class:Mode implements __contains__ against its :attr:~Mode.os_mode form ("b" in handle.mode → True). Reach for self.mode.os_mode when an actual POSIX string is required.

closed property

closed: bool

Stdlib IO[bytes] parity — False while the bound backing is reachable.

Stdlib semantics: closed means "file unusable for I/O." On a cursor the predicate flips only when teardown has dropped the parent reference; on a storage IO it always reads False (the storage owns its own bytes). Matters for pyarrow / pandas / polars / zipfile, which guard every op with an assert not closed.

commit

commit()

Commit current state

rollback

rollback()

Rollback current state

mark_dirty

mark_dirty() -> None

Signal pending mutations — commit on next clean :meth:close.

for_scheme classmethod

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch classmethod

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

to_singleton

to_singleton(ttl: Any = ...) -> 'Singleton'

Promote this instance into the per-class _INSTANCES cache.

Hot listing paths (iterdir / _ls / glob) build children with singleton_ttl=False so the bounded cache doesn't fill up with thousands of short-lived entries. When a caller decides one of those children is worth keeping around (handing it to a long-running worker, returning it from an API), :meth:to_singleton registers self into the cache so the next constructor call with the same key collapses to the same instance.

ttl defaults to the subclass's _SINGLETON_TTL (... = no caching, None = process lifetime, or a seconds count). When a different instance is already cached under this key, that pre-existing one wins and is returned unchanged — the cache is the source of truth.

invalidate_singleton

invalidate_singleton(remove_global: bool = True) -> None

Pop self from the per-class _INSTANCES cache.

Mutating ops on a Singleton-cached object (writes, deletes, schema invalidations on a Databricks table, put_object on an :class:S3Path) want to make sure the next caller asking for the same key gets a fresh build rather than collapsing onto this stale handle — that's what remove_global=True (the default) does. The pop is :meth:identity-guarded: only an entry that still points at self is removed, so a concurrent re-construction that already raced past this thread is left alone.

remove_global=False is a no-op. The keyword exists so subclass invalidators (invalidate_singleton, _invalidate_entity_tag_cache, …) can offer the same switch without branching at the call site.

matches_static

matches_static(
    predicate: "Predicate", *, free_cols: "tuple[str, ...] | None" = None
) -> bool

True iff predicate could match any row given :attr:static_values. Conservative on undecidables (column not in static values, predicate evaluator failure) so the caller still reads.

Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.

free_cols lets a caller that's about to prune the same predicate against N children precompute the free-column tuple once and reuse it — :func:free_columns walks the AST every call, so on a 64-OR predicate (the cache batch lookup shape) the saving is N-1 full walks per iter_children loop. Default None keeps the call site short for one-off prune checks.

options_class classmethod

options_class() -> 'type[O]'

The :class:CastOptions subclass this implementer consumes.

Default :class:CastOptions. Format-specific leaves with their own knobs (Parquet compression, CSV delimiter, …) override.

check_options classmethod

check_options(
    options: "O | None" = None, overrides: "dict | None" = None, **kwargs: Any
) -> O

Validate and merge caller kwargs into a resolved options.

Canonical pattern: a public method passes overrides=locals() and the ...-defaulted entries are stripped, the rest merged.

cleanup

cleanup(wait: 'Any' = False) -> int

Garbage-collect stale state on this backend.

Default no-op (returns 0) — single-file leaves and warehouse-backed tables don't have a sweep concept the client owns. Folder-shaped subclasses override to unlink stale part-* files, throttled by TTL.

wait controls sync vs async dispatch on backends that support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig (or True / a positive timeout) blocks until the sweep finishes; a falsy value (the default) hands the work off to a background thread. Backends without an async path treat both the same.

Returns the number of files / rows removed when known; 0 for fire-and-forget async dispatch or a no-op backend.

optimize

optimize(byte_size: 'int | None' = None, **kwargs: Any) -> int

Repartition / compact this Tabular's storage.

Default implementation is a no-op and returns 0 — single-file leaves (parquet, csv, arrow IPC, …) don't have a compaction concept. Aggregator subclasses (:class:Folder) override this to walk their child leaves and bin-pack small part files into bundles near byte_size. Files already close to the target size are left alone so a repeated call is cheap.

byte_size=None keeps the legacy "collapse every leaf with more than one part into a single file" behavior, which is what the local-cache compaction loop in :class:Session expects. Any extra keyword arguments are accepted and ignored so upstream callers can pass forward-compatible knobs without the base raising.

delete

delete(
    predicate: "PredicateLike" = None,
    *,
    wait: "WaitingConfigArg" = True,
    missing_ok: bool = False,
    delete_staging: bool = True,
    **kwargs: Any
) -> "Table"

Delete rows matching predicate; return this tabular.

predicate is a :class:Predicate from :mod:yggdrasil.execution.expr or a SQL string that parses into one ("id IN (1,2,3)", "price > 100 AND region = 'EU'"). None means "no filter" — every row is removed (DELETE FROM t with no WHERE).

wait / missing_ok / delete_staging are honoured by resource-backed subclasses (e.g. :class:yggdrasil.databricks.table.table.Table, which drops the table asset); the generic row-rewrite path ignores them. Any extra **kwargs (e.g. options=DeltaOptions(...)) flow through to :meth:_delete.

The default implementation reads every batch, drops rows the predicate accepts, and rewrites the leaf with the survivors. Aggregator subclasses (:class:yggdrasil.path.folder.Folder) override to walk children, prune subtrees whose partition bounds make the predicate trivially false, and only rewrite the leaves that actually hold matched rows.

collect_schema

collect_schema(options: 'O | None' = None, **kwargs: Any) -> Schema

Return this Tabular's :class:Schema, caching the first hit.

The cache slot is :attr:_schema_cache; on first call this method stamps the resolved schema into it so subsequent collect_schema calls short-circuit. Writers overwrite the slot via :meth:_persist_schema; lifecycle hooks clear it via :meth:_unpersist_schema.

count

count(options: 'O | None' = None, **kwargs: Any) -> int

Return the number of rows in this tabular.

scan_arrow_batches

scan_arrow_batches(
    options: "O | None" = None, **kwargs: Any
) -> Iterator[pa.RecordBatch]

Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.

The lazy / zero-copy counterpart to :meth:read_arrow_batches, mirroring :meth:read_polars_frame vs :meth:scan_polars_frame. Where read_arrow_batches layers the full options pipeline on every batch — target cast, projection, resample, dedup, row-limit slicing, each of which can copy or re-encode — scan_arrow_batches hands back exactly what the leaf produced, untouched. For an in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular) those batches are views over the held buffers (no copy); for a byte-backed leaf they're the freshly-decoded batches with none of the extra processing copies layered on. Use it when you want the raw Arrow stream and will project / filter downstream yourself.

scan_arrow_table

scan_arrow_table(options: 'O | None' = None, **kwargs: Any) -> pa.Table

Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).

The zero-copy counterpart to :meth:read_arrow_table. Assembles the source batches with :func:pa.Table.from_batches, which references the batch buffers as table chunks rather than copying them — so no cast, no projection, no rechunk memcpy that read_arrow_table performs to coalesce + conform the result. An empty source yields an empty table carrying the bound schema.

The batches must share one schema (the zero-copy contract): read_arrow_table reconciles parts that drifted across writes, scan_arrow_table does not — reach for read_arrow_table when a source's parts are known to be heterogeneous.

scan_arrow_batch_reader

scan_arrow_batch_reader(
    options: "O | None" = None, **kwargs: Any
) -> "pa.RecordBatchReader"

Zero-copy scan as a streaming :class:pa.RecordBatchReader view.

The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps the source batch stream in a reader without the per-batch conform / target-cast pass, so batches flow through as views over the source buffers. The reader's schema is the source's own — taken from the first batch, so it matches the raw views exactly (no collect_schema probe, which on a byte cursor would consume the stream out from under the read). Only the first batch is pulled up front to seed the schema; the rest stay lazy behind the reader.

read_table

read_table(options: 'O | None' = None, **kwargs: Any) -> 'Tabular | None'

Read into an in-memory :class:Tabular.

When options.spark_session is set, reads via :meth:_read_spark_frame and wraps in a :class:Dataset. Otherwise materializes Arrow batches into :class:ArrowTabular. Returns None when empty.

write_table

write_table(obj: Any, options: 'O | None' = None, **kwargs: Any) -> None

Dispatch obj to the best _write_* hook based on its runtime type.

Recognizes another :class:Tabular (drained as a pyarrow record-batch stream), pa.Table / pa.RecordBatch / pa.RecordBatchReader, polars DataFrame / LazyFrame, pandas DataFrame, pyspark DataFrame, list[dict], dict[str, list], and iterables of any of the above. Module-name sniffing keeps optional engine deps out of the import graph — we only touch a frame's API once we've confirmed it's an instance of one we know how to drain.

union

union(other: 'Any', *, mode: 'ModeLike | None' = None) -> 'Tabular'

Return a Tabular representing self UNION ALL other.

mode controls how mismatched schemas are reconciled:

• Mode.IGNORE (default) — keep self's schema; extra columns in other are dropped, missing ones are filled null.
• Mode.APPEND — widen to the superset schema (every field from both sides survives).

Concrete subclasses override :meth:_union for in-place mutation (Arrow batch append, Spark unionByName).

Accepts :class:Tabular, pa.RecordBatch, pa.Table, list[Response], or a Spark DataFrame. None returns self unchanged.

read_spark_dataset

read_spark_dataset(options: 'O | None' = None, **kwargs: Any) -> 'SparkDataset'

Read into a :class:Dataset holder.

Mirrors :meth:read_arrow_dataset for the Spark engine: the return type is a yggdrasil holder rather than the bare engine frame, so callers keep the Tabular surface (chained transforms, persist / insert / schema, …) without an extra wrap at the call site. :class:Dataset overrides :meth:_read_spark_dataset to return itself in place — no materialise round trip when the source already speaks Spark.

read_record_iterator

read_record_iterator(
    options: "O | None" = None, **kwargs: Any
) -> "Iterator[Mapping[str, Any]]"

Stream rows as plain dict. True streaming — the full table never materializes; batch.to_pylist() does the column→row rotation in pyarrow C++ once per batch.

read_records

read_records(options: 'O | None' = None, **kwargs: Any) -> 'Iterator[Any]'

Stream rows as :class:yggdrasil.data.record.Record. Lower per-row allocation than :meth:read_pylist for stable-schema sources — the underlying :class:Schema is materialized once and shared by reference across every record.

unique

unique(by: 'str | Any | Iterable[Any]') -> 'Tabular'

Drop duplicate rows on by; keep first occurrence per key tuple.

Parameters

by One or more column references — :class:str column names, :class:yggdrasil.data.Field instances (resolved via :attr:Field.name), or any iterable mixing the two. Empty / None is a no-op — returns self.

Returns

Tabular A new holder carrying the deduped rows. Spark-shaped inputs (anything whose :meth:_native_spark_frame exposes a :class:pyspark.sql.DataFrame) return a fresh :class:yggdrasil.spark.tabular.Dataset over the spark-side dedup; everything else collects through Arrow and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.

resample

resample(
    on: "str | Any",
    sampling: "int | float | Any",
    *,
    partition_by: "str | Any | Iterable[Any] | None" = None,
    fill_strategy: "str | None" = "ffill"
) -> "Tabular"

Align rows to a fixed time grid on on; one row per bucket.

Parameters

on The time column to resample on — column name (:class:str) or :class:yggdrasil.data.Field. sampling Bucket size. Accepted shapes:

* :class:`int` / :class:`float` — seconds (floats are
  rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
  ``"P1D"``, ``"PT15M"``) parsed via
  :func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.

``sampling <= 0`` is a short-circuit — returns ``self``.

partition_by Entity columns the resample is independent on. None / empty → flat global timeline. Same coercion as :meth:unique's by. fill_strategy How to fill nulls left by the bucket's "first" aggregation. "ffill" (default), "bfill", or "none" / None to disable. See :func:yggdrasil.arrow.ops.fill_arrow_table for the full semantics.

Returns

Tabular Spark-shaped holders return a :class:Dataset over the spark-side resample; everything else returns an :class:ArrowTabular over the arrow-side resample.

select

select(*columns: 'str | Any') -> 'Tabular'

Project to columns and return a new Tabular.

Each entry is a column reference — :class:str, a :class:yggdrasil.data.Field (resolved via :attr:Field.name), or an iterable mixing both. The result preserves the caller's order, which matches both :meth:pyarrow.Table.select and :meth:pyspark.sql.DataFrame.select semantics.

Raises :class:ValueError on an empty selection — a zero- column projection is almost always a caller mistake; pass :class:Schema.empty projections through the cast surface instead.

drop

drop(*columns: 'str | Any') -> 'Tabular'

Return a new Tabular with the named columns removed.

Columns missing from the source are silently ignored — matches Spark's :meth:DataFrame.drop and pyarrow's :meth:Table.drop_columns (when filtered to existing names). An empty argument list is a no-op that returns self.

filter

filter(predicate: 'PredicateLike') -> 'Tabular'

Drop rows where predicate is false.

predicate accepts every shape :meth:yggdrasil.execution.expr.Expression.from_ recognises:

• a SQL predicate string ("x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser;
• a yggdrasil :class:Predicate node (col("x") > 0, :func:is_in, :func:between, …);
• a native engine expression — :class:pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column — lifted via the matching backend.

The predicate is parsed once and dispatched to the typed :meth:_filter hook; the engine-side filter then runs in its native kernel (Arrow C++, Spark Catalyst) so the row scan stays vectorised.

cast

cast(options: 'O | None' = None, **kwargs) -> 'Tabular'

Cast rows, returning a new :class:Tabular.

Accepts a :class:Schema or :class:CastOptions. When options is given, reads to arrow and casts each batch through :meth:CastOptions.cast_arrow_batch.

display

display(n: int = 10, *, max_width: int = 32) -> str

Render the first n rows as an aligned, typed text table.

Columns and their types come from this Tabular's own :meth:collect_schema — the header is two rows: the column names, then their type tags (the project :class:~yggdrasil.data.Field's :meth:Field.short → :meth:DataType.short, recursive for nested types — i64 / str / list<str> / struct<name:str, age:i64>). Columns are separated by │ with a ─┼─ rule; numbers/booleans right-align; nested cell values are compacted to one line. Long values and headers are clipped (cells to max_width, type/name tags to a slightly larger cap) so one long string or column name can't balloon the table. The n rows are pushed down as a row_limit so no more than that is ever read.

print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))

lazy

lazy() -> 'LazyTabular'

Return a :class:LazyTabular wrapping this source.

Transformations on the returned object (select, filter, join, …) accumulate in an :class:ExecutionPlan without touching data. Any read_* call materialises the plan.

joinpath

joinpath(*segments: Any) -> 'IO'

Build a sibling IO at self.url joined with segments.

URL-shaped IOs (:class:LocalPath, remote paths) use this to mint a child path; :class:Memory and other non-URL leaves raise :class:ValueError.

to_url

to_url() -> 'URL'

The canonical :class:URL that addresses this holder.

from_ classmethod

from_(
    obj: Any, *, url: URL | None = None, mode: ModeLike = "rb+", **kwargs
) -> "IO"

Auto-route obj to the right storage / cursor, return an owning IO.

Two shapes share the method:

• Storage subclasses (cls has a :attr:scheme — :class:IO itself, :class:Memory, :class:LocalPath, remote paths). The result is a storage IO that owns its bytes — IO.from_(b"x") → :class:Memory, IO.from_("file://...") → :class:LocalPath.
• Cursor / format-leaf subclasses (cls has no scheme — :class:IO, :class:ParquetFile, :class:CSVFile, …). The result is an owning cursor over a fresh storage parent built from obj.

Recognised input shapes:

• :class:IO of cls — pass through (idempotent).
• :class:IO of a different class — for storage cls, return the underlying parent; for cursor cls, borrow the same parent into a fresh cursor.
• bytes-like (bytes / bytearray / memoryview) — back with a fresh :class:Memory.
• path-like (str / pathlib.Path / URL) — back with the path-shaped storage class for the scheme.
• local file handle — back with :class:LocalPath; lazy read from disk (no drain).
• other file-like — drain into a fresh :class:MemoryStream.

from_url classmethod

from_url(url: URL, **kwargs) -> 'IO'

Create a new IO from a URL.

When cls is abstract (has subclasses but isn't itself constructible — e.g. :class:Path), the URL scheme is resolved through the :class:URLBased registry to a concrete subclass; an unknown scheme raises :class:ValueError instead of producing the obscure "Can't instantiate abstract class" :class:TypeError.

from_bytes classmethod

from_bytes(data: bytes, **kwargs) -> 'IO'

Create a new IO from bytes.

from_holder classmethod

from_holder(
    holder: "IO",
    *,
    owns_holder: bool = False,
    mode: ModeLike = "rb+",
    media_type: Any = None,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Construct a cursor over holder, dispatching to the format leaf.

Resolves the format-specific :class:IO leaf via media_type (when given) or the holder's stamped stat().media_type, and returns an instance of that leaf bound to holder. When no leaf can be resolved, falls back to cls itself.

With auto_open=True (the default) the returned cursor is already acquired, so the caller can immediately read/write without entering a with block. Set auto_open=False to defer the acquire to the caller's with / :meth:acquire.

owns_holder=True hands close-ownership of holder to the returned cursor — closing the cursor closes the holder. The default False keeps the holder's lifetime in the caller's hands; the returned cursor is a non-owning borrow.

class_for_media_type classmethod

class_for_media_type(
    media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"

Resolve a :class:MediaType (or coercible) to its format leaf.

Looks up :attr:MediaType.mime_type's name in :data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet compressed with zstd or snappy still resolves to :class:ParquetFile; the codec layer is the holder's concern.

The returned class is a :class:Tabular subclass — typically a :class:Holder byte-backed leaf, occasionally a non-Holder leaf (:class:Folder, :class:DeltaFolder). Returns default on miss when supplied; otherwise raises :class:KeyError with the list of registered names.

for_holder classmethod

for_holder(
    holder: "IO",
    *,
    media_type: "MediaType | MimeType | str | None" = None,
    default: Any = ...,
    **kwargs: Any
) -> "Tabular"

Build the right format leaf for holder.

Resolution order for the format discriminator:

1. The explicit media_type kwarg, when supplied.
2. holder.stat().media_type — set by the holder from its URL extension, magic-byte sniff, or content-type header.

The resolved class is instantiated as Cls(holder=holder, **kwargs). On lookup miss, falls back to default when supplied; otherwise raises :class:KeyError.

registered_classes classmethod

registered_classes() -> 'dict[str, type]'

Snapshot of the registry — debugging / introspection only.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Slice size bytes from offset as a :class:memoryview.

cursor=True ignores the explicit offset and reads from the holder's internal cursor (:attr:tell), advancing it past the bytes returned. cursor=False (default) keeps the cursor-less positional contract — the cursor is untouched.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's bounds-check uses its own size — avoids a redundant stat probe on remote backings when the cursor has no local size cache, and routes through any subclass _active override (lazy materialization on :class:ZipEntryFile, …).

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Splice data at offset, pre-growing the holder as needed.

size caps the byte count written — size=-1 (default) writes all of data; size>=0 writes min(len(data), size) bytes. Caps via a slice of data (zero-copy on memoryview / bytes), so downstream pipelines that only need the first N bytes of a larger buffer skip the trailing tail.

overwrite declares that this write replaces the holder's tail past offset + size — after the splice, :attr:size is set to offset + size. Callers that currently do truncate(0) followed by write_bytes(...) collapse to a single write_bytes(..., overwrite=True), which on whole-blob remote backends saves a SDK round trip (the atomic upload at offset == 0 already replaces the object — no preceding truncate needed).

Pipeline:

1. Slice data to size if capped.
2. Normalize offset (-1 → append, -N → self.size - N).
3. Pre-grow visible :attr:size to cover the splice via :meth:resize.
4. Hand the normalized (data, offset) to :meth:_write_mv.
5. Truncate tail past offset + n when overwrite.
6. Mark dirty + bump cached mtime if anything was written.

update_stat=False skips the post-write :meth:_touch_stat and :meth:mark_dirty calls. Use it for bulk loops that want a single stat refresh at the end (one :func:time.time call instead of one per write); the caller is then responsible for calling :meth:_touch_stat (or re-statting via the path-side _stat for filesystem backends) once the loop finishes.

Cursor IOs (those wrapping a :attr:parent storage) delegate the whole call through :meth:_active so the parent's resize / bounds-check / dirty-marking fires once, on the backing storage — the cursor only advances its own _pos.

reserve

reserve(n: int) -> None

Pre-grow capacity to at least n bytes.

Capacity-only — does NOT change :attr:size. Idempotent when capacity ≥ n already. Subclasses with no growable capacity layer may treat this as a no-op. Cursor / format-leaf IOs delegate to the bound parent.

resize

resize(n: int) -> int

Grow visible :attr:size to at least n bytes (one-way).

Sister of :meth:truncate, but never shrinks. Used by :meth:write_mv to pre-allocate a known target before the splice so :meth:_write_mv doesn't have to manage size.

• n <= size → no-op, returns current :attr:size.
• n > size → extends with zero-padding via :meth:truncate, returns n.

Subclasses with a native grow-only primitive (capacity hint to a remote upload session, posix_fallocate on local fd) override for the cheaper path; the default works on every backend.

truncate

truncate(size: 'int | None' = None) -> int

Set the visible :attr:size to exactly size bytes.

Shrinks drop the tail; extends zero-pad. Returns the new size.

On a cursor (self._parent is not None), size=None truncates at the current cursor position and the cursor is clamped if it would exceed the post-truncate size. On a storage IO size=None is invalid — pass an explicit byte count.

clear

clear() -> None

Drop the IO's payload entirely.

:class:Memory resets the underlying bytearray to zero bytes (capacity drops too). :class:yggdrasil.io.path.Path unlinks the backing file with missing_ok=True so the operation is idempotent. After :meth:clear, :attr:size reads 0 and the IO is still usable — subsequent writes grow it from scratch.

stat

stat() -> IOStats

Snapshot the holder's metadata into a fresh :class:IOStats.

Delegates to :meth:_stat for the backend-specific fields (kind and the live size for path-bound holders); mutating the returned instance does NOT round-trip onto the holder. Use the holder's own setters / :meth:_touch_stat when you need to update metadata.

touch_mtime

touch_mtime(when: float | None = None) -> None

Stamp the holder's mtime with the current time.

Bulk-write helper — call once after a write loop instead of letting every :meth:write_mv call sample the clock. when accepts an explicit timestamp (e.g. an upstream "Last-Modified" header); None defaults to :func:time.time.

acquire

acquire() -> 'IO'

Bring the IO's backing into the acquired state.

Lifecycle primitive — idempotent. Returns self. :meth:__enter__ calls this; so does :meth:open before constructing its cursor IO.

open

open(
    mode: ModeLike = "rb+",
    *,
    media_type: "MediaType | None" = None,
    owns_holder: bool = False,
    auto_open: bool = True,
    **kwargs: Any
) -> "IO"

Acquire the IO and return a fresh :class:IO cursor over it.

Dispatches to the format-specific :class:IO leaf via the IO's stamped media type (or media_type override), so LocalPath("data.parquet").open() lands on :class:ParquetFile, LocalPath("data.csv").open() on :class:CSVFile, and an unknown / no-media holder falls back to a plain :class:IO.

Pattern::

with LocalPath("/tmp/x.bin").open("wb") as bio:
    bio.write(b"hello")
# path released here.

with LocalPath("data.parquet").open() as bio:
    table = bio.read_arrow_table()  # Tabular surface
# path released here.

The default owns_holder=False returns a non-owning cursor — closing the cursor leaves the parent open, so the caller can mint multiple cursors against the same parent. Pass owns_holder=True to transfer close-ownership of the parent to the cursor (the cursor's close then also closes the parent).

flush

flush() -> None

Push buffered writes to the durable backing.

Cursor IOs forward the flush to their bound parent; storage IOs go through :meth:Disposable.commit (default no-op unless a subclass overrides).

close

close(force: bool = False) -> None

Release the IO; on :attr:temporary, discard pending writes instead of committing them.

On a cursor with owns_holder=True the bound parent is closed too. Preserves the cursor position across the close — a reopen on the same instance lands at the byte the previous transaction left off.

pread

pread(n: int, pos: int, *, cursor: bool = False) -> bytes

Positional read. Returns at most n bytes at pos.

cursor=True reads from the internal cursor instead of pos and advances it past the bytes returned.

pwrite

pwrite(
    data: Union[bytes, bytearray, memoryview],
    pos: int,
    *,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Positionally write. Returns bytes actually written.

update_stat=False defers the post-write stat refresh to the caller — see :meth:write_mv for the bulk-write rationale. cursor=True writes at the internal cursor instead of pos and advances it by the bytes written.

memoryview

memoryview() -> memoryview

View over the holder's visible bytes.

iter_mv

iter_mv(
    chunk_size: int = 256 * 1024,
    *,
    start: int = 0,
    length: Optional[int] = None
) -> Iterator[memoryview]

Yield [start, start+length) in bounded, zero-copy memoryview chunks (default: the whole holder from start).

Each chunk is a :meth:read_mv slice — a view straight into the live in-memory window, or a bounded read for spilled / file-backed storage — so a consumer like http.client can sock.sendall it without a copy, and never more than chunk_size is resident at once. Reads are positional (the cursor is untouched), so the holder can be iterated again — e.g. a connection retry re-sending the same body — by calling this afresh.

read_bytes

read_bytes(size: int = -1, offset: int = 0, *, cursor: bool = False) -> bytes

Read size bytes starting at offset as :class:bytes.

size=-1 reads to EOF; offset accepts negative indices via :func:_resolve_pos (-1 → size, -N → self.size - N). cursor=True reads from the internal cursor and advances it past the bytes returned.

write_bytes

write_bytes(
    data: Any,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: "bool | None" = None,
    cursor: bool = False
) -> int

Splice data at offset. Returns bytes written.

overwrite defaults to None → resolved: a whole-content write from the start (offset == 0, size == -1, no cursor) replaces the object (pathlib write_bytes truncate semantics), so a whole-blob remote backend does it in a single PUT instead of a stat + read-page + upload read-modify-write. A positional / cursor / size-capped write is a splice that preserves the rest, so it resolves to False. Pass an explicit True / False to force either.

size caps the byte count written — size=-1 (default) writes the entire source; size>=0 writes at most size bytes. The cap is forwarded into each type-directed branch so a stream source stops reading after size bytes (no over-pull) and a bytes-like source slices its tail off before dispatching.

overwrite declares that this write replaces every byte from offset onward. The holder ends at offset + bytes_written regardless of its prior size, and whole-blob remote backends collapse the implied truncate(...) + write(...) pair into one SDK call.

Type-directed dispatch — bytes-like payloads (:class:bytes, :class:bytearray, :class:memoryview, and str after UTF-8 encoding) splice through :meth:write_mv; other :class:Holder instances route through :meth:write_holder (size-aware: small payloads write inline, large ones stream); file-like sources (anything exposing .read) drain through :meth:write_stream. Subclasses override :meth:_write_mv, :meth:_write_stream, and / or :meth:_write_holder rather than this dispatch.

read_text

read_text(
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    size: int = -1,
    offset: int = 0,
    cursor: bool = False
) -> str

Decode size bytes at offset as text.

cursor=True reads from the internal cursor and advances it.

write_text

write_text(
    text: str,
    encoding: str = "utf-8",
    errors: str = "strict",
    *,
    offset: int = 0,
    cursor: bool = False
) -> int

Encode text and splice at offset. Returns bytes written.

cursor=True writes at the internal cursor and advances it.

head

head(size: int, *, offset: int = 0) -> bytes

Peek the first size bytes from offset (default 0).

A bounded positional read off the front of the object that leaves the internal cursor (:meth:tell) untouched — head composes with cursor reads without disturbing them. size is clamped to what's available, so a short object (or one shorter than offset + size) returns fewer bytes rather than raising; size < 0 reads from offset to EOF.

tail

tail(size: int) -> bytes

Peek the last size bytes, leaving the cursor untouched.

The end-anchored companion to :meth:head — a bounded positional read off the back of the object. size is clamped to the object's length, so requesting more than exists (or size < 0) returns the whole object. The internal cursor (:meth:tell) is not moved.

readinto

readinto(buffer: Any, *, offset: int = 0, cursor: bool = False) -> int

Fill buffer with bytes starting at offset.

Returns the number of bytes written into buffer — min(len(buffer), self.size - offset). Matches the stdlib :meth:io.RawIOBase.readinto shape. cursor=True reads from the internal cursor and advances it.

On a cursor IO (_parent is not None) the default flips to cursor-anchored — stdlib readinto(buf) then matches the BinaryIO contract.

readline

readline(limit: int = -1, *, offset: int = 0, cursor: bool = False) -> bytes

Read up to the next newline starting at offset.

Returns the line including the trailing \n (or short when EOF lands first). limit >= 0 caps the byte count. cursor=True reads from the internal cursor and advances it past the returned line. On a cursor IO the default flips to cursor-anchored.

readlines

readlines(
    hint: int = -1, *, offset: int = 0, cursor: bool = False
) -> list[bytes]

Read every line from offset to EOF (or until hint bytes).

cursor=True reads from the internal cursor and advances it past the bytes consumed. On a cursor IO the default flips to cursor-anchored.

tell

tell() -> int

Current cursor position.

seek

seek(offset: int, whence: int = 0) -> int

Seek the internal cursor to offset relative to whence.

Mirrors :meth:io.IOBase.seek with two ergonomic deviations:

• seek(-1, SEEK_SET) is a "go to end" sentinel — pairs with read(-1) / "read all". Any other negative SEEK_SET offset raises :class:ValueError.
• SEEK_CUR / SEEK_END with a negative offset that would land before byte 0 clamps to 0 instead of raising.

write_local_path

write_local_path(
    path: PathLike,
    *,
    pos: int = 0,
    n: int = -1,
    chunk_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Load path's bytes into this holder at pos.

n < 0 reads the whole file; n >= 0 caps the source bytes pulled at n. Streams in chunk_size slices so a large file doesn't materialize into memory.

Pre-allocates the holder via :meth:resize when the source size is known up front (n >= 0 or local stat available), so the inner loop only writes — no per-chunk grow.

write_stream

write_stream(
    src: Any,
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Drain a binary source into this holder at offset.

Public entry point: accepts a yggdrasil :class:IO[bytes], a stdlib :class:typing.BinaryIO (io.BytesIO, open(..., "rb"), urllib3 responses, …), or any file-like carrying a .read. Non-:class:IO sources are coerced via :meth:IO.from_ so subclass-side :meth:_write_stream always receives a real :class:IO[bytes].

size caps the byte count drained from src — size=-1 (default) reads to EOF; size>=0 stops at size bytes (no over-pull from the source).

overwrite truncates the holder's tail past offset + bytes_written; whole-blob remote backends get a single atomic PUT instead of an explicit truncate followed by a write.

batch_size is the read/write chunk size for the default streaming path (:data:_COPY_CHUNK, 1 MiB). Tune up for high-throughput remote sinks where the per-call overhead dominates, or down to bound peak memory on a slow consumer.

write_holder

write_holder(
    src: "Holder",
    *,
    offset: int = 0,
    size: int = -1,
    overwrite: bool = False,
    batch_size: int = _COPY_CHUNK,
    cursor: bool = False
) -> int

Splice another :class:Holder's bytes into this one at offset.

Public entry point: validates the inputs, then dispatches to :meth:_write_holder. size caps the byte count pulled from src — size=-1 (default) writes the whole source; size>=0 writes the first size bytes. overwrite truncates the tail past offset + bytes_written (collapses truncate(...) + write_holder(...) into one operation for whole-blob remote backends). batch_size is forwarded to the streaming path for above-threshold payloads.

Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).

upload

upload(src: Any, *, size: int = -1, offset: int = 0) -> 'Holder'

Upload src's bytes into this holder.

Symmetric to :meth:download but indexed from the destination side — dst.upload(src) makes the destination's content equal to the source's.

src accepts any of:

• :class:Holder (incl. any :class:Path subclass) — its bytes are pulled starting at offset.
• :class:IO cursor — offset (if non-zero) seeks before read(); otherwise the cursor's current position is honoured.
• str / :class:os.PathLike — coerced via Path.from_(src) and treated as a holder.

size and offset slice the source: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Slicing forces the whole-payload fast path in :meth:_transfer_to to defer to a bytes copy (the backend-specific shortcuts — shutil.copyfile, write_local_path — don't expose a window).

When self is a :class:Path whose URL ends in a trailing / (directory shape), the source's filename (src.url.name or "download" for nameless holders) is joined onto it. No remote stat is issued — the trailing slash is a purely local, cp-style hint.

Returns the resolved destination so chains like dst.upload(src).read_bytes() work.

Subclasses with a faster move (e.g. local→local via sendfile, local→remote chunked stream) override :meth:_transfer_to, not this method.

download

download(to: Any = None, *, size: int = -1, offset: int = 0) -> 'Holder | IO'

Copy this holder's bytes to a local target.

When to is :data:None, bytes land in the user's ~/Downloads folder under :attr:url.name (or "download" for nameless holders), with browser-style (1) / (2) / … suffixes appended on name conflict. Otherwise to accepts the same shapes as :meth:upload (:class:Holder, :class:IO, str / :class:os.PathLike). size and offset slice this holder: size=-1 (default) reads to EOF, size>=0 caps the byte count, offset is the starting offset. Returns the resolved target.

to_bytes

to_bytes() -> bytes

Full payload as :class:bytes — alias for read_bytes().

getvalue

getvalue() -> bytes

Stdlib :class:io.BytesIO parity — alias for :meth:to_bytes.

decode

decode(encoding: str = 'utf-8', errors: str = 'replace') -> str

Decode the whole payload as text. Cursorless — does not seek.

to_base64

to_base64(urlsafe: bool = True) -> str

Return the payload base64-encoded as an ASCII str.

urlsafe=True (default) uses :func:base64.urlsafe_b64encode — - / _ in place of + / / so the result drops cleanly into a URL or filename. urlsafe=False falls back to the standard alphabet.

xxh3_64

xxh3_64()

Return an :class:xxhash.xxh3_64 instance over the payload.

Always rebuilds an updatable :class:xxhash.xxh3_64 so callers can keep mixing more bytes in if they want. The expensive part — walking the payload — is short-circuited via the cached digest; we just seed a fresh hasher with the cached value's bytes when available.

xxh3_int64

xxh3_int64() -> int

64-bit xxh3 hash of the payload as a signed int64.

xxh3_64 produces an unsigned 64-bit value; downstream Arrow schemas pin the field as int64, so the digest is wrapped into signed range [-2**63, 2**63). Memoized against (_size, _mtime) — which every write path bumps via :meth:_touch_stat — so repeated reads pay the walk once.

remaining_bytes

remaining_bytes() -> int

Bytes from the cursor to EOF on the active payload.

arrow_input_stream

arrow_input_stream() -> '_ArrowInputStreamContext'

Context manager yielding the cheapest :class:pa.NativeFile over the payload.

Local-path holder + no codec → :func:pyarrow.memory_map (zero-copy). Codec-tagged holder → decompress, then wrap in a :class:pa.BufferReader. Anything else → snapshot and wrap. The yielded stream is always a real :class:pa.NativeFile, so the caller hands it directly to pyarrow readers.

arrow_output_stream

arrow_output_stream(*, append: bool = False) -> '_ArrowOutputStreamContext'

Context manager yielding a :class:pa.BufferOutputStream writer.

with bio.arrow_output_stream() as sink: writer(sink). The yielded sink accepts the format encoder's writes against a pure-Arrow in-memory buffer. On a clean exit the encoded bytes are committed to self via :meth:_commit_format_payload, which handles codec compression and the overwrite-vs-append disposition.

appendable

appendable() -> bool

True when writes append at EOF — :data:Mode.APPEND only.

with_media_type

with_media_type(media_type: Any, *, copy: bool = False) -> 'IO'

Stamp media_type onto the bound IO's metadata.

With copy=False (the default), mutates self and returns it. copy=True allocates a fresh holder over the same bytes and returns a new IO over it.

as_media

as_media(media_type: Any = None) -> 'IO'

Return a typed Tabular leaf bound to this buffer's holder.

.. deprecated:: Use :meth:open with a media_type instead — holder.open(media_type=...) dispatches to the same format leaf and returns an acquired cursor. as_media is retained for callers that haven't migrated.

Resolution: explicit media_type wins; otherwise the buffer's stamped media type is used. The leaf borrows the same backing storage so durable bytes are shared without a copy. When self is already an instance of the resolved leaf class, returns self unchanged.

Raises :class:KeyError when no media type can be resolved or the resolved type has no registered Tabular leaf.

fileno

fileno() -> int

Underlying fd if the holder exposes one. Raises otherwise.

read

read(size: int = -1) -> bytes

Read up to size bytes from the cursor, advancing past them.

Stdlib :meth:io.RawIOBase.read semantic: size < 0 / None reads to EOF; otherwise reads up to size bytes, returning fewer at EOF.

Static IOs (:class:Memory, :class:Path) know their full size up front; cap the request at self.size - self._pos before dispatching so the storage's strict read_bytes doesn't trip on an out-of-range window. Streaming IOs (:class:MemoryStream — is_streaming) lazily pull bytes; forward the request unclamped so the storage pulls until it has enough or signals EOF.

readall

readall() -> bytes

Read from cursor to EOF, advancing the cursor.

write

write(b: Any, *, update_stat: bool = True) -> int

Write b at the cursor, advancing it.

Accepts bytes-like, str (UTF-8), io.BytesIO, or any file-like with .read. File-like sources route through :meth:write_stream so backends with an atomic whole-object upload push a single request. The buffer-protocol fallback catches things like :class:pyarrow.Buffer that aren't bytes/bytearray/memoryview but ARE memoryview-able.

read_bytes_u32

read_bytes_u32() -> bytes

Length-prefixed (uint32 LE) bytes blob.

read_str_u32

read_str_u32(encoding: str = 'utf-8') -> str

Length-prefixed UTF-8 string.

json_load

json_load(*, media_type: Any = None, orient: Any = None) -> Any

Parse the buffer, auto-detecting media type and compression.

Resolution order for the media type:

1. Explicit media_type kwarg.
2. Cached :attr:media_type on the IO.
3. Magic-byte sniff via :meth:MediaType.from_io — when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.

If the resolved type carries a codec the buffer is decompressed first and the inner mime is stamped onto the decompressed buffer. JSON / NDJSON / opaque-bytes payloads go through json.loads (or pandas.read_json when orient is set); every other registered format dispatches to its :class:Tabular leaf and returns read_pylist().

decompress

decompress(*, codec: Any = None, copy: bool = True) -> 'IO'

Return a new IO over the decompressed payload.

codec may be a :class:Codec, a codec name ("gzip", "zstd", …), or a :class:MediaType-shaped object whose codec attribute is read. Returns the original buffer when no codec is set / supplied.