yggdrasil.path¶
path ¶
Backend-agnostic filesystem abstractions for yggdrasil.
Public surface¶
- :class:
Path— abstractpathlib.Path-like path - :class:
RemotePath— abstract base for network-backed paths (S3, Databricks, …) - :class:
LocalPath— local :class:Pathbacked by :mod:ossyscalls - :class:
ProxyPathMixin— be a path by delegating to an inner one
Stat-related types live on :mod:yggdrasil.io.io_stats
(:class:IOStats / :class:IOKind).
Concrete remote backends register themselves on import (see
yggdrasil.aws.fs and yggdrasil.databricks.fs).
Path ¶
Path(
data: Any = None,
*,
stat: IOStats | None = None,
url: URL | None = None,
binary: bytes | bytearray | memoryview | None = None,
path: PathLike | None = None,
holder: "IO | None" = None,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
temporary: bool = False,
singleton_ttl: Any = ...,
**kwargs
)
Bases: IO, PathLike, ABC
Abstract URL-addressed byte holder with filesystem semantics.
Two layers, no shared state between them:
- Holder I/O — :meth:
read_mv/ :meth:write_mv/ :meth:truncate/ :meth:clear/ :attr:sizeroute through :meth:_read_mv/ :meth:_write_mv. Backends with native positional I/O (LocalPath viaos.pread/os.pwrite) override these directly; whole-blob backends (RemotePath) splice via a download / re-upload. - Filesystem — :meth:
stat, :meth:exists, :meth:is_file, :meth:is_dir, :meth:iterdir, :meth:mkdir, :meth:unlink, :meth:remove. All thin wrappers over the abstract hooks.
Pure-path manipulation (:attr:parts, :attr:name, :attr:parent,
:attr:suffix, :meth:joinpath, :meth:with_suffix, …) delegates
straight to :attr:url — Path adds no parsing.
Initialize the IO.
Exactly one of url / binary / path / data /
holder determines the seed; the rest are mutually
exclusive (validated in :meth:__new__).
holder= (alias: parent=) borrows an existing IO as
backing storage — every byte primitive then delegates through
:meth:_active. owns_holder=True transfers close-ownership
so closing this IO also closes the parent.
temporary=True marks the IO for self-cleanup on release:
:meth:_release calls :meth:clear so the payload is dropped
when the IO closes. Default False — clears only happen
when the caller asks.
mode follows stdlib :func:open semantics, normalized to
a :class:Mode enum. Side effects fire on :meth:_acquire,
not here: cursor stays at byte 0 until then.
stat lets callers seed the metadata cache (size / mtime /
media_type) when they already know it — saves a backend probe
on the first :meth:stat call.
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
size_known
property
¶
True when reading :attr:size won't trigger a backend probe.
Always true for in-memory IOs (size is a slot). Path IOs
override to True only when their stat cache is warm —
callers that want to short-circuit on an empty buffer
(parquet / arrow IPC / CSV readers checking size == 0)
can guard the check on this predicate so a cold remote path
doesn't pay a HeadObject / get_status /
get_metadata round trip just to discover the file is
non-empty. Cursor / format-leaf IOs delegate to the parent.
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_local_path
property
¶
True when the IO is a path on the local filesystem.
Cursor / format-leaf IOs delegate to the bound parent.
Storage subclasses (:class:LocalPath) override directly.
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
options_class
classmethod
¶
The :class:CastOptions subclass this implementer consumes.
Default :class:CastOptions. Format-specific leaves with
their own knobs (Parquet compression, CSV delimiter, …)
override.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
read_mv ¶
Slice size bytes from offset as a :class:memoryview.
cursor=True ignores the explicit offset and reads from
the holder's internal cursor (:attr:tell), advancing it past
the bytes returned. cursor=False (default) keeps the
cursor-less positional contract — the cursor is untouched.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
bounds-check uses its own size — avoids a redundant stat
probe on remote backings when the cursor has no local size
cache, and routes through any subclass _active override
(lazy materialization on :class:ZipEntryFile, …).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice data at offset, pre-growing the holder as needed.
size caps the byte count written — size=-1 (default)
writes all of data; size>=0 writes
min(len(data), size) bytes. Caps via a slice of
data (zero-copy on memoryview / bytes), so
downstream pipelines that only need the first N bytes of
a larger buffer skip the trailing tail.
overwrite declares that this write replaces the
holder's tail past offset + size — after the splice,
:attr:size is set to offset + size. Callers that
currently do truncate(0) followed by write_bytes(...)
collapse to a single write_bytes(..., overwrite=True),
which on whole-blob remote backends saves a SDK round
trip (the atomic upload at offset == 0 already
replaces the object — no preceding truncate needed).
Pipeline:
- Slice
datatosizeif capped. - Normalize
offset(-1→ append,-N→self.size - N). - Pre-grow visible :attr:
sizeto cover the splice via :meth:resize. - Hand the normalized
(data, offset)to :meth:_write_mv. - Truncate tail past
offset + nwhenoverwrite. - Mark dirty + bump cached mtime if anything was written.
update_stat=False skips the post-write
:meth:_touch_stat and :meth:mark_dirty calls. Use it for
bulk loops that want a single stat refresh at the end (one
:func:time.time call instead of one per write); the caller
is then responsible for calling :meth:_touch_stat (or
re-statting via the path-side _stat for filesystem
backends) once the loop finishes.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
resize / bounds-check / dirty-marking fires once, on the
backing storage — the cursor only advances its own _pos.
resize ¶
Grow visible :attr:size to at least n bytes (one-way).
Sister of :meth:truncate, but never shrinks. Used by
:meth:write_mv to pre-allocate a known target before the
splice so :meth:_write_mv doesn't have to manage size.
n <= size→ no-op, returns current :attr:size.n > size→ extends with zero-padding via :meth:truncate, returnsn.
Subclasses with a native grow-only primitive (capacity hint to
a remote upload session, posix_fallocate on local fd)
override for the cheaper path; the default works on every
backend.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
flush ¶
Push buffered writes to the durable backing.
Cursor IOs forward the flush to their bound parent; storage
IOs go through :meth:Disposable.commit (default no-op
unless a subclass overrides).
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
from_
classmethod
¶
Coerce obj (str / URL / pathlib / Path) into a :class:Path.
When called on the abstract :class:Path, dispatches via the
:class:Holder scheme registry to the subclass registered for
the URL's scheme; defaults to :class:LocalPath for path-shaped
URLs without an explicit scheme. When called on a concrete
subclass, returns an instance of that subclass.
truncate ¶
Resize to exactly n bytes via read-modify-write.
The generic whole-file fallback: download, slice / zero-extend,
re-write. Backends with a native resize (LocalPath
os.ftruncate, RemotePath's single upload) override this.
invalidate_singleton ¶
Drop the cached :class:IOStats and pop self from
_INSTANCES. Single canonical invalidator for paths.
Mutating ops (writes, deletes) call this so the next read on
the same logical path picks up fresh state. Subclasses with
their own caches (schema, table info, column lists) override
and chain super().invalidate_singleton(...).
ls ¶
ls(
*,
recursive: bool = False,
limit: "int | None" = None,
singleton_ttl: Any = False
) -> Iterator["Path"]
Yield children lazily. limit caps how many are produced — the
underlying listing stays incremental, so a bounded ls over a huge
prefix never materialises (or fetches) more than it needs.
unlink ¶
Remove the leaf — pathlib-compatible: refuses directories.
Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises
:class:IsADirectoryError for directories so callers don't
accidentally recursive-delete via unlink. Use :meth:remove
for the directory case. Thin wrapper over :meth:_delete's
path-removal mode.
remove ¶
remove(
recursive: bool = True,
missing_ok: bool = True,
wait: WaitingConfigArg = True,
fresher_than: Optional[TimeLike] = None,
older_than: Optional[TimeLike] = None,
) -> "Path"
Remove this path — the file, or the whole subtree when recursive.
Thin wrapper over :meth:_delete's path-removal mode (the single
deletion primitive). fresher_than / older_than scope the
removal to children inside that mtime window.
wait_until_gone ¶
Block until :meth:exists reports False or wait expires.
Polls the backend with a fresh probe each iteration — the
stat cache is invalidated between checks so a TTL'd hit
can't mask a deletion that landed after the cache was
filled. Useful when a fire-and-forget unlink (e.g.
WarehouseStatementBatch.clear_temporary_resources) means
the caller can't observe completion through the original
operation's return value.
Raises :class:TimeoutError when wait's deadline elapses
and the path is still present.
touch ¶
Create the path as an empty file if it doesn't exist.
write_bytes(b"") short-circuits in the holder fast path
(zero bytes, no flush), which would leave a missing file behind
— open + close around the empty write so the holder actually
materialises the entry on the backing store.
upload_module ¶
Zip a local module / package and write it under this path.
module is anything :func:resolve_module_root accepts —
an importable module name ("yggdrasil.io"), a
:class:os.PathLike pointing at a package directory or an
existing .zip / .whl archive, or a callable
carrying a __module__ attribute. The module is packed
into a deflated zip whose top-level entry is the package
directory itself, so the archive can be added to
sys.path directly (or fed to
:meth:SparkSession.addArtifacts with pyfile=True).
Destination shape on self:
- self names a file with a
.zip/.whlsuffix — archive bytes land at that exact path. - self is anything else — archive lands at
self / <name or "<module>.zip">.
Returns the concrete :class:Path that now holds the
archive. overwrite=False raises
:class:FileExistsError when the destination already
exists.
import_module ¶
import_module(
module_name: str | None = None,
*,
install: bool = True,
cache_dir: "Any" = None
) -> Any
Download a module archive at this path and import it.
Inverse of :meth:upload_module: fetch the archive bytes
at self, drop them on local disk, prepend the archive (or
its extracted parent) to :data:sys.path, and return the
live module via :func:importlib.import_module.
module_name defaults to the archive's stem (filename
minus suffix). cache_dir picks where the archive lands
locally (default: a fresh
:meth:LocalPath.staging_path-style directory).
install=True (the default) preserves the archive on
disk so subsequent imports in the same process hit the
cache. install=False makes the cache-dir lifetime the
caller's problem.
as_media ¶
Wrap this path in the format leaf for its media type.
.. deprecated::
Use :meth:open with a media_type instead —
path.open(media_type=...) already dispatches to the
right format leaf and gives a properly acquired cursor with
lifecycle handling. as_media returns an un-acquired leaf
and is kept only for callers that haven't migrated.
Resolution: explicit media_type first, else the holder's
:class:MediaType (path extension, magic-byte sniff, or
content-type header). The resolved class is looked up in the
:class:Holder format registry and instantiated bound to this
path.
Raises :class:KeyError when the path's media type isn't
registered as a tabular format.
open ¶
Acquire the path and return an :class:IO cursor bound to it.
mode accepts a :class:Mode member, an alias string, or
a stdlib open() mode string. None falls through to
:meth:Holder.open which uses "rb+". Other keyword
arguments (owns_holder, media_type, auto_open,
…) ride through to :meth:Holder.open.
File ¶
File(
data: Any = None,
*,
stat: IOStats | None = None,
url: URL | None = None,
binary: bytes | bytearray | memoryview | None = None,
path: PathLike | None = None,
holder: "IO | None" = None,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
temporary: bool = False,
singleton_ttl: Any = ...,
**kwargs
)
Bases: Path
Path that is known to be a file — skips is_dir/is_file stat calls.
Initialize the IO.
Exactly one of url / binary / path / data /
holder determines the seed; the rest are mutually
exclusive (validated in :meth:__new__).
holder= (alias: parent=) borrows an existing IO as
backing storage — every byte primitive then delegates through
:meth:_active. owns_holder=True transfers close-ownership
so closing this IO also closes the parent.
temporary=True marks the IO for self-cleanup on release:
:meth:_release calls :meth:clear so the payload is dropped
when the IO closes. Default False — clears only happen
when the caller asks.
mode follows stdlib :func:open semantics, normalized to
a :class:Mode enum. Side effects fire on :meth:_acquire,
not here: cursor stays at byte 0 until then.
stat lets callers seed the metadata cache (size / mtime /
media_type) when they already know it — saves a backend probe
on the first :meth:stat call.
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
size_known
property
¶
True when reading :attr:size won't trigger a backend probe.
Always true for in-memory IOs (size is a slot). Path IOs
override to True only when their stat cache is warm —
callers that want to short-circuit on an empty buffer
(parquet / arrow IPC / CSV readers checking size == 0)
can guard the check on this predicate so a cold remote path
doesn't pay a HeadObject / get_status /
get_metadata round trip just to discover the file is
non-empty. Cursor / format-leaf IOs delegate to the parent.
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_local_path
property
¶
True when the IO is a path on the local filesystem.
Cursor / format-leaf IOs delegate to the bound parent.
Storage subclasses (:class:LocalPath) override directly.
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
open ¶
Acquire the path and return an :class:IO cursor bound to it.
mode accepts a :class:Mode member, an alias string, or
a stdlib open() mode string. None falls through to
:meth:Holder.open which uses "rb+". Other keyword
arguments (owns_holder, media_type, auto_open,
…) ride through to :meth:Holder.open.
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Drop the cached :class:IOStats and pop self from
_INSTANCES. Single canonical invalidator for paths.
Mutating ops (writes, deletes) call this so the next read on
the same logical path picks up fresh state. Subclasses with
their own caches (schema, table info, column lists) override
and chain super().invalidate_singleton(...).
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
Coerce obj (str / URL / pathlib / Path) into a :class:Path.
When called on the abstract :class:Path, dispatches via the
:class:Holder scheme registry to the subclass registered for
the URL's scheme; defaults to :class:LocalPath for path-shaped
URLs without an explicit scheme. When called on a concrete
subclass, returns an instance of that subclass.
options_class
classmethod
¶
The :class:CastOptions subclass this implementer consumes.
Default :class:CastOptions. Format-specific leaves with
their own knobs (Parquet compression, CSV delimiter, …)
override.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
read_mv ¶
Slice size bytes from offset as a :class:memoryview.
cursor=True ignores the explicit offset and reads from
the holder's internal cursor (:attr:tell), advancing it past
the bytes returned. cursor=False (default) keeps the
cursor-less positional contract — the cursor is untouched.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
bounds-check uses its own size — avoids a redundant stat
probe on remote backings when the cursor has no local size
cache, and routes through any subclass _active override
(lazy materialization on :class:ZipEntryFile, …).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice data at offset, pre-growing the holder as needed.
size caps the byte count written — size=-1 (default)
writes all of data; size>=0 writes
min(len(data), size) bytes. Caps via a slice of
data (zero-copy on memoryview / bytes), so
downstream pipelines that only need the first N bytes of
a larger buffer skip the trailing tail.
overwrite declares that this write replaces the
holder's tail past offset + size — after the splice,
:attr:size is set to offset + size. Callers that
currently do truncate(0) followed by write_bytes(...)
collapse to a single write_bytes(..., overwrite=True),
which on whole-blob remote backends saves a SDK round
trip (the atomic upload at offset == 0 already
replaces the object — no preceding truncate needed).
Pipeline:
- Slice
datatosizeif capped. - Normalize
offset(-1→ append,-N→self.size - N). - Pre-grow visible :attr:
sizeto cover the splice via :meth:resize. - Hand the normalized
(data, offset)to :meth:_write_mv. - Truncate tail past
offset + nwhenoverwrite. - Mark dirty + bump cached mtime if anything was written.
update_stat=False skips the post-write
:meth:_touch_stat and :meth:mark_dirty calls. Use it for
bulk loops that want a single stat refresh at the end (one
:func:time.time call instead of one per write); the caller
is then responsible for calling :meth:_touch_stat (or
re-statting via the path-side _stat for filesystem
backends) once the loop finishes.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
resize / bounds-check / dirty-marking fires once, on the
backing storage — the cursor only advances its own _pos.
resize ¶
Grow visible :attr:size to at least n bytes (one-way).
Sister of :meth:truncate, but never shrinks. Used by
:meth:write_mv to pre-allocate a known target before the
splice so :meth:_write_mv doesn't have to manage size.
n <= size→ no-op, returns current :attr:size.n > size→ extends with zero-padding via :meth:truncate, returnsn.
Subclasses with a native grow-only primitive (capacity hint to
a remote upload session, posix_fallocate on local fd)
override for the cheaper path; the default works on every
backend.
truncate ¶
Resize to exactly n bytes via read-modify-write.
The generic whole-file fallback: download, slice / zero-extend,
re-write. Backends with a native resize (LocalPath
os.ftruncate, RemotePath's single upload) override this.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
flush ¶
Push buffered writes to the durable backing.
Cursor IOs forward the flush to their bound parent; storage
IOs go through :meth:Disposable.commit (default no-op
unless a subclass overrides).
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
as_media ¶
Wrap this path in the format leaf for its media type.
.. deprecated::
Use :meth:open with a media_type instead —
path.open(media_type=...) already dispatches to the
right format leaf and gives a properly acquired cursor with
lifecycle handling. as_media returns an un-acquired leaf
and is kept only for callers that haven't migrated.
Resolution: explicit media_type first, else the holder's
:class:MediaType (path extension, magic-byte sniff, or
content-type header). The resolved class is looked up in the
:class:Holder format registry and instantiated bound to this
path.
Raises :class:KeyError when the path's media type isn't
registered as a tabular format.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
ls ¶
ls(
*,
recursive: bool = False,
limit: "int | None" = None,
singleton_ttl: Any = False
) -> Iterator["Path"]
Yield children lazily. limit caps how many are produced — the
underlying listing stays incremental, so a bounded ls over a huge
prefix never materialises (or fetches) more than it needs.
unlink ¶
Remove the leaf — pathlib-compatible: refuses directories.
Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises
:class:IsADirectoryError for directories so callers don't
accidentally recursive-delete via unlink. Use :meth:remove
for the directory case. Thin wrapper over :meth:_delete's
path-removal mode.
remove ¶
remove(
recursive: bool = True,
missing_ok: bool = True,
wait: WaitingConfigArg = True,
fresher_than: Optional[TimeLike] = None,
older_than: Optional[TimeLike] = None,
) -> "Path"
Remove this path — the file, or the whole subtree when recursive.
Thin wrapper over :meth:_delete's path-removal mode (the single
deletion primitive). fresher_than / older_than scope the
removal to children inside that mtime window.
wait_until_gone ¶
Block until :meth:exists reports False or wait expires.
Polls the backend with a fresh probe each iteration — the
stat cache is invalidated between checks so a TTL'd hit
can't mask a deletion that landed after the cache was
filled. Useful when a fire-and-forget unlink (e.g.
WarehouseStatementBatch.clear_temporary_resources) means
the caller can't observe completion through the original
operation's return value.
Raises :class:TimeoutError when wait's deadline elapses
and the path is still present.
touch ¶
Create the path as an empty file if it doesn't exist.
write_bytes(b"") short-circuits in the holder fast path
(zero bytes, no flush), which would leave a missing file behind
— open + close around the empty write so the holder actually
materialises the entry on the backing store.
upload_module ¶
Zip a local module / package and write it under this path.
module is anything :func:resolve_module_root accepts —
an importable module name ("yggdrasil.io"), a
:class:os.PathLike pointing at a package directory or an
existing .zip / .whl archive, or a callable
carrying a __module__ attribute. The module is packed
into a deflated zip whose top-level entry is the package
directory itself, so the archive can be added to
sys.path directly (or fed to
:meth:SparkSession.addArtifacts with pyfile=True).
Destination shape on self:
- self names a file with a
.zip/.whlsuffix — archive bytes land at that exact path. - self is anything else — archive lands at
self / <name or "<module>.zip">.
Returns the concrete :class:Path that now holds the
archive. overwrite=False raises
:class:FileExistsError when the destination already
exists.
import_module ¶
import_module(
module_name: str | None = None,
*,
install: bool = True,
cache_dir: "Any" = None
) -> Any
Download a module archive at this path and import it.
Inverse of :meth:upload_module: fetch the archive bytes
at self, drop them on local disk, prepend the archive (or
its extracted parent) to :data:sys.path, and return the
live module via :func:importlib.import_module.
module_name defaults to the archive's stem (filename
minus suffix). cache_dir picks where the archive lands
locally (default: a fresh
:meth:LocalPath.staging_path-style directory).
install=True (the default) preserves the archive on
disk so subsequent imports in the same process hit the
cache. install=False makes the cache-dir lifetime the
caller's problem.
Folder ¶
Folder(
data: Any = None,
*,
path: Any = None,
tabular_parent: "Tabular | None" = None,
yggmeta: bool = True,
**kwargs: Any
)
Bases: Path
:class:Tabular over a directory of tabular files.
Inherits :class:Holder so it registers in the cross-cutting
media-type registry alongside byte-backed leaves. The byte
primitives raise :class:NotImplementedError — a folder is a
logical container, not a positional buffer; navigate via
:meth:iter_children / :meth:read_arrow_batches instead.
:class:Singleton-cached with a 15-minute TTL, keyed on the
backing URL string + the yggmeta flag (see
:meth:_singleton_key). Every Folder(path=p) and every
recursive partition-candidate probe with the same URL collapses
to one live instance, so the on-instance schema cache,
free_columns memo, and Hive static_values cache stay
warm across the whole send_many cache-scan loop without
leaking across long-running workers.
Bind to a folder path. No I/O.
data and path accept the same shape; path wins
when both are supplied. tabular_parent rides through to
the :class:Tabular slot — set by the enclosing folder when
it yields this one as a child.
Partition KV doesn't need a constructor seed: a Hive-shaped
sub-folder at <base>/partition_key=42/ reports
{"partition_key": 42} via :attr:static_values directly
from the bound :attr:path URL — see :attr:static_values
for the parsing contract.
yggmeta (default True) enables the dot-prefixed
sidecar at <path>/.ygg/. :meth:_persist_schema (the
Tabular write hook) dumps the schema there so
:meth:_collect_schema returns the typed :class:Schema
(with every :class:Field tag — partition_by,
cluster_by, primary keys, …) on the next read without
re-opening a data leaf. Disable for ad-hoc folders where the
sidecar would be noise (the sidecar folder itself constructs
with yggmeta=False so we never recurse).
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
size_known
property
¶
True when reading :attr:size won't trigger a backend probe.
Always true for in-memory IOs (size is a slot). Path IOs
override to True only when their stat cache is warm —
callers that want to short-circuit on an empty buffer
(parquet / arrow IPC / CSV readers checking size == 0)
can guard the check on this predicate so a cold remote path
doesn't pay a HeadObject / get_status /
get_metadata round trip just to discover the file is
non-empty. Cursor / format-leaf IOs delegate to the parent.
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
static_values
property
¶
Partition KV parsed from the folder's bound :attr:path URL.
Walks the URL's segments (self.path.url.parts) and picks
every Hive-shaped <col>=<val> chunk — a folder at
<base>/year=2024/month=05/ reports
{"year": 2024, "month": 5} without any constructor seed.
Values are cast to their declared dtype when a parent
:class:Folder (or this folder itself) already has the
schema cached, otherwise the decoded string passes through;
the downstream :meth:matches_static prune is conservative
and just degrades to a row-level filter on undecidable
compares.
Schema lookup walks tabular_parent first so partition
children minted during a read reuse the root folder's
cached schema instead of paying a per-child sidecar load.
Only when no ancestor has the schema cached does the read
fall through to :meth:collect_schema's sidecar / first-
batch path.
The parsed mapping is memoised on the instance — URL parts
are immutable for the bound path and the schema only flips
when :meth:_persist_schema writes a new layout, which
invalidates the cache. Calling once per matches_static
per child (the prune hot path) costs one dict read after
the first invocation.
open ¶
Acquire the path and return an :class:IO cursor bound to it.
mode accepts a :class:Mode member, an alias string, or
a stdlib open() mode string. None falls through to
:meth:Holder.open which uses "rb+". Other keyword
arguments (owns_holder, media_type, auto_open,
…) ride through to :meth:Holder.open.
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Drop the cached :class:IOStats and pop self from
_INSTANCES. Single canonical invalidator for paths.
Mutating ops (writes, deletes) call this so the next read on
the same logical path picks up fresh state. Subclasses with
their own caches (schema, table info, column lists) override
and chain super().invalidate_singleton(...).
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
Coerce obj (str / URL / pathlib / Path) into a :class:Path.
When called on the abstract :class:Path, dispatches via the
:class:Holder scheme registry to the subclass registered for
the URL's scheme; defaults to :class:LocalPath for path-shaped
URLs without an explicit scheme. When called on a concrete
subclass, returns an instance of that subclass.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
read_mv ¶
Slice size bytes from offset as a :class:memoryview.
cursor=True ignores the explicit offset and reads from
the holder's internal cursor (:attr:tell), advancing it past
the bytes returned. cursor=False (default) keeps the
cursor-less positional contract — the cursor is untouched.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
bounds-check uses its own size — avoids a redundant stat
probe on remote backings when the cursor has no local size
cache, and routes through any subclass _active override
(lazy materialization on :class:ZipEntryFile, …).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice data at offset, pre-growing the holder as needed.
size caps the byte count written — size=-1 (default)
writes all of data; size>=0 writes
min(len(data), size) bytes. Caps via a slice of
data (zero-copy on memoryview / bytes), so
downstream pipelines that only need the first N bytes of
a larger buffer skip the trailing tail.
overwrite declares that this write replaces the
holder's tail past offset + size — after the splice,
:attr:size is set to offset + size. Callers that
currently do truncate(0) followed by write_bytes(...)
collapse to a single write_bytes(..., overwrite=True),
which on whole-blob remote backends saves a SDK round
trip (the atomic upload at offset == 0 already
replaces the object — no preceding truncate needed).
Pipeline:
- Slice
datatosizeif capped. - Normalize
offset(-1→ append,-N→self.size - N). - Pre-grow visible :attr:
sizeto cover the splice via :meth:resize. - Hand the normalized
(data, offset)to :meth:_write_mv. - Truncate tail past
offset + nwhenoverwrite. - Mark dirty + bump cached mtime if anything was written.
update_stat=False skips the post-write
:meth:_touch_stat and :meth:mark_dirty calls. Use it for
bulk loops that want a single stat refresh at the end (one
:func:time.time call instead of one per write); the caller
is then responsible for calling :meth:_touch_stat (or
re-statting via the path-side _stat for filesystem
backends) once the loop finishes.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
resize / bounds-check / dirty-marking fires once, on the
backing storage — the cursor only advances its own _pos.
resize ¶
Grow visible :attr:size to at least n bytes (one-way).
Sister of :meth:truncate, but never shrinks. Used by
:meth:write_mv to pre-allocate a known target before the
splice so :meth:_write_mv doesn't have to manage size.
n <= size→ no-op, returns current :attr:size.n > size→ extends with zero-padding via :meth:truncate, returnsn.
Subclasses with a native grow-only primitive (capacity hint to
a remote upload session, posix_fallocate on local fd)
override for the cheaper path; the default works on every
backend.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
flush ¶
Push buffered writes to the durable backing.
Cursor IOs forward the flush to their bound parent; storage
IOs go through :meth:Disposable.commit (default no-op
unless a subclass overrides).
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
as_media ¶
Wrap this path in the format leaf for its media type.
.. deprecated::
Use :meth:open with a media_type instead —
path.open(media_type=...) already dispatches to the
right format leaf and gives a properly acquired cursor with
lifecycle handling. as_media returns an un-acquired leaf
and is kept only for callers that haven't migrated.
Resolution: explicit media_type first, else the holder's
:class:MediaType (path extension, magic-byte sniff, or
content-type header). The resolved class is looked up in the
:class:Holder format registry and instantiated bound to this
path.
Raises :class:KeyError when the path's media type isn't
registered as a tabular format.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
ls ¶
ls(
*,
recursive: bool = False,
limit: "int | None" = None,
singleton_ttl: Any = False
) -> Iterator["Path"]
Yield children lazily. limit caps how many are produced — the
underlying listing stays incremental, so a bounded ls over a huge
prefix never materialises (or fetches) more than it needs.
unlink ¶
Remove the leaf — pathlib-compatible: refuses directories.
Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises
:class:IsADirectoryError for directories so callers don't
accidentally recursive-delete via unlink. Use :meth:remove
for the directory case. Thin wrapper over :meth:_delete's
path-removal mode.
remove ¶
remove(
recursive: bool = True,
missing_ok: bool = True,
wait: WaitingConfigArg = True,
fresher_than: Optional[TimeLike] = None,
older_than: Optional[TimeLike] = None,
) -> "Path"
Remove this path — the file, or the whole subtree when recursive.
Thin wrapper over :meth:_delete's path-removal mode (the single
deletion primitive). fresher_than / older_than scope the
removal to children inside that mtime window.
wait_until_gone ¶
Block until :meth:exists reports False or wait expires.
Polls the backend with a fresh probe each iteration — the
stat cache is invalidated between checks so a TTL'd hit
can't mask a deletion that landed after the cache was
filled. Useful when a fire-and-forget unlink (e.g.
WarehouseStatementBatch.clear_temporary_resources) means
the caller can't observe completion through the original
operation's return value.
Raises :class:TimeoutError when wait's deadline elapses
and the path is still present.
touch ¶
Create the path as an empty file if it doesn't exist.
write_bytes(b"") short-circuits in the holder fast path
(zero bytes, no flush), which would leave a missing file behind
— open + close around the empty write so the holder actually
materialises the entry on the backing store.
upload_module ¶
Zip a local module / package and write it under this path.
module is anything :func:resolve_module_root accepts —
an importable module name ("yggdrasil.io"), a
:class:os.PathLike pointing at a package directory or an
existing .zip / .whl archive, or a callable
carrying a __module__ attribute. The module is packed
into a deflated zip whose top-level entry is the package
directory itself, so the archive can be added to
sys.path directly (or fed to
:meth:SparkSession.addArtifacts with pyfile=True).
Destination shape on self:
- self names a file with a
.zip/.whlsuffix — archive bytes land at that exact path. - self is anything else — archive lands at
self / <name or "<module>.zip">.
Returns the concrete :class:Path that now holds the
archive. overwrite=False raises
:class:FileExistsError when the destination already
exists.
import_module ¶
import_module(
module_name: str | None = None,
*,
install: bool = True,
cache_dir: "Any" = None
) -> Any
Download a module archive at this path and import it.
Inverse of :meth:upload_module: fetch the archive bytes
at self, drop them on local disk, prepend the archive (or
its extracted parent) to :data:sys.path, and return the
live module via :func:importlib.import_module.
module_name defaults to the archive's stem (filename
minus suffix). cache_dir picks where the archive lands
locally (default: a fresh
:meth:LocalPath.staging_path-style directory).
install=True (the default) preserves the archive on
disk so subsequent imports in the same process hit the
cache. install=False makes the cache-dir lifetime the
caller's problem.
iter_children ¶
Yield every non-private direct entry of :attr:path (one level).
Sub-directories come back as a fresh :class:Folder; file entries
resolve through :meth:_leaf_for to a registered :class:Tabular
leaf (non-tabular files are skipped). A missing folder yields nothing.
Plain flat listing — the partition-aware, recursive walk lives in
:meth:iter_leaves (which the read path uses). options is accepted
for signature symmetry but isn't consulted here.
partition_columns ¶
partition_columns(
options: "FolderOptions | None" = None, *, schema: Any = None
) -> "tuple[str, ...]"
Partition column names declared on the folder's schema.
schema overrides the lookup when supplied (write path
passes the just-peeked batch schema). Otherwise the resolved
schema is :meth:collect_schema — which itself reads the
.ygg/schema.arrow sidecar when present and falls back to
the first-batch shape — with options.target as a final
fallback when even the sidecar / first batch is empty (a
fresh folder asked for its layout before any write).
Columns already pinned by :attr:static_values get filtered
out: a partition-key=v leaf would otherwise re-infer
partition_key from its own data and recurse forever into
partition_key=v/partition_key=v/…. :class:Field parses
every tag off the Arrow schema, so the helper just walks
schema.children and picks the partition_by=True ones.
make_child ¶
Mint a fresh tabular leaf bound to a fresh path under :attr:path.
Filename shape: part-{epoch_ms}-{seed}.{ext} where ext
is options.child_media_type.full_extension. The
millisecond timestamp gives lexical-time ordering; a 2-byte
seed (~65k-value space) breaks ties between writes that land
in the same millisecond.
The :class:Tabular leaf is dispatched directly from the
media type via :meth:IO.class_for_media_type, so the
write path doesn't go through the path-extension reverse-
lookup. A media type with no registered leaf falls back to
a raw :class:IO so non-tabular extensions still get a
working write.
Returns a closed leaf. Caller opens it inside a with
block to write bytes.
iter_leaves ¶
Recursively yield the surviving leaf data files under this folder.
Walks the tree itself — a non-recursive :meth:Path.ls at each level —
skipping private entries (dot-prefixed) and empty (0-byte) files, and
recursing into sub-directories the predicate doesn't prune on their
path-level Hive :attr:static_values. Opens no data file: a pure
listing + path-prune, reusable as a file-enumeration pass. The residual
row-level filter stays in :meth:_read_arrow_batches.
Fast get: when the predicate pins this level's partition column to a
finite value set (col == v / col in [...]), the matching
<col>=<val>/ sub-paths are probed directly — a handful of stats
instead of listing (and pruning) every sibling partition.
optimize ¶
optimize(
byte_size: "int | None" = None,
*,
target_media_type: "MediaType | str | Any" = MediaTypes.ARROW_IPC,
tolerance: float = OPTIMIZE_TOLERANCE,
**kwargs: Any
) -> int
Compact small part files into byte_size-shaped bundles.
Walks the tree under :attr:path and at every directory that
holds part files, groups them by combined size and rewrites
each group as a single fresh part. Two flavors of the pass:
byte_size=None(the default and the shape the local-cache compaction loop in :class:Sessioncalls with) — collapses every directory with more than one part into a single file.byte_size=N— first-fit-decreasing bin pack into bins of capacityNbytes. Parts whose size is within±toleranceof N (or already larger) are skipped: they are already "close enough" and rewriting them would just burn IO.
target_media_type (a :class:MediaType or anything
:meth:MediaType.from_ accepts) selects the format the
rewritten parts are encoded in. Defaults to Arrow IPC.
Returns the number of new part files created. Idempotent: a
second call on a tree that's already at target leaves
nothing to do and returns 0.
Memory ¶
Memory(
data: Any = None,
*,
spill_bytes: Any = None,
spill_dir: Optional[str] = None,
spill_path: Optional[Any] = None,
**kwargs
)
Bases: IO
Fully memory-resident byte holder, with optional mmap auto-spill.
Construction shapes (in addition to those inherited from
:class:Holder):
Memory()— empty, zero capacity.Memory(int n)— empty, capacitynreserved.Memory(other_mem)— deep copy of another Memory.
Plus three keyword-only knobs that turn on the spill path:
spill_bytes— threshold in bytes. Once the underlying capacity would exceed this value, the bytearray is migrated to an mmap-backed temp file and dropped. Accepts an int, a :class:yggdrasil.enums.byteunit.ByteUnitmember, or a size string ("128 MB"). Defaults toNone(no spill — pure bytearray mode, the legacy behavior).spill_dir— directory the spill file is created in. Defaults to :func:tempfile.gettempdir. Ignored whenspill_pathis supplied.spill_path— explicit path to use as the spill file. When set, the Memory writes into that exact location and treats it as caller-owned: :meth:clearand close release the mapping but do not unlink the file. Use this to pin spill to a caller-controlled scratch area (tmpfs,/local_disk0, …).
Visible :attr:size and underlying buffer capacity are tracked
separately. reserve(n) grows capacity without moving
:attr:size; :meth:truncate moves :attr:size, zero-padding on
extend.
Implements the five :class:Holder primitives — :meth:_read_mv,
:meth:_write_mv, :meth:reserve, :meth:truncate, :meth:clear
— plus :attr:size and the lazy :meth:stat. Everything else
(positional normalization, bounds checking, pre-grow via
:meth:resize, mark_dirty, bytes/text convenience,
append-at-end pos = -1 semantics) comes from the base class.
Lifecycle / closed-state access¶
A :class:Memory is usable both acquired (inside a with
block or after :meth:acquire) and closed. The choice between
the two is the caller's, and either is correct:
- Contextual opening (preferred for hot paths): wrap usage in
with mem.open() as bio:(orwith mem:). This pins the backing, lets :class:IOaccumulate writes in a single cursor, and gives the spill path a consistent acquire/release window. This is the performant pattern for a sequence of reads/writes. - Direct access on a closed holder: all of :meth:
read_bytes, :meth:write_bytes, :meth:pread, :meth:pwrite, :meth:memoryview, :meth:to_bytes, etc. are still callable against a closed Memory — :class:Memory's acquire/release are no-ops by design (the bytes survive a :meth:close). This is explicitly supported even though it skips the cursor's buffering: each call re-enters the holder cold. Use it for one-shot accesses or when integrating with code that doesn't manage a context.
In short: closed-state access is allowed and correct; contextual opening is the performant path. The choice is the caller's, not the holder's, even if direct access is less performant.
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
size_known
property
¶
True when reading :attr:size won't trigger a backend probe.
Always true for in-memory IOs (size is a slot). Path IOs
override to True only when their stat cache is warm —
callers that want to short-circuit on an empty buffer
(parquet / arrow IPC / CSV readers checking size == 0)
can guard the check on this predicate so a cold remote path
doesn't pay a HeadObject / get_status /
get_metadata round trip just to discover the file is
non-empty. Cursor / format-leaf IOs delegate to the parent.
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
capacity
property
¶
Current allocated capacity (length of the underlying buffer).
For a spilled holder this is the size of the mapped file, which
is always at least :data:_MIN_MMAP_BYTES even when visible
:attr:size is zero — POSIX mmap of an empty file is invalid.
open ¶
open(
mode: ModeLike = "rb+",
*,
media_type: "MediaType | None" = None,
owns_holder: bool = False,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Acquire the IO and return a fresh :class:IO cursor over it.
Dispatches to the format-specific :class:IO leaf via the
IO's stamped media type (or media_type override), so
LocalPath("data.parquet").open() lands on
:class:ParquetFile, LocalPath("data.csv").open() on
:class:CSVFile, and an unknown / no-media holder falls back
to a plain :class:IO.
Pattern::
with LocalPath("/tmp/x.bin").open("wb") as bio:
bio.write(b"hello")
# path released here.
with LocalPath("data.parquet").open() as bio:
table = bio.read_arrow_table() # Tabular surface
# path released here.
The default owns_holder=False returns a non-owning
cursor — closing the cursor leaves the parent open, so the
caller can mint multiple cursors against the same parent.
Pass owns_holder=True to transfer close-ownership of the
parent to the cursor (the cursor's close then also closes
the parent).
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Pop self from the per-class _INSTANCES cache.
Mutating ops on a Singleton-cached object (writes, deletes,
schema invalidations on a Databricks table, put_object on
an :class:S3Path) want to make sure the next caller asking
for the same key gets a fresh build rather than collapsing
onto this stale handle — that's what remove_global=True
(the default) does. The pop is :meth:identity-guarded:
only an entry that still points at self is removed, so
a concurrent re-construction that already raced past this
thread is left alone.
remove_global=False is a no-op. The keyword exists so
subclass invalidators (invalidate_singleton,
_invalidate_entity_tag_cache, …) can offer the same
switch without branching at the call site.
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
Auto-route obj to the right storage / cursor, return an owning IO.
Two shapes share the method:
- Storage subclasses (
clshas a :attr:scheme— :class:IOitself, :class:Memory, :class:LocalPath, remote paths). The result is a storage IO that owns its bytes —IO.from_(b"x")→ :class:Memory,IO.from_("file://...")→ :class:LocalPath. - Cursor / format-leaf subclasses (
clshas noscheme— :class:IO, :class:ParquetFile, :class:CSVFile, …). The result is an owning cursor over a fresh storage parent built from obj.
Recognised input shapes:
- :class:
IOofcls— pass through (idempotent). - :class:
IOof a different class — for storagecls, return the underlying parent; for cursorcls, borrow the same parent into a fresh cursor. - bytes-like (
bytes/bytearray/memoryview) — back with a fresh :class:Memory. - path-like (
str/pathlib.Path/URL) — back with the path-shaped storage class for the scheme. - local file handle — back with :class:
LocalPath; lazy read from disk (no drain). - other file-like — drain into a fresh :class:
MemoryStream.
options_class
classmethod
¶
The :class:CastOptions subclass this implementer consumes.
Default :class:CastOptions. Format-specific leaves with
their own knobs (Parquet compression, CSV delimiter, …)
override.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
read_mv ¶
Slice size bytes from offset as a :class:memoryview.
cursor=True ignores the explicit offset and reads from
the holder's internal cursor (:attr:tell), advancing it past
the bytes returned. cursor=False (default) keeps the
cursor-less positional contract — the cursor is untouched.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
bounds-check uses its own size — avoids a redundant stat
probe on remote backings when the cursor has no local size
cache, and routes through any subclass _active override
(lazy materialization on :class:ZipEntryFile, …).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice data at offset, pre-growing the holder as needed.
size caps the byte count written — size=-1 (default)
writes all of data; size>=0 writes
min(len(data), size) bytes. Caps via a slice of
data (zero-copy on memoryview / bytes), so
downstream pipelines that only need the first N bytes of
a larger buffer skip the trailing tail.
overwrite declares that this write replaces the
holder's tail past offset + size — after the splice,
:attr:size is set to offset + size. Callers that
currently do truncate(0) followed by write_bytes(...)
collapse to a single write_bytes(..., overwrite=True),
which on whole-blob remote backends saves a SDK round
trip (the atomic upload at offset == 0 already
replaces the object — no preceding truncate needed).
Pipeline:
- Slice
datatosizeif capped. - Normalize
offset(-1→ append,-N→self.size - N). - Pre-grow visible :attr:
sizeto cover the splice via :meth:resize. - Hand the normalized
(data, offset)to :meth:_write_mv. - Truncate tail past
offset + nwhenoverwrite. - Mark dirty + bump cached mtime if anything was written.
update_stat=False skips the post-write
:meth:_touch_stat and :meth:mark_dirty calls. Use it for
bulk loops that want a single stat refresh at the end (one
:func:time.time call instead of one per write); the caller
is then responsible for calling :meth:_touch_stat (or
re-statting via the path-side _stat for filesystem
backends) once the loop finishes.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
resize / bounds-check / dirty-marking fires once, on the
backing storage — the cursor only advances its own _pos.
resize ¶
Grow visible :attr:size to at least n bytes (one-way).
Sister of :meth:truncate, but never shrinks. Used by
:meth:write_mv to pre-allocate a known target before the
splice so :meth:_write_mv doesn't have to manage size.
n <= size→ no-op, returns current :attr:size.n > size→ extends with zero-padding via :meth:truncate, returnsn.
Subclasses with a native grow-only primitive (capacity hint to
a remote upload session, posix_fallocate on local fd)
override for the cheaper path; the default works on every
backend.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
flush ¶
Push buffered writes to the durable backing.
Cursor IOs forward the flush to their bound parent; storage
IOs go through :meth:Disposable.commit (default no-op
unless a subclass overrides).
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
as_media ¶
Return a typed Tabular leaf bound to this buffer's holder.
.. deprecated::
Use :meth:open with a media_type instead —
holder.open(media_type=...) dispatches to the same
format leaf and returns an acquired cursor. as_media is
retained for callers that haven't migrated.
Resolution: explicit media_type wins; otherwise the buffer's
stamped media type is used. The leaf borrows the same backing
storage so durable bytes are shared without a copy. When
self is already an instance of the resolved leaf class,
returns self unchanged.
Raises :class:KeyError when no media type can be resolved or
the resolved type has no registered Tabular leaf.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
from_bytearray
classmethod
¶
Construct a :class:Memory aliasing an existing bytearray.
Zero-copy: buf is shared with the returned Memory.
Mutations through :meth:write_mv / :meth:truncate /
:meth:reserve propagate to buf directly. Closing the
returned Memory does not free buf.
Used by :class:IO as the in-memory _owner: the
same bytearray IO mutates directly is exposed through
the :class:Holder interface for backend-agnostic callers.
view-constructed instances cannot spill: the whole point
of an alias is that buf stays the canonical storage.
memoryview ¶
Memoryview over the visible payload (size-bounded).
Override of :meth:Holder.memoryview that aliases the
underlying buffer directly — no copy, no per-byte dispatch.
Works equally on bytearray and mmap backings.
MemoryStream ¶
MemoryStream(
source: SourceLike = None,
*,
content_encoding: Optional[str] = None,
byte_size: int = _DEFAULT_BYTE_SIZE,
spill_threshold: int = _DEFAULT_SPILL_THRESHOLD,
pull_chunk: Optional[int] = None,
**kwargs: Any
)
Bases: Holder
Sliding-window streaming holder with optional spill-to-disk.
Construction::
MemoryStream(
source,
*,
byte_size=2 GiB,
spill_threshold=128 MiB,
pull_chunk=64 KiB,
)
source is the upstream feed (see module docstring for accepted
shapes). byte_size caps total retained bytes (memory + spill);
when retention would exceed it, the oldest bytes are evicted.
spill_threshold caps the in-memory live window; cold bytes
above this go to a tempfile and stay readable until evicted.
pull_chunk is the default size of each pull from the source.
When byte_size <= spill_threshold the spill file is never
created — the holder collapses to a pure in-memory eviction loop,
matching the original single-window shape so small-budget
consumers don't pay tempfile setup cost.
Implements the :class:Holder primitives with absolute-offset
semantics: :attr:size is the highest offset the stream has
reached so far. Reads valid in [spill_start, size); writes
valid in [window_start, size].
window_start
property
¶
Absolute offset of the first byte in the in-memory window.
window_end
property
¶
Absolute offset one past the last byte in the in-memory window
(== :attr:size).
spill_start
property
¶
Absolute offset of the first retained byte.
Equal to :attr:window_start when no spill is active.
Otherwise it sits at the start of the spill region; bytes
before this have been evicted and reads behind it raise.
is_streaming
property
¶
True while EOF hasn't been reached on the source feed.
:attr:size reflects only the bytes pulled so far, so
cursor-anchored readers (:class:IO.read) need to
bypass the standard cap at size clamp until the
source signals EOF.
size_known
property
¶
False while the source can still grow :attr:size.
The base :class:Holder reports True for in-memory holders
because their size is a settled slot. A streaming holder is
different: :attr:size reflects only the bytes pulled so far, so
before the first read it is 0 even though the body is
non-empty. Format leaves (parquet / arrow / csv / ndjson) guard
their read with if self.size_known and self.size == 0: return
as an empty-buffer short-circuit — with the base True that
fired on an un-pulled stream and silently returned zero rows.
Reporting the size as unknown until EOF makes the short-circuit
defer to the actual read (which drives the pull), so a
stream=True response body parses correctly.
read_mv ¶
Read size bytes at absolute offset, pulling from source
as needed. size < 0 reads to EOF.
cursor=True reads from the holder's internal cursor and
advances it past the returned bytes. Reads are valid in
[spill_start, size) — anything behind :attr:spill_start
has been evicted (truly dropped from both memory and spill)
and raises.
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice bytes at offset. Appends past current end extend the
stream and may slide the window; in-window writes overwrite.
cursor=True writes at the internal cursor and advances it.
size>=0 slices the input buffer to at most size
bytes before the splice. overwrite=True truncates the
tail past offset + len(data) after the splice (same
contract as :meth:Holder.write_mv). Writes behind
:attr:window_start raise — the target bytes have
already been evicted.
reserve ¶
Pre-grow the underlying bytearray, capped at :attr:byte_size.
Capacity beyond :attr:byte_size would be evicted on the very
next :meth:_slide_window call, so the cap is the honest
ceiling here.
truncate ¶
Set visible :attr:size to n. Shrinks drop the tail
(in-memory and spill); extends zero-pad in memory.
Truncating below :attr:spill_start raises — those bytes
are evicted and unrecoverable. A truncate that lands inside
the spill region drops the trailing spill bytes and the
whole in-memory window.
memoryview ¶
View over the live window only (not the full stream).
Override of :meth:Holder.memoryview — that one would call
self.read_mv(-1, 0), which raises once :attr:window_start
0 because position 0 is no longer in the window. The window view is the honest answer for a sliding-window holder.
RemotePath ¶
Bases: Path
Abstract :class:Holder for network-backed backends.
Subclasses pick a scheme (s3, dbfs, …), implement the
filesystem hooks plus :meth:_read_mv (ranged / whole-object GET)
and :meth:_upload (atomic whole-object PUT) against their network
client, and override :meth:_stat_uncached for the metadata probe.
Everything else (predicate pins, stat caching, whole-blob read /
write, optional streaming) is inherited from this base.
RemotePath leaves the :class:Singleton machinery
deactivated (_SINGLETON_TTL = ...), the same as
:class:Holder: a leaf object/file path is a cheap redirector
onto a long-lived backend resource (the :class:S3Bucket, the
:class:DatabricksService / client), so there's nothing
expensive to share — and not interning instances means two
callers asking for the same URL get independent stat caches, so a
delete or external mutation observed through one path is never
masked by a sibling's cached snapshot. Only the heavyweight
container resources that genuinely benefit from a shared,
rarely-changing identity — :class:S3Bucket, UC
:class:~yggdrasil.databricks.catalog.UCCatalog /
:class:~yggdrasil.databricks.schema.UCSchema /
:class:~yggdrasil.databricks.volume.Volume — opt back in by
setting their own _SINGLETON_TTL. iterdir-style hot loops
still pass singleton_ttl=False explicitly; a caller that wants
a specific leaf path interned can pass singleton_ttl=None or a
seconds count.
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
size_known
property
¶
True only when the stat cache carries a fresh entry.
Lets ParquetFile / CSVFile / ArrowIPCFile skip a probe
round trip just to short-circuit on size == 0: when the
cache is cold the format reader will trip its own EOF /
empty-file error which the caller catches and translates to
an empty schema. When the cache is warm the cheap size
read fires unchanged.
open ¶
Acquire the path and return an :class:IO cursor bound to it.
mode accepts a :class:Mode member, an alias string, or
a stdlib open() mode string. None falls through to
:meth:Holder.open which uses "rb+". Other keyword
arguments (owns_holder, media_type, auto_open,
…) ride through to :meth:Holder.open.
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
Coerce obj (str / URL / pathlib / Path) into a :class:Path.
When called on the abstract :class:Path, dispatches via the
:class:Holder scheme registry to the subclass registered for
the URL's scheme; defaults to :class:LocalPath for path-shaped
URLs without an explicit scheme. When called on a concrete
subclass, returns an instance of that subclass.
options_class
classmethod
¶
The :class:CastOptions subclass this implementer consumes.
Default :class:CastOptions. Format-specific leaves with
their own knobs (Parquet compression, CSV delimiter, …)
override.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
as_media ¶
Wrap this path in the format leaf for its media type.
.. deprecated::
Use :meth:open with a media_type instead —
path.open(media_type=...) already dispatches to the
right format leaf and gives a properly acquired cursor with
lifecycle handling. as_media returns an un-acquired leaf
and is kept only for callers that haven't migrated.
Resolution: explicit media_type first, else the holder's
:class:MediaType (path extension, magic-byte sniff, or
content-type header). The resolved class is looked up in the
:class:Holder format registry and instantiated bound to this
path.
Raises :class:KeyError when the path's media type isn't
registered as a tabular format.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
ls ¶
ls(
*,
recursive: bool = False,
limit: "int | None" = None,
singleton_ttl: Any = False
) -> Iterator["Path"]
Yield children lazily. limit caps how many are produced — the
underlying listing stays incremental, so a bounded ls over a huge
prefix never materialises (or fetches) more than it needs.
unlink ¶
Remove the leaf — pathlib-compatible: refuses directories.
Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises
:class:IsADirectoryError for directories so callers don't
accidentally recursive-delete via unlink. Use :meth:remove
for the directory case. Thin wrapper over :meth:_delete's
path-removal mode.
remove ¶
remove(
recursive: bool = True,
missing_ok: bool = True,
wait: WaitingConfigArg = True,
fresher_than: Optional[TimeLike] = None,
older_than: Optional[TimeLike] = None,
) -> "Path"
Remove this path — the file, or the whole subtree when recursive.
Thin wrapper over :meth:_delete's path-removal mode (the single
deletion primitive). fresher_than / older_than scope the
removal to children inside that mtime window.
wait_until_gone ¶
Block until :meth:exists reports False or wait expires.
Polls the backend with a fresh probe each iteration — the
stat cache is invalidated between checks so a TTL'd hit
can't mask a deletion that landed after the cache was
filled. Useful when a fire-and-forget unlink (e.g.
WarehouseStatementBatch.clear_temporary_resources) means
the caller can't observe completion through the original
operation's return value.
Raises :class:TimeoutError when wait's deadline elapses
and the path is still present.
touch ¶
Create the path as an empty file if it doesn't exist.
write_bytes(b"") short-circuits in the holder fast path
(zero bytes, no flush), which would leave a missing file behind
— open + close around the empty write so the holder actually
materialises the entry on the backing store.
upload_module ¶
Zip a local module / package and write it under this path.
module is anything :func:resolve_module_root accepts —
an importable module name ("yggdrasil.io"), a
:class:os.PathLike pointing at a package directory or an
existing .zip / .whl archive, or a callable
carrying a __module__ attribute. The module is packed
into a deflated zip whose top-level entry is the package
directory itself, so the archive can be added to
sys.path directly (or fed to
:meth:SparkSession.addArtifacts with pyfile=True).
Destination shape on self:
- self names a file with a
.zip/.whlsuffix — archive bytes land at that exact path. - self is anything else — archive lands at
self / <name or "<module>.zip">.
Returns the concrete :class:Path that now holds the
archive. overwrite=False raises
:class:FileExistsError when the destination already
exists.
import_module ¶
import_module(
module_name: str | None = None,
*,
install: bool = True,
cache_dir: "Any" = None
) -> Any
Download a module archive at this path and import it.
Inverse of :meth:upload_module: fetch the archive bytes
at self, drop them on local disk, prepend the archive (or
its extracted parent) to :data:sys.path, and return the
live module via :func:importlib.import_module.
module_name defaults to the archive's stem (filename
minus suffix). cache_dir picks where the archive lands
locally (default: a fresh
:meth:LocalPath.staging_path-style directory).
install=True (the default) preserves the archive on
disk so subsequent imports in the same process hit the
cache. install=False makes the cache-dir lifetime the
caller's problem.
invalidate_singleton ¶
Drop this path's cached :class:IOStats, schema, and
_INSTANCES entry — see :meth:Path.invalidate_singleton.
A mutation just ran, so the cached metadata is no longer
authoritative; the next read re-probes the backend. Discards any
un-flushed write scratch (callers must :meth:flush first to keep
pending writes).
resize ¶
No-op for remote-backend paths.
:class:Holder.resize would call :meth:truncate to pre-grow
a holder before a positional write. On remote backends every
truncate is a full-object upload, so the pre-grow would
double the network traffic for every write. The upload that
:meth:write_mv runs next will materialize the right size on
its own.
arrow_random_access_file ¶
Yield a pyarrow random-access file backed by ranged _read_mv.
Lets pyarrow readers seek and pull only the bytes they touch — a
Parquet column / row-group projection fetches the footer plus the
projected chunks, instead of snapshotting the whole object the
way :meth:arrow_input_stream does. :class:ParquetFile reaches
for this when a projection is bound and the backend advertises
:attr:SUPPORTS_RANGED_RANDOM_ACCESS (S3, Volumes); a full read
still snapshots. Generic over any holder via _read_mv +
size.
read_byte_range ¶
Read exactly length bytes from offset — a ranged backend fetch.
The explicit byte-range surface for tabular / format readers that
want a specific window (a Parquet footer, an Arrow IPC block) without
snapshotting the whole object. Works whether the holder is opened or
not: an in-flight write scratch is served from disk, otherwise the
subclass :meth:_read_mv issues a ranged GET on backends that
support it. length < 0 reads to EOF.
An explicit non-negative window goes straight to :meth:_read_mv —
no self.size (HEAD) bounds probe, so a footer fetch is a single
ranged GET. A short read near EOF is the caller's to interpret.
read_mv ¶
Read — served from the acquired write scratch when one is in flight, otherwise straight through to the backend.
Inside an acquired window with un-flushed writes the scratch file is
the authoritative view (read-after-write within the handle); without
a scratch this is the base :class:Holder read over the subclass
:meth:_read_mv (ranged where the backend supports it).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Whole-blob write — direct upload when closed, disk-paged when open.
- Closed (un-acquired). A whole-object overwrite from the start
(
offset == 0,overwrite, no cursor; whatwrite_bytes(...)resolves to) is a single :meth:_upload, no stat probe, no read-modify-write — the atomic PUT replaces the object. Positional / partial writes defer to the base :class:Holdersplice (download, splice, re-upload via :meth:_write_mv). - Open (acquired —
with path:/path.open("wb")). The write splices into a temp-file scratch (paging through the OS cache, not piling up in memory); :meth:flush/ release streams the scratch to the backend in one upload.
write_arrow_io ¶
Commit an Arrow-encoded payload directly to the backend.
Accepts a pa.Buffer, bytes, bytearray, or
memoryview and uploads it in one backend call — no
truncate, no stat probe. Tabular IO files (ParquetFile,
ArrowIPCFile, etc.) route through this after the format encoder
finishes so the encoded bytes go straight to the remote object
without intermediate copies. Whole-object replace: any in-flight
write scratch is superseded.
truncate ¶
Resize the object to exactly n bytes.
- Acquired (the
open("wb")truncate-on-acquire, and explicit truncates inside awith): resize the disk scratch; the commit happens once on :meth:flush. An emptytruncate(0)therefore costs no PUT until release — andopen("wb")immediately followed by a write coalesces to a single upload. - Closed: a whole-object upload —
truncate(0)PUTs an empty object;truncate(n > 0)downloads, slices / zero-extends, re-uploads.
flush ¶
Commit the acquired write scratch to the backend in one upload.
The single (streamed) PUT that an open("wb") window produces —
every write() since acquire spliced into the disk scratch, and
this drains it. The scratch streams off disk (bounded memory) on
backends that support it; others read it back for the SDK's
whole-object upload. A no-op when nothing was buffered.
with path.open("wb"): pass still materialises an empty object
(the acquire-time truncate(0) seeded an empty scratch).
LocalPath ¶
Bases: Path
File-backed byte holder for the local filesystem.
Construction shapes (in addition to those inherited from
:class:Path):
LocalPath("/tmp/foo.bin")— bind to a path string.LocalPath(pathlib.Path(...))— bind to apathlibpath.LocalPath(url=URL("file:///…"))— bind to afile://URL.
The fd is opened lazily on the first :meth:open /
with-block / acquire call. Until then the holder is a
URL with a side of os.stat — :meth:stat works; read /
write do not.
closed
property
¶
Stdlib IO[bytes] parity — False while the bound
backing is reachable.
Stdlib semantics: closed means "file unusable for I/O."
On a cursor the predicate flips only when teardown has dropped
the parent reference; on a storage IO it always reads
False (the storage owns its own bytes). Matters for
pyarrow / pandas / polars / zipfile, which guard every op
with an assert not closed.
parent
property
¶
The IO one level up — cursor parent first, else URL parent.
Resolution order:
- The cursor parent (
self._parent, set by :meth:IO.openand by format-leaf construction withparent=/holder=). When set, this IO is a cursor and the parent is its backing storage. - The URL parent — a sibling IO of the same concrete
class at
self.url.parent. Used by URL-shaped storage leaves (:class:Path/ :class:LocalPath/ remote paths) to walk up the filesystem.
Returns None when neither applies (top-level storage
with no URL hierarchy — e.g., :class:Memory, which
overrides :meth:_url_parent to skip the URL branch).
parents
property
¶
Walk the parent chain outward, yielding one IO per step.
Each step follows :attr:parent — cursor parent first, then
URL parent (when applicable), terminating when .parent
returns None. Empty on top-level non-URL storage
(:class:Memory).
size_known
property
¶
True when reading :attr:size won't trigger a backend probe.
Always true for in-memory IOs (size is a slot). Path IOs
override to True only when their stat cache is warm —
callers that want to short-circuit on an empty buffer
(parquet / arrow IPC / CSV readers checking size == 0)
can guard the check on this predicate so a cold remote path
doesn't pay a HeadObject / get_status /
get_metadata round trip just to discover the file is
non-empty. Cursor / format-leaf IOs delegate to the parent.
holder_is_overwrite
property
¶
True when the backing holder was opened in OVERWRITE mode.
Primitives use this to skip append checks: the holder was already truncated so there is no existing data to merge with.
media_type
property
writable
¶
The holder's :class:MediaType, or None if unset.
Resolves lazily on first read: a fresh holder bound only by URL
carries the sentinel ... in :attr:_media_type and runs
:meth:URL.infer_media_type here once, caching the result back
onto the slot. Subsequent reads (and pickling, IOStats
snapshots, codec dispatch, …) hit the cached value.
Cursor IOs (those wrapping a :attr:parent storage) defer to
the parent's stamped media type when their own slot is unset
— the codec / format dispatch on a :class:JSONFile bound to
a gzip-stamped :class:Memory parent needs to see the parent's
media type, not its own (the cursor was constructed bare).
is_streaming
property
¶
True when :attr:size reflects only the bytes pulled so far.
Streaming holders (:class:MemoryStream over a live
source) lazily pull bytes on read; their :attr:size
grows as the cursor advances and may underreport the
eventual total. Static holders (:class:Memory,
:class:Path) know their full size up front so the
default is False.
:class:IO.read checks this flag to decide whether to
cap the requested byte count at :attr:size (static
case — out-of-range reads would raise) or pass the
request through unclamped (streaming case — the holder
pulls until it has enough or EOF).
xxh3_64_digest
property
¶
8-byte big-endian payload digest — equivalent to
xxh3_64().digest() but served from the cached
:meth:xxh3_int64 so callers mixing the digest into a parent
hash don't re-walk the payload.
holder
property
¶
The bound parent IO (cursor case) or self (storage case).
Backwards-compatible alias preserved from the pre-merge
IO.holder property — call sites that drilled through a
cursor to reach its backing storage keep working.
mode
property
¶
The typed :class:Mode enum this buffer was opened with.
pandas / pyarrow / zipfile inspect .mode for substrings like
"b" to dispatch binary vs text reads; those sniffs still work
because :class:Mode implements __contains__ against its
:attr:~Mode.os_mode form ("b" in handle.mode → True).
Reach for self.mode.os_mode when an actual POSIX string is
required.
os_path
property
¶
The local filesystem path as a string. Alias for :meth:full_path.
Always returns the OS-native form — backslashes on Windows,
forward slashes elsewhere — matching :meth:pathlib.Path.__fspath__
and what tools that compare against str(pathlib.Path(...))
expect. The underlying URL.__fspath__ is POSIX-style by
contract; we apply :func:os.path.normpath here so callers
of LocalPath get the platform-canonical shape.
open ¶
Acquire the path and return an :class:IO cursor bound to it.
mode accepts a :class:Mode member, an alias string, or
a stdlib open() mode string. None falls through to
:meth:Holder.open which uses "rb+". Other keyword
arguments (owns_holder, media_type, auto_open,
…) ride through to :meth:Holder.open.
close ¶
Release the IO; on :attr:temporary, discard pending
writes instead of committing them.
On a cursor with owns_holder=True the bound parent is
closed too. Preserves the cursor position across the close
— a reopen on the same instance lands at the byte the
previous transaction left off.
for_scheme
classmethod
¶
Return the :class:URLBased subclass registered for scheme.
Lazy: if no subclass is registered yet, this routes through
:meth:Scheme.path_class which imports the backend module on
demand (firing :meth:__init_subclass__ as a side effect).
Raises :class:ValueError for an unknown scheme and
:class:ImportError when the backend's optional dependencies
aren't installed.
dispatch
classmethod
¶
Build the right :class:URLBased subclass from url.
Looks up the subclass via :meth:for_scheme, then delegates
to that subclass's :meth:from_url. Used as the cross-cutting
entry point when the caller has a URL but doesn't know (or
care) which concrete class owns its scheme.
URL.from_(url).scheme drives the lookup; an empty scheme
falls back to the file:// handler so bare paths work.
from_url
classmethod
¶
Create a new IO from a URL.
When cls is abstract (has subclasses but isn't itself
constructible — e.g. :class:Path), the URL scheme is
resolved through the :class:URLBased registry to a concrete
subclass; an unknown scheme raises :class:ValueError
instead of producing the obscure "Can't instantiate abstract
class" :class:TypeError.
to_singleton ¶
Promote this instance into the per-class _INSTANCES cache.
Hot listing paths (iterdir / _ls / glob) build
children with singleton_ttl=False so the bounded cache
doesn't fill up with thousands of short-lived entries. When a
caller decides one of those children is worth keeping around
(handing it to a long-running worker, returning it from an
API), :meth:to_singleton registers self into the cache
so the next constructor call with the same key collapses to
the same instance.
ttl defaults to the subclass's _SINGLETON_TTL
(... = no caching, None = process lifetime, or a
seconds count). When a different instance is already cached
under this key, that pre-existing one wins and is returned
unchanged — the cache is the source of truth.
invalidate_singleton ¶
Drop the cached :class:IOStats and pop self from
_INSTANCES. Single canonical invalidator for paths.
Mutating ops (writes, deletes) call this so the next read on
the same logical path picks up fresh state. Subclasses with
their own caches (schema, table info, column lists) override
and chain super().invalidate_singleton(...).
class_for_media_type
classmethod
¶
class_for_media_type(
media_type: "MediaType | MimeType | str | Any", *, default: Any = ...
) -> "type"
Resolve a :class:MediaType (or coercible) to its format leaf.
Looks up :attr:MediaType.mime_type's name in
:data:_HOLDER_FORMAT_REGISTRY. Codec is orthogonal — Parquet
compressed with zstd or snappy still resolves to
:class:ParquetFile; the codec layer is the holder's concern.
The returned class is a :class:Tabular subclass — typically a
:class:Holder byte-backed leaf, occasionally a non-Holder
leaf (:class:Folder, :class:DeltaFolder). Returns default
on miss when supplied; otherwise raises :class:KeyError with
the list of registered names.
matches_static ¶
True iff predicate could match any row given
:attr:static_values. Conservative on undecidables (column
not in static values, predicate evaluator failure) so the
caller still reads.
Builds a one-row pyarrow Table from the predicate's free columns that we have static values for, then evaluates the predicate against it — generalises the partition-only prune so any aggregator (folder read, future warehouse file skip) reuses the one helper.
free_cols lets a caller that's about to prune the same
predicate against N children precompute the free-column
tuple once and reuse it — :func:free_columns walks the
AST every call, so on a 64-OR predicate (the cache batch
lookup shape) the saving is N-1 full walks per
iter_children loop. Default None keeps the call
site short for one-off prune checks.
from_
classmethod
¶
Coerce obj (str / URL / pathlib / Path) into a :class:Path.
When called on the abstract :class:Path, dispatches via the
:class:Holder scheme registry to the subclass registered for
the URL's scheme; defaults to :class:LocalPath for path-shaped
URLs without an explicit scheme. When called on a concrete
subclass, returns an instance of that subclass.
options_class
classmethod
¶
The :class:CastOptions subclass this implementer consumes.
Default :class:CastOptions. Format-specific leaves with
their own knobs (Parquet compression, CSV delimiter, …)
override.
check_options
classmethod
¶
Validate and merge caller kwargs into a resolved options.
Canonical pattern: a public method passes overrides=locals()
and the ...-defaulted entries are stripped, the rest merged.
cleanup ¶
Garbage-collect stale state on this backend.
Default no-op (returns 0) — single-file leaves and
warehouse-backed tables don't have a sweep concept the
client owns. Folder-shaped subclasses override to unlink
stale part-* files, throttled by TTL.
wait controls sync vs async dispatch on backends that
support it: a truthy :class:yggdrasil.dataclasses.waiting.WaitingConfig
(or True / a positive timeout) blocks until the sweep
finishes; a falsy value (the default) hands the work off to a
background thread. Backends without an async path treat both
the same.
Returns the number of files / rows removed when known; 0
for fire-and-forget async dispatch or a no-op backend.
optimize ¶
Repartition / compact this Tabular's storage.
Default implementation is a no-op and returns 0 — single-file
leaves (parquet, csv, arrow IPC, …) don't have a compaction
concept. Aggregator subclasses (:class:Folder) override
this to walk their child leaves and bin-pack small part files
into bundles near byte_size.
Files already close to the target size are left alone so a
repeated call is cheap.
byte_size=None keeps the legacy "collapse every leaf with
more than one part into a single file" behavior, which is what
the local-cache compaction loop in :class:Session expects.
Any extra keyword arguments are accepted and ignored so
upstream callers can pass forward-compatible knobs without the
base raising.
delete ¶
delete(
predicate: "PredicateLike" = None,
*,
wait: "WaitingConfigArg" = True,
missing_ok: bool = False,
delete_staging: bool = True,
**kwargs: Any
) -> "Table"
Delete rows matching predicate; return this tabular.
predicate is a :class:Predicate from
:mod:yggdrasil.execution.expr or a SQL string that parses into
one ("id IN (1,2,3)", "price > 100 AND region = 'EU'").
None means "no filter" — every row is removed (DELETE FROM t
with no WHERE).
wait / missing_ok / delete_staging are honoured by
resource-backed subclasses (e.g.
:class:yggdrasil.databricks.table.table.Table, which drops the
table asset); the generic row-rewrite path ignores them. Any extra
**kwargs (e.g. options=DeltaOptions(...)) flow through to
:meth:_delete.
The default implementation reads every batch, drops rows the
predicate accepts, and rewrites the leaf with the survivors.
Aggregator subclasses (:class:yggdrasil.path.folder.Folder)
override to walk children, prune subtrees whose partition bounds
make the predicate trivially false, and only rewrite the leaves
that actually hold matched rows.
collect_schema ¶
Return this Tabular's :class:Schema, caching the first hit.
The cache slot is :attr:_schema_cache; on first call this
method stamps the resolved schema into it so subsequent
collect_schema calls short-circuit. Writers overwrite
the slot via :meth:_persist_schema; lifecycle hooks clear
it via :meth:_unpersist_schema.
count ¶
Return the number of rows in this tabular.
scan_arrow_batches ¶
Zero-copy scan — yield the source's :class:pa.RecordBatch views verbatim.
The lazy / zero-copy counterpart to :meth:read_arrow_batches,
mirroring :meth:read_polars_frame vs :meth:scan_polars_frame.
Where read_arrow_batches layers the full options pipeline on
every batch — target cast, projection, resample, dedup, row-limit
slicing, each of which can copy or re-encode — scan_arrow_batches
hands back exactly what the leaf produced, untouched. For an
in-memory source (:class:~yggdrasil.arrow.tabular.ArrowTabular)
those batches are views over the held buffers (no copy); for a
byte-backed leaf they're the freshly-decoded batches with none of
the extra processing copies layered on. Use it when you want the
raw Arrow stream and will project / filter downstream yourself.
scan_arrow_table ¶
Zero-copy scan into one chunked :class:pa.Table (no rechunk, no cast).
The zero-copy counterpart to :meth:read_arrow_table. Assembles
the source batches with :func:pa.Table.from_batches, which
references the batch buffers as table chunks rather than copying
them — so no cast, no projection, no rechunk memcpy that
read_arrow_table performs to coalesce + conform the result. An
empty source yields an empty table carrying the bound schema.
The batches must share one schema (the zero-copy contract):
read_arrow_table reconciles parts that drifted across writes,
scan_arrow_table does not — reach for read_arrow_table when
a source's parts are known to be heterogeneous.
scan_arrow_batch_reader ¶
Zero-copy scan as a streaming :class:pa.RecordBatchReader view.
The raw-reader counterpart to :meth:read_arrow_batch_reader: wraps
the source batch stream in a reader without the per-batch
conform / target-cast pass, so batches flow through as views over
the source buffers. The reader's schema is the source's own — taken
from the first batch, so it matches the raw views exactly (no
collect_schema probe, which on a byte cursor would consume the
stream out from under the read). Only the first batch is pulled up
front to seed the schema; the rest stay lazy behind the reader.
read_table ¶
Read into an in-memory :class:Tabular.
When options.spark_session is set, reads via
:meth:_read_spark_frame and wraps in a :class:Dataset.
Otherwise materializes Arrow batches into :class:ArrowTabular.
Returns None when empty.
write_table ¶
Dispatch obj to the best _write_* hook based on its runtime type.
Recognizes another :class:Tabular (drained as a pyarrow
record-batch stream), pa.Table / pa.RecordBatch /
pa.RecordBatchReader, polars DataFrame / LazyFrame,
pandas DataFrame, pyspark DataFrame, list[dict],
dict[str, list], and iterables of any of the above.
Module-name sniffing keeps optional engine deps out of the
import graph — we only touch a frame's API once we've
confirmed it's an instance of one we know how to drain.
union ¶
Return a Tabular representing self UNION ALL other.
mode controls how mismatched schemas are reconciled:
Mode.IGNORE(default) — keepself's schema; extra columns in other are dropped, missing ones are filled null.Mode.APPEND— widen to the superset schema (every field from both sides survives).
Concrete subclasses override :meth:_union for in-place
mutation (Arrow batch append, Spark unionByName).
Accepts :class:Tabular, pa.RecordBatch, pa.Table,
list[Response], or a Spark DataFrame.
None returns self unchanged.
read_spark_dataset ¶
Read into a :class:Dataset holder.
Mirrors :meth:read_arrow_dataset for the Spark engine: the
return type is a yggdrasil holder rather than the bare engine
frame, so callers keep the Tabular surface (chained transforms,
persist / insert / schema, …) without an extra wrap
at the call site. :class:Dataset overrides
:meth:_read_spark_dataset to return itself in place — no
materialise round trip when the source already speaks Spark.
read_record_iterator ¶
Stream rows as plain dict. True streaming — the full
table never materializes; batch.to_pylist() does the
column→row rotation in pyarrow C++ once per batch.
read_records ¶
Stream rows as :class:yggdrasil.data.record.Record. Lower
per-row allocation than :meth:read_pylist for stable-schema
sources — the underlying :class:Schema is materialized once
and shared by reference across every record.
unique ¶
Drop duplicate rows on by; keep first occurrence per key tuple.
Parameters¶
by
One or more column references — :class:str column names,
:class:yggdrasil.data.Field instances (resolved via
:attr:Field.name), or any iterable mixing the two. Empty
/ None is a no-op — returns self.
Returns¶
Tabular
A new holder carrying the deduped rows. Spark-shaped
inputs (anything whose :meth:_native_spark_frame
exposes a :class:pyspark.sql.DataFrame) return a fresh
:class:yggdrasil.spark.tabular.Dataset over the
spark-side dedup; everything else collects through Arrow
and returns an :class:yggdrasil.arrow.tabular.ArrowTabular.
resample ¶
resample(
on: "str | Any",
sampling: "int | float | Any",
*,
partition_by: "str | Any | Iterable[Any] | None" = None,
fill_strategy: "str | None" = "ffill"
) -> "Tabular"
Align rows to a fixed time grid on on; one row per bucket.
Parameters¶
on
The time column to resample on — column name
(:class:str) or :class:yggdrasil.data.Field.
sampling
Bucket size. Accepted shapes:
* :class:`int` / :class:`float` — seconds (floats are
rounded to the nearest integer second).
* :class:`datetime.timedelta` — total seconds.
* :class:`str` — ISO-8601 duration (``"PT1H"``,
``"P1D"``, ``"PT15M"``) parsed via
:func:`yggdrasil.data.types.primitive.temporal._parse_iso_duration`.
``sampling <= 0`` is a short-circuit — returns ``self``.
partition_by
Entity columns the resample is independent on. None /
empty → flat global timeline. Same coercion as
:meth:unique's by.
fill_strategy
How to fill nulls left by the bucket's "first" aggregation.
"ffill" (default), "bfill", or "none" /
None to disable. See
:func:yggdrasil.arrow.ops.fill_arrow_table for the
full semantics.
Returns¶
Tabular
Spark-shaped holders return a :class:Dataset over the
spark-side resample; everything else returns an
:class:ArrowTabular over the arrow-side resample.
select ¶
Project to columns and return a new Tabular.
Each entry is a column reference — :class:str, a
:class:yggdrasil.data.Field (resolved via
:attr:Field.name), or an iterable mixing both. The result
preserves the caller's order, which matches both
:meth:pyarrow.Table.select and
:meth:pyspark.sql.DataFrame.select semantics.
Raises :class:ValueError on an empty selection — a zero-
column projection is almost always a caller mistake; pass
:class:Schema.empty projections through the cast surface
instead.
drop ¶
Return a new Tabular with the named columns removed.
Columns missing from the source are silently ignored —
matches Spark's :meth:DataFrame.drop and pyarrow's
:meth:Table.drop_columns (when filtered to existing
names). An empty argument list is a no-op that returns
self.
filter ¶
Drop rows where predicate is false.
predicate accepts every shape
:meth:yggdrasil.execution.expr.Expression.from_
recognises:
- a SQL predicate string (
"x > 0 AND y IS NOT NULL"), parsed by the in-tree SQL parser; - a yggdrasil :class:
Predicatenode (col("x") > 0, :func:is_in, :func:between, …); - a native engine expression —
:class:
pyarrow.compute.Expression, :class:polars.Expr, or :class:pyspark.sql.Column— lifted via the matching backend.
The predicate is parsed once and dispatched to the typed
:meth:_filter hook; the engine-side filter then runs in
its native kernel (Arrow C++, Spark Catalyst) so the row
scan stays vectorised.
cast ¶
Cast rows, returning a new :class:Tabular.
Accepts a :class:Schema or :class:CastOptions. When
options is given, reads to arrow and casts each batch
through :meth:CastOptions.cast_arrow_batch.
display ¶
Render the first n rows as an aligned, typed text table.
Columns and their types come from this Tabular's own
:meth:collect_schema — the header is two rows: the column names,
then their type tags (the project :class:~yggdrasil.data.Field's
:meth:Field.short → :meth:DataType.short, recursive for nested types
— i64 / str / list<str> / struct<name:str, age:i64>).
Columns are separated by │ with a ─┼─ rule; numbers/booleans
right-align; nested cell values are compacted to one line. Long values
and headers are clipped (cells to max_width, type/name tags to a
slightly larger cap) so one long string or column name can't balloon the
table. The n rows are pushed down as a row_limit so no more than
that is ever read.
print(dbc.sql.execute("SELECT * FROM t").display())
print(IO.from_("data.parquet").display(5))
lazy ¶
Return a :class:LazyTabular wrapping this source.
Transformations on the returned object (select, filter,
join, …) accumulate in an :class:ExecutionPlan without
touching data. Any read_* call materialises the plan.
joinpath ¶
Build a sibling IO at self.url joined with segments.
URL-shaped IOs (:class:LocalPath, remote paths) use
this to mint a child path; :class:Memory and other
non-URL leaves raise :class:ValueError.
from_holder
classmethod
¶
from_holder(
holder: "IO",
*,
owns_holder: bool = False,
mode: ModeLike = "rb+",
media_type: Any = None,
auto_open: bool = True,
**kwargs: Any
) -> "IO"
Construct a cursor over holder, dispatching to the format leaf.
Resolves the format-specific :class:IO leaf via media_type
(when given) or the holder's stamped stat().media_type, and
returns an instance of that leaf bound to holder. When no
leaf can be resolved, falls back to cls itself.
With auto_open=True (the default) the returned cursor is
already acquired, so the caller can immediately read/write
without entering a with block. Set auto_open=False to
defer the acquire to the caller's with / :meth:acquire.
owns_holder=True hands close-ownership of holder to the
returned cursor — closing the cursor closes the holder. The
default False keeps the holder's lifetime in the caller's
hands; the returned cursor is a non-owning borrow.
for_holder
classmethod
¶
for_holder(
holder: "IO",
*,
media_type: "MediaType | MimeType | str | None" = None,
default: Any = ...,
**kwargs: Any
) -> "Tabular"
Build the right format leaf for holder.
Resolution order for the format discriminator:
- The explicit media_type kwarg, when supplied.
holder.stat().media_type— set by the holder from its URL extension, magic-byte sniff, or content-type header.
The resolved class is instantiated as Cls(holder=holder,
**kwargs). On lookup miss, falls back to default when
supplied; otherwise raises :class:KeyError.
registered_classes
classmethod
¶
Snapshot of the registry — debugging / introspection only.
read_mv ¶
Slice size bytes from offset as a :class:memoryview.
cursor=True ignores the explicit offset and reads from
the holder's internal cursor (:attr:tell), advancing it past
the bytes returned. cursor=False (default) keeps the
cursor-less positional contract — the cursor is untouched.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
bounds-check uses its own size — avoids a redundant stat
probe on remote backings when the cursor has no local size
cache, and routes through any subclass _active override
(lazy materialization on :class:ZipEntryFile, …).
write_mv ¶
write_mv(
data: memoryview,
offset: int = 0,
*,
size: int = -1,
overwrite: bool = False,
update_stat: bool = True,
cursor: bool = False
) -> int
Splice data at offset, pre-growing the holder as needed.
size caps the byte count written — size=-1 (default)
writes all of data; size>=0 writes
min(len(data), size) bytes. Caps via a slice of
data (zero-copy on memoryview / bytes), so
downstream pipelines that only need the first N bytes of
a larger buffer skip the trailing tail.
overwrite declares that this write replaces the
holder's tail past offset + size — after the splice,
:attr:size is set to offset + size. Callers that
currently do truncate(0) followed by write_bytes(...)
collapse to a single write_bytes(..., overwrite=True),
which on whole-blob remote backends saves a SDK round
trip (the atomic upload at offset == 0 already
replaces the object — no preceding truncate needed).
Pipeline:
- Slice
datatosizeif capped. - Normalize
offset(-1→ append,-N→self.size - N). - Pre-grow visible :attr:
sizeto cover the splice via :meth:resize. - Hand the normalized
(data, offset)to :meth:_write_mv. - Truncate tail past
offset + nwhenoverwrite. - Mark dirty + bump cached mtime if anything was written.
update_stat=False skips the post-write
:meth:_touch_stat and :meth:mark_dirty calls. Use it for
bulk loops that want a single stat refresh at the end (one
:func:time.time call instead of one per write); the caller
is then responsible for calling :meth:_touch_stat (or
re-statting via the path-side _stat for filesystem
backends) once the loop finishes.
Cursor IOs (those wrapping a :attr:parent storage) delegate
the whole call through :meth:_active so the parent's
resize / bounds-check / dirty-marking fires once, on the
backing storage — the cursor only advances its own _pos.
resize ¶
Grow visible :attr:size to at least n bytes (one-way).
Sister of :meth:truncate, but never shrinks. Used by
:meth:write_mv to pre-allocate a known target before the
splice so :meth:_write_mv doesn't have to manage size.
n <= size→ no-op, returns current :attr:size.n > size→ extends with zero-padding via :meth:truncate, returnsn.
Subclasses with a native grow-only primitive (capacity hint to
a remote upload session, posix_fallocate on local fd)
override for the cheaper path; the default works on every
backend.
clear ¶
Drop the IO's payload entirely.
:class:Memory resets the underlying bytearray to zero
bytes (capacity drops too). :class:yggdrasil.io.path.Path
unlinks the backing file with missing_ok=True so the
operation is idempotent. After :meth:clear, :attr:size
reads 0 and the IO is still usable — subsequent writes
grow it from scratch.
stat ¶
Snapshot the holder's metadata into a fresh :class:IOStats.
Delegates to :meth:_stat for the backend-specific fields
(kind and the live size for path-bound holders); mutating
the returned instance does NOT round-trip onto the holder.
Use the holder's own setters / :meth:_touch_stat when you
need to update metadata.
touch_mtime ¶
Stamp the holder's mtime with the current time.
Bulk-write helper — call once after a write loop instead of
letting every :meth:write_mv call sample the clock. when
accepts an explicit timestamp (e.g. an upstream "Last-Modified"
header); None defaults to :func:time.time.
acquire ¶
Bring the IO's backing into the acquired state.
Lifecycle primitive — idempotent. Returns self.
:meth:__enter__ calls this; so does :meth:open before
constructing its cursor IO.
flush ¶
Push buffered writes to the durable backing.
Cursor IOs forward the flush to their bound parent; storage
IOs go through :meth:Disposable.commit (default no-op
unless a subclass overrides).
pread ¶
Positional read. Returns at most n bytes at pos.
cursor=True reads from the internal cursor instead of pos
and advances it past the bytes returned.
pwrite ¶
pwrite(
data: Union[bytes, bytearray, memoryview],
pos: int,
*,
update_stat: bool = True,
cursor: bool = False
) -> int
Positionally write. Returns bytes actually written.
update_stat=False defers the post-write stat refresh to
the caller — see :meth:write_mv for the bulk-write rationale.
cursor=True writes at the internal cursor instead of pos
and advances it by the bytes written.
iter_mv ¶
iter_mv(
chunk_size: int = 256 * 1024,
*,
start: int = 0,
length: Optional[int] = None
) -> Iterator[memoryview]
Yield [start, start+length) in bounded, zero-copy memoryview
chunks (default: the whole holder from start).
Each chunk is a :meth:read_mv slice — a view straight into the live
in-memory window, or a bounded read for spilled / file-backed storage —
so a consumer like http.client can sock.sendall it without a
copy, and never more than chunk_size is resident at once. Reads are
positional (the cursor is untouched), so the holder can be iterated
again — e.g. a connection retry re-sending the same body — by calling
this afresh.
read_bytes ¶
Read size bytes starting at offset as :class:bytes.
size=-1 reads to EOF; offset accepts negative
indices via :func:_resolve_pos (-1 → size,
-N → self.size - N). cursor=True reads from the
internal cursor and advances it past the bytes returned.
write_bytes ¶
write_bytes(
data: Any,
offset: int = 0,
*,
size: int = -1,
overwrite: "bool | None" = None,
cursor: bool = False
) -> int
Splice data at offset. Returns bytes written.
overwrite defaults to None → resolved: a whole-content write
from the start (offset == 0, size == -1, no cursor) replaces the
object (pathlib write_bytes truncate semantics), so a whole-blob
remote backend does it in a single PUT instead of a stat + read-page +
upload read-modify-write. A positional / cursor / size-capped write is a
splice that preserves the rest, so it resolves to False. Pass an
explicit True / False to force either.
size caps the byte count written — size=-1
(default) writes the entire source; size>=0 writes at
most size bytes. The cap is forwarded into each
type-directed branch so a stream source stops reading
after size bytes (no over-pull) and a bytes-like
source slices its tail off before dispatching.
overwrite declares that this write replaces every
byte from offset onward. The holder ends at
offset + bytes_written regardless of its prior size,
and whole-blob remote backends collapse the implied
truncate(...) + write(...) pair into one SDK call.
Type-directed dispatch — bytes-like payloads
(:class:bytes, :class:bytearray, :class:memoryview,
and str after UTF-8 encoding) splice through
:meth:write_mv; other :class:Holder instances route
through :meth:write_holder (size-aware: small payloads
write inline, large ones stream); file-like sources
(anything exposing .read) drain through
:meth:write_stream. Subclasses override
:meth:_write_mv, :meth:_write_stream, and / or
:meth:_write_holder rather than this dispatch.
read_text ¶
read_text(
encoding: str = "utf-8",
errors: str = "strict",
*,
size: int = -1,
offset: int = 0,
cursor: bool = False
) -> str
Decode size bytes at offset as text.
cursor=True reads from the internal cursor and advances it.
write_text ¶
write_text(
text: str,
encoding: str = "utf-8",
errors: str = "strict",
*,
offset: int = 0,
cursor: bool = False
) -> int
Encode text and splice at offset. Returns bytes written.
cursor=True writes at the internal cursor and advances it.
head ¶
Peek the first size bytes from offset (default 0).
A bounded positional read off the front of the object that
leaves the internal cursor (:meth:tell) untouched — head
composes with cursor reads without disturbing them. size
is clamped to what's available, so a short object (or one
shorter than offset + size) returns fewer bytes rather
than raising; size < 0 reads from offset to EOF.
tail ¶
Peek the last size bytes, leaving the cursor untouched.
The end-anchored companion to :meth:head — a bounded
positional read off the back of the object. size is
clamped to the object's length, so requesting more than
exists (or size < 0) returns the whole object. The
internal cursor (:meth:tell) is not moved.
readinto ¶
Fill buffer with bytes starting at offset.
Returns the number of bytes written into buffer —
min(len(buffer), self.size - offset). Matches the
stdlib :meth:io.RawIOBase.readinto shape. cursor=True
reads from the internal cursor and advances it.
On a cursor IO (_parent is not None) the default flips
to cursor-anchored — stdlib readinto(buf) then matches
the BinaryIO contract.
readline ¶
Read up to the next newline starting at offset.
Returns the line including the trailing \n (or short
when EOF lands first). limit >= 0 caps the byte count.
cursor=True reads from the internal cursor and advances
it past the returned line. On a cursor IO the default flips
to cursor-anchored.
readlines ¶
Read every line from offset to EOF (or until hint bytes).
cursor=True reads from the internal cursor and advances it
past the bytes consumed. On a cursor IO the default flips to
cursor-anchored.
seek ¶
Seek the internal cursor to offset relative to whence.
Mirrors :meth:io.IOBase.seek with two ergonomic deviations:
seek(-1, SEEK_SET)is a "go to end" sentinel — pairs withread(-1)/ "read all". Any other negativeSEEK_SEToffset raises :class:ValueError.SEEK_CUR/SEEK_ENDwith a negative offset that would land before byte 0 clamps to 0 instead of raising.
write_local_path ¶
write_local_path(
path: PathLike,
*,
pos: int = 0,
n: int = -1,
chunk_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Load path's bytes into this holder at pos.
n < 0 reads the whole file; n >= 0 caps the source
bytes pulled at n. Streams in chunk_size slices so a
large file doesn't materialize into memory.
Pre-allocates the holder via :meth:resize when the source
size is known up front (n >= 0 or local stat available),
so the inner loop only writes — no per-chunk grow.
write_stream ¶
write_stream(
src: Any,
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Drain a binary source into this holder at offset.
Public entry point: accepts a yggdrasil :class:IO[bytes],
a stdlib :class:typing.BinaryIO (io.BytesIO,
open(..., "rb"), urllib3 responses, …), or any file-like
carrying a .read. Non-:class:IO sources are coerced
via :meth:IO.from_ so subclass-side :meth:_write_stream
always receives a real :class:IO[bytes].
size caps the byte count drained from src —
size=-1 (default) reads to EOF; size>=0 stops at
size bytes (no over-pull from the source).
overwrite truncates the holder's tail past
offset + bytes_written; whole-blob remote backends
get a single atomic PUT instead of an explicit truncate
followed by a write.
batch_size is the read/write chunk size for the
default streaming path (:data:_COPY_CHUNK, 1 MiB).
Tune up for high-throughput remote sinks where the
per-call overhead dominates, or down to bound peak
memory on a slow consumer.
write_holder ¶
write_holder(
src: "Holder",
*,
offset: int = 0,
size: int = -1,
overwrite: bool = False,
batch_size: int = _COPY_CHUNK,
cursor: bool = False
) -> int
Splice another :class:Holder's bytes into this one at offset.
Public entry point: validates the inputs, then dispatches
to :meth:_write_holder. size caps the byte count
pulled from src — size=-1 (default) writes the
whole source; size>=0 writes the first size bytes.
overwrite truncates the tail past
offset + bytes_written (collapses truncate(...) +
write_holder(...) into one operation for whole-blob
remote backends). batch_size is forwarded to the
streaming path for above-threshold payloads.
Subclasses override the private hook to swap in a backend-aware fast path (Workspace / Volumes / S3 can hand the source straight to their atomic-upload SDK call without ever materialising the bytes in Python).
upload ¶
Upload src's bytes into this holder.
Symmetric to :meth:download but indexed from the
destination side — dst.upload(src) makes the
destination's content equal to the source's.
src accepts any of:
- :class:
Holder(incl. any :class:Pathsubclass) — its bytes are pulled starting at offset. - :class:
IOcursor — offset (if non-zero) seeks beforeread(); otherwise the cursor's current position is honoured. str/ :class:os.PathLike— coerced viaPath.from_(src)and treated as a holder.
size and offset slice the source: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Slicing forces the whole-payload
fast path in :meth:_transfer_to to defer to a bytes
copy (the backend-specific shortcuts —
shutil.copyfile, write_local_path — don't expose
a window).
When self is a :class:Path whose URL ends in a
trailing / (directory shape), the source's filename
(src.url.name or "download" for nameless holders)
is joined onto it. No remote stat is issued — the
trailing slash is a purely local, cp-style hint.
Returns the resolved destination so chains like
dst.upload(src).read_bytes() work.
Subclasses with a faster move (e.g. local→local via
sendfile, local→remote chunked stream) override
:meth:_transfer_to, not this method.
download ¶
Copy this holder's bytes to a local target.
When to is :data:None, bytes land in the user's
~/Downloads folder under :attr:url.name (or
"download" for nameless holders), with browser-style
(1) / (2) / … suffixes appended on name conflict.
Otherwise to accepts the same shapes as :meth:upload
(:class:Holder, :class:IO, str / :class:os.PathLike).
size and offset slice this holder: size=-1 (default)
reads to EOF, size>=0 caps the byte count, offset
is the starting offset. Returns the resolved target.
decode ¶
Decode the whole payload as text. Cursorless — does not seek.
to_base64 ¶
Return the payload base64-encoded as an ASCII str.
urlsafe=True (default) uses :func:base64.urlsafe_b64encode
— - / _ in place of + / / so the result drops
cleanly into a URL or filename. urlsafe=False falls back
to the standard alphabet.
xxh3_64 ¶
Return an :class:xxhash.xxh3_64 instance over the payload.
Always rebuilds an updatable :class:xxhash.xxh3_64 so callers
can keep mixing more bytes in if they want. The expensive
part — walking the payload — is short-circuited via the
cached digest; we just seed a fresh hasher with the cached
value's bytes when available.
xxh3_int64 ¶
64-bit xxh3 hash of the payload as a signed int64.
xxh3_64 produces an unsigned 64-bit value; downstream Arrow
schemas pin the field as int64, so the digest is wrapped
into signed range [-2**63, 2**63). Memoized against
(_size, _mtime) — which every write path bumps via
:meth:_touch_stat — so repeated reads pay the walk once.
arrow_input_stream ¶
Context manager yielding the cheapest :class:pa.NativeFile over the payload.
Local-path holder + no codec → :func:pyarrow.memory_map
(zero-copy). Codec-tagged holder → decompress, then wrap in a
:class:pa.BufferReader. Anything else → snapshot and wrap.
The yielded stream is always a real :class:pa.NativeFile,
so the caller hands it directly to pyarrow readers.
arrow_output_stream ¶
Context manager yielding a :class:pa.BufferOutputStream writer.
with bio.arrow_output_stream() as sink: writer(sink). The
yielded sink accepts the format encoder's writes against a
pure-Arrow in-memory buffer. On a clean exit the encoded
bytes are committed to self via
:meth:_commit_format_payload, which handles codec
compression and the overwrite-vs-append disposition.
with_media_type ¶
Stamp media_type onto the bound IO's metadata.
With copy=False (the default), mutates self and returns
it. copy=True allocates a fresh holder over the same bytes
and returns a new IO over it.
as_media ¶
Wrap this path in the format leaf for its media type.
.. deprecated::
Use :meth:open with a media_type instead —
path.open(media_type=...) already dispatches to the
right format leaf and gives a properly acquired cursor with
lifecycle handling. as_media returns an un-acquired leaf
and is kept only for callers that haven't migrated.
Resolution: explicit media_type first, else the holder's
:class:MediaType (path extension, magic-byte sniff, or
content-type header). The resolved class is looked up in the
:class:Holder format registry and instantiated bound to this
path.
Raises :class:KeyError when the path's media type isn't
registered as a tabular format.
read ¶
Read up to size bytes from the cursor, advancing past them.
Stdlib :meth:io.RawIOBase.read semantic: size < 0 /
None reads to EOF; otherwise reads up to size bytes,
returning fewer at EOF.
Static IOs (:class:Memory, :class:Path) know their full
size up front; cap the request at self.size - self._pos
before dispatching so the storage's strict read_bytes
doesn't trip on an out-of-range window. Streaming IOs
(:class:MemoryStream — is_streaming) lazily pull bytes;
forward the request unclamped so the storage pulls until it
has enough or signals EOF.
write ¶
Write b at the cursor, advancing it.
Accepts bytes-like, str (UTF-8), io.BytesIO, or any
file-like with .read. File-like sources route through
:meth:write_stream so backends with an atomic whole-object
upload push a single request. The buffer-protocol fallback
catches things like :class:pyarrow.Buffer that aren't
bytes/bytearray/memoryview but ARE memoryview-able.
json_load ¶
Parse the buffer, auto-detecting media type and compression.
Resolution order for the media type:
- Explicit media_type kwarg.
- Cached :attr:
media_typeon the IO. - Magic-byte sniff via :meth:
MediaType.from_io— when this fires and the IO had no cached media type, the sniffed value is stamped onto the IO so future callers (codec handling, tabular dispatch) see it without re-sniffing.
If the resolved type carries a codec the buffer is
decompressed first and the inner mime is stamped onto the
decompressed buffer. JSON / NDJSON / opaque-bytes payloads go
through json.loads (or pandas.read_json when orient
is set); every other registered format dispatches to its
:class:Tabular leaf and returns read_pylist().
decompress ¶
Return a new IO over the decompressed payload.
codec may be a :class:Codec, a codec name ("gzip",
"zstd", …), or a :class:MediaType-shaped object whose
codec attribute is read. Returns the original buffer when
no codec is set / supplied.
ls ¶
ls(
*,
recursive: bool = False,
limit: "int | None" = None,
singleton_ttl: Any = False
) -> Iterator["Path"]
Yield children lazily. limit caps how many are produced — the
underlying listing stays incremental, so a bounded ls over a huge
prefix never materialises (or fetches) more than it needs.
unlink ¶
Remove the leaf — pathlib-compatible: refuses directories.
Mirrors :meth:pathlib.Path.unlink: succeeds for files, raises
:class:IsADirectoryError for directories so callers don't
accidentally recursive-delete via unlink. Use :meth:remove
for the directory case. Thin wrapper over :meth:_delete's
path-removal mode.
remove ¶
remove(
recursive: bool = True,
missing_ok: bool = True,
wait: WaitingConfigArg = True,
fresher_than: Optional[TimeLike] = None,
older_than: Optional[TimeLike] = None,
) -> "Path"
Remove this path — the file, or the whole subtree when recursive.
Thin wrapper over :meth:_delete's path-removal mode (the single
deletion primitive). fresher_than / older_than scope the
removal to children inside that mtime window.
wait_until_gone ¶
Block until :meth:exists reports False or wait expires.
Polls the backend with a fresh probe each iteration — the
stat cache is invalidated between checks so a TTL'd hit
can't mask a deletion that landed after the cache was
filled. Useful when a fire-and-forget unlink (e.g.
WarehouseStatementBatch.clear_temporary_resources) means
the caller can't observe completion through the original
operation's return value.
Raises :class:TimeoutError when wait's deadline elapses
and the path is still present.
touch ¶
Create the path as an empty file if it doesn't exist.
write_bytes(b"") short-circuits in the holder fast path
(zero bytes, no flush), which would leave a missing file behind
— open + close around the empty write so the holder actually
materialises the entry on the backing store.
upload_module ¶
Zip a local module / package and write it under this path.
module is anything :func:resolve_module_root accepts —
an importable module name ("yggdrasil.io"), a
:class:os.PathLike pointing at a package directory or an
existing .zip / .whl archive, or a callable
carrying a __module__ attribute. The module is packed
into a deflated zip whose top-level entry is the package
directory itself, so the archive can be added to
sys.path directly (or fed to
:meth:SparkSession.addArtifacts with pyfile=True).
Destination shape on self:
- self names a file with a
.zip/.whlsuffix — archive bytes land at that exact path. - self is anything else — archive lands at
self / <name or "<module>.zip">.
Returns the concrete :class:Path that now holds the
archive. overwrite=False raises
:class:FileExistsError when the destination already
exists.
import_module ¶
import_module(
module_name: str | None = None,
*,
install: bool = True,
cache_dir: "Any" = None
) -> Any
Download a module archive at this path and import it.
Inverse of :meth:upload_module: fetch the archive bytes
at self, drop them on local disk, prepend the archive (or
its extracted parent) to :data:sys.path, and return the
live module via :func:importlib.import_module.
module_name defaults to the archive's stem (filename
minus suffix). cache_dir picks where the archive lands
locally (default: a fresh
:meth:LocalPath.staging_path-style directory).
install=True (the default) preserves the archive on
disk so subsequent imports in the same process hit the
cache. install=False makes the cache-dir lifetime the
caller's problem.
staging_path
classmethod
¶
Return a fresh :class:LocalPath over a staging file.
The returned holder is closed (un-acquired) and marked
temporary=True — closing it unlinks the file. Triggers
the once-per-process sweep of stale staging entries (>1 day
old) on the first call of the process; subsequent calls
skip straight to name minting.
Useful as a one-liner for scratch buffers, atomic-write targets that get renamed into place on success, or any "give me a local file I can write to" pattern that doesn't care about the filename.
reserve ¶
No-op — local filesystems have no useful capacity-vs-size
distinction. Files grow on write; posix_fallocate is an
optimization, and most filesystems no-op it anyway.
truncate ¶
Set the file size to exactly n bytes via :func:os.ftruncate.
Shrinks drop the tail; extends zero-pad. Overrides the
:class:Path default (read-modify-write via _read_mv/_write_mv)
with the direct syscall. Transient fd when the holder isn't
acquired so truncate is usable on a fresh path.
ProxyPathMixin ¶
Bases: ABC
Mirror the full :class:Path surface onto an inner path.
Implement :meth:_internal_path; everything else delegates. The
inner path is resolved on every access (the implementation is free
to cache it) so a refreshed / rebuilt inner path is picked up
transparently.