yggdrasil.path.memory_stream

memory_stream

Spill-aware sliding-window streaming :class:Holder.

:class:MemoryStream is a :class:Holder that pulls bytes from a streaming source on demand into an in-memory window plus an optional on-disk spill file. Positions are absolute offsets from the start of the stream.

Retention model

• :attr:spill_threshold (default 128 MiB) caps the live in-memory bytearray. Pulls that would push the buffer past this threshold spill the cold (oldest) bytes to a tempfile; the buffer holds only the recent window.
• :attr:byte_size (default 2 GiB) caps total retained bytes (memory + spill). Pulls that would push retention past this cap evict the oldest bytes — spilled bytes first, then in-memory.
• When byte_size <= spill_threshold, no spill file is ever created; the holder falls back to the legacy single-window eviction shape so small-budget consumers don't pay tempfile overhead.

Reads valid in [spill_start, size); writes valid in [window_start, size] (append or in-place inside the live in-memory portion). Reads or writes that target an evicted offset raise — those bytes are gone.

Sources accepted:

• File-like with .read(n): open(...), :class:io.BytesIO, urllib3.HTTPResponse, requests.Response.raw.
• Bytes-like (:class:bytes / :class:bytearray / :class:memoryview): wrapped internally as a finite source — useful for tests and for reusing the windowed-reader code on already-resident bytes.
• Iterable yielding bytes-like chunks (a generator, a list of frames).
• Callable f(n) -> bytes-like returning at most n bytes; an empty return signals EOF.
• None: empty stream — only manual feeds via :meth:write_bytes add bytes.

MemoryStream

MemoryStream(
    source: SourceLike = None,
    *,
    content_encoding: Optional[str] = None,
    byte_size: int = _DEFAULT_BYTE_SIZE,
    spill_threshold: int = _DEFAULT_SPILL_THRESHOLD,
    pull_chunk: Optional[int] = None,
    **kwargs: Any
)

Bases: Holder

Sliding-window streaming holder with optional spill-to-disk.

Construction::

MemoryStream(
    source,
    *,
    byte_size=2 GiB,
    spill_threshold=128 MiB,
    pull_chunk=64 KiB,
)

source is the upstream feed (see module docstring for accepted shapes). byte_size caps total retained bytes (memory + spill); when retention would exceed it, the oldest bytes are evicted. spill_threshold caps the in-memory live window; cold bytes above this go to a tempfile and stay readable until evicted. pull_chunk is the default size of each pull from the source.

When byte_size <= spill_threshold the spill file is never created — the holder collapses to a pure in-memory eviction loop, matching the original single-window shape so small-budget consumers don't pay tempfile setup cost.

Implements the :class:Holder primitives with absolute-offset semantics: :attr:size is the highest offset the stream has reached so far. Reads valid in [spill_start, size); writes valid in [window_start, size].

byte_size property

byte_size: int

Maximum bytes retained (memory + spill).

spill_threshold property

spill_threshold: int

In-memory window cap; bytes above spill to disk.

window_start property

window_start: int

Absolute offset of the first byte in the in-memory window.

window_end property

window_end: int

Absolute offset one past the last byte in the in-memory window (== :attr:size).

spill_start property

spill_start: int

Absolute offset of the first retained byte.

Equal to :attr:window_start when no spill is active. Otherwise it sits at the start of the spill region; bytes before this have been evicted and reads behind it raise.

has_spill property

has_spill: bool

True iff a spill tempfile is currently active.

eof property

eof: bool

True once the source has signalled EOF.

is_streaming property

is_streaming: bool

True while EOF hasn't been reached on the source feed.

:attr:size reflects only the bytes pulled so far, so cursor-anchored readers (:class:IO.read) need to bypass the standard cap at size clamp until the source signals EOF.

size_known property

size_known: bool

False while the source can still grow :attr:size.

The base :class:Holder reports True for in-memory holders because their size is a settled slot. A streaming holder is different: :attr:size reflects only the bytes pulled so far, so before the first read it is 0 even though the body is non-empty. Format leaves (parquet / arrow / csv / ndjson) guard their read with if self.size_known and self.size == 0: return as an empty-buffer short-circuit — with the base True that fired on an un-pulled stream and silently returned zero rows. Reporting the size as unknown until EOF makes the short-circuit defer to the actual read (which drives the pull), so a stream=True response body parses correctly.

read_mv

read_mv(size: int = -1, offset: int = 0, *, cursor: bool = False) -> memoryview

Read size bytes at absolute offset, pulling from source as needed. size < 0 reads to EOF.

cursor=True reads from the holder's internal cursor and advances it past the returned bytes. Reads are valid in [spill_start, size) — anything behind :attr:spill_start has been evicted (truly dropped from both memory and spill) and raises.

write_mv

write_mv(
    data: memoryview,
    offset: int = 0,
    *,
    size: int = -1,
    overwrite: bool = False,
    update_stat: bool = True,
    cursor: bool = False
) -> int

Splice bytes at offset. Appends past current end extend the stream and may slide the window; in-window writes overwrite.

cursor=True writes at the internal cursor and advances it. size>=0 slices the input buffer to at most size bytes before the splice. overwrite=True truncates the tail past offset + len(data) after the splice (same contract as :meth:Holder.write_mv). Writes behind :attr:window_start raise — the target bytes have already been evicted.

reserve

reserve(n: int) -> None

Pre-grow the underlying bytearray, capped at :attr:byte_size.

Capacity beyond :attr:byte_size would be evicted on the very next :meth:_slide_window call, so the cap is the honest ceiling here.

truncate

truncate(n: int) -> int

Set visible :attr:size to n. Shrinks drop the tail (in-memory and spill); extends zero-pad in memory.

Truncating below :attr:spill_start raises — those bytes are evicted and unrecoverable. A truncate that lands inside the spill region drops the trailing spill bytes and the whole in-memory window.

memoryview

memoryview() -> memoryview

View over the live window only (not the full stream).

Override of :meth:Holder.memoryview — that one would call self.read_mv(-1, 0), which raises once :attr:window_start

0 because position 0 is no longer in the window. The window view is the honest answer for a sliding-window holder.