yggdrasil.io.delta._cache¶
_cache ¶
Two-tier byte cache for immutable Delta log files.
RAM is precious, so the in-memory tier is a tiny byte-bounded LRU (default 4 MB,
holding only the hottest small files — commit JSONs, manifests), backed by a
persistent local-disk store under ~/.cache that holds the bulk (notably the
large checkpoint parquet) and survives process restarts, so repeated CLI runs /
job restarts don't re-GET the same files from S3.
The caller decides per read whether to cache (cache=): only remote,
immutable, version-addressed files are cached (commit JSONs, checkpoint
parquet/manifests, sidecars). A local table reads straight through — it's
already on local disk, so a second copy would be pointless — and so does the
mutable _last_checkpoint pointer (the listing keeps it fresh). All cache ops
are best-effort: a miss or an I/O error on the cache never breaks a read. The one
staleness window is a table dropped and recreated at the same path within the
disk TTL.
Knobs (env): YGG_DELTA_RAM_CACHE_BYTES (default 4 MB),
YGG_DELTA_CACHE_DIR (default $XDG_CACHE_HOME/yggdrasil/delta-log or
~/.cache/...), YGG_DELTA_DISK_CACHE_TTL (seconds, default 1 day; 0
disables disk), YGG_DELTA_DISK_CACHE_MAX_BYTES (default 1 GB).
read_one ¶
Read one log file; cache=False (a local table, or the mutable pointer)
reads straight through. Raises FileNotFoundError if the source is absent.
read_many ¶
Bytes for each path, in order, missing files mapping to None. With
cache (a remote table), RAM/disk hits are resolved first and only the
misses fetched concurrently; without it (a local table) everything is read
concurrently and nothing is cached.