Holders

Every storage implementation is reached through the positional IOBase contract.

Holder: every storage handle

Holder names every IOBase implementation the core ships: the local Buffer, Folder, Path, and File, the fs trio, and the generic Buffered, Text, and Media wrappers. into_text, buffered, and into_media are idempotent, so a dynamic caller can request the optimized view without stacking wrappers.

use yggdryl::holder::Holder;
use yggdryl::holder::local::Folder;

// Generic construction records the location without probing its role.
let directory = Holder::local(Folder::temporary()?.path()?)?;
assert!(matches!(directory, Holder::Path(_)));

let missing = Holder::local(Folder::temporary()?.path()?.join("yggdryl-generic-doc.bin"))?;
assert!(matches!(missing, Holder::Path(_)));

Holder::local returns Holder::Path, the unresolved local::Path role, so construction performs no filesystem call. Holder::buffer, Holder::folder, and Holder::file commit to a role explicitly and also touch nothing. The generic path resolves through the matching specialized handle only when an operation needs to know what is there.

Holder::open first promotes IPC, Parquet, Avro, or plain text into its inferred media wrapper, then opens it. This keeps schema, footer, and dimension caches behind one generic handle—the exact route Python and JavaScript scopes use. JSON, directories, and unknown byte media remain raw.

Walking a tree stays in one type, because every hierarchy accessor returns Holder too.

use yggdryl::holder::Holder;
use yggdryl::IOBase;
use yggdryl::holder::local::Folder;

let root = Holder::folder(Folder::temporary()?.path()?)?;
assert!(root.is_container());

// A child need not exist. Naming one yields a leaf handle, and nothing is created.
let leaf = root.child_by_path("yggdryl-generic-child.bin")?;
assert!(matches!(leaf, Holder::File(_)));
assert!(!leaf.is_container());
assert_eq!(leaf.size(), 0);

IOBase

yggdryl is the crate's one storage abstraction: positional reads and writes over anything that holds bytes.

Python and JavaScript expose one handle class, IOBase, under the names each language already uses for a path. The byte contract and the record methods both cross into the bindings; the role traits, the wrappers, and the streaming adapters stay in Rust, and every section below says which of the three languages reach it.

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let mut handle = Buffer::new();
handle.pwrite(0, b"symbol,price\n")?;
handle.pwrite(13, b"AAPL,1\n")?;
assert_eq!(handle.size(), 20);

// Two reads at different offsets, in any order: there is no shared cursor.
let mut tail = [0_u8; 4];
handle.pread(13, &mut tail)?;
let mut head = [0_u8; 6];
handle.pread(0, &mut head)?;
assert_eq!(&head, b"symbol");
assert_eq!(&tail, b"AAPL");

from yggdryl import IOBase

handle = IOBase.from_bytes()
handle.pwrite(0, b"symbol,price\n")
handle.pwrite(13, b"AAPL,1\n")
assert handle.size == 20

# Two reads at different offsets, in any order: there is no shared cursor.
assert handle.read_range_bytes(13, 4) == b"AAPL"
assert handle.read_range_bytes(0, 6) == b"symbol"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const handle = IOBase.fromBytes()
handle.pwrite(0, Buffer.from('symbol,price\n'))
handle.pwrite(13, Buffer.from('AAPL,1\n'))
assert.equal(handle.size, 20)

// Two reads at different offsets, in any order: there is no shared cursor.
assert.equal(handle.readRangeBytes(13, 4).toString(), 'AAPL')
assert.equal(handle.readRangeBytes(0, 6).toString(), 'symbol')

IOBase::pread and IOBase::pwrite are the only two methods an implementation must supply for bytes; everything else on the trait is derived from them. The bindings expose the derived read_range_bytes/readRangeBytes for the read - a caller-supplied buffer is a Rust shape - and pwrite under its own name for the write. They take an explicit offset rather than sharing a cursor, so a footer-first container such as Parquet reads its index without seeking, and two readers of one handle never interfere.

Three invariants hold for every implementation:

• pread returns a short count only at the end of the value; a read entirely past size returns 0.
• pwrite grows the value when the write extends past the end, and zero-fills the gap an offset beyond the current size creates.
• size never exceeds capacity, and reserve changes only capacity.

Streamed bytes

The canonical Rust entry point is IOBase::pstream_bytes(&self, position: u64, batch_size: usize) -> Result<ByteStream<'_>>. It starts at an explicit decoded byte position and yields owned arrays of at most batch_size bytes. Construction is lazy, never asks for size, and an error is yielded after every successful prefix before the iterator stays fused. batch_size must be non-zero.

use yggdryl::{IOBase, IOCursor};
use yggdryl::holder::Buffer;

let handle = Buffer::from_bytes(b"0123456789".to_vec());
let chunks = handle
    .pstream_bytes(2, 3)?
    .collect::<yggdryl::Result<Vec<_>>>()?;
assert_eq!(chunks, [b"234".to_vec(), b"567".to_vec(), b"89".to_vec()]);

// The cursor form starts at `tell` and advances only when bytes are yielded.
let mut cursor = handle.cursor_at(1);
let first = cursor.stream_bytes(2)?.next().transpose()?.unwrap();
assert_eq!(first, b"12");
assert_eq!(cursor.tell(), 3);

The bindings expose the same lazy iterator with a 65,536-byte default batch:

PythonJavaScript

from yggdryl import IOBase

handle = IOBase.from_bytes(b"0123456789")
assert list(handle.pstream_bytes(2, 3)) == [b"234", b"567", b"89"]

cursor = handle.cursor(1)
stream = cursor.stream_bytes(2)
assert next(stream) == b"12"
assert cursor.tell() == 3

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const handle = IOBase.fromBytes(Buffer.from('0123456789'))
assert.deepEqual(
  [...handle.pstreamBytes(2, 3)].map((part) => part.toString()),
  ['234', '567', '89'],
)

const cursor = handle.cursor(1)
const first = cursor.streamBytes(2).next()
assert.equal(first.value.toString(), '12')
assert.equal(cursor.tell(), 3)

ByteStream also implements std::io::Read, so codecs and parsers fill their own reusable windows without allocating an iterator item first. A coded handle decodes directly from the encoded source; a non-zero position is reached by decoding and discarding the prefix because compression frames are not seekable. The stream does not open the coded handle or retain decoded pages. Through Buffered, it deliberately bypasses the page cache, leaving cached_pages() == 0; use positional reads when retained pages are wanted.

Measured streamed-byte behavior

Criterion measured the same 8 MiB decoded fixture on Windows 11 x86_64, an AMD Ryzen 5 150 (6 cores/12 threads), and rustc 1.96.1 on 2026-08-23. Cells are medians; the stream cases retain no decoded pages.

operation	decoded bytes	plain	gzip	zlib	zstd
first pstream_bytes item	64 KiB	3.42 us	80.96 us	71.94 us	281.24 us
one pread	64 KiB	1.84 us	73.09 us	65.32 us	260.15 us
pstream_bytes drain	8 MiB	0.501 ms	8.405 ms	8.225 ms	14.995 ms
read_all_bytes	8 MiB	1.981 ms	15.265 ms	14.761 ms	21.049 ms
sixteen sequential pread calls	1 MiB	0.037 ms	10.268 ms	8.581 ms	14.412 ms

The last row rebuilds a decoder at every compressed offset. Keep one ByteStream for a scan: it is both faster and bounded-memory. Whole-value reads deliberately trade those properties for one returned Vec<u8>.

Regenerate with:

cargo bench -p yggdryl --bench io --features parquet -- io_pstream --noplot

Built from what you already hold

In Python, IOBase(...) accepts more than a path: callers hold open files and streams more often than the strings that named them, so the constructor takes those directly. A file-like object with a real filesystem name captures the location - nothing is read, per the laziness contract - while a nameless stream such as io.BytesIO captures its content into an in-memory handle. Passing another handle rebuilds it, and passing an in-memory handle captures its content and media type.

import io
import pathlib
import tempfile

from yggdryl import IOBase

# An open file names its own location, so the handle addresses the path.
target = pathlib.Path(tempfile.mkdtemp()) / "quotes.json"
target.write_bytes(b"{}")
with open(target, "rb") as stream:
    handle = IOBase(stream)
assert handle.name == "quotes.json"

# A nameless stream holds only content, so the content is what is taken.
buffered = IOBase(io.BytesIO(b'{"symbol": "AAPL"}'))
buffered.media_type = "application/json"
assert buffered.read_text() == '{"symbol": "AAPL"}'

Laziness

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::{IOKind};
use yggdryl::holder::local;

let path = local::Folder::temporary()?.path()?.join("yggdryl-docs-io-lazy.csv");
let _ = std::fs::remove_file(&path);

// Constructing touches nothing: no file is created, opened, or mapped.
let mut handle = local::File::new(&path)?;
assert!(!handle.exists());

// Reading something absent yields nothing rather than failing.
assert_eq!(handle.size(), 0);
let mut probe = [0_u8; 8];
assert_eq!(handle.pread(0, &mut probe)?, 0);
assert_eq!(handle.kind(), IOKind::Unknown);

// Writing creates the resource, and any parent it needs.
handle.write_all_bytes(b"symbol,price\n")?;
assert_eq!(handle.kind(), IOKind::File);
assert_eq!(handle.read_all_bytes()?, b"symbol,price\n");

handle.close()?;
// Teardown through the abstraction: absence is a no-op success.
handle.remove(false)?;

import pathlib
import tempfile

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())

# Constructing touches nothing: no file is created, opened, or mapped.
handle = IOBase(root / "nested" / "lazy.csv")
assert not handle.exists()

# Reading something absent yields nothing rather than raising.
assert handle.size == 0
assert handle.read_bytes() == b""

# Writing creates the resource, and any parent it needs.
handle.write_text("symbol,price\n")
assert handle.is_file()
assert handle.read_text() == "symbol,price\n"

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))

// Constructing touches nothing: no file is created, opened, or mapped.
const handle = new IOBase(path.join(root, 'nested', 'lazy.csv'))
assert.ok(!handle.exists())

// Reading something absent yields nothing rather than throwing.
assert.equal(handle.size, 0)
assert.equal(handle.readBytes().length, 0)

// Writing creates the resource, and any parent it needs.
handle.writeText('symbol,price\n')
assert.ok(handle.isFile())
assert.equal(handle.readText(), 'symbol,price\n')

fs.rmSync(root, { recursive: true, force: true })

A handle is a description of where bytes would live, not proof that they do. Constructing one never fails for a resource that does not exist yet and never pays for one that is never used; non-existence is resolved at the operation instead. Reads skip, writes create, and truncate/reserve create too. That is why a caller can probe a location without a separate existence check, and why the same code works whether the target is there or not.

Metadata follows the rule: media_type is computed when it is asked for, and re-derived after the bytes change.

Kinds

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::{IOKind};
use yggdryl::holder::local;

assert_eq!(Buffer::new().kind(), IOKind::Memory);
assert!(IOKind::Memory.is_leaf());

let folder = local::Folder::temporary()?;
assert_eq!(folder.kind(), IOKind::Directory);
assert!(folder.is_container());

// Nothing is there, so nothing has decided; a write settles it.
let absent = local::File::new(local::Folder::temporary()?.path()?.join("yggdryl-docs-io-absent.bin"))?;
assert_eq!(absent.kind(), IOKind::Unknown);
assert!(!absent.kind().is_known());

import pathlib
import tempfile

from yggdryl import IOBase

folder = IOBase(pathlib.Path(tempfile.mkdtemp()))
assert folder.is_dir()
assert not folder.is_file()

# Nothing is there, so nothing has decided; a write settles it.
leaf = folder / "ticks.csv"
assert not leaf.exists()
leaf.write_text("symbol\n")
assert leaf.is_file()
assert not leaf.is_dir()

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const folder = new IOBase(fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-')))
assert.ok(folder.isDir())
assert.ok(!folder.isFile())

// Nothing is there, so nothing has decided; a write settles it.
const leaf = folder.joinpath('ticks.csv')
assert.ok(!leaf.exists())
leaf.writeText('symbol\n')
assert.ok(leaf.isFile())
assert.ok(!leaf.isDir())

fs.rmSync(folder.intoPath(), { recursive: true, force: true })

IOKind is the vocabulary every backend answers in: Memory for bytes with no location, File for a leaf that holds bytes, Directory for a container that holds other resources, and Unknown for a location that does not exist yet. A table format adds Table, Namespace, and Catalog - all of them containers, all of them answered by the value that adds the framing rather than by storage, which sees three indistinguishable folders. is_container, is_leaf, and is_known are the questions callers actually ask; the enum is documented with the rest of the shared enums in types.md. The bindings expose the questions rather than the enum, as exists, is_dir, and is_file.

Bytes or rows

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::{MimeType};
use yggdryl::holder::local;

// A leaf answers from its representation, and the two are complements.
let mut notes = Buffer::new();
notes.set_media_type(MimeType::PLAIN_TEXT.into());
assert!(notes.is_atomic());
assert!(!notes.is_tabular());

// The name is enough: nothing has been written to this location yet.
let trades = local::File::new(local::Folder::temporary()?.path()?.join("yggdryl-docs-shape.parquet"))?;
assert!(trades.is_tabular());
assert!(!trades.is_atomic());

// A container is neither one whole byte value nor - with nothing under
// it - a table.
let folder = local::Folder::temporary()?;
assert!(!folder.is_atomic());

import pathlib
import tempfile

from yggdryl import IOBase

root = IOBase(pathlib.Path(tempfile.mkdtemp()))

notes = root / "notes.txt"
assert notes.is_atomic()
assert not notes.is_tabular()

# The name is enough: nothing has been written to this location yet.
trades = root / "trades.parquet"
assert trades.is_tabular()
assert not trades.is_atomic()

assert not root.is_atomic()

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = new IOBase(fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-')))

const notes = root.joinpath('notes.txt')
assert.ok(notes.isAtomic())
assert.ok(!notes.isTabular())

// The name is enough: nothing has been written to this location yet.
const trades = root.joinpath('trades.parquet')
assert.ok(trades.isTabular())
assert.ok(!trades.isAtomic())

assert.ok(!root.isAtomic())

fs.rmSync(root.intoPath(), { recursive: true, force: true })

A handle presents its value one of two ways, and these are how a caller asks which. Atomic is the byte surface - read_all_bytes reads the value whole and write_all_bytes replaces it whole - while tabular is the record surface, read_arrow_reader and its three intent-specific writing siblings. Wherever bytes are held the two are complements; a container that holds neither rows nor one byte value - a plain folder of logs, a namespace, a catalog - answers false to both.

The answers cost as little as they can. The media type settles every leaf and every location nothing has decided yet, without a call into the backing store, so a name that already spells a record encoding is answered from the name. IOKind::Table settles a table format's folder outright, and Namespace and Catalog settle the containers that hold only containers. Only a plain Directory is probed, and the probe stops at the first leaf that settles the question rather than listing the tree - a folder reads as the table beneath it, and a partitioned tree is one table in one encoding.

is_tabular is about the representation, not about this build: a .parquet leaf is tabular whether or not the parquet feature is compiled in. record_options is the call that reports an encoding this build cannot decode, and it names it.

Whole values

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let mut handle = Buffer::new();
handle.write_all_bytes(b"symbol,price\n")?;

// `append_bytes` reports the offset the bytes landed at.
assert_eq!(handle.append_bytes(b"AAPL,1\n")?, 13);
assert_eq!(handle.read_range_bytes(0, 6)?, b"symbol");
// A range past the end yields what exists rather than failing.
assert!(handle.read_range_bytes(100, 4)?.is_empty());
assert_eq!(handle.read_all_bytes()?.len(), 20);

from yggdryl import IOBase

handle = IOBase.from_bytes()
handle.write_bytes(b"symbol,price\n")

# `append_bytes` reports the offset the bytes landed at.
assert handle.append_bytes(b"AAPL,1\n") == 13
assert handle.read_range_bytes(0, 6) == b"symbol"
# A range past the end yields what exists rather than raising.
assert handle.read_range_bytes(100, 4) == b""
assert len(handle.read_bytes()) == 20

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const handle = IOBase.fromBytes()
handle.writeBytes(Buffer.from('symbol,price\n'))

// `appendBytes` reports the offset the bytes landed at.
assert.equal(handle.appendBytes(Buffer.from('AAPL,1\n')), 13)
assert.equal(handle.readRangeBytes(0, 6).toString(), 'symbol')
// A range past the end yields what exists rather than throwing.
assert.equal(handle.readRangeBytes(100, 4).length, 0)
assert.equal(handle.readBytes().length, 20)

read_all_bytes, read_range_bytes, pwrite_all, append_bytes, write_all_bytes, and clear are the conveniences derived from pread/pwrite. Each read and append among them names the core type it answers, because the same verbs also address rows: append_bytes is the byte sibling of append_arrow_reader, and read_range_bytes the ranged half of what read_all_bytes reads whole. is_atomic is how a caller asks which surface a handle is for.

The bindings keep their own runtime spelling for the two whole-value calls - read_bytes/read_text and write_bytes/write_text - and carry read_range_bytes and append_bytes under the core name, camelCased in JavaScript. Over each of those two sits one inferring read_range/append entry point, which the Python and JavaScript pages spell out.

pread_exact is the strict form of pread: it fails, naming the shortfall, when the value ends before the buffer is full.

copy_into moves bytes between two handles in chunks, so neither side is buffered whole, and it carries the media type across. It is copy_into in Python and copyInto in JavaScript.

Digests

read_digest and read_range_digest are derived the same way, and answer a Digest rather than the bytes. Both stream through pstream_bytes and retain one bounded chunk, so memory is flat in the object's size: a 64 GiB file costs one window rather than a copy, and nothing calls read_all_bytes. Because they are derived, every backend and every wrapper inherits them.

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::DigestAlgorithm;

let mut handle = Buffer::new();
handle.write_all_bytes(b"symbol,price\nAAPL,1\n")?;

assert_eq!(
    handle.read_digest(DigestAlgorithm::Xxh3)?,
    DigestAlgorithm::Xxh3.digest(&handle.read_all_bytes()?),
);
assert_eq!(
    handle.read_range_digest(0, 6, DigestAlgorithm::Xxh3)?,
    DigestAlgorithm::Xxh3.digest(b"symbol"),
);
// Absence is emptiness here too: nothing written digests as no bytes.
assert_eq!(
    Buffer::new().read_digest(DigestAlgorithm::Xxh32)?,
    DigestAlgorithm::Xxh32.digest(b""),
);

from yggdryl import IOBase, xxhash

handle = IOBase.from_bytes()
handle.write_bytes(b"symbol,price\nAAPL,1\n")

assert handle.read_digest("xxh3-64") == xxhash.digest(handle.read_bytes(), "xxh3-64")
assert handle.read_range_digest(0, 6) == xxhash.digest(b"symbol", "xxh3-64")
assert IOBase.from_bytes().read_digest("xxh32") == xxhash.digest(b"", "xxh32")

const assert = require('node:assert/strict')
const { IOBase, xxhash } = require('yggdryl')

const handle = IOBase.fromBytes()
const payload = Buffer.from('symbol,price\nAAPL,1\n')
handle.writeBytes(payload)

assert.ok(handle.readDigest('xxh3-64').equals(xxhash.digest(payload, 'xxh3-64')))
assert.ok(handle.readRangeDigest(0, 6).equals(xxhash.digest(Buffer.from('symbol'), 'xxh3-64')))

A range is clamped exactly as read_range_bytes clamps it, a resource that does not exist digests as no bytes, and a container is a typed failure naming the kind - a folder holds no bytes of its own, and which files a folder digest would cover is a convention no format states. xxhash carries the rest: the algorithms, the resumable states, the Hashed<H> wrapper that hashes writes as they land, and the canonical value feed.

Structured values

read_scalar and write_scalar use the handle's media type to select JSON, YAML, or TOML and any outer gzip, zlib, or zstd coding. Reads feed the parser from pstream_bytes, so decoded pages are not retained. An optional Field directs native parsing and casting; omitting it infers the natural value. Rust keeps a typed Struct row as its canonical ordered Scalar::Sequence; Python and JavaScript restore the field names as dictionaries and objects by default. Python cls=Scalar and JavaScript { scalar: true } return that exact core value instead.

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::{Field, Url, Scalar};

let media = Url::from_str("file:///trade.json.gz")?.media_type();
let mut handle = Buffer::new().with_media_type(media);
let value = Scalar::from_record([
    ("quantity", Scalar::from(2_i64)),
    ("symbol", Scalar::from("AAPL")),
])?;
handle.write_scalar(&value)?;

let field = Field::from_str(
    "trade: struct<quantity: int32 not null, symbol: utf8 not null> not null",
)?;
assert_eq!(handle.read_scalar(Some(&field))?[0], Scalar::from(2_i64));

import pathlib
import tempfile

from yggdryl import IOBase, Scalar

path = pathlib.Path(tempfile.mkdtemp()) / "trade.json.gz"
handle = IOBase(path)
handle.write_scalar({"quantity": 2, "symbol": "AAPL"})
field = "trade: struct<quantity: int32 not null, symbol: utf8 not null> not null"
assert handle.read_scalar(field) == {"quantity": 2, "symbol": "AAPL"}
value = handle.read_scalar(field, cls=Scalar)
assert value.kind == "sequence"

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase, Scalar } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-value-'))
const handle = new IOBase(path.join(root, 'trade.json.gz'))
handle.writeScalar({ quantity: 2, symbol: 'AAPL' })
const field = 'trade: struct<quantity: int32 not null, symbol: utf8 not null> not null'
assert.deepEqual(handle.readScalar(field), { quantity: 2, symbol: 'AAPL' })
const value = handle.readScalar({ field, scalar: true })
assert.ok(value instanceof Scalar)
assert.equal(value.kind, 'sequence')

Measured structured-value I/O

Criterion measured one 16,384-record JSON value through the same IOBase methods on Windows 11 x86_64, an AMD Ryzen 5 150 (6 cores/12 threads), and rustc 1.96.1 on 2026-08-23. The table uses Criterion's reported point estimate; each compressed case includes coding and parsing or rendering.

representation	read_scalar	read throughput	write_scalar	write throughput
JSON	71.445 ms	11.880 MiB/s	19.137 ms	44.352 MiB/s
JSON + gzip	81.427 ms	10.424 MiB/s	394.83 ms	2.150 MiB/s
JSON + zlib	79.588 ms	10.665 MiB/s	385.78 ms	2.200 MiB/s
JSON + zstd	78.580 ms	10.801 MiB/s	195.60 ms	4.339 MiB/s

Parsing dominates these reads; the write rows expose each compressor's cost. Regenerate the table with:

cargo bench -p yggdryl --bench io --all-features -- io_value --noplot

Streaming adapters

Rust only

The Python and JavaScript packages expose positional reads and writes, not the std::io adapters over them.

use std::io::{Read, Write};

use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let mut handle = Buffer::new();
handle.writer_at(0).write_all(b"symbol,price\n")?;
handle.append_bytes(b"AAPL,1\n")?;

let mut text = String::new();
handle.reader_at(13).read_to_string(&mut text)?;
assert_eq!(text, "AAPL,1\n");

reader_at and writer_at borrow the handle as a Reader/Writer implementing std::io::Read and std::io::Write. Each adapter advances its own offset, so a second reader started elsewhere is unaffected. Adding and removing a coding moves bytes between two handles through a coding without going through these adapters, and is in all three languages.

Cursors

A handle is positional - pread/pwrite take an offset - so a position is state a caller opts into, not something two readers fight over. A cursor is that position made explicit: tell and seek move it, reads and writes advance it, and two cursors over one resource advance independently.

RustPythonJavaScript

use std::io::Read;

use yggdryl::{IOBase, IOCursor};
use yggdryl::holder::Buffer;

let mut cursor = Buffer::new().cursor();
cursor.write_next(b"symbol,price\n")?;
assert_eq!(cursor.tell(), 13);

cursor.seek_to(7);
let mut word = [0_u8; 5];
cursor.read_exact(&mut word)?; // std::io::Read rides the same position
assert_eq!(&word, b"price");

from yggdryl import IOBase

handle = IOBase.from_bytes()
cursor = handle.cursor()
cursor.write(b"symbol,price\n")

# The write landed on the handle itself; the position is the cursor's.
assert handle.read_bytes() == b"symbol,price\n"
assert cursor.seek(-6, 2) == 7
assert cursor.read(5) == b"price"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const handle = IOBase.fromBytes()
const cursor = handle.cursor()
cursor.write(Buffer.from('symbol,price\n'))

assert.equal(handle.readBytes().toString(), 'symbol,price\n')
cursor.seek(7)
assert.equal(cursor.read(5).toString(), 'price')

In Rust, IOCursor is the trait - tell, seek_to, seek, read_next, write_next - and Cursor<H> is the one wrapper every implementation shares: built by IOBase::cursor/cursor_at, it stays a full handle over the same bytes and implements std::io::Read, Write, and Seek over its own position, so it goes wherever standard readers go; the owned line iterator is built on exactly it. In Python and JavaScript the cursor shares the handle - a write through it is a write there - and follows each language's file conventions: seek(offset, whence) and read(size=-1) in Python, seek, tell, and a position property in JavaScript.

What the bytes are

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::MimeType;

// Nothing names an in-memory buffer, so its type comes from its bytes.
let mut handle = Buffer::from_bytes(br#"{"symbol":"AAPL"}"#.to_vec());
assert_eq!(handle.media_type().base(), &MimeType::JSON);

// It is re-derived after the content changes.
handle.write_all_bytes(b"PAR1payload")?;
assert_eq!(handle.media_type().base(), &MimeType::PARQUET);

from yggdryl import IOBase

# Nothing names an in-memory buffer, so its type comes from its bytes.
handle = IOBase.from_bytes(b'{"symbol":"AAPL"}')
assert str(handle.media_type.base) == "application/json"

# It is re-derived after the content changes.
handle.write_bytes(b"PAR1payload")
assert str(handle.media_type.base) == "application/vnd.apache.parquet"

const assert = require('node:assert/strict')
const { IOBase, MimeType } = require('yggdryl')

// Nothing names an in-memory buffer, so its type comes from its bytes.
const handle = IOBase.fromBytes(Buffer.from('{"symbol":"AAPL"}'))
assert.ok(handle.mediaType.base.equals(MimeType.JSON))

// It is re-derived after the content changes.
handle.writeBytes(Buffer.from('PAR1payload'))
assert.ok(handle.mediaType.base.equals(MimeType.PARQUET))

media_type is the second thing a handle carries, next to the optional url naming where the bytes live. It answers both questions a caller has: what representation the bytes are, and what content codings sit on top.

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::{Codec, MimeType, Url};

// A declared type wins, and the codings it carries are what `codec` reports.
let named = Buffer::new().with_media_type(Url::from_str("file:///trades.json.gz")?.media_type());
assert_eq!(named.media_type().base(), &MimeType::JSON);
assert_eq!(named.codec(), Codec::Gzip);

codec reads the last coding out of the media type, which is how compression is never passed as a separate argument. set_media_type declares one explicitly, which is required for a format that content cannot identify. Both cross into the bindings - codec as a read-only property, media_type as a settable one - and the next section is what a caller does with the coding once it has been read.

Adding and removing a coding

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::{Codec, Url};

let mut plain = Buffer::new().with_media_type(Url::from_str("file:///rows.json")?.media_type());
plain.write_all_bytes(br#"{"symbol":"AAPL"}"#)?;
// Nothing wraps these bytes, so there is nothing to undo.
assert_eq!(plain.codec(), Codec::Identity);

let mut encoded =
    Buffer::new().with_media_type(Url::from_str("file:///rows.json.gz")?.media_type());
assert_eq!(encoded.codec(), Codec::Gzip);

// The coding is an argument here, and the target's name is one place to read it from.
let codec = encoded.codec();
plain.compress_into(&mut encoded, codec)?;
assert_eq!(&encoded.read_all_bytes()?[..2], b"\x1f\x8b");

let mut decoded = Buffer::new();
encoded.decompress_into(&mut decoded)?;
assert_eq!(decoded.read_all_bytes()?, plain.read_all_bytes()?);
assert_eq!(decoded.codec(), Codec::Identity);

import pathlib
import tempfile

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())
plain = IOBase(root / "rows.json")
plain.write_bytes(b'{"symbol":"AAPL"}')

# Nothing wraps these bytes, so there is nothing to undo.
assert plain.codec is None

encoded = IOBase(root / "rows.json.gz")
assert encoded.codec == "gzip"

# The target's name already said gzip, so nothing here repeats it.
assert plain.compress_into(encoded) == encoded.size
assert encoded.read_bytes()[:2] == b"\x1f\x8b"

decoded = IOBase(root / "roundtrip.json")
assert encoded.decompress_into(decoded) == 17
assert decoded.read_bytes() == plain.read_bytes()
assert decoded.codec is None

# A target declaring no coding is refused rather than copied unchanged.
reason = None
try:
    plain.compress_into(IOBase(root / "copy.json"))
except ValueError as error:
    reason = str(error)
assert "expected a target declaring a content coding" in reason
assert not (root / "copy.json").exists()

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
const plain = new IOBase(path.join(root, 'rows.json'))
plain.writeBytes(Buffer.from('{"symbol":"AAPL"}'))

// Nothing wraps these bytes, so there is nothing to undo.
assert.equal(plain.codec, null)

const encoded = new IOBase(path.join(root, 'rows.json.gz'))
assert.equal(encoded.codec, 'gzip')

// The target's name already said gzip, so nothing here repeats it.
assert.equal(plain.compressInto(encoded), encoded.size)
assert.deepEqual([...encoded.readBytes().subarray(0, 2)], [0x1f, 0x8b])

const decoded = new IOBase(path.join(root, 'roundtrip.json'))
assert.equal(encoded.decompressInto(decoded), 17)
assert.equal(decoded.readText(), '{"symbol":"AAPL"}')
assert.equal(decoded.codec, null)

// An in-memory target has no name to declare a coding, so this one is named.
const memory = IOBase.fromBytes()
assert.ok(plain.compressInto(memory, 'zstd') > 0)
assert.equal(memory.codec, 'zstd')

// A target declaring no coding is refused rather than copied unchanged.
assert.throws(
  () => plain.compressInto(new IOBase(path.join(root, 'copy.json'))),
  /expected a target declaring a content coding/,
)
assert.equal(fs.existsSync(path.join(root, 'copy.json')), false)

fs.rmSync(root, { recursive: true, force: true })

codec answers what coding the handle's own name declares - rows.json.gz is gzip - and an in-memory handle answers off the media type it was told to hold. Absence is spelled as absence: Python answers None and JavaScript null, never the string "identity", because the question a caller is asking here is whether there is anything to undo, and a sentinel spelling of "no" is one more value every branch has to know about.

compress_into and decompress_into - compressInto and decompressInto in JavaScript - move every byte from one handle into another and add or remove a coding on the way. The coding defaults to the one a name already declares: the target's for a compress, the source's for a decompress, so writing a rows.json into a rows.json.gz never spells gzip twice, and reading it back needs no argument at all. That works because the transfer records the coding in the target's media type, which is the same place a located handle reads it from. Naming a coding explicitly overrides the default - the escape hatch for an in-memory target, which has no name to declare anything, and for bytes whose name lies about what they hold. level is the shared 0-9 scale.

Both bindings refuse a compress into a target that declares no coding rather than writing the bytes through unchanged, and the refusal names the media type the target does carry. A coding nobody named is a coding nobody can decode by name later, so a silent identity copy is a failure that surfaces one reader downstream instead of here. Rust asks for the coding outright, so the question never arises: there is no name to default from until a caller reads one, which is what the Rust example above does in the open.

None of this sits on the reading path. Record encodings and text codecs already read through the codings a name declares - trades.arrows.gz writes gzipped and reads straight back as batches, text.md's load peels the same coding off a quote.json.gz, and plain-text records decode coded lines as a stream. These two methods are for moving bytes between representations: publishing a plain file as a compressed one, or handing an outside tool a form it can open. What coding sits on the bytes is not something a reader has to think about. The codings themselves are documented per format in coding.md, coding.md, and coding.md.

Open and close

The scoped pair is exposed in all three runtimes: Rust calls open / close directly, Python binds them to a context manager, and JavaScript exposes the same methods (plus Symbol.dispose where the runtime provides it).

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::coding::Coded;
use yggdryl::Codec;

let mut handle = Coded::wrap(Buffer::new(), Codec::Zstd);
assert!(!handle.opened());

handle.open()?;
assert!(handle.opened());
handle.write_all_bytes(b"symbol,price\n")?;

// Closing publishes the pending write and releases the cache.
handle.close()?;
assert!(!handle.opened());

// The handle stays usable; the next read re-materializes.
assert_eq!(handle.read_all_bytes()?, b"symbol,price\n");

import pathlib
import tempfile

from yggdryl import IOBase

path = pathlib.Path(tempfile.mkdtemp()) / "trades.csv"

# `with` is the scoped pair: `__enter__` opens and `__exit__` closes.
with IOBase(path) as handle:
    handle.write_text("symbol,price\n")
    assert handle.opened
    assert not handle.closed

# Closing published the bytes at their exact length, which is what another
# reader needs; the handle stays usable and simply re-materializes.
assert path.stat().st_size == 13
assert IOBase(path).read_text() == "symbol,price\n"

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
const target = path.join(root, 'value.bin')
fs.writeFileSync(target, 'symbol,price\n')

const handle = new IOBase(target)
handle.open()
assert.equal(handle.opened(), true)
handle.close()
assert.equal(handle.closed(), true)

fs.rmSync(root, { recursive: true, force: true })

A handle works without open: every operation materializes what it needs. Calling it moves that cost to a known point and keeps the cached state alive across many small operations instead of re-deriving it per call. Opening an already-open handle is a no-op, and opening a resource that does not exist yet succeeds without creating it - creation still waits for the first write. close flushes and releases what open cached; the handle remains usable afterwards.

The cache is strictly opt-in, and that is the whole contract: open caches, closed fetches. A closed handle re-derives its metadata on every ask, so a resource that changes underneath it is seen immediately; an open one holds what open cached until close, which is what makes many small operations against one resource cheap. Nothing fills the cache as a side effect of an ordinary read, because a cache nobody asked for is how a handle serves a stale answer.

What is cached depends on the implementation. Buffer has nothing to cache, so the trait defaults apply and opened stays false. local::File caches the descriptor and the memory mapping, while Coded retains the decoded value only for an explicitly opened session. Media wrappers retain the metadata their format keeps re-reading: IPC schema and dimensions, the Parquet footer, Avro header and block metadata, or Text's resolved field, coding plan, and dimensions. These are the operations a scoped context binds to, which is why the bindings can map __enter__ / __exit__ and open / close onto the same core lifecycle.

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

target = pathlib.Path(tempfile.mkdtemp()) / "lake" / "trades.parquet"
IOBase(target).overwrite_arrow_table(pa.table({"id": [1, 2], "venue": ["XNAS", "XNYS"]}))

# Metadata-heavy work belongs inside the scope: the schema probe, the
# per-batch reads, and the size checks all reuse what `open` cached, and
# `close` releases it at a known point.
rows = 0
with IOBase(target) as handle:
    field = handle.read_arrow_field()
    for batch in handle.read_arrow_reader():
        rows += batch.num_rows
assert rows == 2

# Outside a scope the same calls still work - each one just fetches fresh,
# which is exactly right for a resource another writer may be changing.
assert IOBase(target).read_arrow_field() == field

Clearing and removing

clear and remove are the lifecycle pair, and what they mean is stated by kind rather than implied by a byte operation. clear empties the contents and keeps the resource: a leaf keeps existing with size 0, a container keeps existing and loses every child recursively. remove deletes the resource itself - completely, whatever "the resource" is for that handle. A wrapping handle removes what it wraps: Gzip::remove deletes the .gz resource rather than discarding a decoded buffer, and a media handle's cached schema or footer goes with it.

Two rules hold on both, and they are the reason the pair exists at all:

• Absence is a no-op success. Neither call is an error on a resource that is not there, and neither creates one. clear is not a write.
• Absence is reached without a probe. An implementation issues the delete and treats the backend's own not-found answer as success. It never calls kind, size, an exists check, or ls first to decide whether to proceed - on a remote backend each of those is a second round trip, and a recursive delete would become a flood of them. Only not-found maps to success; a permission, network, or busy failure stays the typed error it is.

Because absence and successful removal are indistinguishable, remove returns nothing rather than a bool or a count - reporting "did something exist" would force exactly the probe this refuses. recursive is ignored on a leaf, and a container that still has children is refused by name rather than silently recursed into.

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::holder::local::Folder;

let root = Folder::temporary()?.path()?.join(format!("yggdryl-docs-lifecycle-{}", std::process::id()));
let mut folder = Folder::new(&root)?;
folder.truncate(0)?;
folder.child_by_path("a.log")?.write_all_bytes(b"line\n")?;

// Clearing empties the container and keeps it.
folder.clear()?;
assert_eq!(folder.ls(true, false).count(), 0);
assert_eq!(folder.kind(), yggdryl::IOKind::Directory);

// Removing deletes it; a second call succeeds, having done nothing. A
// handle asked for as a container keeps answering `Directory`, because that
// is what it was asked for - the parent's listing is what shows it gone.
let leaf = root.join("nested");
let mut nested = Folder::new(&leaf)?;
nested.truncate(0)?;
assert_eq!(folder.ls(false, false).count(), 1);
nested.remove(false)?;
nested.remove(false)?;
assert_eq!(folder.ls(false, false).count(), 0);
folder.remove(false)?;

// A wrapping handle removes what it wraps, cache included.
let mut coded = yggdryl::coding::gzip::Gzip::new(Buffer::new());
coded.write_all_bytes(b"symbol,price\n")?;
coded.remove(false)?;
assert_eq!(coded.size(), 0);

import pathlib
import tempfile

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())
handle = IOBase(root / "logs")
handle.mkdir()
(handle / "a.log").write_text("line\n")

# Clearing empties the container and keeps it.
handle.clear()
assert list(handle.iterdir()) == []
assert handle.is_dir()

# Removing deletes it; a second call succeeds, having done nothing. A
# handle asked for as a container keeps answering `is_dir`, because that
# is what it was asked for - the parent's listing is what shows it gone.
handle.remove()
handle.remove()
assert list(IOBase(root).iterdir()) == []

# A container that still has children is refused rather than recursed into.
handle.mkdir()
(handle / "a.log").write_text("line\n")
try:
    handle.remove()
except Exception as error:
    assert "children" in str(error)
handle.remove(recursive=True)
assert list(IOBase(root).iterdir()) == []

# The handle stays usable and lazy - a write recreates the resource.
leaf = IOBase(root / "trades.csv")
leaf.write_text("symbol,price\n")
leaf.remove()
assert not leaf.exists()
leaf.write_text("symbol,price\n")
assert leaf.read_text() == "symbol,price\n"

const assert = require('node:assert')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = path.join(os.tmpdir(), `yggdryl-docs-lifecycle-${process.pid}`)
const handle = new IOBase(path.join(root, 'logs'))
handle.mkdir()
handle.joinpath(['a.log']).writeText('line\n')

// Clearing empties the container and keeps it.
handle.clear()
assert.equal([...handle.ls(true, false)].length, 0)

// Removing deletes it; a second call succeeds, having done nothing.
handle.remove()
handle.remove()
assert.equal([...new IOBase(root).ls(false, false)].length, 0)

// The handle stays usable and lazy - a write recreates the resource.
const leaf = new IOBase(path.join(root, 'trades.csv'))
leaf.writeText('symbol,price\n')
leaf.remove()
assert.equal(leaf.exists(), false)
leaf.writeText('symbol,price\n')
assert.equal(leaf.readText(), 'symbol,price\n')
new IOBase(root).remove(true)

Table decides both deliberately rather than inheriting a folder's answers. clear commits one snapshot carrying no data files, so the table still exists with its schema, properties, and history and holds zero rows - deleting files behind the manifests would leave exactly the orphaned state a complete operation must not. remove deletes the table's whole location, metadata tree and data files together, because dropping a table is not emptying it.

Buffer

Rust only

The bindings reach the in-memory implementation through IOBase.from_bytes, not through a Buffer class of their own.

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::MimeType;

let mut handle = Buffer::with_capacity(1_024);
handle.reserve(4_096)?;
assert!(handle.capacity() >= 4_096);
// Reserving changes the allocation, never the length.
assert_eq!(handle.size(), 0);

handle.pwrite(0, b"symbol,price\n")?;
assert_eq!(handle.as_slice(), b"symbol,price\n");

// A format the bytes cannot identify is declared rather than guessed.
let csv = Buffer::from_bytes(handle.into_bytes()).with_media_type(MimeType::CSV.into());
assert_eq!(csv.media_type().base(), &MimeType::CSV);

Buffer is the in-memory implementation, and the one every example and test reaches for. The allocation doubles rather than growing exactly, so appending in many small writes stays amortized constant; reserve pre-sizes it when the final length is known. as_slice, as_mut_slice, and into_bytes reach the bytes directly, and taking a mutable slice discards any inferred media type, because the content's identity may change through it.

A buffer is not stored anywhere, so url reports a synthetic mem: identity naming the process and the allocation - enough to tell two live buffers apart in a log, without pretending the bytes live somewhere. The other implementations in the core are holder.md, the memory-mapped local tree, and holder.md, which puts any existing Arrow filesystem - S3, GCS, Azure, or one you wrote - behind the same trait. Anything else is a sibling module supplying the same three roles, never a change to this one. Two wrapping handles sit over any of them and are handles themselves: Coded below, and the page cache in holder.md.

Coded

Rust only

The Python and JavaScript packages do not expose the compression wrappers.

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::coding::Coded;
use yggdryl::{Codec, Level, MimeType, Url};

let inner = Buffer::new().with_media_type(Url::from_str("file:///trades.arrows.gz")?.media_type());
let mut handle = Coded::wrap(inner, Codec::Gzip).with_level(Level::BEST);

// The wrapper's bytes are decoded, so its media type has the coding removed.
assert_eq!(handle.media_type().base(), &MimeType::ARROW_STREAM);
assert_eq!(handle.media_type().encoding_len(), 0);

let payload = "symbol,price\n".repeat(64).into_bytes();
handle.write_all_bytes(&payload)?;
handle.flush()?;

// Reads decompress; the wrapped handle only ever holds the encoded form.
assert_eq!(handle.read_all_bytes()?, payload);
assert!(handle.handle().size() < payload.len() as u64);

Coded wraps any handle and presents the decoded bytes: reads decompress, writes compress. It is an IOBase itself, so it goes anywhere a handle goes. The per-format aliases in coding.md, coding.md, and coding.md are this type with the codec already chosen.

A content coding is not seekable, which forces two tradeoffs. The decoded value is materialized once and held until close, so positional reads and writes work at all over a compressed payload; and a write is published to the wrapped handle on flush or close, not on every pwrite. into_handle publishes and returns the wrapped handle. Codec::Identity makes the wrapper a pass-through.

Buffered

Rust exposes the typed Buffered<H> wrapper. Python and JavaScript keep their generic IOBase identity and configure the same core cache through buffered(page_size=..., max_bytes=..., ttl=...) or buffered({ pageSize, maxBytes, ttlMs }); repeated calls replace the options without stacking caches.

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let handle = Buffer::from_bytes(vec![4_u8; 4_096]).buffered(BufferedOptions::default());

// The first read fetches the page holding the range; the second is memory.
assert_eq!(handle.read_range_bytes(0, 8)?, [4_u8; 8]);
assert_eq!(handle.read_range_bytes(2_000, 8)?, [4_u8; 8]);
assert_eq!(handle.cached_pages(), 1);

Buffered is the other wrapping handle: it serves reads from fixed-size pages under a byte budget and a time to live, writes through, and pins the value's first and last pages so a footer-first container never re-reads either end. Everything else it mirrors. holder.md is the page.

Roles

Rust only

The bindings expose one handle class rather than the three role traits behind it.

use yggdryl::IOBase;
use yggdryl::{IOKind, MimeType};
use yggdryl::holder::local;

let path = local::Folder::temporary()?.path()?.join("yggdryl-docs-io-folder");
let _ = std::fs::remove_dir_all(&path);
let mut folder = local::Folder::new(&path)?;

// A container holds no bytes: reads are empty, byte writes are refused.
let mut probe = [0_u8; 4];
assert_eq!(folder.pread(0, &mut probe)?, 0);
assert_eq!(folder.size(), 0);
let refused = folder.pwrite(0, b"x").unwrap_err().to_string();
assert!(refused.contains("got the directory"), "{refused}");

// Truncating to zero is the write that brings a container into being.
folder.truncate(0)?;
assert!(folder.exists());
assert_eq!(folder.kind(), IOKind::Directory);
assert_eq!(folder.media_type().base(), &MimeType::DIRECTORY);
assert_eq!(folder.ls(false, false).count(), 0);

std::fs::remove_dir_all(&path)?;

IOBase says how to move bytes. The three role traits say what a resource is, and each declares only what the backend alone knows, pre-implementing the rest as methods the backend's IOBase impl forwards to:

• IOFolder - a container. Declares folder_url, folder_exists, create_folder, list_folder. Pre-implements folder_pread (reads nothing), folder_pwrite (refuses, naming the container), folder_truncate (creates on 0, errors otherwise), folder_media_type (inode/directory), and folder_kind (Directory).
• IOFile - a leaf. Declares file_url and file_exists. Pre-implements file_ls (lists nothing), file_child_by_path (refuses, naming the file), and file_kind (File when it exists, Unknown when it does not).
• IOPath - a location whose role is not resolved yet. Declares path_url, is_folder, is_file. Pre-implements path_exists, path_kind (Directory, File, or Unknown), and path_media_type (the container type, or the one the name implies).

use yggdryl::IOBase;
use yggdryl::{IOKind};
use yggdryl::holder::local;

// A location that arrived from outside answers by looking at what is there.
let existing = local::Path::new(local::Folder::temporary()?.path()?)?;
assert_eq!(existing.kind(), IOKind::Directory);

let undecided = local::Path::new(local::Folder::temporary()?.path()?.join("yggdryl-docs-io-undecided"))?;
assert_eq!(undecided.kind(), IOKind::Unknown);
assert!(undecided.read_all_bytes()?.is_empty());

// A leaf is not a container: it lists nothing and resolves no child.
let leaf = local::File::new(local::Folder::temporary()?.path()?.join("yggdryl-docs-io-leaf.arrows"))?;
assert_eq!(leaf.ls(true, false).count(), 0);
assert!(leaf.child_by_path("nested").is_err());

parent, child_by_path, and ls return types.md's Holder, which is why a walk over a tree needs no type parameter. A resource that cannot contain others lists nothing rather than failing, so a caller can walk without testing each node first. local::Path is the reference IOPath: it resolves by looking, and a byte write is what settles an undecided location into a file. A remote store is the same three roles over a different transport.

Delegating to a wrapped handle

Rust only

A backend implements IOBase in Rust; neither binding can add one.

use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;

/// A wrapper mirrors the handle's bytes rather than owning bytes of its own.
struct Wrapped {
    handle: Buffer,
}

impl IOMedia for Wrapped {
    yggdryl::delegate_iomedia!(handle);
}

impl IOBase for Wrapped {
    yggdryl::delegate_iobase!(handle);
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut wrapper = Wrapped {
        handle: Buffer::new(),
    };
    wrapper.open()?;
    wrapper.write_all_bytes(b"AAPL")?;

    assert_eq!(wrapper.opened(), wrapper.handle.opened());
    assert_eq!(wrapper.read_all_bytes()?, b"AAPL");
    assert_eq!(wrapper.handle.as_slice(), b"AAPL");
    Ok(())
}

delegate_iobase! forwards the storage contract, including open, opened, and close; it does not forward record behavior. delegate_iomedia! independently forwards dimensions, options, Field/reader reads, and typed writes. A wrapper that changes any forwarded method uses the explicit list form and leaves that method out. media.md, media.md, and the compression handles choose the lists that match the state each wrapper owns.

It has two other spellings, for two other shapes:

• delegate_iobase!(handle, except_lifecycle) omits clear, remove, is_atomic, is_tabular, and is_io. A caching wrapper writes the first pair so invalidation is part of the operation; a record-encoding wrapper answers the three surface questions from its representation instead of mirroring the bytes beneath it. Ipc, Parquet, and the text handler are this shape.
• delegate_iobase!(handle: pread, size, ...) names the methods to mirror one by one, for a wrapper that changes one of them - a method cannot be both expanded by the macro and written out underneath it. Buffered is this shape: it owns the two positional primitives, the resize that invalidates, the open/close pair, and the clear/remove pair, and mirrors the rest. A method left out of the list falls back to the trait's own default, which for clear and remove means truncating rather than reaching the resource - so leave them out only when writing them.

Arrow batches

RustPythonJavaScript

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, StringArray};
use yggdryl::arrow;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{DataType, Url};

// A non-null struct Field is the schema.
let schema = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.nullable_field("symbol"),
])?
.required_field("row");

let arrow_schema = schema.clone().into_arrow_schema()?;
let batch = RecordBatch::try_new(
    Arc::clone(&arrow_schema),
    vec![
        Arc::new(Int64Array::from(vec![1, 2])),
        Arc::new(StringArray::from(vec![Some("AAPL"), None])),
    ],
)?;

// The handle's own media type picks the encoding; no format argument is passed.
let mut handle = Buffer::new().with_media_type(Url::from_str("file:///trades.arrows")?.media_type());
let options = handle.record_options()?;

// Overwrite takes a batch reader; the method name fixes the write intent.
handle.overwrite_arrow_reader(arrow::batch_reader(arrow_schema, [batch]), &options)?;
assert_eq!(handle.read_arrow_field(&options)?, schema);
assert_eq!((handle.row_size()?, handle.column_size()?), (2, 2));

// The read path returns one. Batches arrive one at a time, never as a vector.
let mut rows = 0;
for batch in handle.read_arrow_reader(&options)? {
    rows += batch?.num_rows();
}
assert_eq!(rows, 2);

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

# A PyArrow schema is the schema; the binding imports it once at the boundary.
schema = pa.schema([
    pa.field("id", pa.int64(), nullable=False),
    pa.field("symbol", pa.string()),
])
batch = pa.record_batch({"id": [1, 2], "symbol": ["AAPL", None]}, schema=schema)

# The handle's own media type picks the encoding; no format argument is passed.
handle = IOBase(pathlib.Path(tempfile.mkdtemp()) / "trades.arrows")
options = handle.record_options()

# The write path takes a batch reader and nothing else.
handle.overwrite_arrow_batch(batch, options=options)
assert handle.read_arrow_field(options=options).name == "row"

# The read path returns one. Batches arrive one at a time, never as a vector.
rows = sum(part.num_rows for part in handle.read_arrow_reader(options=options))
assert rows == 2

const assert = require('node:assert/strict')
const arrow = require('apache-arrow')
const { BatchReader, Field, IOBase, MimeType, fields } = require('yggdryl')

// A non-null struct Field is the schema.
const schema = fields.struct(
  'row',
  [Field.from('id: int64'), Field.from('symbol: utf8')],
  { nullable: false },
)

const table = new arrow.Table({
  id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
  symbol: arrow.vectorFromArray(['AAPL', null], new arrow.Utf8()),
})

// The handle's own media type picks the encoding; no format argument is passed.
const handle = IOBase.fromBytes()
handle.mediaType = MimeType.ARROW_STREAM
const options = handle.recordOptions()

// The write path takes a batch reader and nothing else.
handle.overwriteArrowReader(BatchReader.from(table), options)
assert.ok(handle.readArrowField(options).equals(schema))

// The read path returns one. Batches arrive one at a time, never as a vector.
let rows = 0
for (const batch of handle.readArrowReader(options)) {
  rows += batch.numRows
}
assert.equal(rows, 2)

Logical dimensions

The media surface exposes two lazy dimensions:

row_size(&self) -> Result<u64>
column_size(&self) -> Result<usize>

They describe the whole logical media, so temporary projection, filter, and row-limit settings do not change either answer. Schema-bearing encodings read headers or footers and never decode row arrays; text streams its configured multiline extractor because only the extractor can identify a record boundary. A folder sums matching leaves, while a table format answers from its current metadata. An explicitly opened media caches the metadata answer until close; a closed handle asks the resource afresh each time. A declared Struct field remains authoritative for column_size, even when the resource has no rows. Python exposes the same values as row_size and column_size properties; JavaScript uses rowSize and columnSize getters.

The same handles expose the core storage role without reconstructing it from is_dir / is_file: Rust returns IOKind, Python's kind property and JavaScript's kind getter return its canonical lowercase name (memory, file, directory, table, namespace, catalog, or unknown). This is a cheap redirection to the native handle and is the observable answer generic dispatch uses.

cargo bench -p yggdryl --bench io --all-features -- io_dimensions compares fresh metadata reads with the explicitly opened cache for both accessors across IPC, Parquet, Avro, and multiline text fixtures.

Canonical record-write signatures

The primitive record surface is one read and three explicit write intents. Each typed shape also has one required-mode dispatcher. The Rust argument order - input, then mode when present, then options - is canonical:

overwrite_arrow_reader(&mut self, reader: BatchReader, options: &RecordOptions) -> Result<()>
append_arrow_reader(&mut self, reader: BatchReader, options: &RecordOptions) -> Result<()>
merge_arrow_reader(&mut self, reader: BatchReader, options: &RecordOptions) -> Result<()>

overwrite_arrow_batch(&mut self, batch: RecordBatch, options: &RecordOptions) -> Result<()>
append_arrow_batch(&mut self, batch: RecordBatch, options: &RecordOptions) -> Result<()>
merge_arrow_batch(&mut self, batch: RecordBatch, options: &RecordOptions) -> Result<()>

overwrite_records(&mut self, records, options: &RecordOptions) -> Result<()>
append_records(&mut self, records, options: &RecordOptions) -> Result<()>
merge_records(&mut self, records, options: &RecordOptions) -> Result<()>

write_arrow_reader(&mut self, reader: BatchReader, mode: IOMode, options: &RecordOptions) -> Result<()>
write_arrow_batch(&mut self, batch: RecordBatch, mode: IOMode, options: &RecordOptions) -> Result<()>
write_records(&mut self, records, mode: IOMode, options: &RecordOptions) -> Result<()>

The mode-selected method validates intent before touching its input, then selects the same-shape explicit method. That keeps a specialized held-batch or native-row adapter authoritative while all of those adapters still converge on the three reader primitives. There is no default mode and no untyped Rust write alias.

cargo bench -p yggdryl --bench io --all-features -- io_write_mode_dispatch measures reader, held-batch, and native-row dispatch under all three modes. Each append and merge sample receives its stored side in Criterion setup, so the timer reports the selected operation rather than fixture construction.

One local Windows x86_64 release run over 4,096 rows (Criterion point estimates; regenerate on the deployment host):

generic dispatcher	overwrite	append	merge
write_arrow_reader	255 us (16.1M rows/s)	703 us (5.83M rows/s)	3.27 ms (1.25M rows/s)
write_arrow_batch	92.0 us (44.5M rows/s)	637 us (6.43M rows/s)	4.14 ms (989k rows/s)
write_records	3.00 ms (1.36M rows/s)	4.44 ms (922k rows/s)	8.96 ms (457k rows/s)

The mode branch itself is shared by all three columns in a row. The larger differences are the selected operation: append re-encodes the stored side, merge indexes it by key, and native records additionally validate and materialize ordered Scalar::Sequence rows.

Bindings keep the input-before-options order and the same intent errors while using their native naming and Arrow holders. Rust requires &RecordOptions; when a binding omits its optional settings value, the binding resolves exactly one default with handle.record_options() before redirecting to Rust. Python makes that one settings value keyword-only:

read_arrow_reader(*, options=None) -> pyarrow.RecordBatchReader
read_records(cls=None, *, options=None) -> Iterator[dict | dataclass]
overwrite|append|merge_arrow_reader(reader, *, options=None) -> None
overwrite|append|merge_arrow_table(table, *, options=None) -> None
overwrite|append|merge_arrow_batch(batch, *, options=None) -> None
overwrite|append|merge_records(records, *, options=None) -> None
write_arrow_reader(reader, mode, *, options=None) -> None
write_arrow_table(table, mode, *, options=None) -> None
write_arrow_batch(batch, mode, *, options=None) -> None
write_records(records, mode, *, options=None) -> None

JavaScript uses the same trailing optional value:

readArrowReader(options?) -> BatchReader
readRecords(options?) -> Iterable<object>
readRecords(cls, options?) -> Iterable<object>
overwrite|append|mergeArrowReader(reader, options?) -> void
overwrite|append|mergeArrowTable(table, options?) -> void
overwrite|append|mergeArrowBatch(batch, options?) -> void
overwrite|append|mergeRecords(records, options?) -> void | Promise<void>
writeArrowReader(reader, mode, options?) -> void
writeArrowTable(table, mode, options?) -> void
writeArrowBatch(batch, mode, options?) -> void
writeRecords(records, mode, options?) -> void | Promise<void>

There are no flattened record-option keywords and no mode inference from merge_by_names: options is the only settings argument in both bindings. Each shape-named adapter rejects a different shape before crossing into the core. A non-empty row source may infer options.field from its decorated field class; an empty source has no class to inspect and therefore requires a field on the supplied options.

overwrite_arrow_reader is the required publication hook for an implementor. The default append and merge implementations cast the incoming stream to options.field once, apply selection and limits once, and then remove the field from a cloned options value before delegating the resulting stream to overwrite. That handoff is deliberate: its rows already have the declared shape, so the publication hook must not cast them to the same field a second time.

options.commit_row_size is the one optional streamed-write publication boundary. Unset, the operation publishes once when the source ends. A non-zero N publishes every complete group of N incoming rows and the final remainder. The splitter slices Arrow batches as views, stops pulling as soon as one cadence is full, and holds only that bounded cadence plus the current input remainder. Shaping still happens once: options.field, selection, and limits are applied before splitting, then both the field and cadence are removed from the delegated options.

Overwrite uses overwrite for the first commit and append for every later one; append and merge retain their intent for every commit. A successful prefix is therefore intentionally visible if conversion, decoding, encoding, or publication fails later. An empty overwrite still publishes its shaped field; empty append and merge remain no-ops. N = 0 is rejected before the input is pulled. Native Rust row conversion is bounded by the smaller of options.batch_row_size (or its default) and commit_row_size, so a failure at row N + 1 cannot erase the completed N-row prefix.

“Publishes once” is one native publication for a leaf or table. A plain folder has no cross-leaf transaction, rename, or compare-and-swap in IOBase, so it keeps the source streaming and publishes each routed leaf independently even when the cadence is unset. A failure can expose only the leaf prefixes already completed; setting commit_row_size additionally bounds how many incoming rows can belong to each visible prefix. An Iceberg folder is redirected first and uses its snapshot commit instead of this per-leaf exception.

The defaults stay streaming. Append chains the stored reader before the incoming reader without collecting either. Merge must index the stored side so it can find a key after its reader has advanced, but it continues to consume and fold the incoming side one batch at a time; it never materializes the incoming reader.

Intent is validated before any input is consumed. Overwrite and append reject a non-empty merge_by_names and direct the caller to merge. Merge requires at least one merge_by_names column and directs a keyless caller to overwrite or append. The key setting supplies row identity only; it never chooses the mode.

IOMedia::read_arrow_reader returns arrow.md's BatchReader, and all three write methods consume one. Together these primitives are the only place an encoding is decoded and encoded; every wider record operation routes through them. arrow::batch_reader is the constructor that turns batches a caller already has - a Vec, an array, a lazily-computed iterator - into one. The primitive core stays batch-native; record, table, and record-batch adapters infer and cast at their runtime boundary, then widen their input into this same reader instead of adding another encoder.

The Rust *_records triplet takes any iterator whose row implements TryInto<Scalar>. A canonical row is one ordered Scalar::Sequence under options.field; a sorted name-to-value Scalar::Record is accepted as input and resolved to that order. The field is required, and there is no separate record/schema model. An infallible Rust struct implements From<Row> for Scalar and receives the standard blanket TryInto implementation. Conversion is lazy, batches hold at most options.batch_row_size rows and never more than options.commit_row_size when a cadence is set, and the resulting exact-schema reader redirects to the same three primitives above.

use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{DataType, MimeType, Scalar};

struct Quote(i32, &'static str);

impl From<Quote> for Scalar {
    fn from(row: Quote) -> Self {
        Scalar::from_sequence([Scalar::from(row.0), Scalar::from(row.1)])
    }
}

let field = DataType::from_fields([
    DataType::Int32.required_field("id"),
    DataType::Utf8.required_field("symbol"),
])?
.required_field("quote");
let mut handle = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
let options = handle.record_options()?.with_field(field);

handle.overwrite_records([Quote(1, "AAPL"), Quote(2, "MSFT")], &options)?;
handle.append_records([Quote(3, "AMD")], &options)?;
handle.merge_records(
    [Quote(2, "MSFT.O")],
    &options.clone().with_merge_by_names(["id"]),
)?;
assert_eq!(
    handle
        .read_arrow_reader(&options)?
        .map(|batch| batch.unwrap().num_rows())
        .sum::<usize>(),
    3,
);

record_options derives the encoding's settings from the handle's media type, so the encoding is never guessed - it is whatever the handle already says it holds. read_arrow_field returns the canonical non-null struct root Field described in types.md. All of this is behind the default arrow feature.

Content coding stays the handle's business. A handle named trades.arrows.zst round-trips compressed through the same methods, because the coding is in its media type and no call takes a coding argument.

RustPythonJavaScript

use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::MimeType;

// An absent resource holds no batches rather than failing to parse.
let empty = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
assert_eq!(
    empty.read_arrow_reader(&empty.record_options()?)?.count(),
    0
);

// An encoding this build does not implement is named rather than guessed.
let csv = Buffer::new().with_media_type(MimeType::CSV.into());
let message = csv.record_options().unwrap_err().to_string();
assert!(message.contains("text/csv"), "{message}");

import pathlib
import tempfile

import pytest

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())

# An absent resource holds no batches rather than failing to parse.
empty = IOBase(root / "absent.arrows")
assert empty.read_arrow_reader().read_all().num_rows == 0

# An encoding this build does not implement is named rather than guessed.
csv = IOBase(root / "trades.csv")
with pytest.raises(ValueError, match="text/csv"):
    csv.record_options()

const assert = require('node:assert/strict')
const { IOBase, MimeType } = require('yggdryl')

// An absent resource holds no batches rather than failing to parse.
const empty = IOBase.fromBytes()
empty.mediaType = MimeType.ARROW_STREAM
assert.equal([...empty.readArrowReader()].length, 0)

// An encoding this build does not implement is named rather than guessed.
const csv = IOBase.fromBytes()
csv.mediaType = MimeType.CSV
assert.throws(() => csv.recordOptions(), /text\/csv/)

The encodings record_options can return are the ones this build carries: Arrow IPC (media.md), Avro (media.md), and plain-text line media (text.md) under the default Arrow feature, plus Apache Parquet under the non-default parquet feature (media.md). Shared settings include the declared field, root name, cast strictness, batch and total limits, commit cadence, compression level, merge keys, selection, and partition filters; IORecordOptions defines them once.

Rows as JavaScript objects

JavaScript example

JavaScript's object-and-constructor behavior is shown below. Python exposes the corresponding lazy read_records mapping/dataclass view and all three record-write intents; Rust accepts typed TryInto<Scalar> rows on writes and keeps its primitive read surface Arrow-native.

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
const handle = new IOBase(path.join(root, 'trades.arrows'))
// Plain objects are rows; `overwriteRecords` widens them into the Arrow stream.
handle.overwriteRecords([
  { id: 1n, venue: 'XNAS' },
  { id: 2n, venue: null },
])

// Plain objects out, streamed batch by batch ...
assert.deepEqual([...handle.readRecords()].map((row) => row.id), [1n, 2n])

// ... or instances of any class whose constructor takes the plain row.
class Trade {
  constructor(row) {
    Object.assign(this, row)
  }
}
const trades = [...handle.readRecords(Trade)]
assert.ok(trades.every((t) => t instanceof Trade))

// An absent resource yields no records rather than raising.
assert.deepEqual([...new IOBase(path.join(root, 'absent.arrows')).readRecords()], [])

fs.rmSync(root, { recursive: true, force: true })

readRecords returns plain objects or passes each one to the constructor it was given. overwriteRecords, appendRecords, and mergeRecords widen rows into the matching explicit reader intent. An absent resource reads as empty and an empty write is a no-op, so neither path needs an existence guard.

Lazy scans

Python only

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

target = pathlib.Path(tempfile.mkdtemp()) / "trades.parquet"
IOBase(target).overwrite_arrow_table(pa.table({"symbol": ["AAPL", "MSFT"], "price": [187.23, 402.11]}))

# A local Parquet leaf becomes the real lazy scan - projection and
# predicate pushdown belong to the engine, and the handle publishes its
# bytes at their exact length first so the foreign reader sees a whole file.
lazy = IOBase(target).scan_polars()
assert lazy.select("symbol").head(10).collect().height == 2

# The pyarrow spelling of the same idea, as a dataset Scanner.
scanner = IOBase(target).scan_arrow()
assert scanner.to_table().num_rows == 2

# Anything a foreign scanner cannot mmap - an in-memory buffer, a
# compressed name, an Arrow stream - streams through the native reader
# instead, so both calls answer for every holder.
memory = IOBase.from_bytes()
memory.media_type = "application/vnd.apache.arrow.stream"
memory.overwrite_arrow_table(pa.table({"symbol": ["AAPL"]}))
assert memory.scan_arrow().to_table().num_rows == 1

scan_polars hands back a polars.LazyFrame and scan_arrow a pyarrow.dataset.Scanner. A plain local Parquet resource is scanned natively; everything else reads through the same native reader every other read uses and arrives as the lazy shape anyway, so callers never branch on where the bytes live.

Column pushdown

RustPythonJavaScript

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, RecordBatchReader, StringArray};
use yggdryl::arrow;
use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{DataType, MimeType};

let stored = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.required_field("symbol"),
    DataType::Utf8.required_field("venue"),
])?
.required_field("row");
let arrow_schema = stored.into_arrow_schema()?;

let batch = RecordBatch::try_new(
    Arc::clone(&arrow_schema),
    vec![
        Arc::new(Int64Array::from(vec![1, 2])),
        Arc::new(StringArray::from(vec!["AAPL", "MSFT"])),
        Arc::new(StringArray::from(vec!["XNAS", "XNAS"])),
    ],
)?;

let mut handle = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
let plain = handle.record_options()?;
handle.overwrite_arrow_reader(arrow::batch_reader(arrow_schema, [batch]), &plain)?;

// One of the three columns, declared as this read's schema.
let wanted = DataType::from_fields([DataType::Int64.required_field("id")])?.required_field("row");

let projected = handle.read_arrow_reader(&plain.clone().with_field(wanted))?;
assert_eq!(projected.schema().fields().len(), 1);
assert_eq!(projected.map(|batch| batch.unwrap().num_columns()).sum::<usize>(), 1);

// The resource is unchanged: it still holds all three.
assert_eq!(handle.read_arrow_field(&plain)?.field_len(), 3);

// A column it does not hold cannot be projected out of it, so the encoding
// reads everything and the cast supplies that column as nulls.
let invented = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.nullable_field("nowhere"),
])?
.required_field("row");
let widened = handle.read_arrow_reader(&plain.with_field(invented))?;
assert_eq!(widened.schema().fields().len(), 2);
assert_eq!(widened.schema().field(1).name(), "nowhere");

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

stored = pa.schema([
    pa.field("id", pa.int64(), nullable=False),
    pa.field("symbol", pa.string(), nullable=False),
    pa.field("venue", pa.string(), nullable=False),
])
batch = pa.record_batch(
    {"id": [1, 2], "symbol": ["AAPL", "MSFT"], "venue": ["XNAS", "XNAS"]},
    schema=stored,
)

handle = IOBase(pathlib.Path(tempfile.mkdtemp()) / "trades.arrows")
handle.overwrite_arrow_batch(batch)

# One of the three columns, declared as this read's schema.
options = handle.record_options()
options.field = pa.schema([pa.field("id", pa.int64(), nullable=False)])

projected = handle.read_arrow_reader(options=options)
assert projected.schema.names == ["id"]
assert projected.read_all().num_columns == 1

# The resource is unchanged: it still holds all three.
assert len(handle.read_arrow_field().dtype) == 3

# A column it does not hold cannot be projected out of it, so the encoding
# reads everything and the cast supplies that column as nulls.
options.field = pa.schema([
    pa.field("id", pa.int64(), nullable=False),
    pa.field("nowhere", pa.string()),
])
widened = handle.read_arrow_reader(options=options)
assert widened.schema.names == ["id", "nowhere"]

const assert = require('node:assert/strict')
const arrow = require('apache-arrow')
const { BatchReader, Field, IOBase, MimeType, fields } = require('yggdryl')

const table = new arrow.Table({
  id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
  symbol: arrow.vectorFromArray(['AAPL', 'MSFT'], new arrow.Utf8()),
  venue: arrow.vectorFromArray(['XNAS', 'XNAS'], new arrow.Utf8()),
})

const handle = IOBase.fromBytes()
handle.mediaType = MimeType.ARROW_STREAM
handle.overwriteArrowReader(BatchReader.from(table))

// One of the three columns, declared as this read's schema.
const wanted = fields.struct('row', [Field.from('id: int64')], { nullable: false })
const options = handle.recordOptions()

const projected = handle.readArrowReader(options.withField(wanted))
assert.equal(projected.field.dtype.length, 1)
assert.equal(projected.intoTable().numCols, 1)

// The resource is unchanged: it still holds all three.
assert.equal(handle.readArrowField().dtype.length, 3)

// A column it does not hold cannot be projected out of it, so the encoding
// reads everything and the cast supplies that column as nulls.
const invented = fields.struct(
  'row',
  [Field.from('id: int64'), Field.from('nowhere: utf8?')],
  { nullable: false },
)
const widened = handle.readArrowReader(options.withField(invented))
assert.equal(widened.field.dtype.length, 2)

The field on the options selects and casts, in one pass over the data. The columns it names that the resource stores are handed to the encoding as its own projection - a Parquet projection mask, an Arrow IPC projection - so the columns it leaves out are skipped rather than read and discarded. media.md is where that also means fewer bytes decoded, because a column chunk is separately addressable; media.md saves the decode and the allocation but still reads the message body, and says so rather than claiming otherwise.

A projection can only drop columns, so the cast does everything else: reordering to the declared order, converting a stored type into the declared one, and filling a column the resource does not hold. Each batch is cast as it is pulled, so nothing is collected to do it. With no declared schema the stored shape is preserved exactly and no cast runs at all.

read_arrow_field answers with the same shape this read produces, so the schema a caller reads and the batches a caller gets can never disagree.

Limiting a read or a write

max_row_size bounds how many result rows flow in total - a count of rows, never a per-row byte cap - and max_byte_size bounds their Arrow in-memory bytes, counted uncompressed. Either bound applies last, after the fixed shaping order - declared schema, then selection, then completion cast, then partition filter - so a limit of ten combined with a filter means the first ten matching rows. A satisfied limit stops pulling, so the rest of the resource is never decoded.

RustPythonJavaScript

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, RecordBatchReader};
use yggdryl::arrow;
use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{DataType, MimeType};

let schema = DataType::from_fields([DataType::Int64.required_field("id")])?
    .required_field("row");
let arrow_schema = schema.into_arrow_schema()?;
let batch = RecordBatch::try_new(
    Arc::clone(&arrow_schema),
    vec![Arc::new(Int64Array::from_iter_values(0..1_000))],
)?;

let mut handle = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
let plain = handle.record_options()?;
handle.overwrite_arrow_reader(arrow::batch_reader(arrow_schema, [batch]), &plain)?;

// Ten result rows, exactly: the batch the bound lands inside is sliced.
let first = handle.read_arrow_reader(&plain.clone().with_max_row_size(10))?;
assert_eq!(first.map(|batch| batch.unwrap().num_rows()).sum::<usize>(), 10);

// Zero is a valid ask: the shaped schema answers, and no batch flows.
let mut none = handle.read_arrow_reader(&plain.clone().with_max_row_size(0))?;
assert_eq!(none.schema().fields().len(), 1);
assert!(none.next().is_none());

// A non-zero byte bound always yields at least one row.
let narrow = handle.read_arrow_reader(&plain.clone().with_max_byte_size(1))?;
assert_eq!(narrow.map(|batch| batch.unwrap().num_rows()).sum::<usize>(), 1);

// A limited write truncates the data the caller offered: three rows land,
// and what the bound cut off is never pulled from the reader.
let mut copy = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
copy.overwrite_arrow_reader(
    handle.read_arrow_reader(&plain)?,
    &plain.clone().with_max_row_size(3),
)?;
let kept = copy.read_arrow_reader(&plain)?;
assert_eq!(kept.map(|batch| batch.unwrap().num_rows()).sum::<usize>(), 3);

import pyarrow as pa

from yggdryl import IOBase

handle = IOBase.from_bytes()
handle.media_type = "application/vnd.apache.arrow.stream"
handle.overwrite_arrow_table(pa.table({"id": list(range(1_000))}))

# Ten result rows, exactly: the batch the bound lands inside is sliced.
ten = handle.record_options()
ten.max_row_size = 10
assert handle.read_arrow_reader(options=ten).read_all().num_rows == 10

# Zero is a valid ask: the shaped schema answers, and no batch flows.
zero = handle.record_options()
zero.max_row_size = 0
empty = handle.read_arrow_reader(options=zero)
assert empty.schema.names == ["id"]
assert empty.read_all().num_rows == 0

# A non-zero byte bound always yields at least one row.
one_byte = handle.record_options()
one_byte.max_byte_size = 1
assert handle.read_arrow_reader(options=one_byte).read_all().num_rows == 1

# A limited write truncates the data the caller offered: three rows land,
# and what the bound cut off is never pulled from the reader.
copy = IOBase.from_bytes()
copy.media_type = "application/vnd.apache.arrow.stream"
first_three = copy.record_options()
first_three.max_row_size = 3
copy.overwrite_arrow_reader(handle.read_arrow_reader(), options=first_three)
assert copy.read_arrow_reader().read_all().num_rows == 3

const assert = require('node:assert/strict')
const arrow = require('apache-arrow')
const { BatchReader, IOBase, MimeType } = require('yggdryl')

const table = new arrow.Table({
  id: arrow.vectorFromArray(
    Array.from({ length: 1000 }, (_, index) => BigInt(index)),
    new arrow.Int64(),
  ),
})

const handle = IOBase.fromBytes()
handle.mediaType = MimeType.ARROW_STREAM
handle.overwriteArrowReader(BatchReader.from(table))
const options = handle.recordOptions()

// Ten result rows, exactly: the batch the bound lands inside is sliced.
assert.equal(handle.readArrowReader(options.withMaxRowSize(10)).intoTable().numRows, 10)

// Zero is a valid ask: the shaped schema answers, and no batch flows.
const empty = handle.readArrowReader(options.withMaxRowSize(0))
assert.equal(empty.field.dtype.length, 1)
assert.equal(empty.intoTable().numRows, 0)

// A non-zero byte bound always yields at least one row.
assert.equal(handle.readArrowReader(options.withMaxByteSize(1)).intoTable().numRows, 1)

// A limited write truncates the data the caller offered: three rows land,
// and what the bound cut off is never pulled from the reader.
const copy = IOBase.fromBytes()
copy.mediaType = MimeType.ARROW_STREAM
copy.overwriteArrowReader(handle.readArrowReader(), options.withMaxRowSize(3))
assert.equal(copy.readArrowReader().intoTable().numRows, 3)

A limit of zero yields a reader with the shaped schema and no batches rather than an error, so "just the schema, cheaply" is a valid ask. The row bound is exact: the batch it lands inside is cut with a slice, a view over the same buffers rather than a copy. The byte bound stops at the last row that keeps the running total at or under the limit, and a non-zero byte limit always yields at least one row rather than silently losing everything to one wide row. When both are set, whichever bound binds first wins.

A limited write truncates the data the caller offered: the bounds cap the incoming reader exactly as they cap a read, and what they cut off is never pulled from it. A limit combined with a non-empty merge_by_names is refused naming both settings, because a truncated merge would update the matched keys it kept and silently drop the rest - corrupting the resource rather than shortening the write.

Appending and merging

Overwrite replaces the resource, which is what an IPC stream or a Parquet file natively supports: each carries one field and one footer. Append retains stored rows. Merge updates matching keys and adds new ones. The called method is always the authority; merge_by_names supplies keys only to the merge method.

RustPythonJavaScript

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, StringArray};
use yggdryl::arrow;
use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{DataType, Url};

let schema = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.nullable_field("symbol"),
])?
.required_field("row");
let arrow_schema = schema.clone().into_arrow_schema()?;
let rows = |ids: Vec<i64>, symbols: Vec<&'static str>| {
    let batch = RecordBatch::try_new(
        Arc::clone(&arrow_schema),
        vec![
            Arc::new(Int64Array::from(ids)),
            Arc::new(StringArray::from(symbols)),
        ],
    )
    .expect("a batch matching the root");
    arrow::batch_reader(batch.schema(), [batch])
};

let mut handle =
    Buffer::new().with_media_type(Url::from_str("file:///trades.arrows")?.media_type());
let options = handle.record_options()?.with_field(schema.clone());

// Overwrite replaces the resource.
handle.overwrite_arrow_reader(rows(vec![1, 2], vec!["AAPL", "MSFT"]), &options)?;

// Appending reads what is there, chains the new batches after it, and rewrites.
handle.append_arrow_reader(rows(vec![3], vec!["NVDA"]), &options)?;
let total: usize = handle
    .read_arrow_reader(&options)?
    .map(|batch| batch.unwrap().num_rows())
    .sum();
assert_eq!(total, 3);

// Merge requires a match key: `2` updates and `9` appends.
let merging = options.clone().with_merge_by_names(["id"]);
handle.merge_arrow_reader(rows(vec![2, 9], vec!["MSFT.O", "AMD"]), &merging)?;
let total: usize = handle
    .read_arrow_reader(&options)?
    .map(|batch| batch.unwrap().num_rows())
    .sum();
assert_eq!(total, 4);

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

schema = pa.schema([
    pa.field("id", pa.int64(), nullable=False),
    pa.field("symbol", pa.string()),
])
rows = lambda ids, symbols: pa.record_batch(
    {"id": ids, "symbol": symbols}, schema=schema
)

handle = IOBase(pathlib.Path(tempfile.mkdtemp()) / "trades.arrows")
options = handle.record_options()
options.field = schema

# No match key: the resource is replaced.
handle.overwrite_arrow_batch(rows([1, 2], ["AAPL", "MSFT"]), options=options)

# Appending reads what is there, chains the new batches after it, and rewrites.
handle.append_arrow_batch(rows([3], ["NVDA"]), options=options)
assert handle.read_arrow_reader(options=options).read_all().num_rows == 3

# A match key merges: `2` is already stored and updates, `9` is new and appends.
merging = handle.record_options()
merging.field = schema
merging.merge_by_names = ["id"]
handle.merge_arrow_batch(rows([2, 9], ["MSFT.O", "AMD"]), options=merging)
assert handle.read_arrow_reader(options=options).read_all().num_rows == 4

const assert = require('node:assert/strict')
const arrow = require('apache-arrow')
const { BatchReader, Field, IOBase, MimeType, fields } = require('yggdryl')

const schema = fields.struct(
  'row',
  [Field.from('id: int64'), Field.from('symbol: utf8?')],
  { nullable: false },
)
const rows = (ids, symbols) =>
  BatchReader.from(
    new arrow.Table({
      id: arrow.vectorFromArray(ids, new arrow.Int64()),
      symbol: arrow.vectorFromArray(symbols, new arrow.Utf8()),
    }),
  )

const handle = IOBase.fromBytes()
handle.mediaType = MimeType.ARROW_STREAM
const options = handle.recordOptions().withField(schema)

// No match key: the resource is replaced.
handle.overwriteArrowReader(rows([1n, 2n], ['AAPL', 'MSFT']), options)

// Appending reads what is there, chains the new batches after it, and rewrites.
handle.appendArrowReader(rows([3n], ['NVDA']), options)
assert.equal(handle.readArrowReader(options).intoTable().numRows, 3)

// A match key merges: `2` is already stored and updates, `9` is new and appends.
const merging = options.withMergeByNames(['id'])
handle.mergeArrowReader(rows([2n, 9n], ['MSFT.O', 'AMD']), merging)
assert.equal(handle.readArrowReader(options).intoTable().numRows, 4)

merge_by_names is a shared record setting, so it means the same thing on every encoding. It is accepted only by merge_arrow_reader, where a non-empty value is required. An overwrite applies a declared field to the incoming rows and then casts the result to the field the resource already stores when it stores one - overwrite replaces rows, not columns, so a caller who really means to change a stored field clears the handle first.

Non-empty names the columns that decide whether two rows are the same row. The key is encoded through Arrow's own row format, so a null key matches another null key exactly and a composite key compares column by column. A key stored more than once has every occurrence updated, because a match key is a rule and not a constraint the stored side was ever checked against; a key arriving more than once lets the last arrival win.

The merge is streamed over the incoming side: one batch is pulled, matched, folded in, and dropped before the next is pulled. What has to be held is the stored side, because updating a row means finding it by key and a reader cannot be rewound to a row it has already yielded.

Appending streams on both sides - the stored batches are chained ahead of the incoming ones and encoded as they arrive - and casts the incoming batches to the target shape first, so data whose schema merely fits is accepted.

Without a commit boundary, one leaf or one table publishes once: nothing reaches that value until its replacement is complete, so a failure leaves it as it was. A plain folder is the explicit exception described above. It has no cross-leaf transaction in IOBase, so already-published leaves remain visible if routing or publication of a later leaf fails.

select_by_names is the companion narrowing setting, and it works on both directions. On a read it yields exactly the named columns of the stored rows, in the order the names are given; on a write - overwrite, merge, or append - it keeps exactly the named columns of the incoming rows, so what it drops can never land. Names match ASCII case-insensitively, the way every cast selects, and a name the rows do not have is an error listing what is there, because a selection is a claim about the rows rather than a wish. An empty list, the default, selects everything.

RustPythonJavaScript

use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::Buffer;
use yggdryl::{arrow, DataType, MimeType};

use arrow_array::{Int64Array, RecordBatch, StringArray};
use std::sync::Arc;
let schema = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.nullable_field("symbol"),
])?
.required_field("row");
let batch = RecordBatch::try_new(
    schema.into_arrow_schema()?,
    vec![
        Arc::new(Int64Array::from(vec![1_i64, 2])),
        Arc::new(StringArray::from(vec![Some("AAPL"), Some("MSFT")])),
    ],
)?;

let mut handle = Buffer::new().with_media_type(MimeType::ARROW_STREAM.into());
let options = handle.record_options()?;
handle.overwrite_arrow_reader(arrow::batch_reader(batch.schema(), [batch]), &options)?;

// A read narrowed to one column yields one column.
let selecting = options.with_select_by_names(["symbol"]);
let first = handle.read_arrow_reader(&selecting)?.next().unwrap()?;
assert_eq!(first.num_columns(), 1);
assert_eq!(first.schema().field(0).name(), "symbol");

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

handle = IOBase(pathlib.Path(tempfile.mkdtemp()) / "orders.arrows")
handle.overwrite_arrow_table(pa.table({"id": [1, 2], "symbol": ["AAPL", "MSFT"]}))

# Record settings live on the one options object shared by every operation.
options = handle.record_options()
options.select_by_names = ["symbol"]
narrowed = handle.read_arrow_reader(options=options).read_all()
assert narrowed.column_names == ["symbol"]

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const arrow = require('apache-arrow')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
const handle = new IOBase(path.join(root, 'orders.arrows'))
handle.overwriteArrowTable(
  new arrow.Table({
    id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
    symbol: arrow.vectorFromArray(['AAPL', 'MSFT'], new arrow.Utf8()),
  }),
)

const narrowed = handle.recordOptions().withSelectByNames(['symbol'])
const table = handle.readArrowReader(narrowed).intoTable()
assert.deepEqual(table.schema.fields.map((field) => field.name), ['symbol'])

fs.rmSync(root, { recursive: true, force: true })

Text records

text/plain uses the same record methods as IPC, Parquet, and Avro. Each physical line becomes url: utf8 and body: binary; setting TextOptions.with_rownum / withRownum inserts rownum: int64 between them and supplies its first value. A flat TextOptions value adds regex rowheader captures, edge stripping, a fixed line separator, and syntax-directed pre-read autotyping. Generic RecordOptions retains the shared timezone accessor.

into_text / intoText retains those options without adding a line iterator, line-only read/write method, or standalone schema builder. Use read_arrow_reader / readArrowReader, read_records / readRecords, and the ordinary overwrite or append methods. Structured text and plain-text records defines the schema, parsing order, errors, examples, and benchmark commands.

Globbing and Hive partitions

A location can name a set rather than one resource, and a Hive path can name the values its rows share. IOBase reads both, so selecting the parts of a lake to rewrite is a listing, not a scan.

RustPythonJavaScript

use yggdryl::IOBase;
use yggdryl::holder::local::Folder;

let root = Folder::temporary()?.path()?.join("yggdryl-doc-lake");
let _ = std::fs::remove_dir_all(&root);
for year in ["2024", "2025"] {
    let leaf = root.join(format!("year={year}")).join("month=01");
    std::fs::create_dir_all(&leaf)?;
    std::fs::write(leaf.join("part-0.parquet"), b"parquet")?;
}

let lake = Folder::new(&root)?;

// A fixed prefix is descended, not listed and filtered.
assert_eq!(lake.glob("year=2024/**/*.parquet", false)?.count(), 1);
assert_eq!(lake.glob("**/*.parquet", false)?.count(), 2);

// Partition filters select the leaves to overwrite or upsert.
let selected: Vec<_> = lake
    .children_where(&[("year", "2024")], false)?
    .collect::<yggdryl::Result<_>>()?;
assert_eq!(selected.len(), 1);
assert_eq!(selected[0].partitions(), vec![
    ("year".to_owned(), "2024".to_owned()),
    ("month".to_owned(), "01".to_owned()),
]);

let _ = std::fs::remove_dir_all(&root);

import pathlib
import tempfile

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp()) / "lake"
for year in ("2024", "2025"):
    leaf = root / f"year={year}" / "month=01"
    leaf.mkdir(parents=True)
    (leaf / "part-0.parquet").write_bytes(b"parquet")

lake = IOBase(root)

assert len(list(lake.glob("year=2024/**/*.parquet"))) == 1
assert len(list(lake.rglob("*.parquet"))) == 2

selected = list(lake.children_where({"year": "2024"}))
assert len(selected) == 1
assert selected[0].partitions == (("year", "2024"), ("month", "01"))

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = path.join(fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-')), 'lake')
for (const year of ['2024', '2025']) {
  const leaf = path.join(root, `year=${year}`, 'month=01')
  fs.mkdirSync(leaf, { recursive: true })
  fs.writeFileSync(path.join(leaf, 'part-0.parquet'), 'parquet')
}

const lake = new IOBase(root)

// A fixed prefix is descended, not listed and filtered.
assert.equal([...lake.glob('year=2024/**/*.parquet')].length, 1)
assert.equal([...lake.rglob('*.parquet')].length, 2)

// Partition filters select the leaves to overwrite or upsert.
const selected = [...lake.childrenWhere({ year: '2024' })]
assert.equal(selected.length, 1)
assert.deepEqual(selected[0].partitions, [
  { column: 'year', value: '2024' },
  { column: 'month', value: '01' },
])

fs.rmSync(root, { recursive: true, force: true })

A location that is a pattern is folder-like before anything touches the backend: kind reports IOKind::Directory, and ls expands the pattern from the fixed root it was split from instead of looking for a directory literally named **.

children_where yields the leaves - never containers - that carry every requested pair. That is what the three record methods use when a handle addresses a folder, and it is there for a caller who wants to reach one partition directly instead.

The pairs are sugar. Each one builds &holder.partition['column'] = 'value' and the whole thing is answered by children_matching, which takes the expression language itself: ranges, null tests, in lists, and every other &holder.* attribute. There is no second filter behind the pairs.

A listing is an iterator

Every listing here yields one entry at a time: ls, glob, rglob, children_matching, and children_where all hand back a Listing, and none of them decides to build a vector on the caller's behalf. That is not a style preference. A folder with a million leaves has to be listable the same way one with three is, and a shape that has to materialize cannot describe a resource larger than memory - the same argument the three record methods already make about batches.

Three consequences a caller can rely on:

• Nothing is touched until the first next. Building a listing costs nothing, and taking three entries from a folder of a hundred thousand costs three. A glob whose fixed prefix names nothing reads no directory beneath it at all.
• The item is a Result, and the iterator is fused after the first one that fails. A listing fails at the failing entry, naming it, without discarding what it already yielded and without spinning against a backend that is already refusing. A caller who wants a vector writes .collect::<Result<Vec<_>>>().
• Order is deterministic. One directory's entries are sorted, and a recursive walk yields each container immediately before the subtree beneath it. The same listing over the same state yields the same sequence twice.

What a recursive walk holds is its frontier - one level's cursor per open depth - never its result. The local backend reads and sorts one directory at a time; an Arrow filesystem answers a whole prefix in one call, because that is the shape an object store already has, and the laziness lives inside that one answer.

A few returns in listing positions stay owned, and each says what bounds it: IOBase::partitions is bounded by one URL's path depth, and a report of what an operation just did - the snapshot ids an expiry removed, the counts a compaction reports - is bounded by the operation. Unbounded by the resource means an iterator; bounded by the act, or already in hand, may be owned.

In Python the listings are ordinary iterators, so iterdir, glob, and rglob behave exactly as pathlib's do - wrap one in list() when you want a sequence, and remember that len() of that list costs the whole walk. In JavaScript they are iterables: for...of walks one, and [...listing] drains it.

Partition pruning and filtering

One option answers the same equality wherever the value lives. filter_partitions names (column, value) pairs, spelled the way partition paths spell them: a folder read prunes - a leaf whose directory names a different value is never listed or decoded - and a column the data carries is filtered row by row, so a path-partitioned lake and a data-partitioned one answer identically.

Both halves are one expression, bound once. The pruning half asks the path (&holder.partition['year'] = '2024') and the filtering half asks the rows (year = 2024), with the text read through the column's own datatype - so a pair on an int32 column is an integer comparison, and the pair ("price", "null") is price is null rather than a comparison against four letters.

RustPythonJavaScript

use yggdryl::media::{IORecordOptions, RecordOptions};
use yggdryl::MimeType;

let options = RecordOptions::for_mime_type(&MimeType::ARROW_STREAM)?
    .with_filter_partitions([("year", "2024"), ("month", "01")]);
// handle.read_arrow_reader(&options)? now reads only the January
// 2024 leaves, and only their matching rows.

import pathlib
import tempfile

import pyarrow as pa

from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp()) / "lake"
IOBase(root / "year=2024" / "month=01" / "trades.arrows").overwrite_arrow_table(
    pa.table({"id": [1, 2]})
)
IOBase(root / "year=2024" / "month=02" / "trades.arrows").overwrite_arrow_table(
    pa.table({"id": [3]})
)

lake = IOBase(root)
options = lake.record_options()
options.filter_partitions = [("year", "2024"), ("month", "01")]
reader = lake.read_arrow_reader(options=options)
assert reader.read_all().num_rows == 2

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
const lake = path.join(root, 'lake')
new IOBase(path.join(lake, 'year=2024', 'month=01', 'trades.arrows'))
  .overwriteRecords([{ id: 1n }, { id: 2n }])
new IOBase(path.join(lake, 'year=2024', 'month=02', 'trades.arrows'))
  .overwriteRecords([{ id: 3n }])

const handle = new IOBase(lake)
const options = handle.recordOptions().withFilterPartitions([
  ['year', '2024'],
  ['month', '01'],
])
assert.equal(handle.readArrowReader(options).intoTable().numRows, 2)

fs.rmSync(root, { recursive: true, force: true })

Writes into a shared folder also smooth concurrent writers: the listing and every whole-leaf rewrite retry a bounded number of times with a short growing pause, so a reader that catches a leaf half-published or two writers racing a replace settle without surfacing a transient error. An append never retries - replaying a torn append would duplicate rows - so it fails honestly instead.

Partition columns in the data

A Hive layout stores a column in the path, so the file under year=2024/month=01 leaves those values out of every row. Addressing the folder rather than the file is what puts them back: the four record primitives resolve the leaves themselves, restore the columns their directory names spell out, and route each row of a write to the leaf its values name.

RustPythonJavaScript

use yggdryl::holder::Holder;
use yggdryl::media::{IORecordOptions, RecordOptions};
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::local::Folder;
use yggdryl::{DataType, MimeType};

let root = Folder::temporary()?.path()?.join("yggdryl-doc-partitioned");
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(root.join("year=2024").join("month=01"))?;

let schema = DataType::from_fields([
    DataType::Int64.required_field("price"),
    DataType::Int32.required_field("year"),
    DataType::Utf8.required_field("month"),
])?
.required_field("row");
let arrow_schema = schema.clone().into_arrow_schema()?;
let batch = arrow_array::RecordBatch::try_new(
    std::sync::Arc::clone(&arrow_schema),
    vec![
        std::sync::Arc::new(arrow_array::Int64Array::from(vec![10, 20])),
        std::sync::Arc::new(arrow_array::Int32Array::from(vec![2024, 2024])),
        std::sync::Arc::new(arrow_array::StringArray::from(vec!["01", "01"])),
    ],
)?;

// The rows carry every column; the write drops the two the path spells out.
let mut lake = Holder::folder(&root)?;
let options = RecordOptions::for_mime_type(&MimeType::ARROW_STREAM)?.with_field(schema.clone());
lake.overwrite_arrow_reader(
    yggdryl::arrow::batch_reader(arrow_schema, [batch]),
    &options,
)?;

// Only `price` reached the leaf; the other two are the directory names.
let leaf = lake.child_by_path("year=2024/month=01/part-0.arrows")?;
assert_eq!(
    leaf.read_arrow_field(&RecordOptions::for_media_type(leaf.media_type())?)?.field_len(),
    1
);

// Reading the folder restores them with their declared types.
let restored = lake
    .read_arrow_reader(&options)?
    .next()
    .expect("one batch")?;
assert_eq!(restored.num_columns(), 3);
assert_eq!(restored.schema().field(1).data_type(), &arrow_schema::DataType::Int32);

let _ = std::fs::remove_dir_all(&root);

import pathlib
import shutil
import tempfile

import pyarrow as pa

from yggdryl import IOBase, RecordOptions

root = pathlib.Path(tempfile.mkdtemp())
(root / "year=2024" / "month=01").mkdir(parents=True)

schema = pa.schema([
    pa.field("price", pa.int64(), nullable=False),
    pa.field("year", pa.int32(), nullable=False),
    pa.field("month", pa.string(), nullable=False),
])
batch = pa.record_batch(
    {"price": [10, 20], "year": [2024, 2024], "month": ["01", "01"]},
    schema=schema,
)

# The rows carry every column; the write drops the two the path spells out.
lake = IOBase(root)
options = RecordOptions("part.arrows")
options.field = schema
lake.overwrite_arrow_batch(batch, options=options)

# Only `price` reached the leaf; the other two are the directory names.
leaf = lake / "year=2024" / "month=01" / "part-0.arrows"
assert len(leaf.read_arrow_field().dtype) == 1

# Reading the folder restores them with their declared types.
restored = lake.read_arrow_reader(options=options).read_all()
assert restored.column_names == ["price", "year", "month"]
assert restored.schema.field("year").type == pa.int32()

shutil.rmtree(root)

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const arrow = require('apache-arrow')
const { BatchReader, Field, IOBase, MimeType, RecordOptions, fields } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-docs-'))
fs.mkdirSync(path.join(root, 'year=2024', 'month=01'), { recursive: true })

const schema = fields.struct(
  'row',
  [Field.from('price: int64'), Field.from('year: int32'), Field.from('month: utf8')],
  { nullable: false },
)
const table = new arrow.Table({
  price: arrow.vectorFromArray([10n, 20n], new arrow.Int64()),
  year: arrow.vectorFromArray([2024, 2024], new arrow.Int32()),
  month: arrow.vectorFromArray(['01', '01'], new arrow.Utf8()),
})

// The rows carry every column; the write drops the two the path spells out.
const lake = new IOBase(root)
const options = RecordOptions.forMimeType(MimeType.ARROW_STREAM).withField(schema)
lake.overwriteArrowReader(BatchReader.from(table), options)

// Only `price` reached the leaf; the other two are the directory names.
const leaf = lake.joinpath('year=2024').joinpath('month=01').joinpath('part-0.arrows')
assert.equal(leaf.readArrowField().dtype.length, 1)

// Reading the folder restores them with their declared types.
const restored = lake.readArrowReader(options).intoTable()
assert.equal(restored.numCols, 3)
assert.equal(restored.schema.fields[1].type.toString(), 'Int32')
assert.deepEqual(restored.getChild('month').toArray(), ['01', '01'])

fs.rmSync(root, { recursive: true, force: true })

The layout is the authority on which columns are partition columns, because nothing in a batch says which of its columns belong in a path. A folder whose leaves already spell out column=value partitions by exactly those columns; a folder that spells out nothing takes the layout from the declared schema, whose partition-marked fields say it; and a folder with neither is one table in one leaf, named after the encoding. So a tree comes into being two ways: address one partition directly to create it, or declare the columns on the schema and let the first write lay the directories out.

Rust

use yggdryl::holder::Holder;
use yggdryl::media::{IORecordOptions, RecordOptions};
use yggdryl::{IOBase, IOMedia};
use yggdryl::holder::local::Folder;
use yggdryl::{DataType, MimeType};

let root = Folder::temporary()?.path()?.join("yggdryl-doc-declared-layout");
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(&root)?;

// Nothing is on disk, so nothing spells a layout. The schema does.
let schema = DataType::from_fields([
    DataType::Int64.required_field("price"),
    DataType::Int32.required_field("year"),
])?
.required_field("row")
.with_partition_fields(&["year"])?;
assert_eq!(schema.partition_field_names().collect::<Vec<_>>(), ["year"]);

let arrow_schema = schema.clone().into_arrow_schema()?;
let batch = arrow_array::RecordBatch::try_new(
    std::sync::Arc::clone(&arrow_schema),
    vec![
        std::sync::Arc::new(arrow_array::Int64Array::from(vec![10, 20])),
        std::sync::Arc::new(arrow_array::Int32Array::from(vec![2024, 2024])),
    ],
)?;

let mut lake = Holder::folder(&root)?;
let options = RecordOptions::for_mime_type(&MimeType::ARROW_STREAM)?.with_field(schema);
lake.overwrite_arrow_reader(
    yggdryl::arrow::batch_reader(arrow_schema, [batch]),
    &options,
)?;

// The directory came from the declaration, and the leaf stores what the
// path does not carry.
assert!(root.join("year=2024").is_dir());

// Reading it back reports the layout without being told it.
let derived = lake.read_arrow_field(
    &RecordOptions::for_mime_type(&MimeType::ARROW_STREAM)?,
)?;
assert_eq!(derived.partition_field_names().collect::<Vec<_>>(), ["year"]);

let _ = std::fs::remove_dir_all(&root);

A declaration that contradicts a stored layout is refused, naming both, because one write cannot mean two trees: a folder already partitioned by year and a schema that marks venue disagree about which columns the leaves are missing, and merging them would leave files whose directory names no longer say what they left out.

A column the data already carries is left alone rather than rewritten from the directory name, so a mismatch between the two stays visible instead of being silently papered over. Without a declared schema the restored values stay text, which is exactly what a directory name holds. A value that is absent is spelled null, which a path cannot distinguish from the four letters - so it is the declared column that decides, and a nullable one reads the text back as a null.

Routing is bounded over the incoming reader: one batch is pulled, split by partition, and written before the next is pulled. The price is paid on the other side, because these encodings rewrite a whole leaf: a partition touched by five batches is rewritten five times. The first batch to reach a leaf performs the caller's operation and the rest append to it, which is what keeps an overwrite an overwrite without buffering the whole write first.

Local file system

The file system as three IOBase handles: a generic location, a directory, and a memory-mapped file.

Rust only

The Python and JavaScript packages do not expose this module yet.

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder};

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-lead-{}.bin", std::process::id()));

let mut file = File::create(&path)?;
file.write_all_bytes(b"AAPL")?;
file.flush()?;

assert_eq!(file.read_all_bytes()?, b"AAPL");

drop(file);
let _ = std::fs::remove_file(&path);

The three roles

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder, Path};
use yggdryl::IOKind;

let root = Folder::temporary()?.path()?.join(format!("yggdryl-doc-roles-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(root.join("nested"))?;
std::fs::write(root.join("a.bin"), b"a")?;

// A container: it holds no bytes of its own, only children.
let folder = Folder::new(&root)?;
assert_eq!(folder.size(), 0);
assert_eq!(folder.ls(false, false).count(), 2);

// A leaf: bytes addressed by offset.
let leaf = File::new(root.join("a.bin"))?;
assert_eq!(leaf.read_all_bytes()?, b"a");

// A location: it answers by looking at what is actually there.
assert_eq!(Path::new(&root)?.kind(), IOKind::Directory);
assert_eq!(Path::new(root.join("a.bin"))?.kind(), IOKind::File);

let _ = std::fs::remove_dir_all(&root);

Folder and File are the two things a file system has; Path is for the case where the caller does not yet know which it holds - a listing entry, a command-line argument, a configuration value. Path resolves once, keeps the implementation it resolved to, and runs every IOBase call through it.

Each role also implements the matching trait from yggdryl - IOFolder, IOFile, IOPath - which pre-implements everything that follows from the role: a container refuses byte writes, a leaf lists nothing and resolves no children, a location reports its IOKind by testing the path. A backend supplies each role's few required members - four for a container, two for a leaf, three for a location - and inherits the rest.

Well-known roots

Folder names the three directories a program may assume. Folder::temporary() is the platform temporary directory. Folder::home() is the current user's home, read from the environment the same way on every platform: HOME first, then USERPROFILE, an unset or empty variable being skipped. Folder::config() is that home joined with .config. Each answers a handle and creates nothing; as everywhere here, a directory comes into being when something is written into it. With neither variable set, home and config fail with a typed absence naming both, for which Error::is_absent is true, so a caller that wants "no home" reads the error instead of guessing a path.

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::Folder;

let temporary = Folder::temporary()?;
assert!(temporary.is_container());
assert!(temporary.url().to_string().starts_with("file:"));

// When a home resolves, the configuration directory is that home joined with `.config`.
match Folder::home() {
    Ok(home) => assert_eq!(Folder::config()?.path()?, home.path()?.join(".config")),
    Err(error) => assert!(error.is_absent()),
}

Laziness

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder};

let root = Folder::temporary()?.path()?.join(format!("yggdryl-doc-lazy-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);

// Constructing touches nothing.
let folder = Folder::new(&root)?;
let mut leaf = File::new(root.join("nested").join("trades.bin"))?;
assert!(!folder.exists());
assert!(!leaf.exists());

// Reading something absent yields nothing - and still creates nothing.
assert_eq!(folder.ls(true, false).count(), 0);
assert!(leaf.read_all_bytes()?.is_empty());
assert_eq!(leaf.size(), 0);
assert!(!root.exists());

// Writing creates the file and every missing parent.
leaf.write_all_bytes(b"trade")?;
leaf.flush()?;
assert!(leaf.exists());
assert_eq!(leaf.read_all_bytes()?, b"trade");

drop(leaf);
let _ = std::fs::remove_dir_all(&root);

Nothing here validates a path up front, so a handle for a file that will exist later is a normal value to hold. The one eager check is the conversion to a canonical file: Url: Folder::new, File::new, and Path::new fail only when the path cannot be expressed as one, and Folder::from_url and Path::from_url fail when the URL is not local.

That URL is the whole state of a Folder: the platform path is derived from it on demand, so a stored path and a stored URL can never disagree.

A write decides an undecided location

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{Folder, Path};
use yggdryl::IOKind;

let root = Folder::temporary()?.path()?.join(format!("yggdryl-doc-decide-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);
std::fs::create_dir_all(&root)?;

// Nothing is there, so nothing has decided what it is.
let mut location = Path::new(root.join("trades.bin"))?;
assert_eq!(location.kind(), IOKind::Unknown);

// A byte write settles it: an undecided location becomes a file.
location.write_all_bytes(b"AAPL")?;
location.flush()?;
assert_eq!(location.kind(), IOKind::File);
assert_eq!(location.read_all_bytes()?, b"AAPL");

// To settle it the other way, say so before writing.
let container = Path::new(root.join("day=2026-08-16"))?;
assert_eq!(container.kind(), IOKind::Unknown);
container.as_directory()?.create()?;
assert_eq!(container.kind(), IOKind::Directory);

drop(location);
let _ = std::fs::remove_dir_all(&root);

Bytes are what distinguish a leaf from a container, so a Path with no resource behind it becomes a file the moment one writes to it. as_directory and as_file are the way to state the intent instead, and both work before anything exists. Folder::create makes the directory and every missing parent; on a container, truncate(0) does the same thing, and any other size is an error, because a container has no bytes to resize.

Walking the tree

Rust

use yggdryl::holder::Holder;
use yggdryl::IOBase;
use yggdryl::holder::local::Folder;

let root = Folder::temporary()?.path()?.join(format!("yggdryl-doc-walk-{}", std::process::id()));
let _ = std::fs::remove_dir_all(&root);

let folder = Folder::new(&root)?;
folder.create()?;

// A child is a handle; writing through it creates the leaf.
let mut leaf = folder.child_by_path("trades.arrows")?;
leaf.write_all_bytes(b"payload")?;
leaf.flush()?;
assert!(matches!(leaf, Holder::File(_)));

// A nested child creates its parent directory on write.
let mut nested = folder.child_by_path("sub/inner.bin")?;
nested.write_all_bytes(b"deep")?;
nested.flush()?;

// Listings are sorted, so two runs agree; recursion reaches the nested leaf.
let names: Vec<String> = folder
    .ls(false, false)
    .map(|entry| {
        let entry = entry?;
        Ok(entry
            .url()
            .and_then(|url| url.file_name())
            .unwrap_or_default()
            .to_owned())
    })
    .collect::<yggdryl::Result<_>>()?;
assert_eq!(names, ["sub", "trades.arrows"]);
assert_eq!(folder.ls(true, false).count(), 3);

// A leaf's parent is the directory holding it.
let parent = leaf.parent().expect("a file has a parent");
assert!(parent.is_container());
assert_eq!(parent.url().unwrap(), folder.url());

drop(leaf);
drop(nested);
let _ = std::fs::remove_dir_all(&root);

ls, child_by_path, and parent return Holder, so one enum walks a whole tree: subdirectories come back as Holder::Folder, files as Holder::File. Children resolve through the URL, so . and .. segments collapse the way they do everywhere else in the crate, and a name with separators in it is a nested child rather than an error.

std::fs::read_dir order is platform-defined; ls sorts, so a listing is reproducible.

The mapping

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder};

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-growth-{}.bin", std::process::id()));

let mut file = File::create(&path)?;
file.pwrite(0, b"trade")?;

// Writing past the mapping remaps at a larger capacity instead of failing.
let bulk = vec![7_u8; 256 * 1024];
file.append_bytes(&bulk)?;
assert_eq!(file.size(), 5 + bulk.len() as u64);
assert!(file.capacity() >= file.size());

// Flushing publishes the logical length, so the file is the bytes, not the mapping.
file.flush()?;
assert_eq!(std::fs::metadata(&path)?.len(), file.size());

drop(file);
let _ = std::fs::remove_file(&path);

size is the logical length and capacity is how much of the file is mapped. Growth is geometric, so a run of appends remaps a logarithmic number of times rather than once per write, which leaves the mapped file longer than the bytes written. flush and close reconcile the two: they release the mapping and then set the file's length. Releasing first is not an optimisation, it is required, because Windows refuses to resize a file while a mapped section is open. Dropping the handle publishes too, but a drop cannot report failure, so call flush when the write must be known to have landed.

Offsets are absolute and a write may start past the end:

Rust

use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder};

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-gap-{}.bin", std::process::id()));

let mut file = File::create(&path)?;
file.pwrite(0, b"ab")?;
file.pwrite(5, b"z")?;

// The gap the offset created is zero-filled.
assert_eq!(file.read_all_bytes()?, b"ab\0\0\0z");

drop(file);
let _ = std::fs::remove_file(&path);

The SIGBUS hazard

The mapping constructor is the only unsafe in the crate, and it is unsafe for a reason no wrapper can remove: the mapping aliases the file's bytes. If another process truncates the file while the mapping is live, touching the lost pages raises SIGBUS - a signal, not an error a Result can carry. File documents that instead of pretending it away.

When the file may change underneath you, take the bytes into memory and work on the copy:

Rust

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::holder::local::{File, Folder};

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-snapshot-{}.bin", std::process::id()));
std::fs::write(&path, b"trade")?;

// The handle - and its mapping - is gone by the time the copy returns.
let mut snapshot = Buffer::new();
File::new(&path)?.copy_into(&mut snapshot)?;

assert_eq!(snapshot.into_bytes(), b"trade");
let _ = std::fs::remove_file(&path);

copy_into transfers in chunks, so neither side is buffered whole, and it carries the source's media type onto the target.

A remote backend is a sibling module

S3, GCS, and Azure are the same three ideas - a location, a container, a leaf - so a new backend is a sibling module supplying the same three roles rather than a change to anything here. Code written against IOBase does not learn which one it got:

Rust

use yggdryl::IOBase;
use yggdryl::holder::Buffer;
use yggdryl::holder::local::{File, Folder};

fn head(handle: &dyn IOBase) -> yggdryl::Result<Vec<u8>> {
    handle.read_range_bytes(0, 4)
}

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-agnostic-{}.bin", std::process::id()));

let mut file = File::create(&path)?;
file.write_all_bytes(b"AAPL,100")?;

let memory = Buffer::from_bytes(b"AAPL,100".to_vec());
assert_eq!(head(&file)?, b"AAPL");
assert_eq!(head(&file)?, head(&memory)?);

drop(file);
let _ = std::fs::remove_file(&path);

That is why ipc and parquet take a handle rather than a path: the same reader runs over a mapped file, an in-memory Buffer, or a compressed handle from gzip, zlib, or zstd.

Private entries

A listing excludes names beginning with a dot unless it is asked for them, so walking a tree does not wander into .git, .venv, or .DS_Store.

use yggdryl::IOBase;
use yggdryl::holder::local::Folder;
use yggdryl::Url;

let root = Folder::temporary()?.path()?.join("yggdryl-doc-private");
std::fs::create_dir_all(root.join(".git"))?;
std::fs::write(root.join("trades.arrows"), b"x")?;

let folder = Folder::new(&root)?;
assert_eq!(folder.ls(false, false).count(), 1);
assert_eq!(folder.ls(false, true).count(), 2);

// The rule is one accessor on the location itself, because every child has one.
assert!(Url::from_str("file:///project/.git")?.is_private());
assert!(!Url::from_str("file:///project/trades.arrows")?.is_private());

std::fs::remove_dir_all(&root)?;

A private directory is not descended into either, so a recursive listing stays out of them entirely.

Filesystems

yggdryl::holder::fs puts any filesystem - S3, GCS, Azure, a local tree, or one you wrote yourself - behind the crate's one storage abstraction, IOBase.

Nothing here implements a transport. FileSystem is a seven-method contract modeled on Arrow's own FileSystem API, so an implementation that already exists - pyarrow.fs, Arrow C++, Arrow Java - maps onto it method for method, and the three roles above it inherit every wrapper the crate already has. A handle over a bucket reads and writes folders and files, streams Arrow records, and composes with Coded, ipc, parquet, and iceberg, with no transport code written in this repository.

Construct from a filesystem and a path

RustPythonJavaScript

use std::sync::Arc;

use yggdryl::holder::fs::{File, MemoryFileSystem};
use yggdryl::IOBase;

let filesystem = Arc::new(MemoryFileSystem::new());
let mut handle = File::from_location(filesystem, "bucket/trades.bin")?;

// Per the laziness contract nothing exists until something is written.
assert!(!handle.exists());
assert_eq!(handle.read_all_bytes()?, b"");

handle.write_all_bytes(b"AAPL")?;
handle.close()?;
assert_eq!(handle.read_all_bytes()?, b"AAPL");

// The handle's identity is a canonical URL naming the filesystem.
assert_eq!(handle.url().to_string(), "memory://bucket/trades.bin");

import tempfile, pathlib
import pyarrow.fs as pafs
from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())
handle = IOBase.from_fs(pafs.LocalFileSystem(), (root / "trades.bin").as_posix())

# Per the laziness contract nothing exists until something is written.
assert not handle.exists()
assert handle.read_bytes() == b""

with handle:
    handle.write_bytes(b"AAPL")

assert handle.read_bytes() == b"AAPL"
assert (root / "trades.bin").read_bytes() == b"AAPL"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

// Arrow JS ships no filesystem, so the handler is the filesystem: the
// same six calls, spelled in camelCase. This one is a Map; an S3 client
// or a caching layer answers the same way.
const files = new Map()
const handler = {
  typeName: 'memory',
  fileInfo: (path) =>
    files.has(path)
      ? { path, kind: 'file', size: BigInt(files.get(path).length) }
      : { path, kind: 'not-found' },
  list: () => [],
  readRange: (path, offset, length) =>
    files.get(path)?.subarray(Number(offset), Number(offset) + length) ?? null,
  writeFull: (path, bytes) => { files.set(path, Buffer.from(bytes)) },
  createDir: () => {},
  deleteFile: (path) => { files.delete(path) },
}

const handle = IOBase.fromFs(handler, 'bucket/trades.bin')

// Per the laziness contract nothing exists until something is written.
assert.equal(handle.exists(), false)
assert.equal(handle.readText(), '')

handle.writeText('AAPL')
handle.close()

assert.equal(handle.readText(), 'AAPL')
assert.equal(String(handle.url), 'memory://bucket/trades.bin')

The first path segment is the authority, which is exactly what a bucket is, so "bucket/key" on an S3 filesystem spells s3://bucket/key. In Python the filesystem may also be inferred from the first argument: IOBase(fs, "bucket/key") means the same as IOBase.from_fs(fs, "bucket/key"), and JavaScript infers the same way with new IOBase(handler, 'bucket/key').

A JavaScript handler belongs to one thread

The handler is called synchronously, in the middle of the native read or write, so it cannot be reached from another thread: a handle used from a Worker refuses with a message saying so rather than queueing work. A worker that needs its own view builds its own handler - only the location string has to travel. This is a named limitation rather than an emulation, because Node-API's only cross-thread call is asynchronous and every method here has to answer now.

The real thing looks like this, and needs credentials and a network, so it is shown rather than run:

Python

from pyarrow.fs import S3FileSystem
from yggdryl import IOBase
from yggdryl.media import iceberg

handle = IOBase.from_fs(S3FileSystem(region="eu-west-1"), "bucket/table")
table = iceberg.Table.open(handle)
rows = table.scan().read_all()

A positional write publishes when the handle closes

An Arrow filesystem replaces whole files. It has no random write - an object store cannot patch five bytes in the middle of an object - while IOBase::pwrite is positional. So a leaf stages its positional mutations in memory and publishes them as exactly one whole-value replacement on flush or close. Until then the stored value is untouched:

RustPythonJavaScript

use std::sync::Arc;

use yggdryl::holder::fs::{FileSystem, File, MemoryFileSystem};
use yggdryl::IOBase;

let filesystem = Arc::new(MemoryFileSystem::new());
filesystem.write_full("bucket/trades.bin", b"stored")?;
let mut handle = File::from_location(filesystem.clone(), "bucket/trades.bin")?;

// Positional writes are pieces of a value, so they stage.
handle.truncate(0)?;
handle.pwrite(0, b"pend")?;
handle.pwrite(4, b"ing")?;

// The handle presents the pending value; the filesystem still has the old one.
assert_eq!(handle.read_all_bytes()?, b"pending");
assert_eq!(filesystem.file_info("bucket/trades.bin")?.size, 6);

handle.close()?;
assert_eq!(filesystem.file_info("bucket/trades.bin")?.size, 7);

import tempfile, pathlib
import pyarrow.fs as pafs
from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())
handle = IOBase.from_fs(pafs.LocalFileSystem(), (root / "staged.bin").as_posix())

handle.pwrite(0, b"pend")
handle.pwrite(4, b"ing")
assert not (root / "staged.bin").exists()

handle.close()
assert (root / "staged.bin").read_bytes() == b"pending"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const files = new Map()
const handler = {
  typeName: 'memory',
  fileInfo: (path) =>
    files.has(path)
      ? { path, kind: 'file', size: BigInt(files.get(path).length) }
      : { path, kind: 'not-found' },
  list: () => [],
  readRange: (path, offset, length) =>
    files.get(path)?.subarray(Number(offset), Number(offset) + length) ?? null,
  writeFull: (path, bytes) => { files.set(path, Buffer.from(bytes)) },
  createDir: () => {},
  deleteFile: (path) => { files.delete(path) },
}

const handle = IOBase.fromFs(handler, 'bucket/staged.bin')
handle.pwrite(0, Buffer.from('pend'))
handle.pwrite(4, Buffer.from('ing'))

// The handle presents the pending value; the filesystem has not been
// asked to store anything yet.
assert.equal(handle.readText(), 'pending')
assert.equal(files.has('bucket/staged.bin'), false)

handle.close()
assert.equal(files.get('bucket/staged.bin').toString(), 'pending')

That is why a file another reader will open is written inside a scope - with in Python, using in JavaScript - which binds to exactly open and close.

A whole-value write needs none of that. write_all_bytes and the ordinary record overwrite or append methods each describe one complete operation, so they publish when they finish. The staging exists to fold many positional writes into one replacement; it is not a mode a caller has to remember to leave.

Reads need none of it. A pread maps straight onto one ranged fetch, so asking for eight bytes of a large object transfers eight bytes rather than the object. What a record encoding does with that is its own business, and parquet currently fetches the value whole - its footer is at the end, and a range-reading reader over pread is the optimization path that page names. Reading one Parquet footer over a bucket therefore still costs a whole object today; the handle is what stops being the reason.

Folders, globs, and partitions

A directory on an object store is a prefix, so existence here is what the filesystem itself reports: the prefix has entries, or a marker exists. Nothing invents marker objects a store would not have written.

RustPythonJavaScript

use std::sync::Arc;

use yggdryl::holder::fs::{FileSystem, Folder, MemoryFileSystem};
use yggdryl::IOBase;

let filesystem = Arc::new(MemoryFileSystem::new());
for year in ["2024", "2025"] {
    let leaf = format!("bucket/year={year}/part-0.parquet");
    filesystem.write_full(&leaf, b"PAR1")?;
}
let lake = Folder::from_location(filesystem, "bucket")?;

assert!(lake.is_container());
assert_eq!(lake.ls(false, false).count(), 2);
assert_eq!(lake.glob("**/*.parquet", false)?.count(), 2);

// A fixed prefix is descended rather than listed and filtered.
assert_eq!(lake.glob("year=2024/**/*.parquet", false)?.count(), 1);

// Hive pairs are read off the location, as they are for any backend.
let selected: Vec<_> = lake
    .children_where(&[("year", "2024")], false)?
    .collect::<yggdryl::Result<_>>()?;
assert_eq!(selected.len(), 1);

import tempfile, pathlib
import pyarrow.fs as pafs
from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp()) / "lake"
for year in ("2024", "2025"):
    leaf = root / f"year={year}"
    leaf.mkdir(parents=True)
    (leaf / "part-0.parquet").write_bytes(b"PAR1")

lake = IOBase.from_fs(pafs.LocalFileSystem(), root.as_posix())

assert lake.is_dir()
assert len(list(lake.iterdir())) == 2
assert len(list(lake.glob("**/*.parquet"))) == 2
assert len(list(lake.children_where({"year": "2024"}))) == 1

# A child still carries the filesystem it came from.
part = lake / "year=2024" / "part-0.parquet"
assert part.read_bytes() == b"PAR1"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

// A directory is a prefix, so `list` derives one from the keys rather
// than storing markers the caller's storage would not have written.
const files = new Map([
  ['bucket/year=2024/part-0.parquet', Buffer.from('PAR1')],
  ['bucket/year=2025/part-0.parquet', Buffer.from('PAR1')],
])
const under = (prefix) =>
  [...files.keys()].filter((name) => prefix === '' || name.startsWith(`${prefix}/`))
const handler = {
  typeName: 'memory',
  fileInfo(path) {
    if (files.has(path)) return { path, kind: 'file', size: BigInt(files.get(path).length) }
    return under(path).length ? { path, kind: 'directory' } : { path, kind: 'not-found' }
  },
  list(path, recursive) {
    const prefix = path === '' ? '' : `${path}/`
    const directories = new Set()
    const found = []
    for (const name of under(path)) {
      const parts = name.slice(prefix.length).split('/')
      for (let depth = 1; depth < parts.length; depth += 1) {
        if (recursive || depth === 1) directories.add(prefix + parts.slice(0, depth).join('/'))
      }
      if (parts.length === 1 || recursive) {
        found.push({ path: name, kind: 'file', size: BigInt(files.get(name).length) })
      }
    }
    for (const name of directories) found.push({ path: name, kind: 'directory' })
    return found
  },
  readRange: (path, offset, length) =>
    files.get(path)?.subarray(Number(offset), Number(offset) + length) ?? null,
  writeFull: (path, bytes) => { files.set(path, Buffer.from(bytes)) },
  createDir: () => {},
  deleteFile: (path) => { files.delete(path) },
}

const lake = IOBase.fromFs(handler, 'bucket')

assert.equal(lake.isDir(), true)
assert.equal([...lake.ls()].length, 2)
assert.equal([...lake.glob('**/*.parquet')].length, 2)
assert.equal([...lake.glob('year=2024/**/*.parquet')].length, 1)
assert.equal([...lake.childrenWhere({ year: '2024' })].length, 1)

// A child still carries the filesystem it came from.
assert.equal(lake.joinpath('year=2024', 'part-0.parquet').readText(), 'PAR1')

Records

The record surface is the same read plus three explicit write intents every handle answers, inherited rather than reimplemented, so the encoding still comes from the media type and never from an argument. Their canonical signatures and intent rules apply unchanged to an Arrow filesystem handle.

RustPythonJavaScript

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch, StringArray};
use yggdryl::holder::fs::{File, MemoryFileSystem};
use yggdryl::media::IORecordOptions;
use yggdryl::{IOBase, IOMedia};
use yggdryl::DataType;

let schema = DataType::from_fields([
    DataType::Int64.required_field("id"),
    DataType::Utf8.nullable_field("symbol"),
])?
.required_field("row");

let batch = RecordBatch::try_new(
    schema.clone().into_arrow_schema()?,
    vec![
        Arc::new(Int64Array::from(vec![1, 2])),
        Arc::new(StringArray::from(vec![Some("AAPL"), None])),
    ],
)?;

let filesystem = Arc::new(MemoryFileSystem::new());
let mut handle = File::from_location(filesystem, "bucket/trades.parquet")?;
let options = handle.record_options()?.with_field(schema.clone());

handle.overwrite_arrow_reader(
    yggdryl::arrow::batch_reader(batch.schema(), [batch]),
    &options,
)?;
handle.close()?;

let rows: usize = handle
    .read_arrow_reader(&options)?
    .map(|batch| batch.unwrap().num_rows())
    .sum();
assert_eq!(rows, 2);
assert_eq!(handle.read_arrow_field(&options)?, schema);

import tempfile, pathlib
import pyarrow as pa
import pyarrow.fs as pafs
import pyarrow.parquet as pq
from yggdryl import IOBase

root = pathlib.Path(tempfile.mkdtemp())
table = pa.table({"id": [1, 2], "symbol": ["AAPL", "MSFT"]})

handle = IOBase.from_fs(pafs.LocalFileSystem(), (root / "trades.parquet").as_posix())
with handle:
    handle.overwrite_arrow_table(table)

assert handle.read_arrow_reader().read_all().num_rows == 2

# What landed is an ordinary Parquet file, so PyArrow reads it back.
assert pq.read_table(root / "trades.parquet").equals(table)

const assert = require('node:assert/strict')
const arrow = require('apache-arrow')
const { BatchReader, IOBase } = require('yggdryl')

const files = new Map()
const handler = {
  typeName: 'memory',
  fileInfo: (path) =>
    files.has(path)
      ? { path, kind: 'file', size: BigInt(files.get(path).length) }
      : { path, kind: 'not-found' },
  list: () => [],
  readRange: (path, offset, length) =>
    files.get(path)?.subarray(Number(offset), Number(offset) + length) ?? null,
  writeFull: (path, bytes) => { files.set(path, Buffer.from(bytes)) },
  createDir: () => {},
  deleteFile: (path) => { files.delete(path) },
}

const table = new arrow.Table({
  id: arrow.vectorFromArray([1n, 2n], new arrow.Int64()),
  symbol: arrow.vectorFromArray(['AAPL', 'MSFT'], new arrow.Utf8()),
})

const handle = IOBase.fromFs(handler, 'bucket/trades.arrows')
handle.overwriteArrowReader(BatchReader.from(table))
handle.close()

assert.equal(handle.readArrowReader().intoTable().numRows, 2)
// The encoding came from the name, never from an argument.
assert.equal(String(handle.mediaType), 'application/vnd.apache.arrow.stream')

A folder handle reads as the partitioned table beneath it, and a folder holding an Iceberg metadata document reads through its snapshots - both are the container behavior every backend inherits.

Composing with the wrappers

Nothing about a foreign filesystem is special to the wrappers, because they only ever see an IOBase. A content coding round trips over a bucket exactly as it does over a file:

Rust only

The Python and JavaScript packages do not expose the compression wrappers. The Iceberg composition below carries its own tabs.

Rust

use std::sync::Arc;

use yggdryl::holder::fs::{File, MemoryFileSystem};
use yggdryl::IOBase;
use yggdryl::coding::Coded;
use yggdryl::{Codec, MimeType};

let filesystem = Arc::new(MemoryFileSystem::new());
let leaf = File::from_location(filesystem.clone(), "bucket/trades.json.gz")?;

let mut coded = Coded::wrap(leaf, Codec::Gzip);
coded.write_all_bytes(br#"{"symbol":"AAPL"}"#)?;
coded.close()?;

// The view presents the decoded value...
assert_eq!(coded.media_type().base(), &MimeType::JSON);
assert_eq!(coded.read_all_bytes()?, br#"{"symbol":"AAPL"}"#);

// ...while what the filesystem holds is gzip.
let stored = File::from_location(filesystem, "bucket/trades.json.gz")?;
assert_eq!(&stored.read_all_bytes()?[..2], &[0x1f, 0x8b]);

An Iceberg table is a folder reached through IOBase only, so a warehouse on a foreign filesystem needs nothing from the table format:

RustPython

use std::sync::Arc;

use arrow_array::{Int64Array, RecordBatch};
use yggdryl::holder::fs::{Folder, MemoryFileSystem};
use yggdryl::media::iceberg::{FormatVersion, PartitionSpec, Table};
use yggdryl::{IOBase, IOMedia};
use yggdryl::DataType;

let schema = DataType::from_fields([DataType::Int64.required_field("id")])?
    .required_field("row");
let batch = RecordBatch::try_new(
    schema.clone().into_arrow_schema()?,
    vec![Arc::new(Int64Array::from(vec![1, 2]))],
)?;

let filesystem = Arc::new(MemoryFileSystem::new());
let root = Folder::from_location(filesystem, "warehouse/trades")?;

let mut table = Table::create(
    root,
    FormatVersion::V2,
    schema,
    PartitionSpec::unpartitioned(),
)?;
table.commit_append(yggdryl::arrow::batch_reader(batch.schema(), [batch]))?;

let options = table.record_options()?;
let rows: usize = table
    .read_arrow_reader(&options)?
    .map(|batch| batch.unwrap().num_rows())
    .sum();
assert_eq!(rows, 2);

import tempfile, pathlib
import pyarrow as pa
import pyarrow.fs as pafs
from yggdryl import IOBase
from yggdryl.media import iceberg

root = pathlib.Path(tempfile.mkdtemp())
table_rows = pa.table({"id": [1, 2], "symbol": ["AAPL", "MSFT"]})

warehouse = IOBase.from_fs(pafs.LocalFileSystem(), (root / "trades").as_posix())
table = iceberg.Table.create(warehouse, table_rows.schema)
table.append(table_rows)

assert table.scan().read_all().num_rows == 2

The two filesystems that ship here

MemoryFileSystem holds everything in one map and is the substrate the tests and benchmarks run on. LocalFileSystem is a thin std::fs mapping whose writes publish through a temporary file and a rename, so a reader never observes a half-written value; it exists to prove the contract against a real operating-system filesystem and to measure the wrapper against a native handle. Neither replaces local, whose memory-mapped File remains the local backend.

Rust only

Both are Rust types. A binding reaches a filesystem through the one its own ecosystem already has - pyarrow.fs in Python, a handler object in JavaScript.

Rust

use std::sync::Arc;

use yggdryl::holder::fs::{File, LocalFileSystem};
use yggdryl::IOBase;
use yggdryl::holder::local::Folder;

let root = Folder::temporary()?.path()?.join(format!("yggdryl-doc-fs-{}", std::process::id()));
std::fs::create_dir_all(&root)?;
let location = root.join("trades.bin").to_string_lossy().replace('\\', "/");

let mut handle = File::from_location(Arc::new(LocalFileSystem::new()), &location)?;
handle.write_all_bytes(b"AAPL")?;
handle.close()?;

assert_eq!(std::fs::read(root.join("trades.bin"))?, b"AAPL");
let _ = std::fs::remove_dir_all(&root);

Bringing your own filesystem

In Rust, implement FileSystem. Seven methods, all synchronous, and the semantics are the ones Arrow already specifies: a path that is not there is Unknown rather than an error, a missing directory lists empty, a read past the end is short, and a write replaces the whole value.

Rust

use std::sync::Arc;

use yggdryl::holder::fs::{FileSystem, FileInfo, File};
use yggdryl::IOBase;
use yggdryl::Result;

/// A filesystem holding exactly one read-only object.
struct OneObject;

impl FileSystem for OneObject {
    fn type_name(&self) -> &str {
        "memory"
    }

    fn file_info(&self, path: &str) -> Result<FileInfo> {
        Ok(if path == "bucket/only.bin" {
            FileInfo::file(path, 5)
        } else {
            FileInfo::not_found(path)
        })
    }

    fn list(&self, _path: &str, _recursive: bool) -> yggdryl::holder::fs::FileInfos {
        yggdryl::holder::fs::FileInfos::new(
            [Ok(FileInfo::file("bucket/only.bin", 5))].into_iter(),
        )
    }

    fn read_range(&self, path: &str, offset: u64, buffer: &mut [u8]) -> Result<usize> {
        if path != "bucket/only.bin" {
            return Ok(0);
        }
        let value = b"AAPL!";
        let offset = offset as usize;
        if offset >= value.len() {
            return Ok(0);
        }
        let count = (value.len() - offset).min(buffer.len());
        buffer[..count].copy_from_slice(&value[offset..offset + count]);
        Ok(count)
    }

    fn write_full(&self, _path: &str, _bytes: &[u8]) -> Result<()> {
        Ok(())
    }

    fn create_dir(&self, _path: &str) -> Result<()> {
        Ok(())
    }

    fn delete_file(&self, _path: &str) -> Result<()> {
        Ok(())
    }
}

let handle = File::from_location(Arc::new(OneObject), "bucket/only.bin")?;
assert_eq!(handle.read_all_bytes()?, b"AAPL!");
assert_eq!(handle.read_range_bytes(1, 3)?, b"APL");

In Python, write a pyarrow.fs.FileSystemHandler and wrap it in pyarrow.fs.PyFileSystem. That is also how an fsspec filesystem arrives, so this one shape covers both:

Python

import pyarrow.fs as pafs
from yggdryl import IOBase

class MyHandler(pafs.FileSystemHandler):
    ...  # get_file_info, get_file_info_selector, open_input_file, open_output_stream, ...

handle = IOBase.from_fs(pafs.PyFileSystem(MyHandler()), "bucket/key.parquet")

A working handler is longer than a documentation page wants, so the complete one lives in python/tests/test_fs.py, where it is exercised end to end - including an Iceberg table whose every byte goes through it.

In JavaScript there is nothing to wrap, because Arrow JS ships no filesystem - so the handler object is the filesystem, and the six methods are the whole contract. Anything a Node program can already reach answers them:

JavaScript

const assert = require('node:assert/strict')
const fs = require('node:fs')
const os = require('node:os')
const path = require('node:path')
const { IOBase } = require('yggdryl')

const root = fs.mkdtempSync(path.join(os.tmpdir(), 'yggdryl-doc-fs-'))

// The same six calls, over node:fs. Absence may throw the way node:fs
// does - the boundary asks what is there and turns ENOENT into the
// contract's empty answer.
const handler = {
  typeName: 'local',
  fileInfo(location) {
    try {
      const stat = fs.statSync(location)
      return {
        path: location,
        kind: stat.isDirectory() ? 'directory' : 'file',
        size: BigInt(stat.size),
      }
    } catch {
      return { path: location, kind: 'not-found' }
    }
  },
  list(location, recursive) {
    const found = []
    for (const entry of fs.readdirSync(location, { withFileTypes: true })) {
      const child = path.posix.join(location, entry.name)
      if (entry.isDirectory()) {
        found.push({ path: child, kind: 'directory' })
        if (recursive) found.push(...this.list(child, true))
      } else {
        found.push({ path: child, kind: 'file', size: BigInt(fs.statSync(child).size) })
      }
    }
    return found
  },
  readRange(location, offset, length) {
    const descriptor = fs.openSync(location, 'r')
    try {
      const buffer = Buffer.alloc(length)
      const read = fs.readSync(descriptor, buffer, 0, length, Number(offset))
      return buffer.subarray(0, read)
    } finally {
      fs.closeSync(descriptor)
    }
  },
  writeFull(location, bytes) {
    fs.mkdirSync(path.posix.dirname(location), { recursive: true })
    fs.writeFileSync(location, bytes)
  },
  createDir: (location) => fs.mkdirSync(location, { recursive: true }),
  deleteFile: (location) => fs.rmSync(location, { force: true }),
}

const handle = IOBase.fromFs(handler, path.posix.join(root, 'lake', 'trades.bin'))
handle.writeText('AAPL')
handle.close()

assert.equal(fs.readFileSync(path.join(root, 'lake', 'trades.bin'), 'utf8'), 'AAPL')
assert.equal(handle.readRangeBytes(1, 3).toString(), 'APL')

fs.rmSync(root, { recursive: true, force: true })

What the wrapper costs

Putting an existing Arrow-compatible filesystem behind IOBase, the only honest question is what the wrapper adds to the transport underneath it. Every row below is the same payload landing in the same place twice: once through an fs handle, once through the native handle (or the language's own filesystem calls) holding those same bytes.

RustPythonJavaScript

cargo bench --bench fs --features "parquet", Criterion medians, 512 KiB payloads and 65,536 rows:

                                  fs      native handle
bytes read_all   (memory)         22.99 us     23.68 us   Buffer
bytes write_all  (memory)         67.00 us     24.85 us   Buffer
bytes pread 4KiB (memory)         63.91 ns     35.05 ns   Buffer
bytes read_all   (local)          25.82 us     23.51 us   local::File
bytes write_all  (local)         212.43 us      1.11 ms   local::File
ipc     write                      4.08 ms      4.44 ms   Buffer
ipc     read                       1.49 ms      1.30 ms   Buffer
parquet write                     18.16 ms     17.84 ms   Buffer
parquet read                       5.17 ms      5.01 ms   Buffer
ls recursive     (local)          61.49 us     92.84 us   local::Folder

The ranged read is the row that matters most, and it is the one stated in nanoseconds. Serving 4 KiB out of a 512 KiB value costs 64 ns, not the 23 us a whole-value read costs, so the handle serves a range without materializing the value. The vtable itself is the 29 ns difference against Buffer: one dynamic call plus a bounds check. This measures the handle, not any reader above it - parquet still fetches its value whole, as that page says.

Whole-value writes are where staging shows. An Arrow filesystem replaces files rather than writing ranges, so a write is buffered and published once - 67 us against Buffer's 25 us for 512 KiB, which is the copy the publication costs. Against the memory-mapped local::File the same write is five times faster (212 us against 1.11 ms), because publishing a whole file through a temporary and a rename beats remapping and resizing a mapping. Neither number makes one backend better than the other; they measure different write shapes, which is exactly why both exist.

Records are within a few percent either way, because the encoding dominates and the wrapper only moves the finished bytes. Listing is faster than the local backend's because one list call answers a recursive walk that std::fs::read_dir has to make per directory.

glob over the same tree shows the descent the contract promises: expanding **/*.parquet across a 16-leaf lake costs 57 us, while year=2024/**/*.parquet costs 23 us, because a fixed prefix is descended rather than listed and filtered.

The baseline is PyArrow's own calls against the same pyarrow.fs.LocalFileSystem - the implementation the wrapper delegates to - so the difference is the vtable crossing and nothing else. fs.py --min-time 0.2 --repeat 7, release wheel, medians:

                        wrapper      PyArrow
bytes write            400.0 us     244.3 us
bytes read             115.2 us      17.6 us
range read (4 KiB)      14.5 us       2.9 us
parquet write            8.16 ms      6.19 ms
parquet read             2.00 ms      1.95 ms
listing (16 entries)    85.5 us      34.4 us

A Parquet read is at parity, because the decode dominates and the boundary moves only finished bytes. Everything smaller is dominated by the crossing itself: each vtable call acquires the GIL and makes a handful of PyArrow calls, which is roughly 12 us of fixed cost, so the 4 KiB range read costs 14.5 us against PyArrow's 2.9 us. That cost is per call, not per byte - the ranged read still reads 4 KiB rather than the 512 KiB object, which is the property that matters on an object store, and it is why the read is 14.5 us rather than the 115 us a whole-value read takes.

A write costs more than PyArrow's because it is a different operation: the wrapper stages the value and publishes it once, which is what makes a positional pwrite API work over a filesystem that only replaces whole files.

JavaScript pays the same shape of cost against node:fs, with the handler crossing the boundary on every call rather than only the handle. bench:fs, release build:

                            wrapper       node:fs
handle from path         107,101/s     257,632/s
write bytes                4,912/s      11,235/s
read bytes                13,193/s      58,399/s
read range (4 KiB)       124,733/s     227,893/s
list children             58,181/s     264,682/s
glob *.parquet             9,372/s      16,900/s
read records            15.6M rows/s   11.6M rows/s (local handle)
write records           10.2M rows/s   22.1M rows/s (local handle)

The ranged read is again the row that carries the claim: it is the fastest byte operation of the three, not the slowest, because it fetches 4 KiB rather than the whole payload. Records read faster than through the local handle because the staged value is already in memory once the first read has fetched it, and slower to write for the same reason a Python write is - the value is staged and published once.

Buffered handles

A page cache that makes any IOBase handle buffered, with the value's first and last pages pinned.

RustPythonJavaScript

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let handle = Buffer::from_bytes(b"symbol,price\nAAPL,1\n".to_vec())
    .buffered(BufferedOptions::default());

assert_eq!(handle.read_range_bytes(0, 6)?, b"symbol");
assert_eq!(handle.read_range_bytes(13, 4)?, b"AAPL");
assert_eq!(handle.cached_pages(), 1);

from yggdryl import IOBase

handle = IOBase.from_bytes(b"symbol,price\nAAPL,1\n")
assert handle.buffered(page_size=64, max_bytes=256, ttl=30.0) is handle
assert handle.read_range_bytes(0, 6) == b"symbol"
assert handle.read_range_bytes(13, 4) == b"AAPL"

const assert = require('node:assert/strict')
const { IOBase } = require('yggdryl')

const handle = IOBase.fromBytes(Buffer.from('symbol,price\nAAPL,1\n'))
assert.equal(handle.buffered({ pageSize: 64, maxBytes: 256, ttlMs: 30_000 }), handle)
assert.deepEqual(handle.readRangeBytes(0, 6), Buffer.from('symbol'))
assert.deepEqual(handle.readRangeBytes(13, 4), Buffer.from('AAPL'))

IOBase::buffered wraps any handle, and what comes back is a handle: Buffered<H> mirrors everything it does not change, so size, url, media_type, kind, parent, child_by_path, and ls answer exactly what the wrapped handle answers. It is invisible except for speed, the same way Coded is invisible except for the coding.

Nothing is cached at construction. Per the laziness contract, a handle is a description of where bytes would live, and the cache only ever holds pages a read asked for. Calling buffered again reconfigures the same cache layer; it never stacks a second one. Rust additionally exposes page-inspection methods used by the detailed examples below.

Pages

use yggdryl::holder::buffered::{Buffered, BufferedOptions};
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

// 256-byte pages, so a 1 KiB value is four of them.
let options = BufferedOptions::default().with_page_size(256);
let handle = Buffered::new(Buffer::from_bytes(vec![7_u8; 1_024]), options);

// A read that misses fetches the whole page holding it, aligned.
assert_eq!(handle.read_range_bytes(300, 4)?.len(), 4);
assert_eq!(handle.cached_pages(), 1);
assert_eq!(handle.cached_bytes(), 256);
assert!(handle.has_cached_page(1));

// A read spanning pages assembles from each of them, caching all it crossed.
assert_eq!(handle.read_range_bytes(100, 600)?.len(), 600);
assert_eq!(handle.cached_pages(), 3);

// The page a given offset lives in is arithmetic, not a lookup.
assert_eq!(handle.options().page_index(700), 2);
assert_eq!(handle.options().page_start(2), 512);

A miss reads page-aligned from the inner handle and copies the requested range out; a hit copies straight from the held page. A read crossing several pages copies each of them into the caller's buffer directly, so nothing is assembled in between.

The three knobs

use std::time::Duration;

use yggdryl::holder::buffered::BufferedOptions;

let options = BufferedOptions::default();
assert_eq!(options.page_size(), 64 * 1024);
assert_eq!(options.max_bytes(), 8 * 1024 * 1024);
assert_eq!(options.ttl(), Duration::from_secs(30));

// A page size is rounded up to a power of two and clamped to 64 ..= 1 GiB.
assert_eq!(BufferedOptions::default().with_page_size(1_000).page_size(), 1_024);
assert_eq!(BufferedOptions::default().with_page_size(0).page_size(), 64);

// A budget below two pages is clamped up to exactly two, never rejected,
// because the two pinned pages have to fit for the cache to work at all.
let tight = BufferedOptions::default().with_page_size(4_096).with_max_bytes(1);
assert_eq!(tight.max_bytes(), 8_192);

// Raising the page size re-applies that clamp to a budget set earlier.
let grown = BufferedOptions::default()
    .with_page_size(1_024)
    .with_max_bytes(4_096)
    .with_page_size(8_192);
assert_eq!(grown.max_bytes(), 16_384);

Page size is the unit a miss fetches. Max bytes is the budget every cached page shares, pinned ones included; over it, the least recently read page leaves first. Time to live is counted from a page's last access, so a page that keeps being read never expires. Expiry is lazy - a lapsed page is discarded when it is touched, and a miss sweeps the table - so nothing here runs a background thread.

Both ends are pinned

The first page, and the page holding the current last byte, are exempt from eviction and from expiry. Both ends of a value are where discovery lives - magic bytes and media-type sniffing at the head, a Parquet footer or an Arrow IPC schema and end-of-stream block at the tail - and they are re-read constantly, so they must never be what a sweep takes.

use yggdryl::holder::buffered::{Buffered, BufferedOptions};
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

// Sixteen pages of value, four pages of budget.
let options = BufferedOptions::default()
    .with_page_size(64)
    .with_max_bytes(4 * 64);
let handle = Buffered::new(Buffer::from_bytes(vec![1_u8; 16 * 64]), options);

// The footer first, then the header: the shape a container is opened with.
handle.read_range_bytes(16 * 64 - 8, 8)?;
handle.read_range_bytes(0, 8)?;

// Then a scan of the middle, four times what the budget can hold.
for page in 1..15 {
    handle.read_range_bytes(page * 64, 8)?;
}

// The budget held throughout, the middle was evicted, and both ends stayed.
assert!(handle.cached_bytes() <= handle.options().max_bytes());
assert!(handle.has_cached_page(0));
assert!(handle.has_cached_page(15));
assert!(!handle.has_cached_page(7));

Two consequences worth stating, because both are observable:

• Pinned pages count toward the budget. That is why max_bytes is clamped to at least two pages: a budget that could not hold both ends would leave the cache thrashing.
• The pin follows the current end. A write or a truncate that moves the size releases the page that used to be last, and the new last page is pinned the next time it is cached. Pinning is a retention guarantee, never a prefetch: a pinned page is still filled lazily, by the first read that wants it.

use yggdryl::holder::buffered::{Buffered, BufferedOptions};
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let options = BufferedOptions::default()
    .with_page_size(64)
    .with_max_bytes(4 * 64);
let mut handle = Buffered::new(Buffer::from_bytes(vec![1_u8; 4 * 64]), options);

// Page 3 ends the value, so it holds a pin.
assert_eq!(handle.read_all_bytes()?.len(), 4 * 64);
assert!(handle.has_cached_page(3));

// A write doubling the value moves the end; page 3 is ordinary again, and a
// scan under budget pressure now evicts it while page 0 stays.
handle.pwrite(8 * 64 - 1, b"z")?;
for page in 4..8 {
    handle.read_range_bytes(page * 64, 8)?;
}
handle.read_range_bytes(5 * 64, 8)?;
handle.read_range_bytes(6 * 64, 8)?;
assert!(handle.has_cached_page(0));
assert!(handle.has_cached_page(7));
assert!(!handle.has_cached_page(3));

Writes are never stale

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let mut handle = Buffer::from_bytes(b"symbol,price\nAAPL,1\n".to_vec())
    .buffered(BufferedOptions::default());
assert_eq!(handle.read_all_bytes()?.len(), 20);

// A write goes straight to the wrapped handle and folds into the pages it
// overlapped, so the read after it can never see the bytes it replaced.
handle.pwrite(13, b"MSFT")?;
assert_eq!(handle.read_range_bytes(13, 4)?, b"MSFT");
assert_eq!(handle.handle().as_slice()[13..17], *b"MSFT");

// Truncating drops every page a resize could have changed, both ways.
handle.truncate(13)?;
assert_eq!(handle.read_all_bytes()?, b"symbol,price\n");
handle.truncate(15)?;
assert_eq!(handle.read_all_bytes()?, b"symbol,price\n\0\0");

pwrite is write-through: the bytes reach the inner handle first, and then every cached page they overlapped is patched with them or dropped. truncate delegates and invalidates everything at or past the new size. flush delegates. close flushes and drops the whole cache - pinned pages included, because closing releases cached state - and leaves a working handle behind that simply fetches again.

clear and remove drop the whole cache too, and they drop it before delegating: a page that outlived either would answer a later read with bytes that are gone, and one that outlived a failed removal would describe a resource whose state is no longer known. That ordering is why they are written out rather than delegated by the macro - a body the macro provides cannot be overridden, and a cache wrapper has to invalidate as part of the call.

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let mut handle = Buffer::from_bytes(vec![3_u8; 4_096]).buffered(BufferedOptions::default());
assert_eq!(handle.read_all_bytes()?.len(), 4_096);
assert_eq!(handle.cached_pages(), 1);

handle.close()?;
assert_eq!(handle.cached_pages(), 0);
assert_eq!(handle.read_range_bytes(0, 4)?, [3, 3, 3, 3]);

The one thing the cache cannot see is a change made behind it - bytes written straight to the inner handle, or to the same file by another process. handle_mut therefore drops every page before it hands the inner handle over, and clear_cache is the same thing said explicitly.

Over a compressed handle

A content coding is not seekable. A closed Coded positional read decodes through the requested range and retains nothing. Wrapping it retains the decoded pages instead, so a hit performs no second decode and a miss restarts only as far as that page.

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::coding::gzip::Gzip;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let payload = "symbol,price\nAAPL,1\n".repeat(512).into_bytes();
let mut source = Gzip::new(Buffer::new());
source.write_all_bytes(&payload)?;
source.flush()?;
let encoded = source.into_handle()?;

// The cache wraps the coding, so the pages it holds are decoded bytes.
let handle = Gzip::new(encoded).buffered(BufferedOptions::default());
assert_eq!(handle.read_range_bytes(0, 6)?, b"symbol");
assert_eq!(handle.read_range_bytes(13, 4)?, b"AAPL");

// Two reads, one page, one decode.
assert_eq!(handle.cached_pages(), 1);
assert_eq!(handle.size(), payload.len() as u64);

Three things follow, and the measurements below cover all of them:

• The order of wrapping matters. Buffered<Coded<_>> caches decoded bytes; Coded<Buffered<_>> caches encoded transport.
• Choose by access shape. pstream_bytes keeps one decoder and zero pages for a scan; Buffered<Coded<_>> retains bounded decoded pages for locality; open materializes once for repeated random access and close releases it.
• A page miss still starts at the frame beginning. Compression has no decoded seek, so later misses cost more than early ones even though none retains the whole value.

Wrapping twice wraps once

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::holder::Holder;
use yggdryl::IOBase;
use yggdryl::holder::Buffer;

let once = Buffer::from_bytes(vec![5_u8; 128]).buffered(BufferedOptions::default());

// `Buffered` has an inherent `buffered`, which wins method resolution, so
// this re-wraps the handle it holds instead of stacking a second cache.
let twice = once.buffered(BufferedOptions::default().with_page_size(512));
assert_eq!(twice.options().page_size(), 512);
assert_eq!(twice.read_range_bytes(0, 4)?, [5, 5, 5, 5]);

// A holder does the same, so a listing entry can be buffered without care.
let held = Holder::buffer(Buffer::from_bytes(vec![5_u8; 128]))
    .buffered(BufferedOptions::default())
    .buffered(BufferedOptions::default());
assert!(matches!(&held, Holder::Buffered(inner) if matches!(inner.handle(), Holder::Buffer(_))));

// `into_handle` gives the wrapped handle back, cache dropped.
let inner: Buffer = twice.into_handle();
assert_eq!(inner.size(), 128);

Cursors ride the cache

use std::io::{Read, Seek, SeekFrom};

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::{IOBase, IOCursor};
use yggdryl::holder::Buffer;

let payload: Vec<u8> = (0..1_024_u32).map(|index| index as u8).collect();
let mut cursor = Buffer::from_bytes(payload)
    .buffered(BufferedOptions::default().with_page_size(256))
    .cursor();

// Sequential reads stream across page boundaries through the cache.
let mut chunk = [0_u8; 300];
cursor.read_exact(&mut chunk)?;
assert_eq!(cursor.tell(), 300);
assert_eq!(chunk[299], 299_u32 as u8);

// A seek to the end lands on the pinned footer page. `IOCursor` and
// `std::io::Seek` both spell `seek`, so this one names the trait it means.
IOCursor::seek(&mut cursor, SeekFrom::End(-4))?;
cursor.read_exact(&mut chunk[..4])?;
assert_eq!(chunk[3], 1_023_u32 as u8);
assert_eq!(cursor.handle().cached_pages(), 3);

cursor and cursor_at come from IOBase itself, so a buffered handle gets the same Read, Write, and Seek implementations every other handle gets - reads and writes just go through the pages.

A file, and what the cache is for

use yggdryl::holder::buffered::BufferedOptions;
use yggdryl::IOBase;
use yggdryl::holder::local::{File, Folder};

let path = Folder::temporary()?.path()?.join(format!("yggdryl-doc-buffered-{}.bin", std::process::id()));
std::fs::write(&path, vec![9_u8; 4_096])?;

let handle = File::new(&path)?.buffered(BufferedOptions::default().with_page_size(1_024));
assert_eq!(handle.size(), 4_096);
assert_eq!(handle.read_range_bytes(2_000, 8)?, [9_u8; 8]);
assert_eq!(handle.cached_pages(), 1);

// The wrapper is the file for every purpose but the reading.
let bare = File::new(&path)?;
assert_eq!(handle.url(), bare.url());

drop(handle);
let _ = std::fs::remove_file(&path);

Over a memory-mapped local file, a pread is already a memcpy out of the page cache the kernel keeps, so wrapping one buys little and costs a lock and a second copy - the measurements below say by how much.

The cache earns its keep where a fetch is not a memcpy, and the core ships two such handles. An fs handle answers every read with one read_range call through the foreign-filesystem vtable - over LocalFileSystem an open, a seek and a read per call, and over an object store a round trip - and a coded handle decodes. Both are the same code as the mapped case, which is the point of the wrapper: what changes is only how much the fetch it removes was worth.

What the cache buys, and what it costs

io_buffered runs three read workloads over one 16 MiB fixture and every handle the core ships, plus a fourth workload over a compressed one. The handles fall into two families, and that split is the whole result:

• Already memory. An in-memory Buffer, a memory-mapped local::File, and an fs handle over MemoryFileSystem. A read is a memcpy - out of a Vec, out of the mapping the kernel already caches, or out of the vtable's own map.
• A fetch per read. An fs handle over LocalFileSystem, where every pread is an open, a seek and a read, and every size is a stat. That is the shape of every object store, and the only such backend the core ships.

random reads 512 bytes at a time inside a 4 MiB hot region that fits the 8 MiB budget; sequential scans the whole 16 MiB in 8 KiB steps, twice the budget, so every page is fetched once and evicted before it is wanted again; footer reads both ends, sweeps 12 MiB of the middle, and reads both ends again.

From one containerized x86_64 Linux run with the group run alone (cargo bench --bench io --features "parquet" -- io_buffered; Criterion, 100 samples, medians). This box's run-to-run spread is wide - a case can move 15% between runs - so read the multiples, never the percentages:

                                     random        sequential        footer
buffer                             10.041 µs        606.58 µs      439.58 µs
file                               28.037 µs        517.80 µs      367.66 µs
buffered  (over file)              75.362 µs      1.9495 ms        507.49 µs
fs_memory                     46.739 µs        734.30 µs      392.83 µs
fs_memory_buffered            77.633 µs      2.0068 ms        503.69 µs
fs_local                     1.0832 ms       2.7574 ms       2.0924 ms
fs_local_buffered             73.668 µs      2.3841 ms        532.18 µs

Over a backend that is already memory, the cache is a cost; over one that fetches, it is worth 4x to 15x. The same code, the same page table, the same pinning:

Workload	fs_local	with the cache
random (a hot region, re-read)	1.0832 ms	73.668 µs	14.7x faster
footer (both ends, big middle)	2.0924 ms	532.18 µs	3.9x faster
sequential (one pass, nothing re-read)	2.7574 ms	2.3841 ms	1.2x faster

The ordering of those three is the useful part. A cache pays where reads repeat - a hot region, or the two ends of a footer-first container - and barely pays where every byte is read exactly once, because a one-pass scan copies each byte twice and reuses none of it. The sequential row is the honest floor: 8 KiB reads through 64 KiB pages, so the cache still turns eight open/seek/read triples into one, and that is worth 16%.

Over the memory-like handles the same cache costs 2.7x (random, against file), because there was no fetch to remove: a hit is a clock read, a lock, a map lookup and a copy against a memcpy that was going to happen anyway. That is the price of not knowing what you were handed, and it is why the wrapper is opt-in rather than automatic.

Two changes during this work moved these numbers, and both are in the diff:

• A hit asks the handle for nothing. read_at used to call size() on every read, for the end-of-value bound and the pin. On fs_local that is a stat per read - the cache paying exactly the cost it exists to remove. The size is now remembered beside the pages and re-asked only when a read runs past what the cache knows. random/fs_local_buffered went 597.88 µs to 73.668 µs and sequential turned from a 1.2x loss into a win.
• Dense page indexes hash with a multiply and a rotation rather than SipHash, and the offset arithmetic shifts rather than divides - which is what the power-of-two page size is for. Together, −20% on the hit case.

What pinning buys cannot be timed over a backend whose re-read is a memcpy, so the target asserts it as a count before any timer starts, over a counting handle: after a 12 MiB middle scan four times wider than the whole budget, re-reading the head and the tail costs zero inner fetches. On fs_local, where a fetch is real, that count is what the footer row's 3.9x is made of.

Over a compressed handle

A content coding is not seekable, so Coded answers a positional read by decoding the value, and which decode it pays depends on whether the handle is open. The coded cases read a 256 KiB gzip value in 64 reads of 4 KiB:

io_buffered/coded/closed      12.182 ms    20.522 MiB/s
io_buffered/coded/open         5.8867 us   41.474 GiB/s
io_buffered/coded/buffered    10.2080 us   23.916 GiB/s

• closed restarts the decoder. Each call now stops after its requested range rather than decoding the whole value, but 64 progressive calls still create 64 decoders and repeatedly discard growing prefixes.
• open serves the one decoded snapshot it explicitly owns. It is the fastest repeated random-access path when holding that complete value is acceptable.
• buffered retains only fetched decoded pages. It turns decoder restarts into page misses and stays within 2x of the opened path for this access pattern.

For a scan, pstream_bytes is the zero-cache choice: it keeps one decoder and leaves cached_pages() == 0. The page cache is for reuse across genuinely positional reads, not a prerequisite for sequential decoding.

The order of wrapping is the useful one: Buffered<Coded<_>> caches the decoded bytes. Coded<Buffered<_>> would cache the compressed bytes and still decode on every read.