Yggdryl

Arrow-native schemas, byte storage, and structured values, implemented once in Rust and exposed to Python and JavaScript as views of the same values.

RustPythonJavaScript

use yggdryl::{DataType, Field};

// A non-null struct field is the schema. There is no separate schema type.
let schema = Field::new(
    "row",
    DataType::from_fields([
        DataType::Int64.required_field("id"),
        DataType::Utf8.nullable_field("symbol"),
    ])?,
    false,
);

assert_eq!(schema.field_len(), 2);
assert_eq!(schema.index_of("symbol"), Some(1));
assert!(!schema.fields()[0].is_nullable());

from yggdryl import DataType, Field

# A datatype argument accepts its own expression, so "int64" needs no wrapper.
schema = Field(
    "row",
    DataType.from_fields(
        [Field("id", "int64", nullable=False), Field("symbol", "utf8")]
    ),
    nullable=False,
)

assert len(schema.dtype) == 2
assert schema.dtype[1].name == "symbol"
assert not schema.dtype[0].nullable

const { DataType, Field } = require('yggdryl')
const assert = require('node:assert/strict')

const schema = new Field(
  'row',
  DataType.fromFields([
    new Field('id', 'int64', false),
    new Field('symbol', 'utf8'),
  ]),
  false,
)

assert.equal(schema.dtype.length, 2)
assert.equal(schema.dtype.getFieldAt(1).name, 'symbol')
assert.equal(schema.dtype.getFieldAt(0).nullable, false)

What is here

A schema is a field. DataType is the logical type tree and Field adds a name, nullability, and metadata. A non-null struct field describes rows, so there is no second schema type to keep in sync, and casting reconciles incoming Arrow data to it.

Storage is one trait. IOBase addresses bytes positionally and lazily: building a handle touches nothing, reading something absent yields nothing, writing creates. An in-memory buffer, a local file or directory, and a compressed view of either are all the same trait, and one enum names every implementation.

Records ride on storage. Any handle reads and writes Arrow batches, choosing Arrow IPC or Parquet from its own media type, and Iceberg reads its schemas as ordinary fields.

Scalars are one tree. JSON, YAML, and TOML share the structured value, and URIs name where any of it lives.

Install

RustPythonJavaScript

[dependencies]
yggdryl = "0.1"

# Parquet and Iceberg are opt-in; everything else is on by default.
# yggdryl = { version = "0.1", features = ["parquet", "iceberg"] }

pip install yggdryl

npm install yggdryl

Start with Getting started, or read the architecture for the shape of the whole thing first.