Skip to content

Getting started

Build one of the three runtimes from this repository and describe your first schema.

Build a runtime

cargo build
cargo test --workspace --features "parquet iceberg"

cd python
python -m venv .venv
.venv/Scripts/python -m pip install maturin pytest pyarrow
.venv/Scripts/python -m maturin develop
.venv/Scripts/python -m pytest

cd node
npm install
npm run build:debug
node --test tests/*.test.js

On Linux and macOS the Python interpreter is .venv/bin/python.

Describe a schema

A non-null struct field is the schema, and its children are the columns.

use yggdryl::{DataType, Field};

let schema = Field::new(
    "trade",
    DataType::from_fields([
        DataType::Int64.required_field("id"),
        DataType::Utf8.nullable_field("symbol"),
        DataType::decimal(18, 4)?.required_field("price"),
    ])?,
    false,
);

assert_eq!(schema.field_len(), 3);
assert_eq!(schema.fields()[2].dtype().to_string(), "decimal64(18,4)");

from yggdryl import DataType, Field

schema = Field(
    "trade",
    DataType.from_fields(
        [
            Field("id", "int64", nullable=False),
            Field("symbol", "utf8"),
            Field("price", DataType.decimal(18, 4), nullable=False),
        ]
    ),
    nullable=False,
)

assert len(schema.dtype) == 3
assert str(schema.dtype[2].dtype) == "decimal64(18,4)"

const { DataType, Field } = require('yggdryl')
const assert = require('node:assert/strict')

const schema = new Field(
  'trade',
  DataType.fromFields([
    new Field('id', 'int64', false),
    new Field('symbol', 'utf8'),
    new Field('price', 'decimal(18,4)', false),
  ]),
  false,
)

assert.equal(schema.dtype.length, 3)
assert.equal(String(schema.dtype.getFieldAt(2).dtype), 'decimal64(18,4)')

Attach metadata

Metadata belongs to the field, and it behaves like the mapping type of each language.

use yggdryl::{DataType, Field};

let mut field = Field::new("symbol", DataType::Utf8, true);
field.insert_metadata("source", "book")?;
field.set_parquet_field_id(7);

assert_eq!(field.get_metadata("source"), Some("book"));
assert_eq!(field.parquet_field_id()?, Some(7));

from yggdryl import Field

field = Field("symbol", "utf8", metadata={"source": "book"})
# Metadata is a mapping on `field.metadata`; subscripting the field itself
# reaches a nested child.
field.metadata["venue"] = "XPAR"
field.set_parquet_field_id(7)

assert field.metadata["source"] == "book"
assert "venue" in field.metadata
assert len(field.metadata) == 3
assert field.parquet_field_id == 7

const { Field } = require('yggdryl')
const assert = require('node:assert/strict')

const field = new Field('symbol', 'utf8', true, { source: 'book' })
field.set('venue', 'XPAR')

assert.equal(field.get('source'), 'book')
assert.ok(field.has('venue'))
assert.equal(field.size, 2)

Where to go next

Concern Page
Logical types, parsing, Arrow projection datatype
Names, nullability, metadata, validation, casting field
Arrow scalars, schema projection, batch readers arrow
Reading and writing bytes anywhere io
Local files and directories local
gzip, zlib, zstd gzip, zlib, zstd
Batches on disk ipc, parquet, iceberg
Naming a resource uri
Scalars and text formats text, json, yaml, toml
Language boundaries Python, JavaScript

Repository checks

cargo fmt --all -- --check
cargo clippy --workspace --all-targets --features "parquet iceberg" -- -D warnings
cargo test --workspace --features "parquet iceberg"
python scripts/check_docs_examples.py
python -m mkdocs build --strict