yggdrasil.databricks.table.auto_loader

auto_loader

On-cluster Auto Loader (cloudFiles) ingestion entry point.

:func:auto_load is what the Databricks job built by :meth:yggdrasil.databricks.table.table.Table.auto_loader actually runs on the cluster, invoked by the ygg databricks table autoload CLI subcommand (from the shipped ygg wheel). It streams files from a source path into a Unity Catalog table via Spark Structured Streaming + Databricks Auto Loader — incremental, exactly-once, schema-evolving — so a table keeps absorbing new files dropped at source without a bespoke pipeline.

Kept as a plain module-level function (typed params) so the ygg databricks table autoload handler stays thin: it just coerces its CLI args and calls in here.

auto_load

auto_load(
    table: str,
    source: str,
    file_format: str = "parquet",
    checkpoint: str = "",
    available_now: bool = True,
    clean_source: bool = False,
    clean_source_retention: str = "8 days",
) -> dict[str, Any]

Ingest files under source into table with Databricks Auto Loader.

Each micro-batch is cast to the target table's schema before the append (yggdrasil's Schema.cast_spark_tabular — the same field casting arrow_insert / spark_insert use): columns are name-matched, missing ones NULL-filled, and types — including nested array<struct> and timestamp zones — cast to the target. The cast runs on every micro-batch as the stream flows, so the write stays schema-stable and tolerant of source drift (extra columns dropped, mismatches cast) rather than failing the stream or evolving the target schema; .toTable keeps Structured Streaming's built-in, checkpoint-coordinated exactly-once.

Parameters:

Name	Type	Description	Default
table	str	Target table, catalog.schema.table.	required
source	str	Cloud path Auto Loader watches (s3://… / /Volumes/…).	required
file_format	str	cloudFiles.format — parquet / json / csv / avro / delta …	'parquet'
checkpoint	str	Streaming checkpoint + schema location. Empty → derived as <table-storage-location>/_ygg_autoloader from DESCRIBE DETAIL so each table gets its own, co-located with its data.	''
available_now	bool	True (default) runs a single Trigger.AvailableNow micro-batch sweep of everything currently at source then stops — the shape a scheduled / file-arrival job wants. False runs a continuous 1-minute micro-batch stream.	True
clean_source	bool	True makes Auto Loader delete each source file once it's been ingested and is older than the retention window (cloudFiles.cleanSource = DELETE) so a staging area doesn't grow without bound. Cleanup runs at the start of a later micro-batch — it does not delete files within the same one-shot AvailableNow sweep that ingests them — so it's a rolling janitor, not immediate. Default False leaves processed files in place.	False
clean_source_retention	str	Retention window for clean_source (cloudFiles.cleanSource.retentionDuration). Databricks requires an interval greater than 7 days; default "8 days".	'8 days'

Returns a small summary dict (table + resolved checkpoint + rows ingested when known) — handy in the job run output / when called locally.