`yggdrasil.url`¶

url ¶

URL `dataclass` ¶

URL(
    scheme: str = "",
    userinfo: str | None = None,
    host: str = "",
    port: int | None = None,
    path: str = "/",
    query: str | None = None,
    fragment: str | None = None,
)

Bases: PathLike

extensions `property` ¶

extensions: list[str]

Return the path's extensions, lowercased, leading dot stripped.

Examples::

URL.from_("/data/file.csv").extensions           == ["csv"]
URL.from_("/data/file.tar.gz").extensions        == ["tar", "gz"]
URL.from_("/data/archive.csv.zst").extensions    == ["csv", "zst"]
URL.from_("/data/README").extensions             == []
URL.from_("/data/.hidden").extensions            == []   # dotfile
URL.from_("/data/.env.local").extensions         == ["local"]

The list ordering is outer-to-inner: for archive.csv.zst you get ["csv", "zst"], matching the codec/media-type refactor convention (outer format first, compression codec last). Leading dotfile marker isn't treated as an extension.

Mirrors :attr:pathlib.PurePosixPath.suffixes but inlined — extensions is read on every codec / media-type dispatch in the IO layer, and PurePosixPath is ~5x slower than the string-level walk this implementation does. The parsed result is memoised on :attr:_extensions_cache so repeat reads collapse to one slot lookup — the cache layer's :meth:Folder._leaf_for walk hits every child URL's extensions on every iter_children.

name `property` ¶

name: str

Last segment of the path, trailing slash stripped.

Examples::

URL.from_("/a/b/c").name   == "c"
URL.from_("/a/b/c/").name  == "c"   # trailing slash ignored
URL.from_("/a/b/").name    == "b"
URL.from_("/").name        == ""
URL.from_("").name         == ""

Inline string handling rather than building self.parts — name is a hot accessor (every IO routing decision) and the list build dominates the cost.

stem `property` ¶

stem: str

The :attr:name with its final extension removed.

Matches :attr:pathlib.PurePosixPath.stem semantics — only the last suffix is stripped, not all of them:

URL.from_("/a/file.csv").stem          == "file"
URL.from_("/a/archive.csv.zst").stem   == "archive.csv"
URL.from_("/a/README").stem            == "README"
URL.from_("/a/.hidden").stem           == ".hidden"  # dotfile
URL.from_("/a/.env.local").stem        == ".env"
URL.from_("/").stem                    == ""
URL.from_("").stem                     == ""

Use :attr:extensions if you need every suffix, or call .stem repeatedly to peel layers.

parent `property` ¶

parent: URL

The URL one path segment up, with query/fragment/authority preserved.

Matches :attr:pathlib.PurePosixPath.parent semantics at the path level — trailing slashes are ignored and the root is its own parent:

URL.from_("https://e.com/a/b/c").parent      # https://e.com/a/b
URL.from_("https://e.com/a/b/c/").parent     # https://e.com/a/b
URL.from_("https://e.com/a").parent          # https://e.com/
URL.from_("https://e.com/").parent           # https://e.com/
URL.from_("/a/b?x=1#frag").parent            # /a?x=1#frag

Query, fragment, scheme, host, userinfo, and port are carried over unchanged. If the inherited query/fragment don't make sense for the parent URL, strip them yourself via :meth:with_query / :meth:with_fragment.

String-level walk rather than PurePosixPath(...).parent — parent is hit by every "walk up the tree" iteration in path registries and listings; PurePosixPath allocates a fresh object each call. The computed parent is memoised on the instance via :attr:_parent_url so the second-and-later access — every iteration of a parent-walk loop — collapses to a slot read.

parents `property` ¶

parents: tuple[URL, ...]

The full ancestry chain, closest first.

A tuple of URLs walking up from :attr:parent to the root. Empty when the URL has no meaningful path:

URL.from_("https://e.com/a/b/c").parents
    # (https://e.com/a/b, https://e.com/a, https://e.com/)
URL.from_("https://e.com/a").parents
    # (https://e.com/,)
URL.from_("https://e.com/").parents
    # ()

Like :attr:parent, every URL in the chain carries the same query, fragment, and authority as self.

media_type `property` ¶

media_type

Inferred :class:MediaType from this URL's extensions.

Memoised on :attr:_media_type_cache — the same URL is hit by every leaf-resolution / codec dispatch in :mod:yggdrasil.io and by the lazy :attr:IO.media_type fallback on every cursor that bound only a URL. None is a real value (URL has no extensions / no registered media type), so the sentinel ... flags "not computed yet".

mime_type `property` ¶

mime_type

The inner :class:MimeType of this URL's :attr:media_type.

For /data/archive.csv.zst this is CSV — the codec wrapper is reported separately via :attr:codec. Returns None when :attr:media_type is None.

codec `property` ¶

codec

The :class:Codec peeled from :attr:media_type, if any.

None for uncompressed URLs (/data/file.csv) and for URLs with no filename. For /data/archive.csv.zst this is the ZSTD codec.

is_memory_address `property` ¶

is_memory_address: bool

True iff this URL is a mem:// in-process handle.

Cheap predicate for dispatch sites that need to branch between real I/O (file, http, s3, ...) and direct object pickup. Pairs with :attr:memory_address and :meth:resolve_memory_address for the actual retrieval.

memory_address `property` ¶

memory_address: int

Integer id() encoded in this mem:// URL.

Raises :class:ValueError if this is not a memory URL or if the path is malformed. Use :attr:is_memory_address to guard the lookup if the scheme is unknown.

is_urlish `classmethod` ¶

is_urlish(value: Any) -> bool

True iff the value is a string or URL-like object.

infer_media_type ¶

infer_media_type(default: 'MediaType' = ...)

Same as :attr:media_type but with a caller-supplied default.

Reuses the :attr:media_type cache when the result would be None (so callers that pass an explicit default for the miss case still hit the slot). The cache itself only holds the default=None answer — None is the value that all other defaults override.

resolve_memory_address ¶

resolve_memory_address() -> object

Dereference this mem:// URL back to the original Python object.

Wraps the module-level :func:resolve_memory_address for ergonomics. The lifetime caveats apply: the caller must have kept a strong reference to the object since the URL was built. See :meth:from_memory_address for the full contract.

from_memory_address `classmethod` ¶

from_memory_address(obj: object) -> 'URL'

Build a mem://<hostname>/<hex_addr> URL pointing at obj.

Encodes id(obj) so the URL can round-trip through code paths that expect a string or :class:URL (cache keys, MediaIO dispatch, pipeline configs). The returned URL is a handle, not a persistent reference: it is valid only within this process and only while the caller holds a strong reference to obj. The host component is the local machine's hostname (cached process-wide) so two memory URLs minted on different hosts never alias even when id() happens to collide across machines.

Lifetime contract:

Same-process, same-interpreter only. No pickling, no cross-process transport, no persistence across restarts.
Caller owns the reference. This constructor does NOT take a reference to obj. If obj is garbage-collected before :meth:resolve_memory_address runs, the address will be reused and resolution returns a different object — or crashes the interpreter if the slot was freed.

See :meth:resolve_memory_address for round-trip retrieval and :attr:is_memory_address for dispatch-site predicates.

Implementation: the rendered path has a leading / so it is a well-formed rooted URL path and bypasses :func:_normalize_path's empty-path coercion. Constructed directly through cls(...) rather than :meth:from_dict because there is nothing to normalize on a hex address — the full :func:_normalize_components pipeline would be wasted work and is in fact unsafe (we don't want anyone deciding to realpath a hex string later).

is_pathish `classmethod` ¶

is_pathish(obj: Any) -> bool

Return True when obj is something :meth:from_ can resolve without resorting to the str(obj) fallback.

Accepts: :class:URL instances, :class:str, :class:pathlib.Path (and any :class:os.PathLike), :class:Mapping (handled by :meth:from_dict), and objects that expose a .url attribute (the duck-typed fallback in :meth:from_).

Rejects None — :meth:from_ raises on None, so it is not pathish. Arbitrary objects without any of the above shapes also return False: :meth:from_ would coerce them via str(obj) and produce nonsense, so calling is_pathish first is the right guard.

This mirrors the is_pathish protocol used elsewhere in the codebase (path_class().is_pathish in MimeType.from_, MediaType.from_), so URL can participate in the same dispatch shape.

from_pathlib `classmethod` ¶

from_pathlib(
    path: Path | str,
    *,
    default_scheme: str | None = None,
    decode: bool = False,
    normalize: bool = True
) -> URL

Build a URL from a pathlib.Path (or a string path).

Expands ~, resolves to an absolute posix path, and prepends a leading slash to Windows drive-letter paths so C:/Users/x becomes /C:/Users/x — the form file URLs use. Passing a string goes through Path first, so from_pathlib("~/data.csv") and from_pathlib(Path("~/data.csv")) produce equal URLs.

Use from_str instead if the input is already a URL string (file:///...) — this method treats its input as a filesystem path, not a URL.

joinpath ¶

joinpath(*segments: Any) -> URL

Append one or more path segments, pathlib-style.

URL("https://e.com/a").joinpath("b", "c") → https://e.com/a/b/c. Scheme, host, userinfo, port, query, and fragment are carried over from self — only the path changes.

Semantics match :class:pathlib.PurePosixPath.joinpath:

A segment starting with / is treated as absolute and resets the accumulated path::
```
URL("/a/b").joinpath("/c")  →  URL("/c")
```
.. is NOT resolved implicitly. URL("/a/b").joinpath("..") yields /a/b/... If you want lexical resolution, run the result through :meth:with_path(..., os_find=True) or handle it at the call site — URL stays syntactic by default.
URL-valued segments contribute only their :attr:path. Use :meth:join instead for RFC 3986 reference resolution (which honours scheme/host on the RHS).
Accepts :class:str, :class:URL, or any :class:os.PathLike. Anything else raises :class:TypeError — url / 42 is almost certainly a bug.

Trailing slashes on self.path are preserved by :class:PurePosixPath semantics: /a/b/ joined with c gives /a/b/c.

match_pattern ¶

match_pattern(pattern: str) -> bool

Glob-match against :attr:name, then against the full URL.

Returns True on either hit. The basename is checked first because *.csv-style filters are the common case and the full URL rendering can be expensive on remote backends (forces :meth:to_string). Uses :func:fnmatch.fnmatch, so the pattern syntax is shell-style: *, ?, [abc], [!abc].

The full rendering is the encoded URL string (scheme + host + path + query + fragment), so patterns like "https://*.com/data/*" work.

matches_patterns ¶

matches_patterns(patterns: Iterable[str] | None) -> bool

Glob-match against any pattern in patterns.

None or an empty iterable returns False — "no patterns" means "no matches", not "everything matches". Callers that want the "include everything when no filter" behaviour should guard at the call site (if not patterns or url.matches_patterns(patterns)).

Basename is checked against every pattern first before the full URL rendering is considered, since :attr:name is cheap and *.ext-style filters are the common case.

is_relative_to ¶

is_relative_to(other: Any) -> bool

True when self.path is equal to or below other.path.

Semantics follow :meth:pathlib.PurePath.is_relative_to at the path level, with an extra URL-aware constraint: when other is a fully-qualified URL (has scheme or host), its scheme and host must match self's. A file under one authority is never "relative to" a file under a different authority.

other is coerced via :meth:from_ so strings, dicts, :class:Path instances, and bare URLs all work::

URL("https://e.com/a/b").is_relative_to("/a")          # True
URL("https://e.com/a/b").is_relative_to("https://e.com/a")  # True
URL("https://e.com/a/b").is_relative_to("https://other.com/a")  # False
URL("https://e.com/a").is_relative_to("https://e.com/a/b")  # False

relative_to ¶

relative_to(other: Any) -> URL

Return a URL whose path is self.path re-rooted under other.

Raises :class:ValueError when self is not under other (check with :meth:is_relative_to first if the relationship is unknown).

The returned URL keeps self's scheme/host/userinfo/port/ query/fragment — only the path changes. The new path is the relative tail, with a leading / so it remains a well-formed URL path (a URL path without a leading slash is ambiguous with urllib.parse). An exact match self == other yields a path of "/".

to_struct_dict ¶

to_struct_dict() -> dict[str, Any]

Flatten into the dict shape that matches :data:URL_STRUCT.

Used by request / response / userinfo serializers to populate a nested URL struct column without each module re-implementing the same field-by-field mapping.

to_pathlib ¶

to_pathlib(*, strict: bool = True) -> Path

Convert this URL to a pathlib.Path.

The inverse of from_pathlib: strips the leading slash from Windows drive-letter paths so /C:/Users/x → C:/Users/x, then hands the result to Path. Does not re-resolve or expand ~ — the round-trip URL.from_pathlib(p).to_pathlib() gives an already-absolute Path, and calling .resolve() on an already-resolved path would force unwanted filesystem I/O (e.g. following symlinks). Callers who want the resolved form can call .resolve() on the returned Path themselves.

By default only the file scheme is accepted and the URL must not carry a query, fragment, host, userinfo, or port — converting those to a Path would silently drop data. Pass strict=False to relax both checks: any scheme is accepted and extra URL components are ignored. Use that mode only when you've already validated the URL shape elsewhere.

Raises:

Type	Description
`ValueError`	if the URL's path is empty, or (when `strict`) if the scheme is not `file` or extra components are present.

Platform note: on Linux, Path("C:/x") yields a PosixPath that treats C: as a directory name — a file:///C:/x URL can only round-trip correctly on Windows. This matches how from_pathlib was specified.

add_param ¶

add_param(
    key: str, value: str | None | Sequence[str | None], replace: bool = True
) -> URL

Add or replace one or more values for key in the query string.

value may be:

a scalar (str or None) — appended as a single (key, v) pair, with None coerced to "",
a list / tuple of scalars — appended as one pair per element. None elements coerce to "".

When replace=True (the default), every existing pair with the same key is removed first. Combined with an empty sequence, this clears the key entirely; combined with a non-empty sequence, it sets the key to exactly those values. With replace=False, new pairs are appended without touching existing ones.

Generators / arbitrary iterables are intentionally not accepted: a single-pass iterator that gets exhausted during validation would be a footgun, and str would be ambiguously iterable. Materialize first if you have one.

add_params ¶

add_params(
    params: Mapping[str, Any] | Iterable[tuple[str, Any]],
    *,
    replace: bool = False
) -> URL

Add or replace multiple (key, value) pairs in one shot.

Values follow the same rules as :meth:add_param: a scalar becomes a single pair, a list / tuple of scalars expands one pair per element. replace=True clears every listed key before appending; the default False appends.

Equivalent to looping with :meth:add_param, but builds a single :class:URL rather than copying once per key.

URLBased ¶

Bases: ABC

Mixin for any class addressable by a :class:URL.

Subclasses declare a class-level scheme: ClassVar[Scheme | None] on the class body; on subclass creation :meth:__init_subclass__ registers the class against that :class:Scheme member in the global :data:_URL_BASED_REGISTRY. The registry is the single source of truth for "what class handles s3://" / "dbfs://" / …; callers either look it up directly (URLBased.for_scheme(...)) or hand a URL to :meth:URLBased.dispatch and let URLBased pick the right subclass.

The two abstract hooks every subclass implements:

:meth:from_url(cls, url, **kwargs) — build an instance of the subclass from a :class:URL. Concrete subclasses typically forward to their own __init__.
:meth:to_url(self) -> URL — render this instance back to its canonical URL form.

Together these make a :class:URLBased round-trippable through a URL: cls.from_url(obj.to_url()) is the identity for any well-behaved subclass.

for_scheme `classmethod` ¶

for_scheme(scheme: Any) -> 'type[URLBased]'

Return the :class:URLBased subclass registered for scheme.

Lazy: if no subclass is registered yet, this routes through :meth:Scheme.path_class which imports the backend module on demand (firing :meth:__init_subclass__ as a side effect).

Raises :class:ValueError for an unknown scheme and :class:ImportError when the backend's optional dependencies aren't installed.

dispatch `classmethod` ¶

dispatch(url: Any, **kwargs: Any) -> 'URLBased'

Build the right :class:URLBased subclass from url.

Looks up the subclass via :meth:for_scheme, then delegates to that subclass's :meth:from_url. Used as the cross-cutting entry point when the caller has a URL but doesn't know (or care) which concrete class owns its scheme.

URL.from_(url).scheme drives the lookup; an empty scheme falls back to the file:// handler so bare paths work.

resolve_memory_address ¶

resolve_memory_address(address: int) -> object

Dereference an integer address back to the Python object.

Uses ctypes.cast(addr, py_object).value — the standard CPython trick. O(1), but a raw dereference: the caller MUST hold a strong reference to the object for the URL's entire lifetime. If the object has been garbage-collected, the slot will be reused and this function returns a different object (or, if the slot was freed and not yet reused, may segfault).

Module-level so non-URL callers (tests, MediaIO dispatch sites) can use it without constructing a URL instance.

hive_cast_value ¶

hive_cast_value(value: Any, dtype: 'pa.DataType | None') -> Any

Cast a :func:hive_decode-d value to dtype, leaving raw on failure.

Used when the partition column's declared dtype is in scope — int64 partition_key lands as :class:int, a timestamp partition as :class:datetime. When dtype is None or the cast raises (un-castable value), the decoded string passes through unchanged — every caller's downstream prune is conservative on undecidable shapes so a no-op cast just forces the row-level filter to run.

Fast path: the common partition dtypes (integers, floats, bool, string, the typed ints we tag on every cached response's partition_key) cast natively with the built-in constructor. Allocating a one-element pa.array and dispatching the cast kernel on every prune check shows up at the top of the cache hot path — the native path is ~50× faster. Anything outside the fast set (timestamps, decimals, lists, …) falls back to the pyarrow round-trip which still handles arbitrary types.

hive_decode ¶

hive_decode(raw: str) -> Any

Inverse of :func:hive_encode — returns the URL-decoded string.

The caller is responsible for casting the result to the partition column's declared dtype (the URL layer doesn't know the schema at parse time; see :func:hive_cast_value for the dtype-aware half).

hive_encode ¶

hive_encode(value: Any) -> str

Encode value as a filesystem-safe Hive partition value.

None → :data:HIVE_DEFAULT_PARTITION matching the Hive / Spark / Delta convention. Everything else is str(value) URL- quoted with the path-separator + = characters reserved so the encoded value can be split back unambiguously on a single = and never collides with a directory boundary.

hive_split ¶

hive_split(name: str) -> 'tuple[str, Any] | None'

Parse a Hive-encoded segment into (column, value).

Returns None when name doesn't match the <col>=<val> convention so the caller can treat the entry as a plain (non- Hive) directory.

yggdrasil.url¶

url ¶

URL dataclass ¶

extensions property ¶

name property ¶

stem property ¶

parent property ¶

parents property ¶

media_type property ¶

mime_type property ¶

codec property ¶

is_memory_address property ¶

memory_address property ¶

is_urlish classmethod ¶

infer_media_type ¶

resolve_memory_address ¶

from_memory_address classmethod ¶

is_pathish classmethod ¶

from_pathlib classmethod ¶

joinpath ¶

match_pattern ¶

matches_patterns ¶

is_relative_to ¶

relative_to ¶

to_struct_dict ¶

to_pathlib ¶

add_param ¶

add_params ¶

URLBased ¶

for_scheme classmethod ¶

dispatch classmethod ¶

resolve_memory_address ¶

hive_cast_value ¶

hive_decode ¶

hive_encode ¶

hive_split ¶

`yggdrasil.url`¶

URL `dataclass` ¶

extensions `property` ¶

name `property` ¶

stem `property` ¶

parent `property` ¶

parents `property` ¶

media_type `property` ¶

mime_type `property` ¶

codec `property` ¶

is_memory_address `property` ¶

memory_address `property` ¶

is_urlish `classmethod` ¶

from_memory_address `classmethod` ¶

is_pathish `classmethod` ¶

from_pathlib `classmethod` ¶

for_scheme `classmethod` ¶

dispatch `classmethod` ¶