`yggdrasil.loki.web`¶

web ¶

Loki on the internet — fetch, browse, tables, images.

Loki reaches the web through yggdrasil's own stack, not a bolted-on client: every request rides :class:~yggdrasil.http_.HTTPSession (its pooling, retry, and response cache), and every tabular body is parsed through the io tabular handlers — :meth:HTTPResponse.to_polars auto-detects CSV / JSON / Parquet / Arrow / XLSX. So "look it up on the internet" and "parse that table" are the same two project abstractions the rest of yggdrasil runs on.

from yggdrasil.loki import web

web.read_text("https://example.com")          # browse → readable text + links
web.read_table("https://…/data.csv")           # → polars DataFrame (any format)
web.read_json("https://api.example.com/x")      # → decoded JSON
web.read_image("https://…/chart.png")           # → bytes + dims + content-type

Browser ¶

Browser(
    *, headless: bool = True, browser: str = "chromium", timeout: float = 30000
)

A headless browser session for interacting with a live page.

Drive a page the way a person would — :meth:goto, :meth:fill a field, :meth:type, :meth:check a box, :meth:select_option, :meth:click a button, :meth:submit, then read the result (:meth:text, :attr:url, :meth:title). Backed by Playwright (Chromium by default), imported only when you open one. Use as a context manager so the browser always closes::

with web.Browser() as b:
    b.goto("https://example.com/login")
    b.fill("#user", "me").fill("#pass", "secret").submit("button[type=submit]")
    print(b.url, b.text())

The action methods return self so calls chain.

fill ¶

fill(selector: str, value: str) -> 'Browser'

Set an input/textarea's value (clears it first).

type ¶

type(selector: str, text: str, *, delay: float = 0) -> 'Browser'

Type text key by key (fires keypress handlers, unlike :meth:fill).

submit ¶

submit(selector: Optional[str] = None) -> 'Browser'

Submit a form — click selector if given, else press Enter.

session ¶

session() -> 'HTTPSession'

The shared (singleton-cached) :class:HTTPSession.

fetch ¶

fetch(
    url: str,
    *,
    params: Optional[dict[str, str]] = None,
    headers: Optional[dict[str, str]] = None,
    timeout: float = 30.0
) -> "HTTPResponse"

GET url through the yggdrasil HTTP session → an :class:HTTPResponse.

Sends a realistic browser header profile (from the http_ user-agent utils) so pages render normally, unless the caller overrides specific headers.

read_table ¶

read_table(
    url: str, *, fmt: Optional[str] = None, **fetch_kwargs: Any
) -> "pl.DataFrame"

Fetch url and parse it into a polars DataFrame via the io handlers.

Handles every tabular format the io layer knows (CSV / JSON / Parquet / Arrow / XLSX). fmt (e.g. "csv") forces the leaf when the URL has no extension and the server mislabels the body.

read_json ¶

read_json(url: str, **fetch_kwargs: Any) -> Any

Fetch url and decode its JSON body.

read_image ¶

read_image(
    url: str, *, save_to: Optional[str] = None, **fetch_kwargs: Any
) -> dict[str, Any]

Fetch an image; report its content type, byte size, and pixel dimensions. Optionally save the bytes to save_to.

read_text ¶

read_text(
    url: str, *, max_chars: int = 4000, **fetch_kwargs: Any
) -> dict[str, Any]

Browse url: fetch it and return readable text + links.

HTML is stripped to readable text (scripts/styles dropped) and its anchors collected — a lightweight browser view. Non-HTML bodies come back as text.

scrape ¶

scrape(
    url: str, *, max_chars: int = 6000, **fetch_kwargs: Any
) -> dict[str, Any]

Scrape a page into structured pieces — title, meta, text, links, tables.

A richer :func:read_text: pulls the <title> and meta description, the readable text, the anchor links, the first HTML <table> as records (if any), and any embedded JSON-LD structured data. Legitimate extraction of a public page's content — it does not defeat access controls.

browser_available ¶

browser_available() -> bool

True when Playwright (and a browser binary) is installed for automation.

A pure probe — never installs. :func:ensure_browser is the install path.

ensure_browser ¶

ensure_browser(*, install: Optional[bool] = None) -> bool

Make the headless browser usable, auto-installing on demand.

Installs the playwright package (via Loki's runtime auto-installer) and then its Chromium binary (python -m playwright install chromium) when either is missing — unless auto-install is disabled (YGG_LOKI_AUTO_INSTALL=0 or install=False), in which case it just probes. Returns whether the browser is ready.

fill_form ¶

fill_form(
    url: str,
    fields: dict[str, str],
    *,
    submit: Optional[str] = None,
    headless: bool = True,
    browser: str = "chromium",
    wait_for: Optional[str] = None,
    screenshot: Optional[str] = None,
    max_chars: int = 4000
) -> dict[str, Any]

Open url, fill fields (CSS selector → value), optionally submit, and return the resulting page state — the high-level "fill in this form".

submit is the selector of the submit button (omit to skip submitting); wait_for waits for a selector to appear after submit (e.g. a results panel); screenshot saves a PNG of the final page.

interact ¶

interact(
    url: str,
    steps: list[dict[str, Any]],
    *,
    headless: bool = True,
    browser: str = "chromium",
    screenshot: Optional[str] = None,
    max_chars: int = 4000
) -> dict[str, Any]

Drive a page through a sequence of interaction steps, return its final state. Each step is one action dict — the page is a thing you operate::

web.interact("https://shop.example/search", [
    {"type": ["#q", "wireless headphones"]},
    {"press": ["#q", "Enter"]},
    {"wait_for": ".results"},
    {"click": ".results a:first-child"},
])

Actions: goto (url), fill/type/select/press ([selector, value]), click/check/wait_for (selector), submit (button selector or null for Enter).

discover_apis ¶

discover_apis(
    url: str, *, limit: int = 40, **fetch_kwargs: Any
) -> dict[str, Any]

Discover the data APIs a page already uses — its underlying endpoints.

Inspects a public page for the data sources it embeds or calls: JSON-LD structured data, <script type="application/json"> payloads (e.g. __NEXT_DATA__), and candidate endpoint URLs referenced in markup/scripts (/api/…, *.json/*.csv, fetch(...) targets). This surfaces the documented/embedded APIs to pull data from — it does not bypass auth; any endpoint that needs credentials still needs them.

yggdrasil.loki.web¶