`yggdrasil.loki.engines.transformers_engine`¶

transformers_engine ¶

Local HuggingFace -backed :class:TokenEngine.

Runs an open-source model on this workstation via the transformers text-generation pipeline — free, private, offline. The default model is sized to the machine (:class:LocalEngine + :mod:yggdrasil.loki.resources): a small Qwen instruct model on a modest CPU box, a larger one as RAM/GPU allow. Override with YGG_LOKI_HF_MODEL (any chat/instruct id) and YGG_LOKI_HF_DEVICE (e.g. "cuda"). When the device is left unset the engine auto-detects an accelerator (:func:yggdrasil.loki.resources.accelerator) — NVIDIA cuda, Intel GPU xpu, or Apple mps — so a local model lands on the GPU instead of the CPU. An Intel NPU (AI Boost) is detected and flagged, but the HF pipeline can't target it directly (use OpenVINO / optimum-intel for that). Available only when transformers + torch are installed.

TransformersEngine ¶

TransformersEngine(
    *,
    model: Optional[str] = None,
    tier: Optional[str] = None,
    device: Optional[str] = None
)

Bases: LocalEngine

Reason with a local HuggingFace model (transformers pipeline).

bootstrap_model `property` ¶

bootstrap_model: str

The default model for this box — the resource-sized row, or the engine's :attr:default_model fallback.

resolve_device ¶

resolve_device() -> Optional[str]

The device to load the pipeline on: an explicit pin (ctor arg / YGG_LOKI_HF_DEVICE) wins; otherwise the best auto-detected accelerator — NVIDIA cuda, Intel GPU xpu, Apple mps — or None (CPU). Lets a local model use the GPU without configuration.

ready ¶

ready(model: Optional[str] = None) -> bool

True when the pipeline for model (resolved if omitted) is loaded.

Lets a caller (the CLI) warn that a turn is about to trigger the slow first load — download weights + build the pipeline — instead of going silent on a CPU box.

warm ¶

warm(model: Optional[str] = None) -> None

Build the model's pipeline ahead of the first turn — best-effort.

Loading a local model is slow and silent (download weights → build the pipeline); the ygg loki REPL calls this on a background thread so the wait overlaps the user picking a session and typing, instead of stalling the first submit. Swallows failures — they're cached in :attr:_FAILED and surfaced on the first real turn.

stream ¶

stream(
    messages: list[dict[str, Any]],
    *,
    system: Optional[str] = None,
    max_tokens: int = DEFAULT_MAX_TOKENS,
    tier: Optional[str] = None,
    **options: Any
) -> Iterator[str]

Generate live, token by token, via TextIteratorStreamer.

Without this the base :meth:stream runs the whole generation in one blocking :meth:complete and yields it at the end — so a slow CPU run prints nothing until it finishes. Here the pipeline runs on a worker thread and feeds a streamer the terminal drains as tokens arrive.

choose_tier ¶

choose_tier(
    messages: Optional[list[dict[str, Any]]] = None,
    system: Optional[str] = None,
) -> str

Adaptive tier for this request: "deep" or "fast".

Sizes on the message content (the actual work, not the fixed system boilerplate) and scans both message and system text for reasoning signals. Long or signalled requests get the deep tier; the rest stay fast. Override for a smarter policy.

resolve_model ¶

resolve_model(
    *,
    messages: Optional[list[dict[str, Any]]] = None,
    system: Optional[str] = None,
    tier: Optional[str] = None
) -> Optional[str]

An explicit pin wins; otherwise the model sized to this workstation.

Local models are resource-bound, so the remote fast/deep cost tier doesn't apply — the box, not the prompt, picks the size.

usage ¶

usage() -> list[Any]

This engine's per-model usage rows from the global meter.

generate ¶

generate(
    prompt: str,
    *,
    system: Optional[str] = None,
    tier: Optional[str] = None,
    **options: Any
) -> str

Convenience: complete a single user prompt → reply text.

generate_stream ¶

generate_stream(
    prompt: str,
    *,
    system: Optional[str] = None,
    tier: Optional[str] = None,
    **options: Any
) -> "Iterator[str]"

Convenience: stream a single user prompt → text chunks.

yggdrasil.loki.engines.transformers_engine¶