`yggdrasil.loki.engines.openvino_engine`¶

openvino_engine ¶

Local OpenVINO -backed :class:TokenEngine — runs a model on the Intel NPU.

Where :class:TransformersEngine runs a model through a torch pipeline (CPU, or an Intel GPU via the XPU torch build), this engine loads the model with optimum-intel <https://github.com/huggingface/optimum-intel>_ + OpenVINO <https://docs.openvino.ai>_ and runs it on an Intel NPU (AI Boost) — the inference accelerator a torch pipeline can't target — falling back to the Intel GPU then CPU. It produces the same HuggingFace text-generation pipeline, so the complete / stream inference path is inherited from :class:TransformersEngine unchanged; only the model loader differs.

Defaults to OpenVINO's pre-converted int4 Qwen2.5 models (small, NPU-friendly, no on-the-fly conversion); a plain HuggingFace id is converted to OpenVINO IR on first load (export=True). Override with YGG_LOKI_OV_MODEL and pin the device with YGG_LOKI_OV_DEVICE (NPU / GPU / CPU). Available only when openvino + optimum are installed and an NPU or GPU is present (CPU-only boxes are better served by the transformers / ollama engines).

OpenVINOEngine ¶

OpenVINOEngine(
    *,
    model: Optional[str] = None,
    tier: Optional[str] = None,
    device: Optional[str] = None
)

Bases: TransformersEngine

Reason with a local model on the Intel NPU via OpenVINO / optimum-intel.

bootstrap_model `property` ¶

bootstrap_model: str

The default model for this box — the resource-sized row, or the engine's :attr:default_model fallback.

available ¶

available() -> bool

True when OpenVINO + optimum are installed and an NPU/GPU is present.

A CPU-only box is left to the transformers / ollama engines — this engine exists for the accelerators a torch pipeline can't reach (chiefly the NPU). Cheap: the package check is find_spec; the device list is memoized.

resolve_device ¶

resolve_device() -> str

The OpenVINO device to run on: an explicit pin wins, else the best present accelerator — NPU first (the whole point), then GPU.

choose_tier ¶

choose_tier(
    messages: Optional[list[dict[str, Any]]] = None,
    system: Optional[str] = None,
) -> str

Adaptive tier for this request: "deep" or "fast".

Sizes on the message content (the actual work, not the fixed system boilerplate) and scans both message and system text for reasoning signals. Long or signalled requests get the deep tier; the rest stay fast. Override for a smarter policy.

resolve_model ¶

resolve_model(
    *,
    messages: Optional[list[dict[str, Any]]] = None,
    system: Optional[str] = None,
    tier: Optional[str] = None
) -> Optional[str]

An explicit pin wins; otherwise the model sized to this workstation.

Local models are resource-bound, so the remote fast/deep cost tier doesn't apply — the box, not the prompt, picks the size.

usage ¶

usage() -> list[Any]

This engine's per-model usage rows from the global meter.

generate ¶

generate(
    prompt: str,
    *,
    system: Optional[str] = None,
    tier: Optional[str] = None,
    **options: Any
) -> str

Convenience: complete a single user prompt → reply text.

stream ¶

stream(
    messages: list[dict[str, Any]],
    *,
    system: Optional[str] = None,
    max_tokens: int = DEFAULT_MAX_TOKENS,
    tier: Optional[str] = None,
    **options: Any
) -> Iterator[str]

Generate live, token by token, via TextIteratorStreamer.

Without this the base :meth:stream runs the whole generation in one blocking :meth:complete and yields it at the end — so a slow CPU run prints nothing until it finishes. Here the pipeline runs on a worker thread and feeds a streamer the terminal drains as tokens arrive.

generate_stream ¶

generate_stream(
    prompt: str,
    *,
    system: Optional[str] = None,
    tier: Optional[str] = None,
    **options: Any
) -> "Iterator[str]"

Convenience: stream a single user prompt → text chunks.

ready ¶

ready(model: Optional[str] = None) -> bool

True when the pipeline for model (resolved if omitted) is loaded.

Lets a caller (the CLI) warn that a turn is about to trigger the slow first load — download weights + build the pipeline — instead of going silent on a CPU box.

warm ¶

warm(model: Optional[str] = None) -> None

Build the model's pipeline ahead of the first turn — best-effort.

Loading a local model is slow and silent (download weights → build the pipeline); the ygg loki REPL calls this on a background thread so the wait overlaps the user picking a session and typing, instead of stalling the first submit. Swallows failures — they're cached in :attr:_FAILED and surfaced on the first real turn.

yggdrasil.loki.engines.openvino_engine¶