llm

OpenAI-compatible LLM transport: chat completions + embeddings.

All config resolves at construction/call time (never at import): constructor arg > env var > default. No retries here — reliability policy (retry-then-skip, cache fallback) belongs to the callers.

ChatClient

A schema-constrained chat completion, against any OpenAI-compatible server -- see the module docstring: config resolves per call/construction, never at import, and reliability policy (retry, degrade) is the caller's.

chat_json(messages, schema)

One chat call, requesting a JSON object matching schema.

Sends reasoning_effort="none" first (see the comment below: chain- of-thought buys nothing for a small schema-bound reply and roughly doubled latency when measured), retrying once without it for a server that rejects the field with a 400 rather than ignoring it.

ChatTarget dataclass

Where chat goes, who makes the call, and where each part came from.

kind is "openai" (this process posts to base_url with api_key) or "hermes" (Hermes' own runtime makes the call -- base_url and api_key are empty and nothing of the credential is ever here). api_key is excluded from repr. *_source are human-readable ("env LLM_BASE_URL", "Hermes (nvidia, ...)", "built-in Ollama default") and are what every unreachable / no-such-model message quotes; origin is the one-phrase form.

origin property

Where this target came from, in one phrase, for error messages.

EmbeddingClient

One embedding vector per call, against any OpenAI-compatible server -- same construction-time config resolution as ChatClient.

embed(text, input_type=None)

The raw embedding for text, untruncated and unnormalized -- embed.truncate_normalize is the caller's job, not this client's. input_type is required by, and only sent to, a NIM server.

embed_many(texts, input_type=None)

Raw embeddings for texts, ONE HTTP request -- measured 7.2x faster than one call per item against live Ollama/embeddinggemma.

The OpenAI-compatible /embeddings response does NOT guarantee data comes back in request order, only that each element carries the index it belongs at. Sorting by it before returning is load-bearing: trusting response order would silently mismatch a vector to the wrong item, corrupting the index in a way nothing downstream could detect.

base_url()

The chat server root, resolved at call time -- see chat_target. Never cached, so a .env change or a new Hermes model takes effect without restarting anything that only imports this module.

chat_api_key()

The chat Bearer key ("" when none), resolved at call time. Callers put it in a header and nowhere else.

chat_client(target=None)

The client for target (default: the resolved one): a ChatClient for an OpenAI-compatible endpoint, a HermesChatClient for a provider only Hermes' runtime can call. Both offer chat_json(messages, schema).

chat_failure_message(detail=None)

The stderr text for a chat run that had to stop.

Says what failed, WHERE the url and model came from, and how to change them -- replacing the bare "is ollama running?" that sent a user with a hosted model to start a daemon they never wanted.

chat_model()

The chat model name, resolved at call time -- see chat_target.

chat_target()

Resolve the chat backend, at call time.

Per field, first match wins:

1. the env var (LLM_BASE_URL, CHAT_MODEL, LLM_API_KEY);
2. the model Hermes is connected to -- ONLY when the host opted in with ATTEST_LLM_FROM_HERMES=1 and LLM_BASE_URL is not set to something of its own (see below). Whatever the provider: a key provider Attestation calls itself, anything else (ChatGPT sign-in, Anthropic, ...) through Hermes' own runtime -- see hermes_model and hermes_bridge;
3. the built-in Ollama default.

Opt-in is off by default, so a non-Hermes user's resolution is exactly what it was. A hosted Research Desk with no chat model on its loopback server is the case it exists for: Hermes' model is the only chat model that machine has.

LLM_BASE_URL / CHAT_MODEL equal to the built-in defaults count as unset for this purpose: .env.sample writes exactly those two lines uncommented and attest install copies it to .env, so on a provisioned box they are boilerplate, not a decision, and honouring them would leave the opt-in with nothing to do. Anything else in LLM_BASE_URL is a decision and is final -- Hermes is not consulted at all.

Raises BackendNotConfigured when opted in and Hermes has no usable model.

default_chat_fn(messages, schema)

The default chat_fn for explain/tagging: a lazily built client, rebuilt whenever what it resolves to changes.

Resolution is per call, not per process: an MCP server lives for a whole session, and a client cached forever would keep calling the model that was connected when it started after the customer chose another in AgentMarkit.

describe_chat()

" model (url from ..., model from ...)" -- or the reason there is none. For attest install --check and every chat failure message: hosts and names only, never a key.

describe_embedding()

The embedding backend, host and model only, and where each came from.

embed_api_key()

The Bearer key for the embedding server.

EMBED_API_KEY when set. Otherwise LLM_API_KEY -- but only when embeddings go to the same server as chat: with EMBED_BASE_URL pointing somewhere else, the chat provider's key is never sent to that other host.

embed_base_url()

Where embeddings come from: EMBED_BASE_URL, else the chat server.

A hosted machine with no GPU can embed on its own CPU (a local Ollama serving embeddinggemma measured 20 items/s on 2 vCPUs, 380 MB resident) while chat goes to the customer's provider -- two servers, which one LLM_BASE_URL could not describe.

Never the model Hermes is connected to (ATTEST_LLM_FROM_HERMES): that is a hosted CHAT model, and sending every title and abstract to it for embedding is a data-egress decision nobody made. Embeddings stay on EMBED_BASE_URL, else LLM_BASE_URL, else the built-in default.

embed_input_types()

Whether the embedding server takes NIM's input_type/truncate fields.

EMBED_INPUT_TYPE (1/true/on or 0/false/off) when set; otherwise on exactly when EMBED_BASE_URL is NVIDIA's hosted API. MEASURED 2026-10-06: nvidia/nemotron-3-embed-1b answers 200 without input_type and ranks at AUC 0.625 on evals/reaction_cases.json against 0.962 with it -- a plain OpenAI-shaped call passes every check and ranks barely above chance.

embed_model()

The configured embedding model name, resolved at call time -- see base_url.

llm_timeout()

Seconds one chat request may take: ATTEST_LLM_TIMEOUT, bounded to 10..600, default 120. Read at call time; both the direct client and the Hermes bridge use it, so their deadlines agree. A slow reasoning model raises it; nothing below 10 s is a usable model call.

load_env()

Load .env (repo root first, then cwd-upward search); real env always wins.

Called only from process entry points (cli.main, mcp_server.main) — never from library imports, so tests stay dotenv-free.

Any OTHER entry point must call this itself. A standalone script that imports attestation and skips it gets DEFAULT_CHAT_MODEL rather than the model in .env, silently and with no error — a one-off re-tagging script did exactly that on 2026-08-11 and ran against the wrong model until the banner it printed gave it away.