LLM Models, Providers, and Cost

Omnigent's pitch is one line in the README: "Use any model. A first-party API key, a Claude/ChatGPT subscription, or any compatible gateway. All first-class." (README.md:36-37). This doc is about the machinery that makes that sentence true — how a model is described, how a provider is resolved, how reasoning effort is normalized, and how dollars are metered across a fleet of heterogeneous wrapped agents and fed back into spend-cap policies.

The interesting tension: Omnigent is a meta-harness. It does not run the model itself most of the time — it wraps native CLIs (Claude Code, Codex, Cursor, Goose, Qwen, Kimi, …) and SDK harnesses. So "model selection" is really "convince someone else's binary to run the model I picked, then meter what it spent." That constraint shapes every design choice below.

1. Two model paths, one abstraction

Omnigent has its own multi-provider LLM client (omnigent/llms/) and it wraps native agents that bring their own clients. Knowing which is which explains the rest.

• In-house client (omnigent/llms/client.py). Presents the OpenAI Responses API as its public surface and routes to provider adapters with Chat Completions as the internal lingua franca. Used for the orchestrator's own LLM calls (the judge, summarizers, brain turns on the claude-sdk path). It replaced litellm (omnigent/llms/LLMCLIENT.md:5).
• Wrapped native agents. The real coding work runs inside claude-native, codex-native, cursor-native, etc. Omnigent does not make their API calls — it configures their model and reads back their usage.

The in-house client routes by a "provider/model-name" string (omnigent/llms/routing.py:51-75). No prefix defaults to openai. The provider table is explicit:

openai, anthropic, gemini, bedrock, vertex, databricks, groq,
deepseek, xai, openrouter, ollama, moonshot

(omnigent/llms/routing.py:19-32). Bedrock / Vertex / Databricks carry None for base URL because they need connection params (region, project, workspace) supplied at call time, not a static endpoint.

Each adapter translates Chat Completions ↔ the provider's native API (Anthropic Messages, Gemini generateContent, Bedrock Converse, …) — the adapter map is documented in omnigent/llms/LLMCLIENT.md:42-58. The public interface is text/structured JSON, not provider-native tool-use plumbing exposed to callers — same provider-agnostic instinct Strix had, but built on the Responses shape rather than XML.

Why two paths matter for cost Cost metering has to cover BOTH. The in-house client meters from its own response usage (it controls the call). The wrapped agents can't be metered that way — Omnigent never sees their HTTP traffic — so it harvests usage from each CLI's own reporting channel (hooks, transcripts, headless result.usage) and normalizes it server-side. Section 5.

2. Provider / auth-mode support matrix

The "use any model" claim decomposes into provider kinds, the kind: discriminator on a providers: entry in ~/.omnigent/config.yaml (omnigent/onboarding/provider_config.py:97-134). Each kind answers how it authenticates, which in turn decides whether Omnigent can enumerate its models and how it meters cost.

Two design notes worth stealing:

1. Defaults are per-family, not per-provider (omnigent/onboarding/provider_config.py:9-14). A Claude default and a Codex default coexist — at most one default per family. The code explicitly cites Zed's per-feature pointers, Continue's per-role models, and Goose's lead/worker as prior art (omnigent/onboarding/provider_config.py:9-14, 953-957). This is the right shape for a meta-harness: "which Claude" and "which GPT" are independent knobs.

2. Secrets are resolved lazily, per family actually consumed (omnigent/onboarding/provider_config.py:340-362). A provider can declare an anthropic and an openai family; only the one the active harness uses forces its env var to exist. api_key_ref: keychain:<name> keeps secrets out of the config file entirely (omnigent/onboarding/provider_config.py:413-463).

For your project — EU AI Act compliance platform: Per-family, kind-typed provider config with a non-secret credential descriptor (ResolvedCredential, omnigent/onboarding/provider_config.py:380-410, describe_active_credential at :1188) is exactly the "what model, via what auth, from what source — without leaking the secret" record an Article 12 logging story wants. The readout names keychain:anthropic or claude CLI login, never the token.

3. The model catalog: deterministic enumeration

omnigent/model_catalog.py backs the sys_list_models runner builtin, so an orchestrator agent can ask "what can each of my sub-agents actually run?" For every sub-agent (plus a "self" row for the brain), it resolves the provider the spawn path would actually use and enumerates that provider's live model listing (omnigent/model_catalog.py:593-618).

Enumeration is deterministic per provider kind (omnigent/model_catalog.py:13-25):

flowchart TD
    spec[sub-agent spec + harness] --> rp[resolve_model_provider]
    rp -->|kind=databricks| db[GET /api/2.0/serving-endpoints<br/>filter chat LLMs]
    rp -->|kind=key + anthropic| an[GET /v1/models<br/>x-api-key]
    rp -->|kind=key/gateway/local| oc[GET /v1/models<br/>bearer]
    rp -->|kind=subscription| st[curated static list<br/>verified=false]
    rp -->|unresolvable| none[source=none<br/>dead-worker preflight]
    db --> filt[family filter:<br/>claude harness keeps claude ids,<br/>codex keeps gpt ids, pi keeps all]
    an --> filt
    oc --> filt
    st --> filt
    filt --> row[ModelListing: source, verified, models, note]

Key properties:

• Provenance is first-class. Every listing carries source (gateway / openai-compatible / anthropic-api / static / none) and a verified bool — false for curated/static (omnigent/model_catalog.py:148-166). A subscription login can't be verified, and the tool says so in the note.
• source="none" doubles as a dead-worker preflight signal (omnigent/model_catalog.py:25). If a sub-agent's provider can't be resolved, the row says "dispatches to this worker cannot run here" before you spawn it (omnigent/model_catalog.py:713-722).
• Failures never leak secrets. Raw exception text can embed an auth_command with credentials, so failures map to redacted categories (omnigent/model_catalog.py:667-695) before reaching the LLM-visible tool output.
• Listings cached 5 min, keyed by a credential fingerprint (sha256 prefix, never the secret — omnigent/model_catalog.py:207-247), so a turn that fans out many dispatches doesn't re-fetch per call.

The family token rule is the spine of cross-harness routing: a model id containing claude is the claude family; gpt/codex is openai; everything else is other (omnigent/model_catalog.py:261-276). The same rule lives in model_override.model_family_mismatch — and they cite each other, so the catalog filter and the dispatch guard agree by construction.

4. Model override: pushing a choice into someone else's binary

This is the meta-harness's hardest model problem. The user (or the advisor) picks a model; Omnigent has to make a native CLI it doesn't control honor it.

omnigent/model_override.py is the validation + normalization layer for a choice that crosses a spawn boundary. The persisted override reaches:

• native CLIs as a --model argv element at terminal launch, and
• SDK harnesses as a HARNESS_<H>_MODEL env var (omnigent/model_override.py:1-8, 230-249).

Because that string lands on a command line, validation is paranoid: a conservative charset where the first char must be alphanumeric so the value can never read as a CLI flag (--model --evil), with a 256-char cap (omnigent/model_override.py:16-63). Defense against argv injection, not just typos.

Three transforms make a single id work across heterogeneous backends:

1. Family guard (model_family_mismatch, omnigent/model_override.py:110-156). Single-vendor harnesses reject the wrong family loud at the dispatch gate, before spawn. claude-sdk only runs claude ids; codex only runs GPT ids (the codex Databricks gateway dropped the chat wire that was the only path to Claude — so a codex×Claude dispatch is genuinely broken and must fail loud, :74-83). antigravity is Gemini-native and rejects Claude/GPT and any databricks- id (:84-104, :147-155). pi and openai-agents are multi-model and accept anything validated.

2. Provider-local normalization (normalize_model_for_provider, omnigent/model_override.py:192-227). A Databricks-gateway child gets a bare claude-opus-4-8 rewritten to databricks-claude-opus-4-8; a vendor-direct (API-key / subscription) child gets the databricks- prefix stripped. Both are mechanical, prefix-only; anything ambiguous passes through and the fail-loud harness error remains the safety net.

3. Plumbing gate (harness_supports_model_override, :230-249). Returns whether the override actually reaches the harness process. The runner threads it through per-harness: cursor-native applies it as --model, pi-native ignores it (omnigent/runner/app.py:498-517), and codex routes it into config.toml / --model (omnigent/runner/app.py:769-822). The runner re-validates on every read (omnigent/runner/app.py:705-720).

Does Omnigent override the underlying agent's model? Yes — when the harness exposes a seam for it (native --model, SDK env var, or, for claude-sdk, a per-turn set_model with no respawn). Where no seam exists (e.g. Kimi has no per-spawn provider flag, omnigent/onboarding/provider_config.py:161-164), it can't, and the config layer says so rather than silently failing.

5. Cost metering across heterogeneous agents

The whole point of the meta-harness is "many agents, one wallet." Cost has to be summed across providers and harnesses that each report differently. Omnigent runs two metering paths that never overlap on a session:

flowchart TD
    subgraph relay["Relay / in-house client (DELTA semantics)"]
        r1[client.responses.create] --> r2[response.usage]
        r2 --> r3[_usage_observer.notify]
        r3 --> r4[_accumulate_session_usage<br/>adds per-response deltas]
    end
    subgraph native["Native harnesses (SET semantics)"]
        n1[claude-native: Claude Code cost.total_cost_usd] --> n3
        n2[codex-native: tokenUsage.total] --> n3
        ncur[cursor-native: stop hook per-turn tokens] --> n3
        n3[POST external_session_usage] --> n4[_persist_native_cumulative_usage<br/>SET, monotonic clamp]
    end
    r4 --> price[fetch_model_pricing + compute_llm_cost<br/>cache-read priced separately]
    n4 --> price
    price --> total[session_usage.total_cost_usd]
    total --> badge[session.usage SSE → web badge / TUI]
    total --> policy[cost_budget policy gate]
    total --> daily[per-user daily rollup]

The split is explicit (omnigent/server/routes/sessions.py:2947-2958): the relay path adds per-response deltas; native harnesses report cumulative totals, so the native persist uses SET, not add.

What each native harness contributes:

• claude-native forwards Claude Code's own cost.total_cost_usd — exact billing, used directly, so the badge matches /cost in the Claude TUI (omnigent/server/routes/sessions.py:2963-2967). It also forwards a separate policy_cost_usd (max(displayed, live transcript estimate)) so the budget gate reflects in-flight sub-agent spend while the displayed figure is frozen (:2968-2974).
• codex-native forwards tokenUsage.total (input/output/cached); cost is computed from those via fetch_model_pricing when no dollar figure is given (:2975-2983).
• cursor-native was thought impossible to meter — its SQLite store and transcripts carry no tokens. The fix: cursor-agent ships a Claude-Code-style stop hook that fires in the interactive TUI with per-turn token counts (docs/cursor-native-cost-tracking.md:24-46). A stdlib-only recorder appends one JSONL line per turn; a runner poller sums them and POSTs the same external_session_usage contract — no server or frontend change (docs/cursor-native-cost-tracking.md:52-73).

Three correctness details that generalize:

• Cache-read tokens are split out and priced at the cache-read rate, not the full input rate (omnigent/server/routes/sessions.py:3028-3037, 2979-2983). Reported input totals are inclusive of cache reads; the split keeps input_tokens as the non-cached remainder.
• Monotonic clamp on the native write (omnigent/server/routes/sessions.py:3016-3027). The external_session_usage event is posted with the session owner's own bearer token, so a client could replay a falsified low cost to reset the budget gate to ~0. Monotonicity makes a downward report a no-op.
• "Priced ⟺ total_cost_usd key present." When a model isn't in the pricing catalog (e.g. Cursor's composer-2.5, or claude-4-sonnet whose cursor id doesn't match claude-sonnet-4-5), tokens still populate but the dollar figure shows "—" (docs/cursor-native-cost-tracking.md:105-126). Omnigent deliberately forwards the raw cursor id rather than guess a version alias — a wrong version means a wrong rate, worse than "—".

Caching note

Omnigent's in-house llms/ adapters do not set cache_control explicitly — there is no ephemeral/cache_control marker anywhere under omnigent/llms/. Caching is delegated: the wrapped CLIs (claude-native et al.) manage their own prompt caching, and Omnigent's job is only to observe it — harvesting cache_read / cache_creation token counts and pricing them at the cache rate (omnigent/server/routes/sessions.py:2876-2935). So: caching = delegated/observe-only, not explicit. (cache_control does appear in the native forwarders, but only as pass-through of the CLIs' own markers.)

6. Reasoning effort, normalized across providers

omnigent/reasoning_effort.py is a small, sharp module: a canonical superset of effort values and per-provider validated subsets.

EFFORT_VALUES   = {none, minimal, low, medium, high, xhigh, max}
OPENAI_EFFORTS  = {none, minimal, low, medium, high, xhigh}   # codex, openai-agents
ANTHROPIC_EFFORTS = {low, medium, high, xhigh, max}           # claude
GEMINI_EFFORTS  = {low, medium, high}                         # antigravity

(omnigent/reasoning_effort.py:9-18).

Normalization here means validation against the provider's actual range, not silent remapping. validate_effort raises on an unsupported value with a provider-named message listing the supported set (:21-43); the native-path variant raises a non-retryable PermanentLLMError so a bad effort fails fast rather than burning retries (:46-55). The in-house client wires this in at the OpenAI Responses boundary (omnigent/llms/client.py:187-193).

The contrast with the per-provider subsets is the design lesson: there is no universal "reasoning effort" — max is meaningful for Claude, absent for OpenAI's Responses effort; Gemini tops out at high. A meta-harness can't pretend otherwise, so it keeps a superset for the UI and validates down to each provider's truth.

7. Cost feeds spend-cap policy (the loop closes)

The metered total isn't just a badge — it gates execution. cost_budget (omnigent/policies/builtins/cost.py) reads event.context.usage.total_cost_usd (the cumulative total from Section 5, omnigent/policies/builtins/cost.py:30-37, 116-131) and the active model (:133-145), and enforces at two phases: request (before the LLM turn — so even text-only turns are budgeted) and tool_call (the native PreToolUse block point, :80-83).

Two gates:

• Soft ask_thresholds_usd — the first time spend crosses each checkpoint, the turn is parked for approval (ASK). Approval is remembered in session_state so each checkpoint asks at most once; a decline re-asks (omnigent/policies/builtins/cost.py:446-467).
• Hard max_cost_usd — a downgrade gate, not a stop. Once over budget, DENY while the session is still on an expensive model (fable/opus/gpt-5 by default, with -mini/-nano carve-outs, omnigent/policies/builtins/cost.py:103-113), telling the user to /model to something cheaper. Switch off the expensive tier and it's allowed again (omnigent/policies/builtins/cost.py:430-445). Unknown model → fail closed, blocked until the user picks a knowable cheaper one (:249-285).

A per-user daily variant rolls the same logic up across all of an owner's sessions for the UTC day (user_daily_cost_budget, omnigent/policies/builtins/cost.py:521-645).

Where it surfaces in a terminal: omnigent/native_cost_popup.py renders the ASK as a tmux display-popup over a native TUI pane and resolves the same server elicitation the web ApprovalCard uses — whichever surface answers first wins, the other clears (omnigent/native_cost_popup.py:1-30, 286-333).

Optional: the per-turn cost advisor

Beyond the hard/soft caps, an opt-in advisor (omnigent/cost_plan.py + omnigent/runner/cost_advisor.py) runs an LLM judge over each user turn and picks ONE brain model sized to the turn's difficulty (cheap/medium/expensive tier → a concrete model, omnigent/cost_plan.py:95-130). In optimize mode it applies the pick (claude-sdk only, via per-turn set_model, omnigent/runner/cost_advisor.py:69-73); in advise mode it records a shadow verdict. A user's explicit /model pin always beats the advisor (omnigent/runner/cost_advisor.py:18-21). The verdict is serialized into one cost_control.plan conversation label (omnigent/cost_plan.py:138-172), and the server rejects client writes to the cost_control.* namespace (omnigent/cost_plan.py:43-48) — telemetry the runner owns, not the client.

For your project — Swisscheese: This is the cost-control spine for parallel agents. The metering sums spend across N heterogeneous workers into one total_cost_usd; the budget gate downgrades (cheaper model) rather than killing the run, which keeps a fan-out alive while bounding it; the per-user daily rollup caps a developer's total across every concurrent session. For a writer→reviewers→fixer pipeline, expensive_models lets you reserve Opus for the writer and force reviewers to a cheaper tier once a budget checkpoint trips.

For your project — EU AI Act compliance platform: The two-channel metering (exact billing where the CLI reports it; token-priced otherwise) plus the "priced ⟺ key present" contract and the monotonic, owner-token-authenticated usage writes give an audit story: a per-session, per-model, cache-aware usage+cost record that can't be silently clawed back. That is the usage-logging substrate Articles 12/13 ask for.

8. What's clever, summarized

• Two model paths, one metering target. In-house Responses client and wrapped native agents both feed one total_cost_usd.
• Per-family defaults, mirroring Zed/Continue/Goose — "which Claude" and "which GPT" are independent.
• Model override survives a spawn boundary with argv-injection-proof validation, family guard at the dispatch gate, and prefix-mechanical provider-local normalization.
• Deterministic catalog with provenance (verified flag, source), where source="none" is a dead-worker preflight.
• Cursor cost via a TUI stop hook — metering a CLI that "couldn't" report.
• Cache-read tokens split and priced separately; usage writes monotonic to defend the budget gate against forged low-cost replays.
• Reasoning effort: superset + per-provider validated subset, fail-loud on unsupported, not silently remapped.
• Budget as downgrade gate, not kill switch — DENY only while on an expensive tier; /model cheaper to continue.
• Optional per-turn brain-model advisor sizing model to query difficulty, shadow or apply, always beaten by a user pin.

9. What to be wary of

• Metering depends on each CLI's reporting fidelity. A wrapped agent that under-reports (or whose model id doesn't match the pricing catalog) shows tokens but no dollars — the budget gate can't budget what it can't price (omnigent/policies/builtins/cost.py:36-37). Cursor's composer-2.5 is the live example.
• Override needs a harness seam. Kimi has no per-spawn provider flag, so Omnigent can't thread a model through it (omnigent/onboarding/provider_config.py:161-164) — routing lives out-of-band in ~/.kimi/config.toml.
• Subscription model lists are curated and verified=false — they can drift from what a given plan actually serves (omnigent/model_catalog.py:756-772).
• A single very expensive turn can overshoot before the next cost refresh (omnigent/policies/builtins/cost.py:44-46); the gate is turn-boundary, not mid-turn.
• Cache-write tokens aren't separately priced on the native path — they stay in the input bucket at the full input rate (docs/cursor-native-cost-tracking.md:130-132).