llm: routes that name a host, a backup and a window, and a profile that reads them
`llm.json` replaces `OLLAMA_BACKENDS_FILE`. It has targets (host and key
environment variable) and routes, where a route is an ordered chain of
`model @ target` entries, each with its own `context_window` and options. The
`default` route now starts on `glm-5.3-flash:cloud` and backs it with
`qwen3.6:35b-128k` on the home host, both at 131072 tokens, so a failover no
longer changes the budget the compressor and `num_ctx` are accounted against.
Embeddings get their own `embedding` route instead of a pinned host.

The engine classifies a failure (network, quota, auth, server, missing model),
cools that entry and moves the call down the chain; a background probe watches
only degraded targets and lets the next turn return to the primary once it is
healthy — it never interrupts a running turn. A profile joins a route with
`llm_route`, and the loader now actually reads it: `llm_route` and
`context_window` were declared on `AgentProfile` and written into every
`config.json` while the loader skipped both, the same silent no-op that
`is_admin_only` used to be, so every profile ran the legacy model list at the
global window. The writer emits them too, or an admin edit would erase the
route from disk.

The turn's resolved entry travels to the UI: the first chunk is stamped with
model and target, `model_info` carries `fallback` when a backup answered, and
the chat header shows a chip only then. `serving_primary` compares the target
as well as the model name, because a route may run one model on two hosts.
`/health` gains an `llm` block with the active entry, target states and missing
models.

`deploy/ollama/` keeps the local model's definition honest: MTP off (slower at
every context on this host), `presence_penalty 0` (the base model's 1.5 breaks
agentic tool calls), and `num_gpu 40` — Ollama's automatic fit claims a 128k
f16 slot fits and then dies mid-prefill with CUDA OOM; 40 of 42 layers leaves
~400 MiB and decodes at ~77 tok/s, 38 layers collapses to 48, and 39 OOMs.
q8_0 KV measured 37 tok/s against 70 on f16 here, so the cache stays f16.
1 parent f026ef0 commit 4cb27d3fbf0b25ffb7180edbd56466e232ee2e76
@Eugene Sukhodolskiy Eugene Sukhodolskiy authored 22 hours ago
Showing 60 changed files
View
.env.example
View
.env.navi_code.example
View
deploy/ollama/Modelfile.qwen-128k 0 → 100644
View
deploy/ollama/README.md 0 → 100644
View
docs/api.md
View
docs/architecture.md
View
docs/config.md
View
docs/mechanics.md
View
docs/profiles.md
View
llm.json 0 → 100644
View
navi/api/routes/admin.py
View
navi/api/routes/health.py
View
navi/config.py
View
navi/core/agent.py
View
navi/core/container.py
View
navi/core/events.py
View
navi/core/registry.py
View
navi/core/subagent_runner.py
View
navi/exceptions.py
View
navi/llm/base.py
View
navi/llm/config.py 0 → 100644
View
navi/llm/fallback.py
View
navi/llm/health.py 0 → 100644
View
navi/llm/ollama.py
View
navi/memory/_embeddings.py
View
navi/profiles/assistant/config.json
View
navi/profiles/base.py
View
navi/profiles/designer_3d/config.json
View
navi/profiles/developer/config.json
View
navi/profiles/discuss/config.json
View
navi/profiles/dispatcher/config.json
View
navi/profiles/loader.py
View
navi/profiles/modeler_3d/config.json
View
navi/profiles/navi_code/config.json
View
navi/profiles/server_admin/config.json
View
navi/tools/_internal/base.py
View
tests/unit/api/test_health.py
View
tests/unit/core/test_agent.py
View
tests/unit/core/test_registry.py
View
tests/unit/llm/test_config.py 0 → 100644
View
tests/unit/llm/test_fallback.py
View
tests/unit/llm/test_health.py 0 → 100644
View
tests/unit/llm/test_model_stamping.py
View
tests/unit/llm/test_ollama.py
View
tests/unit/memory/test_store.py
View
tests/unit/profiles/test_profile_loader.py
View
webclient/dist/assets/index-Bf7mBiaD.js 100644 → 0
Not supported
View
webclient/dist/assets/index-DPNYPAd0.css 0 → 100644
View
webclient/dist/assets/index-WziZgnPy.js 0 → 100644
Not supported
View
webclient/dist/assets/index-uqZyYvXb.css 100644 → 0
View
webclient/dist/index.html
View
webclient/dist/sw.js
View
webclient/src/components/chat/ChatHeader.vue
View
webclient/src/components/ui/ModelChip.vue 0 → 100644
View
webclient/src/composables/useWebSocket.js
View
webclient/src/i18n/messages/en.js
View
webclient/src/i18n/messages/ru.js
View
webclient/src/i18n/messages/uk.js
View
webclient/src/stores/chat.js
View
webclient/tests/unit/stores/chat.test.js