# MCP (Model Context Protocol) How Navi connects to external MCP servers, how their tools are exposed, and how per-user credentials (BYOK) work. ## Configs Each server lives in its own file `mcp_servers.d/.json`: ```json { "transport": "streamable_http", "url": "https://creds.example.com/mcp", "headers": { "Authorization": "Bearer ${NAVI_MCP_GNEXUS_CREDS_TOKEN}" }, "groups": { "creds": ["search_secrets", "get_secret", "reveal_secret"] }, "instructions": "MANDATORY: ...", "user_key": { "header": "Authorization", "prefix": "Bearer " } } ``` | Field | Applies to | Meaning | |---|---|---| | `transport` | all | `stdio` \| `sse` \| `streamable_http` | | `command`, `args`, `env`, `cwd` | `stdio` | Subprocess launch | | `url`, `headers` | `sse`, `streamable_http` | Endpoint and HTTP headers | | `groups` | all | Named tool groups profiles reference instead of listing tools | | `instructions` | all | Overlay injected next to the server's own initialize instructions | | `user_key` | all | Declares the per-user credential slot (BYOK); omit for key-less servers | The legacy monolithic `mcp_servers.json` is auto-migrated to `mcp_servers.d/` on first load. Edits take effect on server restart or a `reload_tools`-triggered `McpManager.reload_all()`. **Heads-up for admin config edits:** `PUT /admin/agents/mcp-servers` round-trips the whole model — write back the config you read, or a `user_key` slot added by hand is silently erased. ## Secrets `mcp_servers.d/*.json` is tracked in git, so a credential written into one is in history for good — and `GET /admin/mcp/config` hands the same value to the admin client. A config therefore never carries a live credential: it names the variable that holds it. ```json "headers": { "Authorization": "Bearer ${NAVI_MCP_GNTODO_TOKEN}" } ``` - **Only `headers` and `env` values are scanned**, and only the braced `${NAME}` form — a bare `$NAME` is left alone, so header values that contain `$` survive. - **The value comes from the process environment over the project's `.env`** (chmod 600, untracked; environment variable beats the file, as with `Settings`). A blank `NAME=` in `.env` counts as missing. `.env` is re-read on every (re)connect, so a rotated token is picked up without a restart — the canonical store for the value itself is **gnexus-creds** (see the platform's secrets rules; `deploy/env.template` lists the variable names). - **A missing variable fails the connection** with `ValueError` naming the server and the variable, rather than dropping the header — a server may answer an unauthenticated request as an anonymous user instead of returning 401, so silently dropping would fail open. `McpManager.load_all` isolates a failing server, so this shows up as one broken entry in `/mcp/status`. - **The explicit escape hatch is `${NAME:-fallback}`** (fallback may be empty). It is for a *setting* that tunes a server — a SearXNG URL — where being unset costs one backend instead of every tool the server provides. Writing the fallback down is the point: it is a decision in the config, not an accident of a variable nobody set. A credential never takes this form. - **Resolution happens at transport open** (`navi/mcp/client.py`), next to the project-relative path resolution. This is what keeps every stored and serialised config placeholder-only: `save_mcp_servers` — reached by `create_mcp_server` and by `PUT /admin/mcp/config` — cannot write a secret back into a tracked file, and a GET→PUT round-trip is lossless. A user key (`user_key`) is injected first and overwrites the header, so a per-user credential simply replaces the placeholder — no substitution happens. ### `navi-web` settings The search/browse server has no user key, but it still has runtime settings of its own, and they reach it through the same placeholder mechanism. A stdio server does **not** inherit the Navi process's environment: the SDK passes a safe allowlist (`HOME`, `PATH`, `USER`, …) plus the config's `env` block, so a variable that is only in `.env` is invisible to it unless the config maps it. | Config `env` key | `.env` variable | Meaning | |---|---|---| | `NAVI_WEB_MCP_TRANSPORT` | — | `stdio`, declared inline | | `NAVI_WEB_SEARXNG_URL` | `SEARXNG_URL` (via `${SEARXNG_URL:-}`) | Self-hosted SearXNG base URL — the primary backend | | `NAVI_WEB_BRAVE_API_KEY` | `NAVI_MCP_NAVI_WEB_BRAVE_KEY` (via `${NAVI_MCP_NAVI_WEB_BRAVE_KEY:-}`) | Optional third backend. Empty means off; the key is optional, so the placeholder carries a fallback (see above) | | `NAVI_WEB_DDG_REGION` | — | DuckDuckGo region, default `ua-uk` | Without `NAVI_WEB_SEARXNG_URL` the server starts straight at the `ddgs` merge — the engines pinned in `search._DDG_BACKENDS` (DuckDuckGo, Brave, Mojeek, Yahoo, Wikipedia, Grokipedia; Yandex is excluded on purpose). It is slower and the first thing to be rate-limited. The `${SEARXNG_URL:-}` fallback is what keeps a host with no SearXNG working: an unset value costs the backend, not the server's three tools. `navi.config.Settings` has no search fields: `SEARXNG_URL` is read here, not by Navi itself. Two details in the SearXNG request are deliberate. The `engines` list is pinned (`google,duckduckgo`) because the instance's own defaults are mostly suspended, and **bing is excluded**: from a host it cannot match, Bing answers with trending/homepage blocks, and a non-empty list of noise otherwise wins the chain over a working fallback. The `language` is `all`, not a fixed tag — a pinned `uk-UA` made the instance's DuckDuckGo engine return CAPTCHA on every request (`kl=ua-uk`) and skewed English queries. The same trap exists in `ddgs` v9: `backend` takes engine names, and an unknown name does not fail — it silently means "all engines". #### Batching is the latency lever `web_search` takes one query or up to three, `web_view` one URL or up to three; several run concurrently, so the call costs the slowest item instead of the sum — three pages measured 17.3 s one by one and 1.9 s together, three searches 3.5 s and 0.6 s. The caps live in the server: three items, deduplicated, extras dropped with a note in the output. The model is pointed at the feature twice — by the tool descriptions, which travel in every request, and by `summary` in `mcp_servers.d/navi-web.json`, which is the line that reaches the system prompt (`instructions` carries the long form for `tool_manual`). The wording is a trigger, not a speed plea: batch when the lookups are already known to be independent, never a follow-up whose query depends on the previous answer. A per-item deadline is deliberate. One hung backend or page is named in the output rather than erasing what the others already returned, and the tool raises — `isError` in the journal — only when nothing at all came back. This is not Navi's `parallel_tool_calls`: that flag runs a whole batch of independent tool calls concurrently for *every* tool of a profile, `terminal` included. The server-side fan-out is scoped to these two tools, and its caps and budgets are enforced where the work happens. #### What `web_view` hands back The page is rendered in Chromium, then `trafilatura` picks the main content out by text density and the server renders that fragment as markdown: `#` headings, fenced code with its own indentation, table rows joined by `|`. This replaced `innerText` after a tag kill-list, which could not tell navigation from prose — 462 of the asyncio page's 1018 lines were shorter than 40 characters — collapsed tables into one line per thousand characters (news.ycombinator.com came back as four lines), marked no headings and dropped every link. Links become a `Links on this page` appendix rather than `[label](url)` inline: at most 40, absolute, deduplicated, with self-anchors and `hide`/`vote`/`login` chrome filtered out. Inline links cost 3.5k characters on the docs page alone, and `trafilatura` drops the `#` from any heading that carries an anchor when they are on. Text past 20k characters is cut at a paragraph boundary, and the note names the sections that did not fit. A page whose content *is* its thread — a forum, a Q&A, a code review — comes back as a bare header when comments are excluded, so when the extractor keeps less than a quarter of the page's visible text it runs again with comments. If it still finds nothing worth sending, the tool falls back to the visible-text read; `metadata.extractor` says which path answered (`trafilatura` or `innerText`), and `metadata.links` counts the appendix. #### The server's own venv Each in-repo server runs from `mcp-servers//.venv`, which is separate from Navi's and is what `mcp_servers.d/.json` points `command` at: `./mcp-servers/navi-web/.venv/bin/python -m app.mcp_server` with its working directory there. Navi's venv does not carry those dependencies, so a dependency added to `mcp-servers//pyproject.toml` needs an install into that venv — `deploy/sync-mcp-venvs.sh` does it for every server, and `deploy/install.sh` calls it. It is stamped by a hash of each `pyproject.toml`, so an unchanged server is skipped; run it on its own after a `git pull` (see [`deploy/UPDATE.md`](../deploy/UPDATE.md)). ```bash bash deploy/sync-mcp-venvs.sh # all servers, no-op for unchanged ones sudo systemctl restart navi ``` For a one-off install outside a deploy checkout, the same thing by hand: ```bash VIRTUAL_ENV=mcp-servers/navi-web/.venv uv pip install -e mcp-servers/navi-web ``` Skipping it is silent: the module does not import, `McpManager` cannot connect, and `build_tool_list` reports the server's tools as missing — the model simply never sees them. `/admin/mcp/status` and the `MCP server %r failed to connect` line in the journal are where it shows up. ## Runtime - `McpManager` (`navi/mcp/manager.py`) holds one `McpClient` per server, reconnects dead ones in a 30 s health-check loop, and is the single `call_tool` gateway. Connection headers/env are fixed at transport open (`navi/mcp/client.py`) — a shared client cannot switch credentials per call. - Tool names follow `mcp____`; profiles select servers via `tools.agent.mcp` / `tools.subagent.mcp` group maps. Registered MCP tools are process-wide, regardless of which profile uses them. - `McpTool.execute` forwards `ctx.user_id` (falling back to the `current_user_id` ContextVar) into `manager.call_tool(..., user_id=...)`. ## Per-user keys (BYOK) A config with a `user_key` slot lets each user bring their own credential: ```json "user_key": { "header": "Authorization", "prefix": "Bearer " } ``` or, for stdio servers: ```json "user_key": { "env": "API_KEY" } ``` Exactly one destination (validator `navi/mcp/config.py::McpUserKey`); a `header` is only valid on HTTP transports, an `env` only on stdio. Resolution flow (`McpManager._client_for`), given the caller's `user_id` and `role`: 1. The config has **no** `user_key` slot → the default shared client. The resolver and the database are never touched. 2. The config **has** a slot and the caller has a saved key → a per-user `McpClient` clone built from `McpServerConfig.with_user_key(key)` (a deep copy with the key injected into the header/env destination). Clones are cached per key snapshot: a changed key transparently rebuilds the client. 3. The config **has** a slot and the caller has **no** key: - `role == "admin"` → the default shared client with the credential from the config file. An admin's calls behave byte-for-byte as before BYOK existed — the shared credential is theirs. - any other role → **refused**, with a `RuntimeError` naming the server and pointing at Settings → MCP keys. There is no quiet fallback to the owner's credential: that fallback is what made six of the owner's servers reachable by every account. `KeyResolver.resolve` reads the user's key (30 s in-memory TTL; every failure — DB error, missing `NAVI_AUTH_ENCRYPTION_KEY` — logs a warning and resolves to `None`, i.e. "no key", which for a non-admin is a refusal rather than a fallback). ### Visibility — a gated server is absent, not merely refused A server the manager would refuse must not appear in the tool list either, or the model sees a tool it can never call, retries it, and records a broken tool instead of a boundary. `build_tool_list` is synchronous while the key store is not, so the allowed set is computed **once per run** and only read afterwards: - `Agent._apply_mcp_scope` (called at the top of `run_stream`) asks `McpManager.visible_servers(role, user_id)` and puts the answer in the `current_allowed_mcp_servers` ContextVar (`navi/tools/_internal/base.py`). It returns two tokens and both are reset on the stop and max-iterations exit paths; the normal-completion path returns without resetting, which is harmless because every run sets the variables again before anything reads them. - `build_tool_list` (`navi/core/tool_utils.py`) skips any server outside that set. - `role == "admin"`, or no MCP layer at all → the ContextVar is `None`, meaning "no restriction", and nothing changes for the owner. - Fail closed: if `visible_servers` raises, the set is empty rather than unrestricted. **Why a server is absent is part of the prompt.** Absence alone left the model knowing the server existed, holding none of its tools, and having nothing to tell the user. `_apply_mcp_scope` therefore also derives the withheld servers — `byok_servers() − allowed`, the complement of what it kept, not a second read of the key store that could disagree with the first — and puts them in `current_keyless_mcp_servers`. `ContextBuilder._mcp_context_msg` names each of them in the `[MCP servers]` system message, followed by the reason and a pointer at Settings → MCP keys, so the assistant can explain the missing key itself instead of silently lacking the tools. The variable defaults to empty, so a run that never set it — a peer run out of a fresh API task — claims nothing. Saving a key in the panel takes effect on the next turn — the resolver's `on_change` callback clears the manager's visibility cache along with the affected clients. Cache bounds: hard LRU cap of 32 per-user clients; per-user clients idle longer than 30 minutes are reaped by the health-check tick. Per-user clients are outside the health-check's reconnect logic — they reconnect lazily with backoff when next called. `disconnect_all` (reload/shutdown) tears user clients down as well. Key changes (PUT/DELETE) fire the resolver's `on_change` callback, which drops affected per-user clients and the visibility cache immediately. ### Storage `mcp_user_keys` table (postgres, `navi/mcp/_ddl.py`), Fernet-encrypted via `NAVI_AUTH_ENCRYPTION_KEY` (`navi/mcp/keystore.py::McpKeyStore`). One row per `(user_id, server_name)`; deleting a user cascades. Keys never leave the backend through logs or REST responses. Anon mode (`NAVI_AUTH_ENABLED=false`) is one shared local user — its saved key effectively becomes a system-level override, and requires `NAVI_AUTH_ENCRYPTION_KEY` to be set (the Fernet encryptor refuses an empty key). ## REST Settings UI surface — see [`api.md`](api.md#mcp-user-keys-byok): `GET /mcp-keys` (every profile-referenced server, keyed ones flagged, no key values), `PUT /mcp-keys/{server}` (save/invalidate), `DELETE /mcp-keys/{server}` (reset to default). The webclient lists all of them: rows for servers without a `user_key` slot are read-only context, and only slotted servers get a key input. Servers that declare an `Authorization`-style header carry a shared default in the config — as a `${...}` placeholder, not a literal (see [Secrets](#secrets)) — which an admin keeps using when they have no personal key, and which an ordinary user cannot reach at all.