tools, mcp: a description is a hook, not a manual ...

With 24 tools in a profile the schemas are the largest fixed block of every
request, and they are paid for whether or not the tool is called. Eleven of
them carried a manual's worth of prose in their `description` — examples,
error tables, "common mistakes", the mechanics of the file areas. Reading
them against the manuals first, the manuals already held all of it, and
richer: this was duplication sitting in the one place the model cannot avoid
reading, not knowledge that had nowhere else to live. So the descriptions
became selection hooks — what the tool is for, when to pick it over its
neighbour, and the one rule that cannot wait — and the detail moved to
tool_manual("<tool>"), which costs nothing until the tool is actually in
play.

filesystem, spawn_agent, schedule_recall, manage_recall, scratchpad, plan,
share_file, content_publish, reflect, todo and peer. Descriptions drop from
18.4 KB to 11.5 KB, those tools' schemas from 25.5 KB to 16.8 KB, and a
profile's whole native toolset from 38.3 KB to 29.6 KB (server_admin,
developer; secretary 33.7 → 25.0, navi_code 32.9 → 25.9, tool_developer
37.9 → 29.2, modeler_3d 33.4 → 24.8). Stated as a budget rather than a byte
count: roughly 4k tokens of the model's window per request, back.

Two things were deliberately not shortened. The filesystem edit ladder
(`edit` → `edit_lines` → `smart_edit`, last resort, costs an LLM call) and
todo's mandatory `validation` on `done` both exist to prevent an extra
round trip; a shorter description there would trade tokens for turns. What
todo lost is the prose around the rule, not the rule. `enabled.json` was
left alone too: weather, gmail and get_current_datetime are opt-in and the
DB shows 12, 16 and 40 calls, so they are opt-in tools that are actually
used.

peer had no manual at all, which is how this surfaced: its description was
the only documentation the tool had. manuals/peer.md is now written from
the tool and its /peer route — the ask/status/list actions, the fact that
the hive is only a phone book and the question travels peer-to-peer, the
one-concurrent-answer semaphore, the two independent recursion guards, and
the six error codes.

The MCP server instructions leave the system prompt the same way. They were
8 KB of always-present prose per request, most of it workflow detail the
model only needs when it is about to use that server. Each server now
contributes one line — a new optional `summary` field in mcp_servers.d/*.json,
defaulting to the first sentence of its `instructions` — and
tool_manual("<server>") returns the full text plus the tool names the server
declares. server_admin's MCP block goes from 8.3 KB to 1.3 KB, secretary's
from 8.1 to 1.2. tool_manual learned to answer for a server (a tool name
still wins over a server of the same name, and dash/underscore spelling is
tolerated), and the index names the servers alongside the tools.

One judgement call worth recording: gnexus-book's and navi-web's
instructions end in an absolute "NEVER bypass these tools with filesystem,
terminal or code_exec" rule. Moving that on demand would have been a
behavioural regression dressed as a token saving, so those two `summary`
fields carry the prohibition verbatim alongside the hook, and it stays
always visible.

docs/tools.md gains a section on the split (a description is paid for every
request, a manual only when the tool is used), which is where the next
person will look before trimming a description back into a manual.

Full suite green (1577 passed, 1 skipped). No frontend change and no new
dependency, so the deploy is a pull and a restart.

agent: a profile switch now survives the turn that made it ...

switch_profile repoints the session with one narrow UPDATE, and the run's own
save() — which lands right after every tool batch, the switch's own batch
included — wrote its stale in-memory profile back over it. The post-turn
reload then read the row it had just clobbered, saw no change, and left the
turn bound to the old profile's tools: the agent called exactly the tools
switch_profile had just advertised ("Newly available: ssh_exec, terminal, …")
and got "tool not found" for every one of them. Prod shows both halves — the
session that reported two switches kept the profile it started with, and
agent.profile_reloaded appears zero times in the whole boot.

The reload now compares against the profile the run is bound to
(bound_profile_id), which set_profile cannot move, instead of against
session.profile_id, which the store either rewrites or mutates to the new
value — making the old guard false in both branches by construction. And
save() leaves the profile_id column alone on conflict: set_profile owns it
after creation, the same way name/created_at are owned elsewhere.

A tool the live tool_map does not hold is now logged (tool.not_found, with
profile and live tool count). It previously left no trace at all in the
journal, which is why this took a database dig to find.

Three tests, each verified to fail without its fix: the next LLM call is
offered the new profile's tools, save() does not rewrite profile_id, and a
missing tool is logged.

tools: list_tools defaults to the profile the run executes as ...

Asking "what tools do I have?" required the agent to know its own profile id
— and to be right about it after a switch_profile had moved the session. The
run knows: run_stream now publishes the active profile id (re-published when
switch_profile reloads the run) alongside the model, SubAgentRunner publishes
its own rather than inheriting the parent's, and ToolContext carries it into
every execute() call.

list_tools reads that when profile_id is omitted, so the bare call answers the
question, and it says which profile it assumed ("current profile") so the
answer cannot be mistaken for another profile's.

scope='agent'|'subagent' selects which of the profile's two toolsets to list —
the set spawn_agent would hand a sub-agent running that profile, which is a
different list and until now had no way to be inspected.

Item D of the list_tools plan, kept separate from the accuracy/compactness work.

docs: sync with the plan rework and compression pipeline ...

- planning: plan tool flow (agent-invoked, no pre-turn gate), confirmation
  by COMPLEXITY, think=False + 20k-char Phase 1 window, retired flags
  removed from profiles/api docs
- compression: all four triggers through compress_and_save_session,
  real-baseline gates, head+tail summary input, _plan_compression shared
  decision, actual config defaults (0.90 threshold, 6000 summary tokens,
  32k input cap)

docs: actualize compression mechanics + real-token baseline ...

mechanics.md: rewrite the Context Compression catalog (adaptive turn
importance, intra-turn fallback, would_compress guard, token-budget
hard-truncate, per-message view truncation, profile-aware overrides,
real-token baseline estimator, meta-summary, archive-on-compress,
CompressionStarted/ContextCompressed events); fix Agent Loop rows
(run() delegates to run_stream, planning entry condition, pre/mid-turn
gates, check_context_size, forced /compact, estimate chars//3 in
compressor.py); fix WS image limits (8 / 50MB) and add /compact control
message; rebuild the Undocumented Mechanics Summary from current ❌ rows;
refresh the cross-reference index.

config.md: LLM_STREAM_FIRST_CHUNK_TIMEOUT 180→90; add
CONTEXT_MESSAGE_TOKEN_BUDGET.

agent.md: run() runs planning; full run_ephemeral signature + compact_stream
entry point; cwd ContextVar; planning entry condition; loop pseudocode
(midturn compression, check_context_size, ModelInfo, real baseline);
streaming guard 90s; workers carry profile; pre-turn estimate-based gate;
per-message view truncation.

sessions.md: full Message flag table (is_context/is_display/is_compression_critical
…); three compression trigger points + forced /compact; real-baseline
estimate; check_context_size guard; expanded algorithm (adaptive partitioning,
intra-turn fallback, meta-summary, 24k cap, 4000-char critical preview,
token-budget hard-truncate fallback, archive-on-compress, baseline clear);
compression events; SessionStore archive methods + list_page profile_id.

Co-Authored-By: Claude <noreply@anthropic.com>

docs: document bounded autonomy + navi-code TUI features ...

- profiles.md: new "Bounded autonomy" section (scope_boundary_enabled,
  observe_skips_plan_enabled) + memory-facts scope filter; navi_code section
  and config.json example updated
- mechanics.md: Agent Loop / Planning Pipeline / Context Builder rows for
  scope boundary, observe short-circuit, MODE classification, scope filter
- navi_code.md: bounded autonomy note in profile section
- navi_code_cli.md: --raw and -h/--help params, multiline input, planning
  and spawn_agent cards, nested subagent styling, planning_status/plan_ready/
  turn_thinking rendering, tui/renderers/ in dev section

Co-Authored-By: Claude <noreply@anthropic.com>

Fix MCP names and profile API format consistency ...

- Replace stale mcp__navi_web__* / mcp__navi_3d__* names with canonical
  mcp__navi-web__* / mcp__navi-3d__* across prompts and key_tools.
- Update /agents/profiles and /admin/profiles endpoints to expose
  tools.agent / tools.subagent instead of deprecated enabled_tools fields.
- Update docs/mechanics.md to reference the new tools structure.
- Archive stale docs/visual.html.

Co-Authored-By: Claude <noreply@anthropic.com>

Add NAVI_AUTH_ENABLED switch for optional auth ...

- Add navi_auth_enabled setting (default true) to navi/config.py and .env.example
- When disabled, treat every request as anonymous admin user (id='anonymous')
- Create/update fixed anonymous navi_users row on startup
- Bypass OAuth/cookie/API-token resolution in navi/auth/deps.py
- Update /auth/status to return {enabled, configured}
- Log security warning on startup when auth is disabled
- Update webclient: skip fetchMe/login screen, show Local mode footer,
  expose /admin link, warn in API keys panel
- Rebuild webclient production bundle
- Add unit and integration tests for no-auth mode
- Update docs: auth.md, config.md, api.md, api_tokens.md, sessions.md,
  websocket.md, mechanics.md, index.md

Co-Authored-By: Claude <noreply@anthropic.com>

Migrate MCP tool naming from mcp:server:tool to mcp__server__tool ...

The colon separator (mcp:server:tool) confuses many LLMs during
tool-calling because colons appear in schemas and URLs. Switch to
double-underscore separator (mcp__server__tool) for robust parsing.

Key changes:
- navi/mcp/tools.py: add build_mcp_name(), parse_mcp_name(), is_mcp_tool()
- navi/core/tool_executor.py: update _resolve_tool() with new helpers
  and legacy colon fallback for old sessions
- navi/core/tool_utils.py, subagent_runner.py: use build_mcp_name()
- navi/api/routes/{admin,agents}.py: prefix via build_mcp_name()
- navi/tools/{list_tools,reload_tools}.py: migrated
- All profile configs + system_prompt.txt: replace mcp: with mcp__
- manuals/{model_3d,lint_scad,render_3d,spawn_agent}.md: updated
- mcp_servers.d/gnexus-book.json: instructions updated
- docs/{api,profiles,tools,mechanics,visual.html}: updated
- tests: test_tool_executor.py and test_mcp.py aligned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix review issues: KV-store NULL, SVG messages, filesystem tests + docs ...

Bugs fixed:
- filesystem.py: add missing `import re` (grep was broken in production)
- image_view.py: consistent SVG rejection message for URL and file paths
- store/__init__.py: normalize user_id None→'' to prevent duplicate rows
  in unique constraint for anonymous sessions; add DDL migration for
  existing NULL values

Tests:
- Add 10 unit tests for filesystem copy, grep, diff operations

Documentation:
- agent.md: document streaming guard wrapper, system prompt caching,
  ContextVar restoration in subagents
- tools.md: document middleware hooks
- websocket.md: document image upload limits and concurrent run guard
- store.md: document user_id normalization
- mechanics.md: mark newly-documented mechanics as documented

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add comprehensive mechanics catalog (docs/mechanics.md) ...

Master catalog of all mechanisms in the Navi project:
- 80+ mechanics catalogued across all subsystems
- Each with description, config flags, source files, documentation status
- 50 undocumented mechanics explicitly listed
- Cross-reference index for feature planning
- Updated docs/index.md to link the catalog

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>