Add client-side image resize and server-side image token budgeting ...

Client (useFileUpload.js):
- Resize images to max 1024px on longest side via Canvas before upload
- JPEG quality 0.9, preserves EXIF orientation
- Reduces typical phone photos from 8 MB → ~400-900 KB

Server (websocket.py):
- Increase limit to 8 images, 50 MB total payload

Server (agent.py):
- Before adding user message to session.context, estimate how many images
  fit in remaining context (500 tok/img, up to 80% of max_context_tokens)
- Images that don't fit are saved as resized JPEGs to session_files/
- References to saved images are appended to the user message text
  so Navi can view them later via image_view tool
- session.messages keeps ALL images for full UI history

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix MCP tool spinner bug: match tool_started → tool_call by tool_call_id ...

- Add tool_call_id field to ToolStarted and ToolEvent dataclasses
- Pass tc.id as tool_call_id from agent.py, subagent_runner.py, and tool_executor.py
- Update frontend chat.js onToolStarted/onToolCall to match cards by toolCallId
  with fallback to name-matching for backward compatibility

Closes spinner issue where LLM short name ("search_docs") didn't match
resolved MCP name ("mcp__gnexus_book__search_docs").

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix MCP tool registration at startup ...

Root cause: _on_mcp_server_connected was defined and registered AFTER
mcp_manager.load_all(), so startup connections never registered their
tools into the ToolRegistry.

Fix: explicitly iterate connected clients after build_default_registries()
creates tool_registry, and register their MCP tools before starting the
health-check loop. The callback stays for future reconnects.

Also adds diagnostic warning in build_tool_list when tools are missing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add debug logging to trace MCP tool registration / resolution ...

Logs added to:
- _on_mcp_server_connected: shows how many tools are registered per server
- build_tool_list: lists missing tool names and counts
- _resolve_tool: dumps tool_map keys when a lookup fails

This is diagnostic — will be removed once the root cause is found.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add MCP health check loop, auto-reconnect, and system toast notifications ...

Backend:
- McpManager: keep all configured servers in the pool even if connect()
  fails at startup; health-check loop (30s) tries to reconnect dead servers
  and verifies live ones with list_tools()
- McpManager: set_on_server_connected callback re-registers tools when a
  dead server comes back online
- McpClient: add mark_disconnected() for silent drop detection
- McpStatusTool: skip list_tools() for disconnected servers
- Orchestrator: broadcast mcp_status_update to all WebSocket sessions
- New event type McpStatusUpdate with server_name, status, tool_count

Webclient:
- useWebSocket: handle mcp_status_update → toast.success/toast.error

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix recall race, ContextVar leaks, dead code, and recall duplication ...

- run_recall: wrap busy check + create_run in session_lock to prevent
  race between scheduler and websocket handler
- run_recall: save ContextVar tokens and reset in finally to avoid
  leaking user context into subsequent background tasks
- websocket.py: reset user ContextVars in finally after run completes
- orchestrator.py: remove dead set_notify / _notify abstraction
- orchestrator.py: extract _finalize_recall to deduplicate success /
  MaxIterationsReached / Exception finalization blocks

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Unify in-memory session state in AgentSessionOrchestrator ...

Replace scattered _runs + _busy_sessions + _session_sockets with a
single _sessions: dict[str, SessionState] on the orchestrator.

- SessionState dataclass holds run, busy_event, and websockets
- _session_sockets module-level global removed from websocket.py;
  socket tracking moved into orchestrator (add/remove_websocket)
- Event bus subscriber _on_recall_update moved into orchestrator
- Per-session asyncio.Lock added to protect concurrent-run guard
- _cleanup() auto-removes empty SessionState entries

Tests updated to reference _sessions instead of legacy _runs.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Pass explicit ToolContext to tools instead of hidden ContextVars ...

Add ToolContext dataclass (session_id, event_sink, stop_event, model,
user_id, user_role, user_info) and thread it through the execution chain:
Agent._execute_tools_with_sink → ToolExecutor → tool.execute().

All ~25 tools updated to accept ctx parameter. Tools that previously
read ContextVar now prefer ctx when provided, falling back to
ContextVar for backward compatibility.

Tests updated to pass ToolContext explicitly — no more test fixtures
that set current_session_id / current_user_id ContextVars.

ContextVar setters remain as fallback for non-tool consumers
(ai_helper, context_builder, planning) and will be removed in a
follow-up refactor.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor profile tool config to explicit agent/subagent structure ...

Replaces the confusing mix of enabled_tools + mcp_servers + subagent_tools
with a single explicit structure:

    tools: {
      agent:   {native: [...], mcp: {server: [groups]}},
      subagent:{native: [...], mcp: {server: [groups]}}
    }

Why:
- Old fields mixed native and MCP names (mcp__server__tool) in one list,
  making it impossible to tell at a glance what a subagent actually gets.
- subagent_runner.py had 25 lines of runtime MCP filtering logic that
  was hard to follow and error-prone.

Changes:
- AgentProfile: add ToolConfig / ToolScopeConfig pydantic models,
  keep old fields (enabled_tools, mcp_servers, subagent_tools) for
  auto-migration via _migrate_tools validator.
- loader.py: read new "tools" key, auto-migrate legacy configs.
- agent.py: _tool_list now accepts ToolScopeConfig.
- subagent_runner.py: simplified — profile.get_subagent_tools() returns
  the exact scope, no runtime filtering needed.
- context_builder.py, list_tools.py, spawn_agent.py: updated to use
  profile.get_agent_tools() / get_subagent_tools().
- All 6 profile config.json files migrated to new schema.
- Secretary subagent now explicitly gets navi-web MCP tools for web search.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix planning: change plan-follow-up role from system to user ...

After injecting the plan as an assistant message into session.context,
the previous code appended a system message saying "Plan is ready.
Execute it now..."  Many instruct-tuned models treat their own
assistant message as a completed response, and a trailing system
instruction is easy to ignore.

Changing the follow-up to role="user" makes the model see:
  assistant: plan
  user: "Execute this plan..."
which obligates the model to produce a new assistant response —
the tool-calling execution phase.

The follow-up message is appended only to session.context (LLM
context) and never to session.messages, so it is invisible in the
chat UI.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix stop button responsiveness and shutdown CancelledError ...

Agent loop (_execute_tools_with_sink):
- Poll stop_event every 1s while draining the event sink via asyncio.wait_for
- When stopped, cancel the tool task, yield a synthetic ToolEvent failure,
  append a cancellation message to session, yield StreamStopped, and return
- Pass stop_event into _execute_tools_with_sink call site

Subagent runner:
- Check stop_event at the start of each tool in turn_tool_calls loop
- Returns early with ("", False) when stopped mid-batch

McpManager.disconnect_all():
- Disconnect clients sequentially instead of asyncio.gather
- Handle asyncio.CancelledError per-client to avoid shutdown traceback

AppContainer.shutdown():
- Catch BaseException instead of Exception for MCP and DB cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix token counting: show only completion tokens, not cumulative prompt+completion ...

The token_count displayed next to assistant messages was summing
prompt_tokens + completion_tokens across ALL tool-calling iterations,
giving hundreds of thousands of tokens for multi-turn conversations.

Now:
- token_count (coins icon) = only completion tokens generated by the model
- context_tokens (ContextBar) = only prompt tokens (context size sent to LLM)

This gives users a realistic measure of how much the model actually generated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate MCP tool naming from mcp:server:tool to mcp__server__tool ...

The colon separator (mcp:server:tool) confuses many LLMs during
tool-calling because colons appear in schemas and URLs. Switch to
double-underscore separator (mcp__server__tool) for robust parsing.

Key changes:
- navi/mcp/tools.py: add build_mcp_name(), parse_mcp_name(), is_mcp_tool()
- navi/core/tool_executor.py: update _resolve_tool() with new helpers
  and legacy colon fallback for old sessions
- navi/core/tool_utils.py, subagent_runner.py: use build_mcp_name()
- navi/api/routes/{admin,agents}.py: prefix via build_mcp_name()
- navi/tools/{list_tools,reload_tools}.py: migrated
- All profile configs + system_prompt.txt: replace mcp: with mcp__
- manuals/{model_3d,lint_scad,render_3d,spawn_agent}.md: updated
- mcp_servers.d/gnexus-book.json: instructions updated
- docs/{api,profiles,tools,mechanics,visual.html}: updated
- tests: test_tool_executor.py and test_mcp.py aligned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

SubAgentRunner: filter mcp_servers against subagent_tools whitelist ...

When a profile defines subagent_tools (strict whitelist for sub-agents),
MCP servers were still expanded unconditionally, granting sub-agents access
to MCP tools not listed in the whitelist.  Now:

- If subagent_tools contains mcp:xxx entries, only those specific MCP tools
  are passed to build_tool_list.
- If subagent_tools is non-empty but contains no mcp: entries, mcp_servers
  is set to None — sub-agents get no MCP tools at all.
- If subagent_tools is empty (fallback to enabled_tools), full mcp_servers
  is kept for backward compatibility.

400 passed, 1 skipped

FallbackOllamaBackend: do not blacklist single server, empty file fallback ...

- When only one Ollama server is configured, LLMConnectionError no longer
  adds it to the dead-server blacklist. This fixes the bug where a
  transient failure permanently blocked all requests until server restart.
- LLMModelNotFoundError on a single server is also not blacklisted.
- _discover_backends now falls back to settings.ollama_host when the
  ollama_backends_file is empty, missing, or returns no valid servers.
- Added unit tests covering single-server no-blacklist, multi-server
  blacklist, model-not-found no-blacklist, and empty-file fallback.

400 passed, 1 skipped

McpTool: auto-inject session_id + normalize navi-3d paths ...

- McpTool.execute() now forces the real session_id from current_session_id
  ContextVar, preventing LLM hallucinations of wrong UUIDs (ghost-session bug).
- For navi-3d MCP server, source_path/output_path are normalized to basename
  to prevent double path nesting when the LLM passes full relative paths.
- Updated MCP tool descriptions to ask for filenames only.
- Added system prompt instructions in context_builder and subagent_runner
  reminding the model to pass bare filenames to navi-3d tools.

396 passed, 1 skipped

Fix UnboundLocalError: create mcp_manager before build_default_registries ...

The previous commit passed mcp_manager to build_default_registries but
left the instantiation after the call, causing UnboundLocalError at
runtime. Move McpManager() creation before the registry build and
remove the now-obsolete post-hoc _mcp_manager patching loop.

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

DRY: unify tool execution in ToolExecutor._execute_one() ...

Three methods (_run_single_tool, _execute_tool_calls,
_execute_tool_calls_streaming) duplicated identical logic:
resolve → middleware → execute → image extraction → build message.

- Extract canonical _execute_one(tc, tool_map) -> (ToolEvent, Message, image_msg)
- All three public methods now delegate to _execute_one
- Public signatures unchanged — no test or caller changes needed

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Eliminate cross-registry patching in registry.py via proper creation order ...

- Reorder build_default_registries so profiles and cp_registry are created
  BEFORE tools that need them
- All tools now receive full dependencies at construction time:
  ListToolsTool(profile_registry=profiles, mcp_manager=mcp_manager)
  SpawnAgentTool(backend_registry=backends, ...)
  ReloadToolsTool(cp_registry=cp_registry, mcp_manager=mcp_manager)
- Remove all post-creation patch lines (_profile_registry, _backend_registry, _cp_registry)
- Add mcp_manager parameter to build_default_registries
- Update create_container to pass mcp_manager

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract single shared Database pool, eliminate 4 duplicated pool creations ...

- Create navi/db.py::Database managing one asyncpg pool
- KvStore, PgSessionStore, MemoryStore, RecallScheduler now accept pool in constructor
- AppContainer holds Database, shutdown closes one pool instead of 4
- create_container creates one pool and passes it to all stores
- All tests updated to set _initialized=True on fakes to skip DDL

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract WebSocket business logic into AgentSessionOrchestrator ...

- Create navi/core/orchestrator.py with AgentSessionOrchestrator and SessionRun
- Orchestrator owns _runs, _busy_sessions, Agent creation, run_agent(), run_recall()
- Transport-agnostic: accepts notify callback from WebSocket handler
- WebSocket handler (websocket.py) now only does serialization/deserialization
- _fire_recall delegates to orchestrator.run_recall() instead of inline logic
- recall_scheduler_loop now accepts orchestrator parameter
- AppContainer gains .orchestrator field, created in create_container()
- deps.py: add get_orchestrator()
- Update integration tests for scheduler_loop and websocket unit tests

All 392 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix PgSessionStore import in container.py ...

`PgSessionStore` lives in `navi.core.pg_session_store`, not `navi.core.session`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Replace global lazy singletons with explicit AppContainer + lifespan ...

- Create navi/core/container.py with AppContainer dataclass and create_container()
- Rewrite navi/api/deps.py: remove module-level singletons, add _container global
  fallback + set_container(), use _resolve_container() for all getters
- Replace @app.on_event with @asynccontextmanager lifespan in main.py
- Update routes to use Depends(get_scheduler) instead of calling get_scheduler()
- Fix FastAPI body parsing bug: remove dataclass parameters from Depends getters
  (FastAPI was treating AppContainer sub-dependencies as Body params, forcing
  embed=True on all endpoint body params and causing 422 errors)
- Update websocket.py to use _resolve_container() instead of get_registries()
- Update integration test fixtures to build AppContainer and call set_container()
- Remove obsolete tests/unit/test_startup.py (tests removed _on_startup)
- Fix test_scheduler_loop.py fixture (get_registries no longer exists)

All 387 tests pass (excluding websocket hang tests).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 5-7: Extract async generators from run_stream, unify run() as wrapper ...

- _compression_events_preturn / _compression_events_midturn
- _consume_stream (uses StreamState)
- _execute_tools_with_sink
- run() is now a thin wrapper around run_stream() collecting StreamEnd
- Remove dead imports (json, LLMChunk)
- Mark god-object decomposition complete in architecture_weak_spots.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 4: Extract SubAgentRunner from run_ephemeral() ...

- Create navi/core/subagent_runner.py with full sub-agent loop logic
- Move _iter_stream_guarded to navi/core/stream_guard.py
- Move _check_context_size to ContextCompressor.check_context_size()
- Extract build_tool_list() and load_user_enabled_tools() to tool_utils.py
- Agent.run_ephemeral() becomes a thin wrapper delegating to SubAgentRunner
- Remove ~310 lines from agent.py
- All existing run_ephemeral tests pass unchanged

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 3: Extract AntiStallMonitor from run_stream() ...

- Create navi/core/anti_stall.py with AntiStallMonitor class
- Encapsulates stall detection (todo progress + repeated tool calls)
- Encapsulates adaptive re-plan (failed todo step detection)
- Provides init() / pre_turn() / post_turn() two-phase interface
- Remove ~50 lines of stall/replan logic from agent.py run_stream()
- Remove _todo_status_snapshot and _todo_failed_steps helpers from agent.py
- Update AgentTurnContext: remove stall fields (now live in AntiStallMonitor)
- Add 13 unit tests for pre_turn and post_turn behavior

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 2: Extract AgentTurnContext dataclass from run_stream() ...

Move 10 turn-level local variables from run_stream() into AgentTurnContext:
- turn_start, tool_call_count, turn_tokens, subagent_tokens
- stall_no_todo, stall_repeat_tools, prev_tool_sigs
- known_failed, replan_msg, injected_fact_ids

This makes run_stream() readable and prepares the ground for
AntiStallMonitor (Step 3) which will consume this context.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract ContextCompressor, fix STL viewer, expand test suite, add architecture audit docs ...

- Extract ContextCompressor from agent.py (Step 1 of god-object refactor)
- Add retry + hard-truncate fallback logic to ContextCompressor
- Add unit tests: agent loop (14), compressor (18), KV store (8),
  auth encrypt (3), auth deps (13), todo/scratchpad/image_view/memory
- Fix WebGL STL viewer: allow-same-origin sandbox + graceful fallback
- Add CompressionStarted event and client-side compression notice
- Add docs/architecture_weak_spots.md and plan_01_god_object_agent.md
- Update test_events.py and test_agent_context_size.py for new logic

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enhance native toolset and add persistent KV store ...

- Add PostgreSQL-backed KvStore (navi/store/) for session-scoped data.
- Migrate todo and scratchpad from in-memory dicts to KvStore.
- Filesystem: add copy, grep, diff actions; compress description.
- CodeExec: remove language param, expose working_dir in schema.
- ImageView: resize to 1024px JPEG + Content-Type guard for URLs.
- Memory list: return distinct categories instead of all facts.
- SSH: add scp action with upload/download support.
- Update CLAUDE.md (Postgres-only), docs/tools.md, add docs/store.md.
- Fix agent/planning/context_builder async signatures for todo helpers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>