Refactor profile tool config to explicit agent/subagent structure ...

Replaces the confusing mix of enabled_tools + mcp_servers + subagent_tools
with a single explicit structure:

    tools: {
      agent:   {native: [...], mcp: {server: [groups]}},
      subagent:{native: [...], mcp: {server: [groups]}}
    }

Why:
- Old fields mixed native and MCP names (mcp__server__tool) in one list,
  making it impossible to tell at a glance what a subagent actually gets.
- subagent_runner.py had 25 lines of runtime MCP filtering logic that
  was hard to follow and error-prone.

Changes:
- AgentProfile: add ToolConfig / ToolScopeConfig pydantic models,
  keep old fields (enabled_tools, mcp_servers, subagent_tools) for
  auto-migration via _migrate_tools validator.
- loader.py: read new "tools" key, auto-migrate legacy configs.
- agent.py: _tool_list now accepts ToolScopeConfig.
- subagent_runner.py: simplified — profile.get_subagent_tools() returns
  the exact scope, no runtime filtering needed.
- context_builder.py, list_tools.py, spawn_agent.py: updated to use
  profile.get_agent_tools() / get_subagent_tools().
- All 6 profile config.json files migrated to new schema.
- Secretary subagent now explicitly gets navi-web MCP tools for web search.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix planning: change plan-follow-up role from system to user ...

After injecting the plan as an assistant message into session.context,
the previous code appended a system message saying "Plan is ready.
Execute it now..."  Many instruct-tuned models treat their own
assistant message as a completed response, and a trailing system
instruction is easy to ignore.

Changing the follow-up to role="user" makes the model see:
  assistant: plan
  user: "Execute this plan..."
which obligates the model to produce a new assistant response —
the tool-calling execution phase.

The follow-up message is appended only to session.context (LLM
context) and never to session.messages, so it is invisible in the
chat UI.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix stop button responsiveness and shutdown CancelledError ...

Agent loop (_execute_tools_with_sink):
- Poll stop_event every 1s while draining the event sink via asyncio.wait_for
- When stopped, cancel the tool task, yield a synthetic ToolEvent failure,
  append a cancellation message to session, yield StreamStopped, and return
- Pass stop_event into _execute_tools_with_sink call site

Subagent runner:
- Check stop_event at the start of each tool in turn_tool_calls loop
- Returns early with ("", False) when stopped mid-batch

McpManager.disconnect_all():
- Disconnect clients sequentially instead of asyncio.gather
- Handle asyncio.CancelledError per-client to avoid shutdown traceback

AppContainer.shutdown():
- Catch BaseException instead of Exception for MCP and DB cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix token counting: show only completion tokens, not cumulative prompt+completion ...

The token_count displayed next to assistant messages was summing
prompt_tokens + completion_tokens across ALL tool-calling iterations,
giving hundreds of thousands of tokens for multi-turn conversations.

Now:
- token_count (coins icon) = only completion tokens generated by the model
- context_tokens (ContextBar) = only prompt tokens (context size sent to LLM)

This gives users a realistic measure of how much the model actually generated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate MCP tool naming from mcp:server:tool to mcp__server__tool ...

The colon separator (mcp:server:tool) confuses many LLMs during
tool-calling because colons appear in schemas and URLs. Switch to
double-underscore separator (mcp__server__tool) for robust parsing.

Key changes:
- navi/mcp/tools.py: add build_mcp_name(), parse_mcp_name(), is_mcp_tool()
- navi/core/tool_executor.py: update _resolve_tool() with new helpers
  and legacy colon fallback for old sessions
- navi/core/tool_utils.py, subagent_runner.py: use build_mcp_name()
- navi/api/routes/{admin,agents}.py: prefix via build_mcp_name()
- navi/tools/{list_tools,reload_tools}.py: migrated
- All profile configs + system_prompt.txt: replace mcp: with mcp__
- manuals/{model_3d,lint_scad,render_3d,spawn_agent}.md: updated
- mcp_servers.d/gnexus-book.json: instructions updated
- docs/{api,profiles,tools,mechanics,visual.html}: updated
- tests: test_tool_executor.py and test_mcp.py aligned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

SubAgentRunner: filter mcp_servers against subagent_tools whitelist ...

When a profile defines subagent_tools (strict whitelist for sub-agents),
MCP servers were still expanded unconditionally, granting sub-agents access
to MCP tools not listed in the whitelist.  Now:

- If subagent_tools contains mcp:xxx entries, only those specific MCP tools
  are passed to build_tool_list.
- If subagent_tools is non-empty but contains no mcp: entries, mcp_servers
  is set to None — sub-agents get no MCP tools at all.
- If subagent_tools is empty (fallback to enabled_tools), full mcp_servers
  is kept for backward compatibility.

400 passed, 1 skipped

FallbackOllamaBackend: do not blacklist single server, empty file fallback ...

- When only one Ollama server is configured, LLMConnectionError no longer
  adds it to the dead-server blacklist. This fixes the bug where a
  transient failure permanently blocked all requests until server restart.
- LLMModelNotFoundError on a single server is also not blacklisted.
- _discover_backends now falls back to settings.ollama_host when the
  ollama_backends_file is empty, missing, or returns no valid servers.
- Added unit tests covering single-server no-blacklist, multi-server
  blacklist, model-not-found no-blacklist, and empty-file fallback.

400 passed, 1 skipped

McpTool: auto-inject session_id + normalize navi-3d paths ...

- McpTool.execute() now forces the real session_id from current_session_id
  ContextVar, preventing LLM hallucinations of wrong UUIDs (ghost-session bug).
- For navi-3d MCP server, source_path/output_path are normalized to basename
  to prevent double path nesting when the LLM passes full relative paths.
- Updated MCP tool descriptions to ask for filenames only.
- Added system prompt instructions in context_builder and subagent_runner
  reminding the model to pass bare filenames to navi-3d tools.

396 passed, 1 skipped

Fix UnboundLocalError: create mcp_manager before build_default_registries ...

The previous commit passed mcp_manager to build_default_registries but
left the instantiation after the call, causing UnboundLocalError at
runtime. Move McpManager() creation before the registry build and
remove the now-obsolete post-hoc _mcp_manager patching loop.

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

DRY: unify tool execution in ToolExecutor._execute_one() ...

Three methods (_run_single_tool, _execute_tool_calls,
_execute_tool_calls_streaming) duplicated identical logic:
resolve → middleware → execute → image extraction → build message.

- Extract canonical _execute_one(tc, tool_map) -> (ToolEvent, Message, image_msg)
- All three public methods now delegate to _execute_one
- Public signatures unchanged — no test or caller changes needed

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Eliminate cross-registry patching in registry.py via proper creation order ...

- Reorder build_default_registries so profiles and cp_registry are created
  BEFORE tools that need them
- All tools now receive full dependencies at construction time:
  ListToolsTool(profile_registry=profiles, mcp_manager=mcp_manager)
  SpawnAgentTool(backend_registry=backends, ...)
  ReloadToolsTool(cp_registry=cp_registry, mcp_manager=mcp_manager)
- Remove all post-creation patch lines (_profile_registry, _backend_registry, _cp_registry)
- Add mcp_manager parameter to build_default_registries
- Update create_container to pass mcp_manager

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract single shared Database pool, eliminate 4 duplicated pool creations ...

- Create navi/db.py::Database managing one asyncpg pool
- KvStore, PgSessionStore, MemoryStore, RecallScheduler now accept pool in constructor
- AppContainer holds Database, shutdown closes one pool instead of 4
- create_container creates one pool and passes it to all stores
- All tests updated to set _initialized=True on fakes to skip DDL

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract WebSocket business logic into AgentSessionOrchestrator ...

- Create navi/core/orchestrator.py with AgentSessionOrchestrator and SessionRun
- Orchestrator owns _runs, _busy_sessions, Agent creation, run_agent(), run_recall()
- Transport-agnostic: accepts notify callback from WebSocket handler
- WebSocket handler (websocket.py) now only does serialization/deserialization
- _fire_recall delegates to orchestrator.run_recall() instead of inline logic
- recall_scheduler_loop now accepts orchestrator parameter
- AppContainer gains .orchestrator field, created in create_container()
- deps.py: add get_orchestrator()
- Update integration tests for scheduler_loop and websocket unit tests

All 392 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix PgSessionStore import in container.py ...

`PgSessionStore` lives in `navi.core.pg_session_store`, not `navi.core.session`.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Replace global lazy singletons with explicit AppContainer + lifespan ...

- Create navi/core/container.py with AppContainer dataclass and create_container()
- Rewrite navi/api/deps.py: remove module-level singletons, add _container global
  fallback + set_container(), use _resolve_container() for all getters
- Replace @app.on_event with @asynccontextmanager lifespan in main.py
- Update routes to use Depends(get_scheduler) instead of calling get_scheduler()
- Fix FastAPI body parsing bug: remove dataclass parameters from Depends getters
  (FastAPI was treating AppContainer sub-dependencies as Body params, forcing
  embed=True on all endpoint body params and causing 422 errors)
- Update websocket.py to use _resolve_container() instead of get_registries()
- Update integration test fixtures to build AppContainer and call set_container()
- Remove obsolete tests/unit/test_startup.py (tests removed _on_startup)
- Fix test_scheduler_loop.py fixture (get_registries no longer exists)

All 387 tests pass (excluding websocket hang tests).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 5-7: Extract async generators from run_stream, unify run() as wrapper ...

- _compression_events_preturn / _compression_events_midturn
- _consume_stream (uses StreamState)
- _execute_tools_with_sink
- run() is now a thin wrapper around run_stream() collecting StreamEnd
- Remove dead imports (json, LLMChunk)
- Mark god-object decomposition complete in architecture_weak_spots.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 4: Extract SubAgentRunner from run_ephemeral() ...

- Create navi/core/subagent_runner.py with full sub-agent loop logic
- Move _iter_stream_guarded to navi/core/stream_guard.py
- Move _check_context_size to ContextCompressor.check_context_size()
- Extract build_tool_list() and load_user_enabled_tools() to tool_utils.py
- Agent.run_ephemeral() becomes a thin wrapper delegating to SubAgentRunner
- Remove ~310 lines from agent.py
- All existing run_ephemeral tests pass unchanged

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 3: Extract AntiStallMonitor from run_stream() ...

- Create navi/core/anti_stall.py with AntiStallMonitor class
- Encapsulates stall detection (todo progress + repeated tool calls)
- Encapsulates adaptive re-plan (failed todo step detection)
- Provides init() / pre_turn() / post_turn() two-phase interface
- Remove ~50 lines of stall/replan logic from agent.py run_stream()
- Remove _todo_status_snapshot and _todo_failed_steps helpers from agent.py
- Update AgentTurnContext: remove stall fields (now live in AntiStallMonitor)
- Add 13 unit tests for pre_turn and post_turn behavior

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 2: Extract AgentTurnContext dataclass from run_stream() ...

Move 10 turn-level local variables from run_stream() into AgentTurnContext:
- turn_start, tool_call_count, turn_tokens, subagent_tokens
- stall_no_todo, stall_repeat_tools, prev_tool_sigs
- known_failed, replan_msg, injected_fact_ids

This makes run_stream() readable and prepares the ground for
AntiStallMonitor (Step 3) which will consume this context.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract ContextCompressor, fix STL viewer, expand test suite, add architecture audit docs ...

- Extract ContextCompressor from agent.py (Step 1 of god-object refactor)
- Add retry + hard-truncate fallback logic to ContextCompressor
- Add unit tests: agent loop (14), compressor (18), KV store (8),
  auth encrypt (3), auth deps (13), todo/scratchpad/image_view/memory
- Fix WebGL STL viewer: allow-same-origin sandbox + graceful fallback
- Add CompressionStarted event and client-side compression notice
- Add docs/architecture_weak_spots.md and plan_01_god_object_agent.md
- Update test_events.py and test_agent_context_size.py for new logic

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enhance native toolset and add persistent KV store ...

- Add PostgreSQL-backed KvStore (navi/store/) for session-scoped data.
- Migrate todo and scratchpad from in-memory dicts to KvStore.
- Filesystem: add copy, grep, diff actions; compress description.
- CodeExec: remove language param, expose working_dir in schema.
- ImageView: resize to 1024px JPEG + Content-Type guard for URLs.
- Memory list: return distinct categories instead of all facts.
- SSH: add scp action with upload/download support.
- Update CLAUDE.md (Postgres-only), docs/tools.md, add docs/store.md.
- Fix agent/planning/context_builder async signatures for todo helpers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

cleanup: remove deprecated tools and orphaned memory tools ...

Removed (no profile used them, no cross-dependencies):
- write_tool.py, delete_tool.py, test_tool.py
- memory_save.py, memory_search.py, memory_forget.py

Updated:
- navi/core/registry.py — removed imports and registrations
- navi/tools/__init__.py — removed imports
- docs/tools.md — removed references, updated self-extension section to MCP
- navi/tools/tool_manual.py — updated example to create_mcp_server

Profile fixes:
- developer: +tool_manual, +ssh_exec
- discuss: +list_profiles
- tool_developer: +mcp_servers navi-web

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix(recall): stabilize scheduled callback system and improve UX ...

Backend fixes:
- stop_session now stops headless recall runs via _busy_sessions dict
- _fire_recall sets user ContextVars so tools work correctly
- MaxIterationsReached treated as success, not failure
- skip_next_recall uses GREATEST(trigger_at, now) for overdue recalls
- schedule_recall rejects past trigger times
- timezone offset double-adjustment fixed for aware datetimes
- _fire_recall registers _AgentRun for reconnect/replay support
- session_sync race with stream_start fixed

Frontend improvements:
- Recall banner moved to ChatHeader with live Cancel/Skip buttons
- Recall messages styled with is_recall flag and badge
- Real-time recall updates via WebSocket (recall_update events)
- Recall filter moved to sessions-header as toggle button
- Session list shows clock icon for sessions with pending recall
- Empty state messages for empty/filtered session lists
- Fixed missing api import in ChatHeader.vue

Tests:
- Updated scheduler_loop tests for _busy_sessions dict change

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor: move tool helper modules into _internal subpackage ...

Moves non-tool infrastructure out of navi/tools/ root so that only
actual tool classes live there:

  base.py           → _internal/base.py
  loader.py         → _internal/loader.py
  middleware.py     → _internal/middleware.py
  logging_middleware.py → _internal/logging_middleware.py
  _time_parser.py   → _internal/time_parser.py

All imports updated across core/, api/, mcp/, tools/, and tests/.
No proxy files remain in navi/tools/ root.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: wire test_mcp_tool into MCP manager startup injection ...

The startup loop in main.py assigns _mcp_manager only to tools that
have the attribute, but test_mcp_tool never declared it and mcp_status
used _manager instead of _mcp_manager — so both received None forever.

Changes:
- test_mcp_tool: add __init__(mcp_manager) with _mcp_manager, fallback
  to module-level import if startup wiring skipped
- mcp_status: rename _manager → _mcp_manager, same fallback
- registry.py: register create_mcp_server and test_mcp_tool builtins
- main.py: include test_mcp_tool in startup wiring loop
- client.py: add 5s timeout to disconnect cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

refactor: migrate MCP config to directory-based mcp_servers.d ...

Replaces monolithic mcp_servers.json with per-server files in
mcp_servers.d/<name>.json. Filename stem becomes the server name.

- load_mcp_servers() reads directory globs, auto-migrates legacy file
- save_mcp_servers() writes per-server files, cleans up stale ones
- Update docstrings in manager.py and context_builder.py

Auto-migration: existing servers (gnexus-book, navi-3d, navi-web) plus
new ones (project_health, time_toolkit) now live in mcp_servers.d/.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add session search across messages with backend + frontend ...

Backend:
- GET /sessions now accepts `search` query param
- _session_summary computes match_indices and match_preview from messages
- pg_session_store: messages ILIKE added to search_list and count_all
- In-memory store: search_list also filters by message content

Frontend:
- AppSidebar: search toggle icon + debounced input, Ctrl+K shortcut
- SessionItem: highlight matching text, show match_preview from search
- SessionList: pass searchQuery to SessionItem
- SessionsStore: searchQuery/searchActive state, setSearch/clearSearch
- API layer: getSessions accepts search param
- MessageList: scroll to target message + brightness flash animation
- ChatStore: loadSession accepts targetMessageIndex, scrollToMessageIndex ref
- CSS: msg-flash keyframe animation in app.scss
- Tests updated for new getSessions signature

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Persist uploaded files in messages, live file tree updates, and UI polish ...

Backend:
- Add `files` field to `Message` model so uploaded file metadata survives page refresh
- Pass `files` through websocket handler → `agent.run_stream` / `agent.run`
- `list_tools`: make `profile_id` required; return error instead of all-tools fallback

Webclient:
- Call `fetchFiles()` after successful file upload for immediate Files tab update
- Live refresh file tree on filesystem (write/edit/append/mkdir/rm/cp/mv), terminal, and code_exec tool calls
- Add manual refresh button (desktop) and pull-to-refresh (mobile) to Files tab
- Fix live link updates: move regex creation inside per-message loop to avoid lastIndex state leak
- Restore full profile name text next to avatar in ChatHeader; hide avatar in header
- Fix mobile ArtifactsPanel: collapse tab text labels so close button fits with 3 tabs

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>