Fix review issues: KV-store NULL, SVG messages, filesystem tests + docs ...

Bugs fixed:
- filesystem.py: add missing `import re` (grep was broken in production)
- image_view.py: consistent SVG rejection message for URL and file paths
- store/__init__.py: normalize user_id None→'' to prevent duplicate rows
  in unique constraint for anonymous sessions; add DDL migration for
  existing NULL values

Tests:
- Add 10 unit tests for filesystem copy, grep, diff operations

Documentation:
- agent.md: document streaming guard wrapper, system prompt caching,
  ContextVar restoration in subagents
- tools.md: document middleware hooks
- websocket.md: document image upload limits and concurrent run guard
- store.md: document user_id normalization
- mechanics.md: mark newly-documented mechanics as documented

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix scratchpad get_section user_id and filesystem diff newline bugs ...

- get_section: use _uid() instead of None so multi-user lookups match
  the user_id-scoped writes done by the scratchpad tool execute()
- filesystem _diff: remove erroneous [l + \"\\n\"...] wrappers around
  splitlines() output; difflib.unified_diff already handles line lists

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enhance native toolset and add persistent KV store ...

- Add PostgreSQL-backed KvStore (navi/store/) for session-scoped data.
- Migrate todo and scratchpad from in-memory dicts to KvStore.
- Filesystem: add copy, grep, diff actions; compress description.
- CodeExec: remove language param, expose working_dir in schema.
- ImageView: resize to 1024px JPEG + Content-Type guard for URLs.
- Memory list: return distinct categories instead of all facts.
- SSH: add scp action with upload/download support.
- Update CLAUDE.md (Postgres-only), docs/tools.md, add docs/store.md.
- Fix agent/planning/context_builder async signatures for todo helpers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

cleanup: remove deprecated tools and orphaned memory tools ...

Removed (no profile used them, no cross-dependencies):
- write_tool.py, delete_tool.py, test_tool.py
- memory_save.py, memory_search.py, memory_forget.py

Updated:
- navi/core/registry.py — removed imports and registrations
- navi/tools/__init__.py — removed imports
- docs/tools.md — removed references, updated self-extension section to MCP
- navi/tools/tool_manual.py — updated example to create_mcp_server

Profile fixes:
- developer: +tool_manual, +ssh_exec
- discuss: +list_profiles
- tool_developer: +mcp_servers navi-web

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix(recall): stabilize scheduled callback system and improve UX ...

Backend fixes:
- stop_session now stops headless recall runs via _busy_sessions dict
- _fire_recall sets user ContextVars so tools work correctly
- MaxIterationsReached treated as success, not failure
- skip_next_recall uses GREATEST(trigger_at, now) for overdue recalls
- schedule_recall rejects past trigger times
- timezone offset double-adjustment fixed for aware datetimes
- _fire_recall registers _AgentRun for reconnect/replay support
- session_sync race with stream_start fixed

Frontend improvements:
- Recall banner moved to ChatHeader with live Cancel/Skip buttons
- Recall messages styled with is_recall flag and badge
- Real-time recall updates via WebSocket (recall_update events)
- Recall filter moved to sessions-header as toggle button
- Session list shows clock icon for sessions with pending recall
- Empty state messages for empty/filtered session lists
- Fixed missing api import in ChatHeader.vue

Tests:
- Updated scheduler_loop tests for _busy_sessions dict change

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor: move tool helper modules into _internal subpackage ...

Moves non-tool infrastructure out of navi/tools/ root so that only
actual tool classes live there:

  base.py           → _internal/base.py
  loader.py         → _internal/loader.py
  middleware.py     → _internal/middleware.py
  logging_middleware.py → _internal/logging_middleware.py
  _time_parser.py   → _internal/time_parser.py

All imports updated across core/, api/, mcp/, tools/, and tests/.
No proxy files remain in navi/tools/ root.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: wire test_mcp_tool into MCP manager startup injection ...

The startup loop in main.py assigns _mcp_manager only to tools that
have the attribute, but test_mcp_tool never declared it and mcp_status
used _manager instead of _mcp_manager — so both received None forever.

Changes:
- test_mcp_tool: add __init__(mcp_manager) with _mcp_manager, fallback
  to module-level import if startup wiring skipped
- mcp_status: rename _manager → _mcp_manager, same fallback
- registry.py: register create_mcp_server and test_mcp_tool builtins
- main.py: include test_mcp_tool in startup wiring loop
- client.py: add 5s timeout to disconnect cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: filesystem empty path validation and terminal timeout defaults ...

filesystem:
- Reject empty string paths in _check_path to prevent writing to cwd
- Clear error message when 'write' action uses 'destination' instead of 'path'

terminal:
- Reduce default timeout from 300s to 20s
- Clamp user-provided timeout between 1 and 300s
- Update parameter description to mention max and common use-cases

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: add create_mcp_server and test_mcp_tool builtins ...

create_mcp_server:
- Scaffolds mcp-servers/<name>/ directory with venv, pyproject.toml,
  app/__init__.py, and annotated mcp_server.py template
- Runs pip install -e . in isolated thread to avoid event-loop issues
- Validates template with python -m py_compile

test_mcp_tool:
- Execute a single MCP tool call for isolated testing
- Detailed diagnostics: not connected, empty response, timeout, crash
- Fallback to module-level _mcp_manager if startup wiring skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

refactor: rewrite tool_developer profile for MCP server development ...

Complete migration from old-style user tools (tools/*.py) to MCP servers.

Profile changes:
- Replace write_tool, delete_tool, test_tool with create_mcp_server,
  test_mcp_tool, mcp_status, reload_tools
- Update system_prompt.txt with 10-step MCP workflow
- Rewrite subagent_system_prompt.txt with canonical server format,
  absolute rules, smoke-test procedures
- Remove mcp_servers mapping from config (no longer bound to
  external servers)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: admin API endpoints for individual MCP server CRUD ...

Add granular REST endpoints for managing single MCP servers:
- GET  /admin/mcp/config/{server_name}
- PUT  /admin/mcp/config/{server_name}
- DELETE /admin/mcp/config/{server_name}

Bulk endpoints remain for backward compatibility.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

refactor: migrate MCP config to directory-based mcp_servers.d ...

Replaces monolithic mcp_servers.json with per-server files in
mcp_servers.d/<name>.json. Filename stem becomes the server name.

- load_mcp_servers() reads directory globs, auto-migrates legacy file
- save_mcp_servers() writes per-server files, cleans up stale ones
- Update docstrings in manager.py and context_builder.py

Auto-migration: existing servers (gnexus-book, navi-3d, navi-web) plus
new ones (project_health, time_toolkit) now live in mcp_servers.d/.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add session search across messages with backend + frontend ...

Backend:
- GET /sessions now accepts `search` query param
- _session_summary computes match_indices and match_preview from messages
- pg_session_store: messages ILIKE added to search_list and count_all
- In-memory store: search_list also filters by message content

Frontend:
- AppSidebar: search toggle icon + debounced input, Ctrl+K shortcut
- SessionItem: highlight matching text, show match_preview from search
- SessionList: pass searchQuery to SessionItem
- SessionsStore: searchQuery/searchActive state, setSearch/clearSearch
- API layer: getSessions accepts search param
- MessageList: scroll to target message + brightness flash animation
- ChatStore: loadSession accepts targetMessageIndex, scrollToMessageIndex ref
- CSS: msg-flash keyframe animation in app.scss
- Tests updated for new getSessions signature

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Persist uploaded files in messages, live file tree updates, and UI polish ...

Backend:
- Add `files` field to `Message` model so uploaded file metadata survives page refresh
- Pass `files` through websocket handler → `agent.run_stream` / `agent.run`
- `list_tools`: make `profile_id` required; return error instead of all-tools fallback

Webclient:
- Call `fetchFiles()` after successful file upload for immediate Files tab update
- Live refresh file tree on filesystem (write/edit/append/mkdir/rm/cp/mv), terminal, and code_exec tool calls
- Add manual refresh button (desktop) and pull-to-refresh (mobile) to Files tab
- Fix live link updates: move regex creation inside per-message loop to avoid lastIndex state leak
- Restore full profile name text next to avatar in ChatHeader; hide avatar in header
- Fix mobile ArtifactsPanel: collapse tab text labels so close button fits with 3 tabs

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Defensive image cleanup in Ollama backend to prevent 'unknown format' errors ...

- _clean_base64_image() strips data URI prefix and validates non-empty result
- _to_ollama_messages() filters out invalid/empty images before sending to Ollama
- Prevents 500 errors when models receive malformed or unsupported image data

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add session file directory listing endpoint and Files tab in ArtifactsPanel ...

Backend:
- New GET /sessions/{session_id}/files endpoint (recursive, max depth 10)
- Includes hidden files, returns {path, size, is_dir, modified_at}
- Path traversal protection via resolve() + relative_to(base)
- list_session_files() helper in session_files.py

Webclient:
- Third "Files" tab in ArtifactsPanel alongside Artifacts and Links
- Tree display with depth-based indentation
- File type icons by extension (image, video, code, etc.)
- Download and inline-open actions per file
- Fetch on session load/reload via chatStore.fetchFiles()

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add MCP admin endpoints for config, reconnect, status, test, and profile group management ...

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add profile_id support to list_tools ...

ListToolsTool now accepts an optional `profile_id` parameter. When
provided, it returns only the tools enabled for that specific profile
(enabled_tools + user tools + resolved MCP tools). This helps the agent
preview what a profile can do before switching.

- Updated ListToolsTool.execute() with profile-aware filtering
- Patched _profile_registry in build_default_registries
- Patched _mcp_manager in main.py startup loop

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix gnexus-book MCP instructions and add old-format fallback in tool executor ...

- Updated mcp_servers.json gnexus-book instructions to use mcp:gnexus-book:
  prefix instead of the old mcp_gnexus-book_ format.
- Added fallback in tool_executor.py for old underscore format
  (mcp_server_tool → mcp:server:tool) so transitional models still work.
- Added unit test for the old-format fallback.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Generate session titles in the user's language ...

Adds "Respond in the same language as the user's messages" to the
session title generator system prompt so titles match the conversation
language instead of defaulting to English.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix profile MCP server groups to use group names instead of tool names ...

Profile configs were listing individual tool names (web_view, http_request)
in mcp_servers.navi-web instead of group names (browse, request). This caused
resolve_group() to return empty results, so MCP tools never appeared in the
profile's resolved tool list.

Also fixes modeler_3d stale tool references in system_prompt.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix agent.py _tool_list to use colon-delimited MCP names ...

The main registry and API routes were migrated, but Agent._tool_list
still built names with the old underscore format. This meant MCP tools
were silently dropped from the tool list passed to the LLM, causing
"tool not found" when the model correctly called mcp:server:tool names.

Also updates stale docstring in McpTools.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Rename MCP tools to mcp:server:tool format and restore human-readable names ...

- Core naming: mcp_server_tool → mcp:server:tool (colon-delimited)
- navi-web tools: search/view/request → web_search/web_view/http_request
- navi-3d tools: compile_scad/render_stl/lint_scad (unchanged names)
- Updated all profile configs, system prompts, docs, manuals, tests
- Added new lint_scad.md manual
- Fixed modeler_3d prompt stale references (scad_lint, model_3d, render_3d)
- All 240 tests pass

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Clean up remaining web_search / web_view / http_request references ...

- Deleted stale source files: navi/tools/web_search.py, web_view.py,
  http_request.py (no longer imported or registered).
- Updated system_prompt.txt in all profiles (secretary, server_admin,
  discuss, modeler_3d subagent) to reference mcp_navi-web_* tool names.
- Updated key_tools in developer, secretary, server_admin config.json.
- Updated docs (tools.md, profiles.md, api.md) to reflect MCP migration.

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Rename navi-web MCP tools and compact search output ...

- web_search → search, web_view → view, http_request → request
  This removes the redundant 'web' prefix from tool names in the
  navi-web MCP server, making them cleaner: mcp_navi-web_search etc.

- Compact search output: output string now contains only
  [Source] + [N] Title — URL, without duplicating the body snippet.
  Full results with body/description remain in metadata for
  programmatic use.

- Updated mcp_servers.json groups and all profile configs.

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate web tools (web_search, web_view, http_request) to navi-web MCP server ...

New MCP server: mcp-servers/navi-web/
- app/search.py    — SearXNG primary, DDG fallback, Brave tertiary
- app/browse.py    — Playwright headless browser page extraction
- app/request.py   — httpx raw HTTP requests
- app/mcp_server.py — FastMCP entry point with stdio transport
- pyproject.toml   — deps: mcp, httpx, ddgs, playwright, pydantic

Changes:
- Added "navi-web" block to mcp_servers.json (stdio transport)
- Removed web_search, web_view, http_request from built-in registry
- Removed them from navi/tools/__init__.py
- Removed them from enabled_tools / subagent_tools in all profiles
- Added "navi-web": ["search", "browse", "request"] to mcp_servers
  in all profiles (developer, secretary, server_admin, tool_developer,
  modeler_3d, discuss)

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate 3D modeling tools to standalone MCP server navi-3d ...

Phase 0 — Fix MCP error propagation:
- McpClient.call_tool now returns (output, is_error) tuple
- McpTool.execute sets success=False when MCP reports isError=True
- Fixes silent failures where MCP errors looked like success

Phase 1 — Create mcp-server-navi-3d:
- New standalone MCP server in mcp-servers/navi-3d/
- Tools: compile_scad, render_stl, lint_scad
- Session-scoped path resolution via SESSION_FILES_DIR env
- Anti-escape validation for security
- Includes tests/unit/test_scad_analyze.py

Phase 2 — Remove from Navi core:
- Deleted navi/tools/{model_3d,render_3d,scad_lint}.py
- Removed from registry.py builtins and navi/tools/__init__.py
- Updated pyproject.toml testpaths to exclude mcp-servers/

Phase 3 — Wire into Navi config:
- Added navi-3d block to mcp_servers.json (SSE on :8002)
- Updated modeler_3d profile: mcp_servers + updated system_prompt
- All old tool names replaced with mcp_navi-3d_* equivalents

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Increase max_iterations across all profiles ...

- developer: 45 → 100
- discuss: 12 → 30
- modeler_3d: 50 → 70
- secretary: 35 → 70
- server_admin: 40 → 100
- tool_developer: 35 → 100

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>