navi_code: leverage context-org/planning tools in system prompt ...

Wire the KV-backed context/planning tools into the profile so the agent
uses them instead of relying on lossy conversation memory under compression.

- Workflow Plan: create a todo for non-trivial tasks.
- Workflow Test & verify: record the verification in the todo `validation`
  field when marking done (structural form of "never claim done without
  verification").
- Replace "Context drift recovery" with "Working state & memory":
  - todo: plan + verification tracking.
  - scratchpad: durable working memory across compression (sections
    goal/findings/errors/artifacts); read before final report.
  - Sub-agent handoff: write context to the scratchpad `context_transfer`
    section before spawn_agent — it is injected into the sub-agent
    automatically (the sub-agent does not inherit short-term memory).
  - schedule_recall: continue after the iteration limit, offload heavy
    work headlessly, poll builds/logs, chain multi-phase work.
  - reflect: selectively, before complex plans or when stuck (3 LLM calls).
  - memory: global cross-project facts, not a scratchpad/docs substitute.
  - Drift recovery folded into a closing paragraph aligned with the
    goal_anchoring machinery.

TUI visualization for schedule_recall is a follow-up task.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: fix review findings 1-5 in system prompt ...

- spawn_agent default profile: correct the false "omit profile_id to use
  this developer profile" — omitting profile_id inherits navi_code (the
  parent), not developer; set profile_id explicitly to pick another profile.
- Safety Rules: scope strict confirmation to destructive/irreversible
  operations (rm, delete, wholesale write overwrite, drop table, force-push);
  state explicitly that routine edit/edit_lines do not require confirmation.
- Workflow Understand: soften the hard "start with docs/index.md" to a soft
  "read notes/docs first (see NAVI.md and Documentation sections below)" so
  it no longer conflicts with the NAVI.md-first entry point.
- Documentation: add a branch for projects with no docs/ — propose creating
  one (with user approval) or record in NAVI.md that docs are absent/not
  needed.
- Execution environment: document terminal action="run" for one-off commands
  (tests, git status, lint, py_compile); persistent terminals only for
  long-running processes.

Findings 6 (subagent briefing toolset), 7 (subagent delegation wording),
8 (planning/context-org tools) deferred for later work.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: docs as living spec + NAVI.md hints file ...

Replace the Project knowledge section with two clearer sections:

- Documentation: docs/ is the project's living specification, not just
  human reference — keep it current with code, and for non-trivial changes
  align docs to the intended end state before implementing (docs-first).
- NAVI.md: a lightweight, pointer-shaped hints file at project root (not a
  source of truth) that tells the agent where to look; capped at ~150 lines
  with a defined structure (Project / Commands / Where to start / Docs index
  / Gotchas / Open decisions), kept distinct from docs/ (spec) and memory
  (global cross-project facts).

Instruction-only: the agent reads NAVI.md via filesystem; no auto-inject.
Applies to navi_code only.

Co-Authored-By: Claude <noreply@anthropic.com>

filesystem: make edit/edit_lines the default, smart_edit a last-resort fallback ...

Agent overused smart_edit (whole-file LLM call, weak context). Rewrite the
FilesystemTool description decision tree so edit (exact text) is the default and
edit_lines (by line numbers) is the deterministic option; smart_edit is reserved
for genuinely semantic changes that cannot be expressed as exact text or line
numbers. Strengthen the old_not_unique hint to steer back to edit/edit_lines
before smart_edit. Add an "Editing policy" section to the navi_code and
developer profile prompts reinforcing the same priority.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: bounded autonomy — scope boundary + observe-vs-act ...

navi_code had unwanted "free flight": an observe request ("look at a
directory") triggered the full Phase 3 plan with milestones + auto-todo,
and goal_anchoring then drove the agent to finish those steps, climbing
into sibling projects and executing milestone docs it found.

Two toggleable, default-off profile flags (on for navi_code):

- scope_boundary_enabled: injects a standing system message keeping the
  agent within the literally requested scope; forbids acting on
  discovered TODO/roadmap/milestone docs (report only).
- observe_skips_plan_enabled: Phase 1 classifies MODE: observe|act; an
  observe request skips Phase 2/3 — no multi-step plan, no auto-todo, no
  "execute step by step" prompt. The agent just gathers info and answers.
  Independent of force_plan (observe on the first message still skips).

Free flight stays reproducible by flipping both flags off.

Co-Authored-By: Claude <noreply@anthropic.com>

profiles: add gemma4:12b-it-qat-128k as top-priority model ...

New local QAT model added at the head of the model priority list across
all profiles, so it is preferred first with cloud fallback.

compressor: structured summaries, profile-aware compression, adaptive keep_recent ...

- Replace free-form summary with strict Markdown template (Goal, Active Files,
  Decisions, Completed Work, Pending Work/Todo, Errors, Key Values).
- Keep filesystem/code_exec/terminal tool results and messages with
  is_compression_critical=True verbatim during compression instead of 300-char truncation.
- Make compression profile-aware: AgentProfile gains compression_keep_recent,
  compression_max_tokens, compression_prompt_file. navi_code uses dedicated
  compression prompt and larger keep_recent/max_tokens.
- Adaptive partition_messages(): important turns (user corrections, errors,
  critical tools) survive longer; filler/social turns compress sooner.
- Increase default context_summary_max_tokens from 3000 to 4000.
- Propagate active profile changes to ContextCompressor and SubAgentRunner.

Co-Authored-By: Claude <noreply@anthropic.com>

Fix MCP names and profile API format consistency ...

- Replace stale mcp__navi_web__* / mcp__navi_3d__* names with canonical
  mcp__navi-web__* / mcp__navi-3d__* across prompts and key_tools.
- Update /agents/profiles and /admin/profiles endpoints to expose
  tools.agent / tools.subagent instead of deprecated enabled_tools fields.
- Update docs/mechanics.md to reference the new tools structure.
- Archive stale docs/visual.html.

Co-Authored-By: Claude <noreply@anthropic.com>

Update Navi Code profile: config, prompts, docs, and env example ...

- Sync navi_code profile tools with current codebase (memory, schedule_recall, manage_recall, switch_profile, list_profiles)

- Remove image_view and MCP tools from navi_code; keep local terminal focus

- Update system prompt and persona for terminal-first coding assistant

- Update docs/navi_code.md, docs/profiles.md, docs/config.md for new tool set

- Refresh .env.navi_code.example with current config variables

- Remove stale CLI prototype directories (cli/, navi_code_cli/)

Docs: actualize navi_code tool lists and key_tools ...

- docs/navi_code.md and docs/profiles.md now correctly list native tools,
  MCP navi-web tools, and excluded tools.
- image_view is documented as enabled for the agent (loads and describes
  images for the model, not shown to the user).
- web_search/http_request are documented as MCP navi-web tools, not native
  disabled tools.
- navi/profiles/navi_code/config.json: updated full_description.key_tools to
  match actual resolved tool names.

Tests: 459 passed, 1 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: Phase 1 — terminal-first profile, default profile mechanism, env/persona, tests ...

- Add navi_code profile (terminal-first coding assistant)
- Add NAVI_DEFAULT_PROFILE_ID setting and POST /sessions default profile fallback
- Add persona_navi_code.txt and .env.navi_code.example
- Add docs/plan_navi_code.md phased plan
- Update tests to verify default-profile selection in no-auth mode

Co-Authored-By: Claude <noreply@anthropic.com>

Clarify terminal description param and profile prompts ...

- terminal.py: stronger description for the 'description' field so
  the model knows it is REQUIRED for action=open and what to write
- server_admin and developer system prompts: add a short section
  explaining persistent terminal usage (open/close/list/status/send_input)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enable gnexus-creds MCP in profiles that already use gnexus-book ...

- discuss: add gnexus-creds read group to agent tools
- server_admin: add gnexus-creds read+write groups to agent and subagent tools

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Prioritise gemma4 and add MCP tools to subagent scopes ...

- modeler_3d: gemma4:31b-cloud first (vision-capable), drop glm-5.1
- modeler_3d subagent: add navi-3d + navi-web MCP tools
- Adjust subagent tool scopes across profiles for explicit MCP access

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor profile tool config to explicit agent/subagent structure ...

Replaces the confusing mix of enabled_tools + mcp_servers + subagent_tools
with a single explicit structure:

    tools: {
      agent:   {native: [...], mcp: {server: [groups]}},
      subagent:{native: [...], mcp: {server: [groups]}}
    }

Why:
- Old fields mixed native and MCP names (mcp__server__tool) in one list,
  making it impossible to tell at a glance what a subagent actually gets.
- subagent_runner.py had 25 lines of runtime MCP filtering logic that
  was hard to follow and error-prone.

Changes:
- AgentProfile: add ToolConfig / ToolScopeConfig pydantic models,
  keep old fields (enabled_tools, mcp_servers, subagent_tools) for
  auto-migration via _migrate_tools validator.
- loader.py: read new "tools" key, auto-migrate legacy configs.
- agent.py: _tool_list now accepts ToolScopeConfig.
- subagent_runner.py: simplified — profile.get_subagent_tools() returns
  the exact scope, no runtime filtering needed.
- context_builder.py, list_tools.py, spawn_agent.py: updated to use
  profile.get_agent_tools() / get_subagent_tools().
- All 6 profile config.json files migrated to new schema.
- Secretary subagent now explicitly gets navi-web MCP tools for web search.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate MCP tool naming from mcp:server:tool to mcp__server__tool ...

The colon separator (mcp:server:tool) confuses many LLMs during
tool-calling because colons appear in schemas and URLs. Switch to
double-underscore separator (mcp__server__tool) for robust parsing.

Key changes:
- navi/mcp/tools.py: add build_mcp_name(), parse_mcp_name(), is_mcp_tool()
- navi/core/tool_executor.py: update _resolve_tool() with new helpers
  and legacy colon fallback for old sessions
- navi/core/tool_utils.py, subagent_runner.py: use build_mcp_name()
- navi/api/routes/{admin,agents}.py: prefix via build_mcp_name()
- navi/tools/{list_tools,reload_tools}.py: migrated
- All profile configs + system_prompt.txt: replace mcp: with mcp__
- manuals/{model_3d,lint_scad,render_3d,spawn_agent}.md: updated
- mcp_servers.d/gnexus-book.json: instructions updated
- docs/{api,profiles,tools,mechanics,visual.html}: updated
- tests: test_tool_executor.py and test_mcp.py aligned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

cleanup: remove deprecated tools and orphaned memory tools ...

Removed (no profile used them, no cross-dependencies):
- write_tool.py, delete_tool.py, test_tool.py
- memory_save.py, memory_search.py, memory_forget.py

Updated:
- navi/core/registry.py — removed imports and registrations
- navi/tools/__init__.py — removed imports
- docs/tools.md — removed references, updated self-extension section to MCP
- navi/tools/tool_manual.py — updated example to create_mcp_server

Profile fixes:
- developer: +tool_manual, +ssh_exec
- discuss: +list_profiles
- tool_developer: +mcp_servers navi-web

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

refactor: rewrite tool_developer profile for MCP server development ...

Complete migration from old-style user tools (tools/*.py) to MCP servers.

Profile changes:
- Replace write_tool, delete_tool, test_tool with create_mcp_server,
  test_mcp_tool, mcp_status, reload_tools
- Update system_prompt.txt with 10-step MCP workflow
- Rewrite subagent_system_prompt.txt with canonical server format,
  absolute rules, smoke-test procedures
- Remove mcp_servers mapping from config (no longer bound to
  external servers)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix profile MCP server groups to use group names instead of tool names ...

Profile configs were listing individual tool names (web_view, http_request)
in mcp_servers.navi-web instead of group names (browse, request). This caused
resolve_group() to return empty results, so MCP tools never appeared in the
profile's resolved tool list.

Also fixes modeler_3d stale tool references in system_prompt.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Rename MCP tools to mcp:server:tool format and restore human-readable names ...

- Core naming: mcp_server_tool → mcp:server:tool (colon-delimited)
- navi-web tools: search/view/request → web_search/web_view/http_request
- navi-3d tools: compile_scad/render_stl/lint_scad (unchanged names)
- Updated all profile configs, system prompts, docs, manuals, tests
- Added new lint_scad.md manual
- Fixed modeler_3d prompt stale references (scad_lint, model_3d, render_3d)
- All 240 tests pass

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Clean up remaining web_search / web_view / http_request references ...

- Deleted stale source files: navi/tools/web_search.py, web_view.py,
  http_request.py (no longer imported or registered).
- Updated system_prompt.txt in all profiles (secretary, server_admin,
  discuss, modeler_3d subagent) to reference mcp_navi-web_* tool names.
- Updated key_tools in developer, secretary, server_admin config.json.
- Updated docs (tools.md, profiles.md, api.md) to reflect MCP migration.

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Rename navi-web MCP tools and compact search output ...

- web_search → search, web_view → view, http_request → request
  This removes the redundant 'web' prefix from tool names in the
  navi-web MCP server, making them cleaner: mcp_navi-web_search etc.

- Compact search output: output string now contains only
  [Source] + [N] Title — URL, without duplicating the body snippet.
  Full results with body/description remain in metadata for
  programmatic use.

- Updated mcp_servers.json groups and all profile configs.

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate web tools (web_search, web_view, http_request) to navi-web MCP server ...

New MCP server: mcp-servers/navi-web/
- app/search.py    — SearXNG primary, DDG fallback, Brave tertiary
- app/browse.py    — Playwright headless browser page extraction
- app/request.py   — httpx raw HTTP requests
- app/mcp_server.py — FastMCP entry point with stdio transport
- pyproject.toml   — deps: mcp, httpx, ddgs, playwright, pydantic

Changes:
- Added "navi-web" block to mcp_servers.json (stdio transport)
- Removed web_search, web_view, http_request from built-in registry
- Removed them from navi/tools/__init__.py
- Removed them from enabled_tools / subagent_tools in all profiles
- Added "navi-web": ["search", "browse", "request"] to mcp_servers
  in all profiles (developer, secretary, server_admin, tool_developer,
  modeler_3d, discuss)

Tests: 240 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate 3D modeling tools to standalone MCP server navi-3d ...

Phase 0 — Fix MCP error propagation:
- McpClient.call_tool now returns (output, is_error) tuple
- McpTool.execute sets success=False when MCP reports isError=True
- Fixes silent failures where MCP errors looked like success

Phase 1 — Create mcp-server-navi-3d:
- New standalone MCP server in mcp-servers/navi-3d/
- Tools: compile_scad, render_stl, lint_scad
- Session-scoped path resolution via SESSION_FILES_DIR env
- Anti-escape validation for security
- Includes tests/unit/test_scad_analyze.py

Phase 2 — Remove from Navi core:
- Deleted navi/tools/{model_3d,render_3d,scad_lint}.py
- Removed from registry.py builtins and navi/tools/__init__.py
- Updated pyproject.toml testpaths to exclude mcp-servers/

Phase 3 — Wire into Navi config:
- Added navi-3d block to mcp_servers.json (SSE on :8002)
- Updated modeler_3d profile: mcp_servers + updated system_prompt
- All old tool names replaced with mcp_navi-3d_* equivalents

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Increase max_iterations across all profiles ...

- developer: 45 → 100
- discuss: 12 → 30
- modeler_3d: 50 → 70
- secretary: 35 → 70
- server_admin: 40 → 100
- tool_developer: 35 → 100

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Clarify knowledge persistence prompts

Add deterministic line-based file editing (edit_lines), rating UI fix, and session refresh ...

- filesystem.py: add edit_lines action (deterministic line ops via operations array)
           + numbered param for read (1-based line numbers in output)
           + clarify four editing modes in tool description
- chat.js: fix rating IDs for streaming messages (assign h_ ID on stream_end)
- SessionList.vue: mobile pull-to-refresh with PTR_THRESHOLD=80
- AppSidebar.vue: desktop refresh button next to Conversations header
- planning.py: knowledge source assessment in Phase 1
- debug panel: MCP servers tab + resolved tools per profile
- NAVI.md: reposition as neutral quick-reference

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor MCP integration: server groups in profiles ...

- mcp_servers.json: add 'groups' (read/write/admin) for gnexus-book
- AgentProfile: new 'mcp_servers' field (server_name -> group list)
- Profile loader: parse and persist 'mcp_servers' in config.json
- Agent._tool_list(): expands mcp_servers into concrete tool names
  via McpManager.resolve_group(), wildcard '*' supported
- /agents/profiles API: includes 'mcp_servers' in response
- Profiles no longer list individual mcp_ tools in 'enabled_tools'
- discuss: gnexus-book read group
- server_admin: gnexus-book read+write+admin groups

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>