Автономность мелких моделей: OUTPUT DISCIPLINE, milestone-todo, перехват финала хода ...

Два последовательных захода над одной проблемой — мелкие модели (12–30B) в navi_code
теряют автономность на трудных шагах: «остановился поболтать» вместо действия и
слишком большое расстояние между пунктами плана.

Заход 1 (З1–З3):
- З1: OUTPUT DISCIPLINE в системном промпте navi_code — «act, don't announce»,
  без few-shot антипримеров. Контракт хода: ответ без tool_calls = конец хода, поэтому
  объявление намерения текстом убивает автономность; правила заставляют вызывать
  инструмент в том же ходе.
- З2: плоский todo + метка группы milestone + декомпозиция. _parse_plan_steps
  возвращает list[tuple[milestone, text]]; milestone — метка группировки (не сущность,
  без статуса), «done» вычисляется при рендеринге; подшаги = больше плоских шагов
  (без вложенности). TUI side-panel группирует по milestone (плоский фолбэк при пустом
  milestone). Plan depth: max 15→20 + правило декомпозиции.
- З3: adaptive re-plan «длинный шаг» — nudge «разбей шаг» при in_progress ≥ порога
  итераций без смены todo (порог 4, раньше общего anti-stall warning на 8).

Заход 2 (шаги 1–3, после cloud-теста 31b vs 12b):
Корневая структурная причина: весь спасательный механизм (anti-stall warning с явным
предложением reflect, adaptive re-plan) живёт только внутри tool-цикла — nudge
инжектируется в pre_turn *следующей* итерации, которой при «остановился поболтать»
нет (ход закрылся по return до post_turn). 31b застревала через «продолжаю
tool-итерации» → дожала до warning → спаслась; 12b — через «замолчала текстом» → мимо
всех nudge.

- Шаг 1: перехват финала хода. Если модель выдала bare-text, но в todo есть открытые
  шаги (pending/in_progress) и лимит не исчерпан — НЕ эмитить StreamEnd, а сохранить
  ассистентский текст в session.context, поставить системный nudge и continue
  (без StreamEnd, без workers — консистентно с multi-iteration tool-турами). Счётчик
  final_interceptions на AgentTurnContext, лимит final_intercept_limit (default 2),
  эскалация жёсткости (мягкий → «second stop»). has_open_steps в todo.py: пустой
  todo → False (защита casual-сообщений), failed/skipped терминальны. Профильные
  флаги final_intercept_enabled/limit в base.py + loader + admin.
- Шаг 2: жёсткий reflect-триггер в промпте — «~3 tool attempts on the same step
  without progress → call reflect IN THIS TURN (tool call, not reasoning aloud)».
- Шаг 3: открыть replan для застревания — «call replan when reflect showed the whole
  approach is dead (not one failed step, but the approach itself)».

Тесты: 874 passed, 1 skipped. Новые — has_open_steps (5), final intercept (5),
milestone-группировка, adaptive long-step nudge, парсер шагов с milestone-маркером.

cloud: navi_code model → gemma4:31b-cloud для тестирования догадки (31b признала
застревание, 12b — нет); .env cloud-host уже gitignored.

Co-Authored-By: Claude <noreply@anthropic.com>

permissions: remove broken client-side gate; add backend design doc ...

The terminal-TUI permission gate was a race: _execute_tools_with_sink
emits ToolStarted and starts the tool on the backend immediately, so by
the time the client dialog appeared the destructive action had already
run -- 'deny' could only stop the session, not un-execute the tool. It
also existed only in the terminal TUI (webclient/android had nothing),
and the system-prompt 'Strict Confirmation' nudge was an unreliable
duplicate.

Remove the broken machinery for a clean slate and plan the real
replacement from zero:

- delete clients/terminal/tui/permissions.py, screens/permission_dialog.py
  and their tests
- strip tui_app.py (engine, _deny_tool, _show_permission_dialog,
  _confirm_shell_command, _stop_session_worker, on_permission_request,
  tool_started gate) and the PermissionRequest TUI event; !cmd now runs
  directly (user-typed, no agent gate)
- drop the 'Strict Confirmation' prompt rule from navi_code/developer/
  tool_developer
- add docs/permissions.md: authoritative backend gate design (engine +
  registry + event + endpoint + agent gate + postgres policy + global
  rules config, sub-agents gated) -- design only, not yet implemented
- update docs/index.md, navi_code_cli.md, profiles.md, testing.md

Until Phase 1 lands there is no destructive-action confirmation -- a
conscious period without the false security of the old race-gate.

replan: integrated mid-task re-planning tool ...

Add a replan tool that re-runs the planner over the live session context
+ todo + scratchpad when the plan's structure is stale due to discoveries
(NOT failed steps -- that stays [Adaptive re-plan]). Integrated approach:
PlanningEngine.run gains is_replan/replan_context (suppresses DIRECT
shortcut and observe-skip, frames Phase 1 as a revision); ReplanRunner
packs reason/goal/todo/findings/errors and captures PlanReady; the tool
is exposed to navi_code/developer/tool_developer via a
current_replan_runner ContextVar set per-iteration in run_stream (correct
after switch_profile). New plan replaces the todo. Lazy events import
breaks the navi.tools -> navi.core -> navi.tools cycle.

profiles: port context-org/planning instructions to developer & tool_developer ...

Port the context-organization and planning-machinery framing accumulated
in navi_code to the developer and tool_developer profiles, adapted per
profile:

- Working state & memory (todo/scratchpad/context_transfer/schedule_recall/
  reflect/memory) replaces the stale "Context drift recovery" section.
- System signals subsection names the runtime-injected messages each profile
  actually receives: [Goal anchor], [Anti-stall warning], [Adaptive re-plan]
  (adaptive_replan is on for both), [Iteration N/M]. [Scope boundary] is
  omitted — scope_boundary_enabled is off on these profiles.
- Reading & searching, Editing policy (tool_developer), Safety Rules, Git
  discipline, and Project environments (isolated venv) added.
- developer: Workflow rewritten planner-aware with an observe carve-out
  (observe_skips_plan now on); Project knowledge replaced by docs-first
  Documentation; sub-agent briefing gains context_transfer + restricted-toolset
  bullets.
- tool_developer: keeps its MCP-specific 10-step workflow and prerequisites;
  sub-agent briefing gains context_transfer + restricted-toolset bullets
  (sub-agent cannot reload_tools/test_mcp_tool/mcp_status — run those inline).
- config: observe_skips_plan_enabled=true on both, so observe requests skip
  phase3 (no plan/todo for read/explain/inspect) — enables the Workflow
  observe carve-out and saves an LLM call on info requests.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: prefer isolated project envs and bootstrap NAVI.md when absent ...

Add a "Project environments" section steering the agent toward the
project's existing isolated env (venv/uv/node_modules/target) — use it,
don't duplicate or bypass it — and only create a project-local one when
deps are needed and none exists; never install system-wide (system-wide
changes still require explicit confirmation). Also nudge the agent to
create NAVI.md when it's absent after real orientation work on a
non-trivial task, seeding the pointer structure so the next session
starts oriented.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: enable planning phase 2 (structured review) ...

phase1 already emits REFLECT: yes|no, but with phase2 off the flag was a
dead-end. Enabling phase2 runs a Critic/Pragmatist/Detailer review pass
before phase3 whenever phase1 flags the task as complex (REFLECT: yes),
feeding "Plan Adjustments" into the execution plan. Skipped for simple
tasks and subagents (gated on `needs_reflect and not is_subagent`), so
the extra LLM call only lands on complex parent-agent work.

Resolves F8 machinery-gap G.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: frame planning machinery in system prompt (F8) ...

Workflow "Plan" no longer duplicates the planner (phase1/3 already build a
structured plan and auto-populate todo); adds an observe/act carve-out so
observe requests don't get pushed to create a todo. New "System signals
you'll see" subsection names the runtime-injected messages ([Goal anchor],
[Scope boundary], [Anti-stall warning], [Iteration N/M]) so the model
recognises them as machinery and responds correctly — and corrects the
factual error that the goal anchor reads scratchpad `goal` (it reads the
original request + live todo). Closing paragraph warns that thinking isn't
re-injected and the plan's per-step executor assignments are lost to
compression, so conclusions/assignments should go in scratchpad.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: give the sub-agent operational wisdom (editing/reading/todo) ...

By default (inherit_system_prompt=False) the sub-agent got only the 17-line
subagent_system_prompt — none of the parent's editing/reading/context
discipline. So a sub-agent with a clean context would still burn it: pulling
whole files into read, over-using smart_edit, not tracking steps. The clean
context is the sub-agent's main advantage — keep it clean.

Expand subagent_system_prompt.txt with a compact operational core:
- Editing: prefer edit/edit_lines (deterministic); smart_edit as last resort
  (extra LLM call, reads the whole file) — the same policy the parent now has.
- Reading: info before read, offset/limit to the region, grep/find/query to
  locate — don't read a file just to search it.
- Track steps with todo and record findings in scratchpad (durable within the
  run; context can be compressed).

inherit_system_prompt stays False: the full parent prompt carries orchestration
and spawn_agent sections that are irrelevant to a sub-agent and references
tools it lacks. The sub-agent keeps its own focused, self-contained prompt.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: brief sub-agent on its restricted toolset + context_transfer ...

The sub-agent briefing section told the model to "give the sub-agent
everything it needs" but never said which tools the sub-agent lacks or how
context actually reaches it — so briefings could ask the sub-agent to use
memory, spawn further agents, or switch profiles, all of which it can't.

- context_transfer: write the context the sub-agent needs (files, snippets,
  how to verify) into the scratchpad `context_transfer` section before
  spawning — it is injected automatically; the sub-agent does not inherit
  short-term memory or conversation history.
- Restricted toolset: the sub-agent has todo/scratchpad/reflect/filesystem/
  code_exec/terminal/list_tools but NOT memory/switch_profile/spawn_agent/
  schedule_recall/manage_recall — brief it to use only what it has (e.g.
  record findings in scratchpad, not memory).

The profile list is not duplicated — spawn_agent's own description carries it.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: leverage context-org/planning tools in system prompt ...

Wire the KV-backed context/planning tools into the profile so the agent
uses them instead of relying on lossy conversation memory under compression.

- Workflow Plan: create a todo for non-trivial tasks.
- Workflow Test & verify: record the verification in the todo `validation`
  field when marking done (structural form of "never claim done without
  verification").
- Replace "Context drift recovery" with "Working state & memory":
  - todo: plan + verification tracking.
  - scratchpad: durable working memory across compression (sections
    goal/findings/errors/artifacts); read before final report.
  - Sub-agent handoff: write context to the scratchpad `context_transfer`
    section before spawn_agent — it is injected into the sub-agent
    automatically (the sub-agent does not inherit short-term memory).
  - schedule_recall: continue after the iteration limit, offload heavy
    work headlessly, poll builds/logs, chain multi-phase work.
  - reflect: selectively, before complex plans or when stuck (3 LLM calls).
  - memory: global cross-project facts, not a scratchpad/docs substitute.
  - Drift recovery folded into a closing paragraph aligned with the
    goal_anchoring machinery.

TUI visualization for schedule_recall is a follow-up task.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: fix review findings 1-5 in system prompt ...

- spawn_agent default profile: correct the false "omit profile_id to use
  this developer profile" — omitting profile_id inherits navi_code (the
  parent), not developer; set profile_id explicitly to pick another profile.
- Safety Rules: scope strict confirmation to destructive/irreversible
  operations (rm, delete, wholesale write overwrite, drop table, force-push);
  state explicitly that routine edit/edit_lines do not require confirmation.
- Workflow Understand: soften the hard "start with docs/index.md" to a soft
  "read notes/docs first (see NAVI.md and Documentation sections below)" so
  it no longer conflicts with the NAVI.md-first entry point.
- Documentation: add a branch for projects with no docs/ — propose creating
  one (with user approval) or record in NAVI.md that docs are absent/not
  needed.
- Execution environment: document terminal action="run" for one-off commands
  (tests, git status, lint, py_compile); persistent terminals only for
  long-running processes.

Findings 6 (subagent briefing toolset), 7 (subagent delegation wording),
8 (planning/context-org tools) deferred for later work.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: docs as living spec + NAVI.md hints file ...

Replace the Project knowledge section with two clearer sections:

- Documentation: docs/ is the project's living specification, not just
  human reference — keep it current with code, and for non-trivial changes
  align docs to the intended end state before implementing (docs-first).
- NAVI.md: a lightweight, pointer-shaped hints file at project root (not a
  source of truth) that tells the agent where to look; capped at ~150 lines
  with a defined structure (Project / Commands / Where to start / Docs index
  / Gotchas / Open decisions), kept distinct from docs/ (spec) and memory
  (global cross-project facts).

Instruction-only: the agent reads NAVI.md via filesystem; no auto-inject.
Applies to navi_code only.

Co-Authored-By: Claude <noreply@anthropic.com>

filesystem: make edit/edit_lines the default, smart_edit a last-resort fallback ...

Agent overused smart_edit (whole-file LLM call, weak context). Rewrite the
FilesystemTool description decision tree so edit (exact text) is the default and
edit_lines (by line numbers) is the deterministic option; smart_edit is reserved
for genuinely semantic changes that cannot be expressed as exact text or line
numbers. Strengthen the old_not_unique hint to steer back to edit/edit_lines
before smart_edit. Add an "Editing policy" section to the navi_code and
developer profile prompts reinforcing the same priority.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: bounded autonomy — scope boundary + observe-vs-act ...

navi_code had unwanted "free flight": an observe request ("look at a
directory") triggered the full Phase 3 plan with milestones + auto-todo,
and goal_anchoring then drove the agent to finish those steps, climbing
into sibling projects and executing milestone docs it found.

Two toggleable, default-off profile flags (on for navi_code):

- scope_boundary_enabled: injects a standing system message keeping the
  agent within the literally requested scope; forbids acting on
  discovered TODO/roadmap/milestone docs (report only).
- observe_skips_plan_enabled: Phase 1 classifies MODE: observe|act; an
  observe request skips Phase 2/3 — no multi-step plan, no auto-todo, no
  "execute step by step" prompt. The agent just gathers info and answers.
  Independent of force_plan (observe on the first message still skips).

Free flight stays reproducible by flipping both flags off.

Co-Authored-By: Claude <noreply@anthropic.com>

profiles: add gemma4:12b-it-qat-128k as top-priority model ...

New local QAT model added at the head of the model priority list across
all profiles, so it is preferred first with cloud fallback.

compressor: structured summaries, profile-aware compression, adaptive keep_recent ...

- Replace free-form summary with strict Markdown template (Goal, Active Files,
  Decisions, Completed Work, Pending Work/Todo, Errors, Key Values).
- Keep filesystem/code_exec/terminal tool results and messages with
  is_compression_critical=True verbatim during compression instead of 300-char truncation.
- Make compression profile-aware: AgentProfile gains compression_keep_recent,
  compression_max_tokens, compression_prompt_file. navi_code uses dedicated
  compression prompt and larger keep_recent/max_tokens.
- Adaptive partition_messages(): important turns (user corrections, errors,
  critical tools) survive longer; filler/social turns compress sooner.
- Increase default context_summary_max_tokens from 3000 to 4000.
- Propagate active profile changes to ContextCompressor and SubAgentRunner.

Co-Authored-By: Claude <noreply@anthropic.com>

Fix MCP names and profile API format consistency ...

- Replace stale mcp__navi_web__* / mcp__navi_3d__* names with canonical
  mcp__navi-web__* / mcp__navi-3d__* across prompts and key_tools.
- Update /agents/profiles and /admin/profiles endpoints to expose
  tools.agent / tools.subagent instead of deprecated enabled_tools fields.
- Update docs/mechanics.md to reference the new tools structure.
- Archive stale docs/visual.html.

Co-Authored-By: Claude <noreply@anthropic.com>

Update Navi Code profile: config, prompts, docs, and env example ...

- Sync navi_code profile tools with current codebase (memory, schedule_recall, manage_recall, switch_profile, list_profiles)

- Remove image_view and MCP tools from navi_code; keep local terminal focus

- Update system prompt and persona for terminal-first coding assistant

- Update docs/navi_code.md, docs/profiles.md, docs/config.md for new tool set

- Refresh .env.navi_code.example with current config variables

- Remove stale CLI prototype directories (cli/, navi_code_cli/)

Docs: actualize navi_code tool lists and key_tools ...

- docs/navi_code.md and docs/profiles.md now correctly list native tools,
  MCP navi-web tools, and excluded tools.
- image_view is documented as enabled for the agent (loads and describes
  images for the model, not shown to the user).
- web_search/http_request are documented as MCP navi-web tools, not native
  disabled tools.
- navi/profiles/navi_code/config.json: updated full_description.key_tools to
  match actual resolved tool names.

Tests: 459 passed, 1 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: Phase 1 — terminal-first profile, default profile mechanism, env/persona, tests ...

- Add navi_code profile (terminal-first coding assistant)
- Add NAVI_DEFAULT_PROFILE_ID setting and POST /sessions default profile fallback
- Add persona_navi_code.txt and .env.navi_code.example
- Add docs/plan_navi_code.md phased plan
- Update tests to verify default-profile selection in no-auth mode

Co-Authored-By: Claude <noreply@anthropic.com>

Clarify terminal description param and profile prompts ...

- terminal.py: stronger description for the 'description' field so
  the model knows it is REQUIRED for action=open and what to write
- server_admin and developer system prompts: add a short section
  explaining persistent terminal usage (open/close/list/status/send_input)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enable gnexus-creds MCP in profiles that already use gnexus-book ...

- discuss: add gnexus-creds read group to agent tools
- server_admin: add gnexus-creds read+write groups to agent and subagent tools

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Prioritise gemma4 and add MCP tools to subagent scopes ...

- modeler_3d: gemma4:31b-cloud first (vision-capable), drop glm-5.1
- modeler_3d subagent: add navi-3d + navi-web MCP tools
- Adjust subagent tool scopes across profiles for explicit MCP access

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor profile tool config to explicit agent/subagent structure ...

Replaces the confusing mix of enabled_tools + mcp_servers + subagent_tools
with a single explicit structure:

    tools: {
      agent:   {native: [...], mcp: {server: [groups]}},
      subagent:{native: [...], mcp: {server: [groups]}}
    }

Why:
- Old fields mixed native and MCP names (mcp__server__tool) in one list,
  making it impossible to tell at a glance what a subagent actually gets.
- subagent_runner.py had 25 lines of runtime MCP filtering logic that
  was hard to follow and error-prone.

Changes:
- AgentProfile: add ToolConfig / ToolScopeConfig pydantic models,
  keep old fields (enabled_tools, mcp_servers, subagent_tools) for
  auto-migration via _migrate_tools validator.
- loader.py: read new "tools" key, auto-migrate legacy configs.
- agent.py: _tool_list now accepts ToolScopeConfig.
- subagent_runner.py: simplified — profile.get_subagent_tools() returns
  the exact scope, no runtime filtering needed.
- context_builder.py, list_tools.py, spawn_agent.py: updated to use
  profile.get_agent_tools() / get_subagent_tools().
- All 6 profile config.json files migrated to new schema.
- Secretary subagent now explicitly gets navi-web MCP tools for web search.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate MCP tool naming from mcp:server:tool to mcp__server__tool ...

The colon separator (mcp:server:tool) confuses many LLMs during
tool-calling because colons appear in schemas and URLs. Switch to
double-underscore separator (mcp__server__tool) for robust parsing.

Key changes:
- navi/mcp/tools.py: add build_mcp_name(), parse_mcp_name(), is_mcp_tool()
- navi/core/tool_executor.py: update _resolve_tool() with new helpers
  and legacy colon fallback for old sessions
- navi/core/tool_utils.py, subagent_runner.py: use build_mcp_name()
- navi/api/routes/{admin,agents}.py: prefix via build_mcp_name()
- navi/tools/{list_tools,reload_tools}.py: migrated
- All profile configs + system_prompt.txt: replace mcp: with mcp__
- manuals/{model_3d,lint_scad,render_3d,spawn_agent}.md: updated
- mcp_servers.d/gnexus-book.json: instructions updated
- docs/{api,profiles,tools,mechanics,visual.html}: updated
- tests: test_tool_executor.py and test_mcp.py aligned

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

cleanup: remove deprecated tools and orphaned memory tools ...

Removed (no profile used them, no cross-dependencies):
- write_tool.py, delete_tool.py, test_tool.py
- memory_save.py, memory_search.py, memory_forget.py

Updated:
- navi/core/registry.py — removed imports and registrations
- navi/tools/__init__.py — removed imports
- docs/tools.md — removed references, updated self-extension section to MCP
- navi/tools/tool_manual.py — updated example to create_mcp_server

Profile fixes:
- developer: +tool_manual, +ssh_exec
- discuss: +list_profiles
- tool_developer: +mcp_servers navi-web

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

refactor: rewrite tool_developer profile for MCP server development ...

Complete migration from old-style user tools (tools/*.py) to MCP servers.

Profile changes:
- Replace write_tool, delete_tool, test_tool with create_mcp_server,
  test_mcp_tool, mcp_status, reload_tools
- Update system_prompt.txt with 10-step MCP workflow
- Rewrite subagent_system_prompt.txt with canonical server format,
  absolute rules, smoke-test procedures
- Remove mcp_servers mapping from config (no longer bound to
  external servers)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix profile MCP server groups to use group names instead of tool names ...

Profile configs were listing individual tool names (web_view, http_request)
in mcp_servers.navi-web instead of group names (browse, request). This caused
resolve_group() to return empty results, so MCP tools never appeared in the
profile's resolved tool list.

Also fixes modeler_3d stale tool references in system_prompt.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>