navi_code: leverage context-org/planning tools in system prompt ...

Wire the KV-backed context/planning tools into the profile so the agent
uses them instead of relying on lossy conversation memory under compression.

- Workflow Plan: create a todo for non-trivial tasks.
- Workflow Test & verify: record the verification in the todo `validation`
  field when marking done (structural form of "never claim done without
  verification").
- Replace "Context drift recovery" with "Working state & memory":
  - todo: plan + verification tracking.
  - scratchpad: durable working memory across compression (sections
    goal/findings/errors/artifacts); read before final report.
  - Sub-agent handoff: write context to the scratchpad `context_transfer`
    section before spawn_agent — it is injected into the sub-agent
    automatically (the sub-agent does not inherit short-term memory).
  - schedule_recall: continue after the iteration limit, offload heavy
    work headlessly, poll builds/logs, chain multi-phase work.
  - reflect: selectively, before complex plans or when stuck (3 LLM calls).
  - memory: global cross-project facts, not a scratchpad/docs substitute.
  - Drift recovery folded into a closing paragraph aligned with the
    goal_anchoring machinery.

TUI visualization for schedule_recall is a follow-up task.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: fix review findings 1-5 in system prompt ...

- spawn_agent default profile: correct the false "omit profile_id to use
  this developer profile" — omitting profile_id inherits navi_code (the
  parent), not developer; set profile_id explicitly to pick another profile.
- Safety Rules: scope strict confirmation to destructive/irreversible
  operations (rm, delete, wholesale write overwrite, drop table, force-push);
  state explicitly that routine edit/edit_lines do not require confirmation.
- Workflow Understand: soften the hard "start with docs/index.md" to a soft
  "read notes/docs first (see NAVI.md and Documentation sections below)" so
  it no longer conflicts with the NAVI.md-first entry point.
- Documentation: add a branch for projects with no docs/ — propose creating
  one (with user approval) or record in NAVI.md that docs are absent/not
  needed.
- Execution environment: document terminal action="run" for one-off commands
  (tests, git status, lint, py_compile); persistent terminals only for
  long-running processes.

Findings 6 (subagent briefing toolset), 7 (subagent delegation wording),
8 (planning/context-org tools) deferred for later work.

Co-Authored-By: Claude <noreply@anthropic.com>

navi_code: docs as living spec + NAVI.md hints file ...

Replace the Project knowledge section with two clearer sections:

- Documentation: docs/ is the project's living specification, not just
  human reference — keep it current with code, and for non-trivial changes
  align docs to the intended end state before implementing (docs-first).
- NAVI.md: a lightweight, pointer-shaped hints file at project root (not a
  source of truth) that tells the agent where to look; capped at ~150 lines
  with a defined structure (Project / Commands / Where to start / Docs index
  / Gotchas / Open decisions), kept distinct from docs/ (spec) and memory
  (global cross-project facts).

Instruction-only: the agent reads NAVI.md via filesystem; no auto-inject.
Applies to navi_code only.

Co-Authored-By: Claude <noreply@anthropic.com>

filesystem edit: render red/green diff in TUI without changing model context ...

edit is now the primary editing method, yet its output was a dry one-line
status ("Edited …: replaced X B with Y B") while edit_lines/smart_edit showed a
highlighted unified diff. Carry the unified diff in ToolResult.metadata["diff"]
(kept out of the model-facing output, so the agent's context is unchanged) and
render it in the TUI: dim summary line plus highlighted diff (green +, red -,
dim @@). Falls back to plain text when no diff metadata is present.

Co-Authored-By: Claude <noreply@anthropic.com>

filesystem: make edit/edit_lines the default, smart_edit a last-resort fallback ...

Agent overused smart_edit (whole-file LLM call, weak context). Rewrite the
FilesystemTool description decision tree so edit (exact text) is the default and
edit_lines (by line numbers) is the deterministic option; smart_edit is reserved
for genuinely semantic changes that cannot be expressed as exact text or line
numbers. Strengthen the old_not_unique hint to steer back to edit/edit_lines
before smart_edit. Add an "Editing policy" section to the navi_code and
developer profile prompts reinforcing the same priority.

Co-Authored-By: Claude <noreply@anthropic.com>

compression: profile-aware worker + real-token baseline estimator ...

Item 2 — thread the active profile into CompressionWorker so
compress_context applies per-profile overrides (compression_keep_recent,
compression_max_tokens, compression_prompt_file). navi_code now compresses
with keep_recent=12 instead of the global 8.

Item 1 — estimate the next LLM call's context from the *real* prompt_tokens
of the previous call (bulk) plus a heuristic delta for messages appended
since, replacing the chars//3 estimate that undercounts code-heavy tool
output and fired midturn compression too late (Navi kept working until the
window was exhausted). Baseline is recorded after each stream and cleared
after compression; check_context_size and the midturn gate use it, with a
heuristic fallback when no baseline exists or the context shrank.

Co-Authored-By: Claude <noreply@anthropic.com>

compression: fix auto-compress no-op on few-huge-messages + honest status ...

Root cause: the compression gate (should_compress) measures tokens, but the
partition measures message/turn count, and CompressionStarted was emitted
before the attempt. For navi_code's "few very large messages" shape (one big
file read = 1 user + assistant + 1 huge tool result, 66k tokens in 3 messages)
the gate fired, the UI showed "compression", but partition returned
to_summarize=[] -> compress_context None -> nothing shrank. The agent kept
going until the window overflowed. It wasn't running "during" compression —
there was no compression, just a no-op the user mistook for one.

A. Per-message head/tail truncation in context_builder.build(): oversized
   tool/assistant messages (over context_message_token_budget, 0=num_ctx//6)
   are capped head+marker+tail in the LLM view only (model_copy — stored
   history and reloads are never affected). A single huge tool result can no
   longer alone blow the window; user/system messages are never truncated.

B. Token-budget hard-truncate fallback in compress_session: when partition
   no-ops but tokens exceed the threshold, drop oldest turns to num_ctx*0.5.
   _hard_truncate is now token-aware (was a fixed message-count floor that
   no-oped on <=6 messages even when huge). New would_compress() predicts
   compress_session's real outcome with no LLM call.

C. Honest CompressionStarted: _compression_events_midturn/_preturn emit it
   only after would_compress() confirms the partition (or token-budget
   fallback) can actually shrink the stored context — no more "compression"
   status with no ContextCompressed to follow.

Bonus: post-turn CompressionWorker now passes keep_recent_messages=
max(12, context_keep_recent*2), matching the midturn path, so a single long
autonomous turn compresses post-turn too (was always a no-op).

Tests (+14): would_compress agreement, token-budget fallback, token-aware
hard_truncate, build() truncation (preserves user, no mutation, head+tail),
agent no-CompressionStarted-when-nothing-to-compress, worker single-long-turn.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: force context compression via typed /compact control message ...

Forced /compact previously sent a chat message, which ran a full agent turn
that only produced summary text instead of running the real context compressor.
Worse, even when wired correctly, forced compact always reported "Nothing to
compact yet — the context is still small" regardless of context size: the typical
navi_code shape is a single long autonomous turn (1 user message + many tool
iterations = one turn), and partition_messages finds nothing to summarize when
turns <= keep_recent. The midturn auto-compress path already bypassed this via
keep_recent_messages (intra-turn split), but compact_stream passed
keep_recent_messages=None, so the fallback was disabled.

Changes:
- WS protocol: {"type":"compact"} control message (distinct from {"type":
  "message"}); rejected while an agent turn is active to avoid racing the agent.
- Agent.compact_stream: forced compression that bypasses the token threshold
  but still runs the real compressor; passes keep_recent_messages=max(12,
  context_keep_recent*2) so a single long turn compresses via intra-turn split
  (mirrors midturn auto-compress). Raises NothingToCompactError when context is
  genuinely too small.
- Orchestrator.run_compact + clear_run: broadcast agent events to subscribers,
  end with done marker, surface NothingToCompactError as an error event.
- Terminal client: ws_client.send accepts str|dict; CompactCommand enqueues
  {"type":"compact"}; TUI distinguishes forced compact (no stream_start) from
  in-turn auto-compress via the _streaming flag.
- Tests: compact_stream (incl. single-long-turn regression), WS handler
  dispatch/rejection, run_compact event/error broadcasting, ws_client send,
  compact command.

Co-Authored-By: Claude <noreply@anthropic.com>

tui: show the changed step's text in the todo update call card ...

The todo update call card (→ todo · #3 → done) now shows the text of the
step being changed, plus the validation when present. The LLM's update args
carry only the index, not the task text, so the text is read from the
current plan row before the tool runs and attached to ToolStarted.metadata
(client-rendering only, backward-compatible).

Backend:
- events.ToolStarted gains a metadata dict (mirrors ToolEvent) → to_wire.
- navi/tools/todo: step_text_for_update(index, ctx) resolves the step text
  via _sid/_uid (so sub-agent isolation holds), started_metadata_for_call
  wraps it for both emit sites.
- agent.py (parent) and subagent_runner.py (sub-agent) enrich ToolStarted
  via the shared helper before emitting.

Renderer:
- TodoStartedRenderer update card reads msg.metadata.step_text → shows the
  step text + validation; falls back to 'no validation' on history replay.
- Removed the step-text line from the result card (it now lives in the call
  card) — result card is back to the compact 'plan · X/N done' summary.

Co-Authored-By: Claude <noreply@anthropic.com>

Isolate sub-agent todo + render live todo in TUI side panel ...

Phase 1 — sub-agent todo isolation (backend):
- Add current_todo_session_id ContextVar; subagent_runner scopes it to the
  sub-agent's ephemeral run id so its auto-populated plan and todo updates
  land in an isolated KV row instead of clobbering the parent session's todo
  (which the parent's goal-anchoring reads every iteration).
- todo._sid() and planning.set_tasks prefer current_todo_session_id; the
  parent run leaves it unset, so all existing todo consumers (anti-stall,
  goal anchor, get_progress_message) behave exactly as before.

Phase 2 — live todo in the TUI right column:
- TodoUpdated event + emit it from the agent loop after planning auto-populate
  and after each tool-execution turn.
- GET /sessions/{id}/todos reads the parent session's todo KV row (explicit
  user_id/session_id, optional injected kv).
- api.get_todos + TodoList/TodoPanel widgets: status-coloured markers
  (pending dim, in_progress accent+bold, done success, failed error, skipped
  dim), progress header, scrollable panel below the auto-height info block.
- Hybrid delivery: REST seeds the panel on attach/switch, todo_updated WS
  events update it live.

Sub-agent todos as nested sub-lists is deferred to a later phase.

Co-Authored-By: Claude <noreply@anthropic.com>

tui: show the currently-served model in the status panel ...

The status panel's Model line was fed the global ollama_default_model, not the
session/profile model, and the server never told the client which model
actually served a call. Now:

- Backends stamp the resolved model onto LLMChunk (first chunk) / LLMResponse.
  The fallback backend reports the model that survived its server+model
  priority list (may differ from the profile's first choice).
- New ModelInfo event ({"type":"model_info","model":...}) emitted once per
  turn from agent._consume_stream, re-emitted only when the model changes
  across iterations. Additive WS event — old clients ignore it.
- TUI: attach_session/switch fetch the profile's configured model (first of
  profile.model) via api.get_profile_model so the panel shows a value before
  the first request; model_info then refines it to the actually-served model.
  Not forwarded to the chat panel. raw CLI prints "[model] ...".

Co-Authored-By: Claude <noreply@anthropic.com>

agent: cwd-aware memory fact filter for bounded autonomy ...

Long-term memory stored context-dependent path facts (e.g. "project_root
→ /home/.../navi-1") as global user facts. search_facts injected them into
any session, so when working in another project the agent was told "the
project root is navi-1" and drifted there.

When scope_boundary_enabled and a session cwd is set, _memory_facts_msg now
drops facts whose value is an absolute path outside the session cwd tree.
Facts are kept when working inside that path (then they are correct), and
non-path/relative facts always pass. Free flight stays reproducible by
toggling the flag off. No facts deleted, extractor untouched.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: bounded autonomy — scope boundary + observe-vs-act ...

navi_code had unwanted "free flight": an observe request ("look at a
directory") triggered the full Phase 3 plan with milestones + auto-todo,
and goal_anchoring then drove the agent to finish those steps, climbing
into sibling projects and executing milestone docs it found.

Two toggleable, default-off profile flags (on for navi_code):

- scope_boundary_enabled: injects a standing system message keeping the
  agent within the literally requested scope; forbids acting on
  discovered TODO/roadmap/milestone docs (report only).
- observe_skips_plan_enabled: Phase 1 classifies MODE: observe|act; an
  observe request skips Phase 2/3 — no multi-step plan, no auto-todo, no
  "execute step by step" prompt. The agent just gathers info and answers.
  Independent of force_plan (observe on the first message still skips).

Free flight stays reproducible by flipping both flags off.

Co-Authored-By: Claude <noreply@anthropic.com>

fix(ws): include session_id/profile_id in session_sync, guard renderer ...

The server sent {"type": "session_sync"} without session_id/profile_id,
crashing the terminal client (render.py did None[:8]). Add the fields to
both session_sync sends and guard the renderer against a missing id.

Co-Authored-By: Claude <noreply@anthropic.com>

session store: lazy persistence — no empty sessions in DB ...

POST /sessions no longer inserts a DB row. The Session is registered in an
in-memory _pending registry on PgSessionStore and only upserted on the first
save() (first user message / meaningful state change). Empty sessions that
never receive a message never reach the DB and vanish on server restart or
via the hourly pending sweep.

- pg_session_store: _pending dict + lock; create() registers, get() checks
  _pending first, save() upserts the sessions row (INSERT ... ON CONFLICT)
  and pops _pending so the session_messages FK is satisfied; sweep_pending()
  drops abandoned entries; pending_sweep_loop() background task.
- main.py: start/cancel pending_sweep_loop in lifespan.
- tests: 7 new tests for create/get/save/list/sweep semantics; updated
  existing save() test comments for the upsert.

profiles: add gemma4:12b-it-qat-128k as top-priority model ...

New local QAT model added at the head of the model priority list across
all profiles, so it is preferred first with cloud fallback.

Navi Code: stop via Esc + project cwd propagation ...

- TUI: Esc stops active stream cooperatively via POST /sessions/{id}/stop
- TUI: render stream_stopped as status message
- CLI/WebSocket: send shell cwd in client->server message field
- Orchestrator stores cwd in session.session_metadata
- ContextBuilder injects [Working directory] into LLM context
- Agent sets current_working_directory ContextVar per turn
- tools/base: ToolContext gains cwd field
- filesystem/terminal/code_exec resolve relative paths against session cwd
- Add bin/navi-code wrapper for PATH symlink; document in README.md
- Update docs/websocket.md and tests

Full pytest: 544 passed, 1 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>

compressor: structured summaries, profile-aware compression, adaptive keep_recent ...

- Replace free-form summary with strict Markdown template (Goal, Active Files,
  Decisions, Completed Work, Pending Work/Todo, Errors, Key Values).
- Keep filesystem/code_exec/terminal tool results and messages with
  is_compression_critical=True verbatim during compression instead of 300-char truncation.
- Make compression profile-aware: AgentProfile gains compression_keep_recent,
  compression_max_tokens, compression_prompt_file. navi_code uses dedicated
  compression prompt and larger keep_recent/max_tokens.
- Adaptive partition_messages(): important turns (user corrections, errors,
  critical tools) survive longer; filler/social turns compress sooner.
- Increase default context_summary_max_tokens from 3000 to 4000.
- Propagate active profile changes to ContextCompressor and SubAgentRunner.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: skip planning for casual greetings + strengthen DIRECT shortcut in Phase 1 ...

- Add fast _is_casual_message heuristic in navi/core/agent.py. Greetings and
  social chat (e.g. 'привет', 'как дела', 'hi', 'thanks') bypass planning even
  on the first session message, unless planning_mandatory is enabled.
- Strengthen Phase 1 planning prompt in navi/core/planning.py: explicitly
  require DIRECT output for greetings, simple questions, and one-step
  instructions.
- Fix broken ContextCompressed import in navi/core/compressor.py.
- Add unit tests covering the new heuristic and DIRECT prompt.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code TUI: fix input box layout, command palette duplicate IDs, status renderer, and WS input loop ...

- clients/terminal/tui/widgets/input_box.py: switch Horizontal to Vertical with width: 100% for Input so it renders and accepts input in real terminals; add refresh on Input.Changed.
- clients/terminal/tui/screens/command_palette.py: remove fixed ListItem IDs to avoid DuplicateIds on fast filter.
- clients/terminal/tui/chat_model.py + renderers/status.py + widgets/chat_panel.py: render backend status events as dim system messages instead of raw dicts.
- clients/terminal/tui/ws_bridge.py: start NaviWebSocketClient.input_loop so enqueued user messages are actually sent to the backend.
- clients/terminal/tui/tui_app.py: focus InputBox synchronously in on_mount so typing works immediately.
- tests/clients/test_tui_app.py: regression test for visible input text.

Co-Authored-By: Claude <noreply@anthropic.com>

Fix MCP names and profile API format consistency ...

- Replace stale mcp__navi_web__* / mcp__navi_3d__* names with canonical
  mcp__navi-web__* / mcp__navi-3d__* across prompts and key_tools.
- Update /agents/profiles and /admin/profiles endpoints to expose
  tools.agent / tools.subagent instead of deprecated enabled_tools fields.
- Update docs/mechanics.md to reference the new tools structure.
- Archive stale docs/visual.html.

Co-Authored-By: Claude <noreply@anthropic.com>

Update Navi Code profile: config, prompts, docs, and env example ...

- Sync navi_code profile tools with current codebase (memory, schedule_recall, manage_recall, switch_profile, list_profiles)

- Remove image_view and MCP tools from navi_code; keep local terminal focus

- Update system prompt and persona for terminal-first coding assistant

- Update docs/navi_code.md, docs/profiles.md, docs/config.md for new tool set

- Refresh .env.navi_code.example with current config variables

- Remove stale CLI prototype directories (cli/, navi_code_cli/)

Docs: actualize navi_code tool lists and key_tools ...

- docs/navi_code.md and docs/profiles.md now correctly list native tools,
  MCP navi-web tools, and excluded tools.
- image_view is documented as enabled for the agent (loads and describes
  images for the model, not shown to the user).
- web_search/http_request are documented as MCP navi-web tools, not native
  disabled tools.
- navi/profiles/navi_code/config.json: updated full_description.key_tools to
  match actual resolved tool names.

Tests: 459 passed, 1 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: Phase 1 — terminal-first profile, default profile mechanism, env/persona, tests ...

- Add navi_code profile (terminal-first coding assistant)
- Add NAVI_DEFAULT_PROFILE_ID setting and POST /sessions default profile fallback
- Add persona_navi_code.txt and .env.navi_code.example
- Add docs/plan_navi_code.md phased plan
- Update tests to verify default-profile selection in no-auth mode

Co-Authored-By: Claude <noreply@anthropic.com>

Fix review issues for NAVI_AUTH_ENABLED feature ...

- Return 503 from /auth/login and /auth/callback when NAVI_AUTH_ENABLED=false
- Return model_copy() of _ANONYMOUS_USER to prevent accidental mutation
- Clean up __import__(datetime) in auth DDL
- Reorder FakeChatSession definition before use in tests
- Remove dead monkeypatch code in no-auth integration fixture
- Add unit test for get_current_user_ws in no-auth mode
- Add integration tests rejecting OAuth endpoints in no-auth mode

Co-Authored-By: Claude <noreply@anthropic.com>

Add NAVI_AUTH_ENABLED switch for optional auth ...

- Add navi_auth_enabled setting (default true) to navi/config.py and .env.example
- When disabled, treat every request as anonymous admin user (id='anonymous')
- Create/update fixed anonymous navi_users row on startup
- Bypass OAuth/cookie/API-token resolution in navi/auth/deps.py
- Update /auth/status to return {enabled, configured}
- Log security warning on startup when auth is disabled
- Update webclient: skip fetchMe/login screen, show Local mode footer,
  expose /admin link, warn in API keys panel
- Rebuild webclient production bundle
- Add unit and integration tests for no-auth mode
- Update docs: auth.md, config.md, api.md, api_tokens.md, sessions.md,
  websocket.md, mechanics.md, index.md

Co-Authored-By: Claude <noreply@anthropic.com>

Fix spawn_agent profile selection: stronger prompt + alias fallback ...

- Rewrite PROFILE SELECTION description to explicitly say: when the
  plan specifies a profile, you MUST pass profile_id. Only omit when
  the plan does NOT specify a profile.
- Add JSON examples showing correct profile_id usage.
- Add fallback for 'profile' parameter name (models sometimes use
  'profile' instead of 'profile_id' when the description says
  'Profile to use').\n\nCo-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Fix scratchpad tool: rename op→action, add examples, improve error messages ...

- Rename primary parameter from 'op' to 'action' for consistency with
  all other tools (terminal, filesystem, todo, etc.). Legacy 'op' still
  works as a fallback to avoid breaking old calls in compressed context.
- Add JSON examples directly in the tool description so the model sees
  the exact structure to produce.
- Improve all error messages to include the correct JSON example,
  making it obvious what the model did wrong.
- Add manuals/scratchpad.md for tool_manual support.
- Update and expand tests for new syntax and error cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Remove offline_access from OAuth scopes ...

gnexus-auth does not support offline_access yet (SUPPORTED scopes are
only openid, email, profile, roles, permissions). Requesting it caused
invalid_scope error and login failures.

Keep refresh token support as-is — tokens are still issued and
refreshed, just bound to the SSO session lifetime.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Fix OAuth callback to handle missing code and error responses ...

FastAPI validated code/state as required query params before the
handler body ran, so any OAuth error response (e.g. invalid_scope
from offline_access, user denied consent) produced an opaque 422
"Field required" instead of a readable error message.

- Make code, state, error, error_description all optional with defaults
- Detect OAuth error responses and return a clear 400 with the error
- Guard state[:8] slicing when state may be None

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>