navi: a token count that matches the model, a type for every file, and a socket that admits it is dead ...

Compression decided everything in chars // 3, and on this workload that sits
~1.8x below reality — 101k real prompt tokens against a ~56k estimate — so the
trigger, the target, the truncate budgets and the pre-call size check all ran
in a unit half the size of the real one. TokenCalibration now fits
overhead + chars / chars_per_token per model from the prompt_eval_count that
every call already reports, and a profile declares its own context_window
(sent to Ollama as num_ctx) instead of every model being held to OLLAMA_NUM_CTX.

Superseded summaries were never demoted in the database: save() only updates
flags for rows in session.messages, a summary is is_display=False, so it lived
in session.context alone and every reload resurrected it — prod sessions
carried up to 15 live summaries. PgSessionStore.set_context_flag() writes the
flip for those rows directly.

Five extension tables disagreed about what a file is. navi/content_types.py is
the one detector — signature, then extension, then a text probe — it decides
the MIME the download route serves, and the session file list now carries the
type the viewers branch on.

The idle receive loop answers client pings and sends its own heartbeat. A
half-open socket keeps readyState at OPEN and never fires close, so traffic
coming back is the only proof the connection still works.

subagents, secrets: an honest result contract, and two leaks closed ...

A pass over the sub-agent system, plus the two findings from the previous one.

Leaks:

- `redact_args` / `is_sensitive_key` (navi/tools/_internal/redact.py) mask
  credential-shaped keys at every log site that prints tool arguments — by key
  name, for all tools, so a future MCP tool is covered without declaring
  anything. ssh_exec passwords were sitting in the prod journal in plaintext.
- `mcp_servers.d/*.json` are tracked in git and carried the scaffolding
  machine's absolute paths, so navi-3d/navi-web silently vanished from the
  toolset off that machine. They now hold project-relative paths, resolved in
  `McpClient.open_transport` — not at load time, because create_mcp_server
  round-trips every config file through save_mcp_servers.

Sub-agent result contract:

- `run_ephemeral` returns a `SubAgentOutcome` (text + status: ok, timeout,
  max_iterations, thinking_stall, user_stop, context_overflow) instead of
  `(text, bool)`. Every exit routes through `SubAgentRunner._finish`, so token
  accounting is never dropped and a partial run always carries a progress
  report — the stop-after-stream path used to return a bare string.
- `spawn_agent` renders its header from the status. Before this, every short
  run reached the parent as "hit iteration limit", and the parent repeats that
  diagnosis to the user — a timeout was reported as an iteration limit.
- `ContextTooLargeError` no longer kills a sub-agent: the parent compresses its
  own context, a sub-agent has no session to compress into, so it stops with
  `context_overflow` and hands back what it did rather than surfacing as
  "Sub-agent failed: …" with the whole run discarded.
- The result is capped at 8000 chars (head and tail kept) so a long run cannot
  flood the parent's context, and tool arguments in the progress report go
  through `redact_args`.
- The navi-3d calling convention left the runner for the server's own tool
  schemas — a generic runner should not know one MCP server's path shapes.
- Validation moved inside spawn_agent's try: a bad argument is a clean
  ToolResult the model can read, not an exception in the parent's loop.

Tests: 1636 passed, 1 skipped. New: SubAgentOutcome paths and headers, result
truncation, context overflow, clean argument failures.

tools: list_tools defaults to the profile the run executes as ...

Asking "what tools do I have?" required the agent to know its own profile id
— and to be right about it after a switch_profile had moved the session. The
run knows: run_stream now publishes the active profile id (re-published when
switch_profile reloads the run) alongside the model, SubAgentRunner publishes
its own rather than inheriting the parent's, and ToolContext carries it into
every execute() call.

list_tools reads that when profile_id is omitted, so the bare call answers the
question, and it says which profile it assumed ("current profile") so the
answer cannot be mistaken for another profile's.

scope='agent'|'subagent' selects which of the profile's two toolsets to list —
the set spawn_agent would hand a sub-agent running that profile, which is a
different list and until now had no way to be inspected.

Item D of the list_tools plan, kept separate from the accuracy/compactness work.

e2e fixes: tasks tool in profiles, bg timeout lift, stepIcon fix, queue semantics docs ...

E2E findings addressed:
- tasks tool was registered but not in any profile's tools.agent.native —
  added to all six profiles (agent could not check/wait/cancel bg tasks)
- detached terminal/code_exec/ssh_exec runs without explicit timeout are
  lifted to 300s: foreground defaults (20/30/60s) marked long commands
  'completed' with partial output while the process still ran
- ToolCard.vue: define stepIcon(status) — template referenced it but the
  function was missing (render crash on task_update step)
- message_queued reachability documented: WS read loop is sequential, the
  queue path is only reachable from a second socket/headless recall

agent parallelism: background tools, bg spawn_agent, parallel tool batches, message queue ...

- backgroundable tools (terminal/ssh_exec/peer/spawn_agent/code_exec) detach
  via TaskManager with per-session/global/spawn caps, rate limit and TTL
- completion delivery both ways: task_update event (out-of-band) + pending
  notes injected into the next turn; tasks tool (list/check/wait/cancel)
- parallel tool-call batches (profile-level gate, off by default) with
  ToolStarted up-front, tagged event mux, call-order results, one save,
  plus dangling tool_call repair on session load
- user message queue instead of busy error: message_queued frame,
  back-to-back drain on the same socket, headless fallback on disconnect
- pin mcp<2 (v2 renames FastMCP with breaking API changes)
- docs: tasks.md (new), websocket/api/agent/config updates, tasks manual,
  spawn_agent background param, persona contract section

docs: sync with the plan rework and compression pipeline ...

- planning: plan tool flow (agent-invoked, no pre-turn gate), confirmation
  by COMPLEXITY, think=False + 20k-char Phase 1 window, retired flags
  removed from profiles/api docs
- compression: all four triggers through compress_and_save_session,
  real-baseline gates, head+tail summary input, _plan_compression shared
  decision, actual config defaults (0.90 threshold, 6000 summary tokens,
  32k input cap)

docs: actualize compression mechanics + real-token baseline ...

mechanics.md: rewrite the Context Compression catalog (adaptive turn
importance, intra-turn fallback, would_compress guard, token-budget
hard-truncate, per-message view truncation, profile-aware overrides,
real-token baseline estimator, meta-summary, archive-on-compress,
CompressionStarted/ContextCompressed events); fix Agent Loop rows
(run() delegates to run_stream, planning entry condition, pre/mid-turn
gates, check_context_size, forced /compact, estimate chars//3 in
compressor.py); fix WS image limits (8 / 50MB) and add /compact control
message; rebuild the Undocumented Mechanics Summary from current ❌ rows;
refresh the cross-reference index.

config.md: LLM_STREAM_FIRST_CHUNK_TIMEOUT 180→90; add
CONTEXT_MESSAGE_TOKEN_BUDGET.

agent.md: run() runs planning; full run_ephemeral signature + compact_stream
entry point; cwd ContextVar; planning entry condition; loop pseudocode
(midturn compression, check_context_size, ModelInfo, real baseline);
streaming guard 90s; workers carry profile; pre-turn estimate-based gate;
per-message view truncation.

sessions.md: full Message flag table (is_context/is_display/is_compression_critical
…); three compression trigger points + forced /compact; real-baseline
estimate; check_context_size guard; expanded algorithm (adaptive partitioning,
intra-turn fallback, meta-summary, 24k cap, 4000-char critical preview,
token-budget hard-truncate fallback, archive-on-compress, baseline clear);
compression events; SessionStore archive methods + list_page profile_id.

Co-Authored-By: Claude <noreply@anthropic.com>

docs: update docs for tools.agent/tools.subagent and Navi Code TUI ...

- profiles.md: document tools.agent/tools.subagent structure, deprecate enabled_tools
- tools.md: move web tools to MCP section, fix mcp__navi-web__* names, add create_mcp_server/test_mcp_tool/mcp_status/recall tools
- api.md: update /agents/profiles and /admin/profiles/{id} response schemas, fix MCP examples
- navi_code.md: correct navi_code tool list (MCP disabled)
- navi_code_cli.md: add /thinking, /compact, correct TUI layout
- index.md: add navi_code, terminal client, pydantic-settings
- testing.md: refresh test directory layout, replace stale coverage roadmap
- agent.md: mention tools.agent/tools.subagent in loop

Co-Authored-By: Claude <noreply@anthropic.com>

Fix review issues: KV-store NULL, SVG messages, filesystem tests + docs ...

Bugs fixed:
- filesystem.py: add missing `import re` (grep was broken in production)
- image_view.py: consistent SVG rejection message for URL and file paths
- store/__init__.py: normalize user_id None→'' to prevent duplicate rows
  in unique constraint for anonymous sessions; add DDL migration for
  existing NULL values

Tests:
- Add 10 unit tests for filesystem copy, grep, diff operations

Documentation:
- agent.md: document streaming guard wrapper, system prompt caching,
  ContextVar restoration in subagents
- tools.md: document middleware hooks
- websocket.md: document image upload limits and concurrent run guard
- store.md: document user_id normalization
- mechanics.md: mark newly-documented mechanics as documented

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Audit and fix docs/ for accuracy against current codebase ...

Package A (API + WebSocket + Sessions):
- api.md: add missing endpoints (health/embed, auth/status, agents/prompts,
  agents/mcp_servers, sessions/content, 7 admin endpoints). Fix response
  schemas, auth requirements, query params. Update path formats.
- websocket.md: fix planning_status.phase type (int 1|2|3), add metadata
  to tool_call, message_index to stream_end, context_tokens/max_context_tokens
  to context_compressed.
- sessions.md: fix context_compression_threshold default (0.80→0.70),
  add list_page method, add metadata to Message flags.

Package B (Config + Profiles + Tools + Agent + Memory):
- config.md: remove stale ANTHROPIC_API_KEY description, add 12 missing
  env vars (embedding, web search, terminal_user_allowed_commands,
  context_providers_dir, gnauth fields, output_reserve_tokens).
  Fix OPENAI_BASE_URL type.
- profiles.md: fix max_iterations default (20→10), add top_k/top_p/num_thread,
  is_admin_only, context_providers, mcp_servers, subagent_think_enabled.
  Clarify subagent_tools fallback and step_validation_enabled.
- tools.md: replace 3 separate memory tools with unified `memory`, add
  `mcp_status`.
- agent.md: fix run_ephemeral return type (str→tuple), clarify DB reads.
- memory.md: fix search_facts/get_all_facts param order, fix MemoryStore
  init description.

236 tests passing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Allow subagents to use selected profiles

Add structured planning review and adaptive depth

Add NAVI.md navigation hub and update core docs ...

NAVI.md: lightweight project entry point for Navi — server command,
key paths, doc map with query instructions, tool manual index.
All profiles read this; detailed content stays in docs/.

docs/agent.md: rewrite to cover 3-phase planning, all 10 thinking
mechanics flags, adaptive replan, anti-stall, goal anchoring.

docs/profiles.md: update AgentProfile fields (all flags), correct
profile list (secretary/server_admin/developer), JSON config format,
auto-discovery instead of manual registration.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add backend documentation ...

10 markdown files covering all backend subsystems: architecture,
agent loop, tool system, sessions, WebSocket protocol, profiles,
memory, config, and API reference. Structured for AI readability
with a single entry point (docs/index.md) and cross-references.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>