compression: fix auto-compress no-op on few-huge-messages + honest status ...

Root cause: the compression gate (should_compress) measures tokens, but the
partition measures message/turn count, and CompressionStarted was emitted
before the attempt. For navi_code's "few very large messages" shape (one big
file read = 1 user + assistant + 1 huge tool result, 66k tokens in 3 messages)
the gate fired, the UI showed "compression", but partition returned
to_summarize=[] -> compress_context None -> nothing shrank. The agent kept
going until the window overflowed. It wasn't running "during" compression —
there was no compression, just a no-op the user mistook for one.

A. Per-message head/tail truncation in context_builder.build(): oversized
   tool/assistant messages (over context_message_token_budget, 0=num_ctx//6)
   are capped head+marker+tail in the LLM view only (model_copy — stored
   history and reloads are never affected). A single huge tool result can no
   longer alone blow the window; user/system messages are never truncated.

B. Token-budget hard-truncate fallback in compress_session: when partition
   no-ops but tokens exceed the threshold, drop oldest turns to num_ctx*0.5.
   _hard_truncate is now token-aware (was a fixed message-count floor that
   no-oped on <=6 messages even when huge). New would_compress() predicts
   compress_session's real outcome with no LLM call.

C. Honest CompressionStarted: _compression_events_midturn/_preturn emit it
   only after would_compress() confirms the partition (or token-budget
   fallback) can actually shrink the stored context — no more "compression"
   status with no ContextCompressed to follow.

Bonus: post-turn CompressionWorker now passes keep_recent_messages=
max(12, context_keep_recent*2), matching the midturn path, so a single long
autonomous turn compresses post-turn too (was always a no-op).

Tests (+14): would_compress agreement, token-budget fallback, token-aware
hard_truncate, build() truncation (preserves user, no mutation, head+tail),
agent no-CompressionStarted-when-nothing-to-compress, worker single-long-turn.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: force context compression via typed /compact control message ...

Forced /compact previously sent a chat message, which ran a full agent turn
that only produced summary text instead of running the real context compressor.
Worse, even when wired correctly, forced compact always reported "Nothing to
compact yet — the context is still small" regardless of context size: the typical
navi_code shape is a single long autonomous turn (1 user message + many tool
iterations = one turn), and partition_messages finds nothing to summarize when
turns <= keep_recent. The midturn auto-compress path already bypassed this via
keep_recent_messages (intra-turn split), but compact_stream passed
keep_recent_messages=None, so the fallback was disabled.

Changes:
- WS protocol: {"type":"compact"} control message (distinct from {"type":
  "message"}); rejected while an agent turn is active to avoid racing the agent.
- Agent.compact_stream: forced compression that bypasses the token threshold
  but still runs the real compressor; passes keep_recent_messages=max(12,
  context_keep_recent*2) so a single long turn compresses via intra-turn split
  (mirrors midturn auto-compress). Raises NothingToCompactError when context is
  genuinely too small.
- Orchestrator.run_compact + clear_run: broadcast agent events to subscribers,
  end with done marker, surface NothingToCompactError as an error event.
- Terminal client: ws_client.send accepts str|dict; CompactCommand enqueues
  {"type":"compact"}; TUI distinguishes forced compact (no stream_start) from
  in-turn auto-compress via the _streaming flag.
- Tests: compact_stream (incl. single-long-turn regression), WS handler
  dispatch/rejection, run_compact event/error broadcasting, ws_client send,
  compact command.

Co-Authored-By: Claude <noreply@anthropic.com>

tui: show the changed step's text in the todo update call card ...

The todo update call card (→ todo · #3 → done) now shows the text of the
step being changed, plus the validation when present. The LLM's update args
carry only the index, not the task text, so the text is read from the
current plan row before the tool runs and attached to ToolStarted.metadata
(client-rendering only, backward-compatible).

Backend:
- events.ToolStarted gains a metadata dict (mirrors ToolEvent) → to_wire.
- navi/tools/todo: step_text_for_update(index, ctx) resolves the step text
  via _sid/_uid (so sub-agent isolation holds), started_metadata_for_call
  wraps it for both emit sites.
- agent.py (parent) and subagent_runner.py (sub-agent) enrich ToolStarted
  via the shared helper before emitting.

Renderer:
- TodoStartedRenderer update card reads msg.metadata.step_text → shows the
  step text + validation; falls back to 'no validation' on history replay.
- Removed the step-text line from the result card (it now lives in the call
  card) — result card is back to the compact 'plan · X/N done' summary.

Co-Authored-By: Claude <noreply@anthropic.com>

Isolate sub-agent todo + render live todo in TUI side panel ...

Phase 1 — sub-agent todo isolation (backend):
- Add current_todo_session_id ContextVar; subagent_runner scopes it to the
  sub-agent's ephemeral run id so its auto-populated plan and todo updates
  land in an isolated KV row instead of clobbering the parent session's todo
  (which the parent's goal-anchoring reads every iteration).
- todo._sid() and planning.set_tasks prefer current_todo_session_id; the
  parent run leaves it unset, so all existing todo consumers (anti-stall,
  goal anchor, get_progress_message) behave exactly as before.

Phase 2 — live todo in the TUI right column:
- TodoUpdated event + emit it from the agent loop after planning auto-populate
  and after each tool-execution turn.
- GET /sessions/{id}/todos reads the parent session's todo KV row (explicit
  user_id/session_id, optional injected kv).
- api.get_todos + TodoList/TodoPanel widgets: status-coloured markers
  (pending dim, in_progress accent+bold, done success, failed error, skipped
  dim), progress header, scrollable panel below the auto-height info block.
- Hybrid delivery: REST seeds the panel on attach/switch, todo_updated WS
  events update it live.

Sub-agent todos as nested sub-lists is deferred to a later phase.

Co-Authored-By: Claude <noreply@anthropic.com>

tui: show the currently-served model in the status panel ...

The status panel's Model line was fed the global ollama_default_model, not the
session/profile model, and the server never told the client which model
actually served a call. Now:

- Backends stamp the resolved model onto LLMChunk (first chunk) / LLMResponse.
  The fallback backend reports the model that survived its server+model
  priority list (may differ from the profile's first choice).
- New ModelInfo event ({"type":"model_info","model":...}) emitted once per
  turn from agent._consume_stream, re-emitted only when the model changes
  across iterations. Additive WS event — old clients ignore it.
- TUI: attach_session/switch fetch the profile's configured model (first of
  profile.model) via api.get_profile_model so the panel shows a value before
  the first request; model_info then refines it to the actually-served model.
  Not forwarded to the chat panel. raw CLI prints "[model] ...".

Co-Authored-By: Claude <noreply@anthropic.com>

agent: cwd-aware memory fact filter for bounded autonomy ...

Long-term memory stored context-dependent path facts (e.g. "project_root
→ /home/.../navi-1") as global user facts. search_facts injected them into
any session, so when working in another project the agent was told "the
project root is navi-1" and drifted there.

When scope_boundary_enabled and a session cwd is set, _memory_facts_msg now
drops facts whose value is an absolute path outside the session cwd tree.
Facts are kept when working inside that path (then they are correct), and
non-path/relative facts always pass. Free flight stays reproducible by
toggling the flag off. No facts deleted, extractor untouched.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: bounded autonomy — scope boundary + observe-vs-act ...

navi_code had unwanted "free flight": an observe request ("look at a
directory") triggered the full Phase 3 plan with milestones + auto-todo,
and goal_anchoring then drove the agent to finish those steps, climbing
into sibling projects and executing milestone docs it found.

Two toggleable, default-off profile flags (on for navi_code):

- scope_boundary_enabled: injects a standing system message keeping the
  agent within the literally requested scope; forbids acting on
  discovered TODO/roadmap/milestone docs (report only).
- observe_skips_plan_enabled: Phase 1 classifies MODE: observe|act; an
  observe request skips Phase 2/3 — no multi-step plan, no auto-todo, no
  "execute step by step" prompt. The agent just gathers info and answers.
  Independent of force_plan (observe on the first message still skips).

Free flight stays reproducible by flipping both flags off.

Co-Authored-By: Claude <noreply@anthropic.com>

session store: lazy persistence — no empty sessions in DB ...

POST /sessions no longer inserts a DB row. The Session is registered in an
in-memory _pending registry on PgSessionStore and only upserted on the first
save() (first user message / meaningful state change). Empty sessions that
never receive a message never reach the DB and vanish on server restart or
via the hourly pending sweep.

- pg_session_store: _pending dict + lock; create() registers, get() checks
  _pending first, save() upserts the sessions row (INSERT ... ON CONFLICT)
  and pops _pending so the session_messages FK is satisfied; sweep_pending()
  drops abandoned entries; pending_sweep_loop() background task.
- main.py: start/cancel pending_sweep_loop in lifespan.
- tests: 7 new tests for create/get/save/list/sweep semantics; updated
  existing save() test comments for the upsert.

Navi Code: stop via Esc + project cwd propagation ...

- TUI: Esc stops active stream cooperatively via POST /sessions/{id}/stop
- TUI: render stream_stopped as status message
- CLI/WebSocket: send shell cwd in client->server message field
- Orchestrator stores cwd in session.session_metadata
- ContextBuilder injects [Working directory] into LLM context
- Agent sets current_working_directory ContextVar per turn
- tools/base: ToolContext gains cwd field
- filesystem/terminal/code_exec resolve relative paths against session cwd
- Add bin/navi-code wrapper for PATH symlink; document in README.md
- Update docs/websocket.md and tests

Full pytest: 544 passed, 1 skipped.

Co-Authored-By: Claude <noreply@anthropic.com>

compressor: structured summaries, profile-aware compression, adaptive keep_recent ...

- Replace free-form summary with strict Markdown template (Goal, Active Files,
  Decisions, Completed Work, Pending Work/Todo, Errors, Key Values).
- Keep filesystem/code_exec/terminal tool results and messages with
  is_compression_critical=True verbatim during compression instead of 300-char truncation.
- Make compression profile-aware: AgentProfile gains compression_keep_recent,
  compression_max_tokens, compression_prompt_file. navi_code uses dedicated
  compression prompt and larger keep_recent/max_tokens.
- Adaptive partition_messages(): important turns (user corrections, errors,
  critical tools) survive longer; filler/social turns compress sooner.
- Increase default context_summary_max_tokens from 3000 to 4000.
- Propagate active profile changes to ContextCompressor and SubAgentRunner.

Co-Authored-By: Claude <noreply@anthropic.com>

agent: skip planning for casual greetings + strengthen DIRECT shortcut in Phase 1 ...

- Add fast _is_casual_message heuristic in navi/core/agent.py. Greetings and
  social chat (e.g. 'привет', 'как дела', 'hi', 'thanks') bypass planning even
  on the first session message, unless planning_mandatory is enabled.
- Strengthen Phase 1 planning prompt in navi/core/planning.py: explicitly
  require DIRECT output for greetings, simple questions, and one-step
  instructions.
- Fix broken ContextCompressed import in navi/core/compressor.py.
- Add unit tests covering the new heuristic and DIRECT prompt.

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code TUI: fix input box layout, command palette duplicate IDs, status renderer, and WS input loop ...

- clients/terminal/tui/widgets/input_box.py: switch Horizontal to Vertical with width: 100% for Input so it renders and accepts input in real terminals; add refresh on Input.Changed.
- clients/terminal/tui/screens/command_palette.py: remove fixed ListItem IDs to avoid DuplicateIds on fast filter.
- clients/terminal/tui/chat_model.py + renderers/status.py + widgets/chat_panel.py: render backend status events as dim system messages instead of raw dicts.
- clients/terminal/tui/ws_bridge.py: start NaviWebSocketClient.input_loop so enqueued user messages are actually sent to the backend.
- clients/terminal/tui/tui_app.py: focus InputBox synchronously in on_mount so typing works immediately.
- tests/clients/test_tui_app.py: regression test for visible input text.

Co-Authored-By: Claude <noreply@anthropic.com>

Fix MCP names and profile API format consistency ...

- Replace stale mcp__navi_web__* / mcp__navi_3d__* names with canonical
  mcp__navi-web__* / mcp__navi-3d__* across prompts and key_tools.
- Update /agents/profiles and /admin/profiles endpoints to expose
  tools.agent / tools.subagent instead of deprecated enabled_tools fields.
- Update docs/mechanics.md to reference the new tools structure.
- Archive stale docs/visual.html.

Co-Authored-By: Claude <noreply@anthropic.com>

Move terminal_manager to _internal subpackage ...

- terminal_manager is an internal helper, not a tool
- Update imports in terminal.py, container.py, test_terminal.py

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix terminal review issues: security, lifecycle, reactivity ...

- TerminalManager.open now accepts exec_tokens to use create_subprocess_exec
  for restricted commands instead of always using shell
- Fix kill/terminate order: SIGTERM first, SIGKILL fallback
- Pop closed sessions from _sessions dict to prevent memory leak
- Add terminal_manager.shutdown() to AppContainer.shutdown()
- Wait for reader tasks in foreground open before returning output
- Add _MAX_TERMINALS_PER_SESSION limit (10)
- Wrap cleanup_idle tasks in _close_one_safe with error logging
- send_input catches BrokenPipeError/ConnectionResetError specifically
- Foreground terminals auto-close after gathering output
- Vue reactivity: replace terminals object immutably instead of mutating
- onTerminalClosed marks matching tool card as no longer pending
- Update tests for new behavior (foreground auto-close, max limit)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add persistent multi-session terminal tool with background support ...

- New TerminalManager module: named subprocess sessions per Navi session,
  background readers, event-sink streaming, idle auto-cleanup
- Refactor terminal tool to multi-action: run, open, close, list,
  status, send_input
- Add TerminalOutputDelta and TerminalClosed events for streaming
- Wire TerminalManager into AppContainer, orchestrator, and registry
- Persist session_metadata in Session model and pg_session_store
- Close all session terminals on session delete
- Webclient: handle terminal_output/terminal_closed WS events,
  display live terminal output in tool cards
- Update unit tests for new terminal actions

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix delta loss during thinking-to-text transition and streaming orphaning ...

Backend:
- agent.py _consume_stream: replace elif with if so a single chunk can
  carry both the final thinking fragment AND the first text delta.
  Fixes the 'first token lost after tools' bug.

Frontend:
- chat.js reloadSession: bail out early when streaming.value is true.
  Replacing messages.value mid-stream orphans streamingMsg, breaking
  Vue reactivity for all subsequent deltas and tool cards.
- useWebSocket.js session_sync handler: add !chat.streaming guard
  before calling reloadSession.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix 19 issues found in full codebase review ...

Backend:
- Stop session auth bypass: require auth for owned sessions, reject anonymous with 401
- upload_file: stream chunks directly to disk instead of buffering in RAM
- MCP config: validate name against path traversal regex
- auth deps: cleanup stale refresh locks periodically
- auth routes: expire mobile auth states after 10 min to prevent unbounded growth
- compressor: meta-summarize existing summaries before compression; preserve assistant content when tool_calls present; rewrite hard_truncate to keep whole turns
- orchestrator: configurable WS replay buffer size; async cleanup/remove_websocket/clear_busy; fix run_recall ContextVar order to avoid deadlock on _build_agent failure; await cleanup in finally
- agent: persist image_msg in session.messages; remove archived messages from session after archive; remove duplicate StreamStopped yield on tool stop
- websocket: try/except around create_task with cleanup on failure; await remove_websocket

Frontend:
- App.vue: hashchange listener lifecycle in onMounted/onUnmounted
- MessageList.vue: passive scroll, flash timeout cleanup, archive scroll snapshot
- InputBar.vue: 300 ms debounce on draft save to localStorage
- SessionList.vue: remove :key from DynamicScroller to avoid remount jitter

Tests: 422 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add meta-summary for multi-level compression ...

When to_summarize contains multiple existing summary messages whose
combined length exceeds 8000 chars (~1/3 of max summarizer input),
run a quick meta-summary pass first to consolidate them into a single
compact summary before the main compression. This prevents information
loss when repeated compressions stack up long summary chains.

- _meta_summarize(): fast LLM pass (think=False, max_tokens=1500)
- compress_context(): detects >1 long summaries and triggers meta pass
- Graceful fallback: if meta-summary fails, continue with raw summaries
- 3 new unit tests: consolidation, skipped for short summaries, failure fallback

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add archive message pagination, configurable WS replay buffer ...

Backend:
- Add archive_threshold to Session model and getSession response
- Add next_before_seq to archive endpoint for cursor pagination
- Make WS replay buffer size configurable via WS_REPLAY_BUFFER_SIZE

Webclient:
- Add getArchivedMessages API function
- Add archive pagination state and loadArchivedMessages to chat store
- MessageList: auto-load older messages on scroll-to-top with scroll
  position preservation and loading spinner

Docs: update config.md with new env vars

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Wire archive trigger into agent after compression ...

After _do_compress_and_save finishes, if the total persisted message count
(db_next_sequence) exceeds session_messages_window (default 1000), the agent
now calls archive_old_messages() to move older rows into
session_messages_archive.

Adds session_messages_window config and unit tests for archive SQL.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add session message archive table + global sequence_number tracking ...

Schema:
- session_messages_archive — identical structure, stores old messages.
- sessions.next_sequence    — monotonic seq counter per session.
- sessions.archive_threshold — split point between hot and archive.

Behaviour:
- get() / _build_sessions() load only seq >= archive_threshold (hot).
- save() UPDATEs existing rows (seq >= 0) and INSERTs new ones (seq = -1)
  with auto-assigned sequence_number = next_sequence, next_sequence+1, ...
- archive_old_messages() moves a batch to archive and bumps the threshold.

This keeps the hot table bounded so list/get RAM stays flat regardless
of total session history size.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Implement delta-save for session messages ...

Replace full DELETE/INSERT with efficient delta writes:
- Track db_message_count on Session (how many rows already persisted).
- On save(): UPDATE mutable flags for existing rows, DELETE only extras
  (race guard), INSERT new messages via executemany.
- Reduces DB write amplification from O(N) to O(delta) per turn.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Review fixes: restore _build_sessions, fix flags, search filter, tests ...

- Restored _load_messages_map and _build_sessions helpers that were
  accidentally dropped in Phase 5 (list_all/list_page/search_list
  called them but they were missing, causing NameError at runtime)
- _build_sessions now filters messages by is_display so list methods
  return consistent display-only history like get()
- count_all/search_list EXISTS subquery now filters to is_display=true
  so search only matches visible chat messages
- Updated pg_session_store docstring to remove stale dual-write claim
- compressor summary_msg now defaults to is_display=False
- Added unit tests for message flags (agent, compressor, planning)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Phase 5: Cleanup legacy JSON dual-write, remove duplicated class ...

- Removed JSON column writes from create() and save()
- Removed duplicated PgSessionStore class block that leaked in during Phase 4
- _serialize/_deserialize retained for boot migration helper only
- All session I/O now goes exclusively through session_messages table

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Phase 4: Read list/search from session_messages, drop JSON fallback ...

- list_all / list_page / search_list now load messages via _build_sessions()
  which batch-loads session_messages in a single query
- count_all search uses EXISTS (SELECT 1 FROM session_messages ...) instead
  of ILIKE on the JSON text column
- Removed _row_to_session and all legacy JSON column reads from list paths
- get() continues to load session_messages directly (Phase 3)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Phase 3: Build context from session_messages table ...

- get() always reads from session_messages, never falls back to JSON columns
- Load all rows in a single query, then split by is_display / is_context flags
- messages[] and context[] now share the same Python objects,
  enabling reliable id() matching in the agent compression path

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Phase 2: Dual-write with is_context/is_display flags on Message ...

- Message model gets is_context and is_display bools
- PgSessionStore.save() writes flags directly to session_messages
- Agent sets is_context=False on display-only messages, is_display=False on context-only
- Planning: plan context msg is_display=False, plan marker is_context=False
- Compression: summarized messages get is_context=False, summary added to messages with is_display=False
- Tests updated for extra user display+context messages per turn

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add subagent progress report on failure ...

When a subagent stops (timeout, max iterations, thinking stall, user stop),
it now returns a structured progress report built from its local message
context, so the parent agent knows what tools were called and what was
accomplished before the stop.

- Add _build_progress_report() to SubAgentRunner
- Report includes: turn number, assistant text, tool calls with results
- Prepended to result_text for every stop reason (completed also gets it)
- Updated test_run_ephemeral_complete to expect the report prefix

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add client-side image resize and server-side image token budgeting ...

Client (useFileUpload.js):
- Resize images to max 1024px on longest side via Canvas before upload
- JPEG quality 0.9, preserves EXIF orientation
- Reduces typical phone photos from 8 MB → ~400-900 KB

Server (websocket.py):
- Increase limit to 8 images, 50 MB total payload

Server (agent.py):
- Before adding user message to session.context, estimate how many images
  fit in remaining context (500 tok/img, up to 80% of max_context_tokens)
- Images that don't fit are saved as resized JPEGs to session_files/
- References to saved images are appended to the user message text
  so Navi can view them later via image_view tool
- session.messages keeps ALL images for full UI history

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>