Enable realtor profile to create listings via vmk_data create_listing_tool ...

- Persist all inline images to session_files/ and expose public URLs in user context

- Add vmk_data MCP group 'create' with create_listing_tool

- Enable 'create' group in realtor profile config

- Extend realtor system prompt with listing creation workflow and form usage

Co-Authored-By: Claude <noreply@anthropic.com>

Add navi_ui form component with client-side validation ...

- Add Form UI component schema in navi/mcp/ui_server/components/form.py
- Support text, textarea, number, email, url, select, multiselect, checkbox, date fields
- Implement hidden user message mode for form submissions (is_display=False, is_context=True)
- Add WebSocket form_submit handling and _start_agent_run helper
- Render forms client-side with real-time JS validation in webclient
- Submit once guard via localStorage and replace form with read-only summary
- Provide wsSend via Vue provide/inject in ChatArea
- Add tests for backend form component, websocket form_submit, and frontend Form.vue
- Update docs/tools.md and docs/websocket.md
- Build production webclient dist

Co-Authored-By: Claude <noreply@anthropic.com>

Fix delta loss during thinking-to-text transition and streaming orphaning ...

Backend:
- agent.py _consume_stream: replace elif with if so a single chunk can
  carry both the final thinking fragment AND the first text delta.
  Fixes the 'first token lost after tools' bug.

Frontend:
- chat.js reloadSession: bail out early when streaming.value is true.
  Replacing messages.value mid-stream orphans streamingMsg, breaking
  Vue reactivity for all subsequent deltas and tool cards.
- useWebSocket.js session_sync handler: add !chat.streaming guard
  before calling reloadSession.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix 19 issues found in full codebase review ...

Backend:
- Stop session auth bypass: require auth for owned sessions, reject anonymous with 401
- upload_file: stream chunks directly to disk instead of buffering in RAM
- MCP config: validate name against path traversal regex
- auth deps: cleanup stale refresh locks periodically
- auth routes: expire mobile auth states after 10 min to prevent unbounded growth
- compressor: meta-summarize existing summaries before compression; preserve assistant content when tool_calls present; rewrite hard_truncate to keep whole turns
- orchestrator: configurable WS replay buffer size; async cleanup/remove_websocket/clear_busy; fix run_recall ContextVar order to avoid deadlock on _build_agent failure; await cleanup in finally
- agent: persist image_msg in session.messages; remove archived messages from session after archive; remove duplicate StreamStopped yield on tool stop
- websocket: try/except around create_task with cleanup on failure; await remove_websocket

Frontend:
- App.vue: hashchange listener lifecycle in onMounted/onUnmounted
- MessageList.vue: passive scroll, flash timeout cleanup, archive scroll snapshot
- InputBar.vue: 300 ms debounce on draft save to localStorage
- SessionList.vue: remove :key from DynamicScroller to avoid remount jitter

Tests: 422 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Wire archive trigger into agent after compression ...

After _do_compress_and_save finishes, if the total persisted message count
(db_next_sequence) exceeds session_messages_window (default 1000), the agent
now calls archive_old_messages() to move older rows into
session_messages_archive.

Adds session_messages_window config and unit tests for archive SQL.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Phase 2: Dual-write with is_context/is_display flags on Message ...

- Message model gets is_context and is_display bools
- PgSessionStore.save() writes flags directly to session_messages
- Agent sets is_context=False on display-only messages, is_display=False on context-only
- Planning: plan context msg is_display=False, plan marker is_context=False
- Compression: summarized messages get is_context=False, summary added to messages with is_display=False
- Tests updated for extra user display+context messages per turn

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add client-side image resize and server-side image token budgeting ...

Client (useFileUpload.js):
- Resize images to max 1024px on longest side via Canvas before upload
- JPEG quality 0.9, preserves EXIF orientation
- Reduces typical phone photos from 8 MB → ~400-900 KB

Server (websocket.py):
- Increase limit to 8 images, 50 MB total payload

Server (agent.py):
- Before adding user message to session.context, estimate how many images
  fit in remaining context (500 tok/img, up to 80% of max_context_tokens)
- Images that don't fit are saved as resized JPEGs to session_files/
- References to saved images are appended to the user message text
  so Navi can view them later via image_view tool
- session.messages keeps ALL images for full UI history

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix MCP tool spinner bug: match tool_started → tool_call by tool_call_id ...

- Add tool_call_id field to ToolStarted and ToolEvent dataclasses
- Pass tc.id as tool_call_id from agent.py, subagent_runner.py, and tool_executor.py
- Update frontend chat.js onToolStarted/onToolCall to match cards by toolCallId
  with fallback to name-matching for backward compatibility

Closes spinner issue where LLM short name ("search_docs") didn't match
resolved MCP name ("mcp__gnexus_book__search_docs").

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Pass explicit ToolContext to tools instead of hidden ContextVars ...

Add ToolContext dataclass (session_id, event_sink, stop_event, model,
user_id, user_role, user_info) and thread it through the execution chain:
Agent._execute_tools_with_sink → ToolExecutor → tool.execute().

All ~25 tools updated to accept ctx parameter. Tools that previously
read ContextVar now prefer ctx when provided, falling back to
ContextVar for backward compatibility.

Tests updated to pass ToolContext explicitly — no more test fixtures
that set current_session_id / current_user_id ContextVars.

ContextVar setters remain as fallback for non-tool consumers
(ai_helper, context_builder, planning) and will be removed in a
follow-up refactor.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor profile tool config to explicit agent/subagent structure ...

Replaces the confusing mix of enabled_tools + mcp_servers + subagent_tools
with a single explicit structure:

    tools: {
      agent:   {native: [...], mcp: {server: [groups]}},
      subagent:{native: [...], mcp: {server: [groups]}}
    }

Why:
- Old fields mixed native and MCP names (mcp__server__tool) in one list,
  making it impossible to tell at a glance what a subagent actually gets.
- subagent_runner.py had 25 lines of runtime MCP filtering logic that
  was hard to follow and error-prone.

Changes:
- AgentProfile: add ToolConfig / ToolScopeConfig pydantic models,
  keep old fields (enabled_tools, mcp_servers, subagent_tools) for
  auto-migration via _migrate_tools validator.
- loader.py: read new "tools" key, auto-migrate legacy configs.
- agent.py: _tool_list now accepts ToolScopeConfig.
- subagent_runner.py: simplified — profile.get_subagent_tools() returns
  the exact scope, no runtime filtering needed.
- context_builder.py, list_tools.py, spawn_agent.py: updated to use
  profile.get_agent_tools() / get_subagent_tools().
- All 6 profile config.json files migrated to new schema.
- Secretary subagent now explicitly gets navi-web MCP tools for web search.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix stop button responsiveness and shutdown CancelledError ...

Agent loop (_execute_tools_with_sink):
- Poll stop_event every 1s while draining the event sink via asyncio.wait_for
- When stopped, cancel the tool task, yield a synthetic ToolEvent failure,
  append a cancellation message to session, yield StreamStopped, and return
- Pass stop_event into _execute_tools_with_sink call site

Subagent runner:
- Check stop_event at the start of each tool in turn_tool_calls loop
- Returns early with ("", False) when stopped mid-batch

McpManager.disconnect_all():
- Disconnect clients sequentially instead of asyncio.gather
- Handle asyncio.CancelledError per-client to avoid shutdown traceback

AppContainer.shutdown():
- Catch BaseException instead of Exception for MCP and DB cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix token counting: show only completion tokens, not cumulative prompt+completion ...

The token_count displayed next to assistant messages was summing
prompt_tokens + completion_tokens across ALL tool-calling iterations,
giving hundreds of thousands of tokens for multi-turn conversations.

Now:
- token_count (coins icon) = only completion tokens generated by the model
- context_tokens (ContextBar) = only prompt tokens (context size sent to LLM)

This gives users a realistic measure of how much the model actually generated.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 5-7: Extract async generators from run_stream, unify run() as wrapper ...

- _compression_events_preturn / _compression_events_midturn
- _consume_stream (uses StreamState)
- _execute_tools_with_sink
- run() is now a thin wrapper around run_stream() collecting StreamEnd
- Remove dead imports (json, LLMChunk)
- Mark god-object decomposition complete in architecture_weak_spots.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 4: Extract SubAgentRunner from run_ephemeral() ...

- Create navi/core/subagent_runner.py with full sub-agent loop logic
- Move _iter_stream_guarded to navi/core/stream_guard.py
- Move _check_context_size to ContextCompressor.check_context_size()
- Extract build_tool_list() and load_user_enabled_tools() to tool_utils.py
- Agent.run_ephemeral() becomes a thin wrapper delegating to SubAgentRunner
- Remove ~310 lines from agent.py
- All existing run_ephemeral tests pass unchanged

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 3: Extract AntiStallMonitor from run_stream() ...

- Create navi/core/anti_stall.py with AntiStallMonitor class
- Encapsulates stall detection (todo progress + repeated tool calls)
- Encapsulates adaptive re-plan (failed todo step detection)
- Provides init() / pre_turn() / post_turn() two-phase interface
- Remove ~50 lines of stall/replan logic from agent.py run_stream()
- Remove _todo_status_snapshot and _todo_failed_steps helpers from agent.py
- Update AgentTurnContext: remove stall fields (now live in AntiStallMonitor)
- Add 13 unit tests for pre_turn and post_turn behavior

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Step 2: Extract AgentTurnContext dataclass from run_stream() ...

Move 10 turn-level local variables from run_stream() into AgentTurnContext:
- turn_start, tool_call_count, turn_tokens, subagent_tokens
- stall_no_todo, stall_repeat_tools, prev_tool_sigs
- known_failed, replan_msg, injected_fact_ids

This makes run_stream() readable and prepares the ground for
AntiStallMonitor (Step 3) which will consume this context.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract ContextCompressor, fix STL viewer, expand test suite, add architecture audit docs ...

- Extract ContextCompressor from agent.py (Step 1 of god-object refactor)
- Add retry + hard-truncate fallback logic to ContextCompressor
- Add unit tests: agent loop (14), compressor (18), KV store (8),
  auth encrypt (3), auth deps (13), todo/scratchpad/image_view/memory
- Fix WebGL STL viewer: allow-same-origin sandbox + graceful fallback
- Add CompressionStarted event and client-side compression notice
- Add docs/architecture_weak_spots.md and plan_01_god_object_agent.md
- Update test_events.py and test_agent_context_size.py for new logic

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add inherit_system_prompt and is_subagent_only mechanisms ...

- inherit_system_prompt: subagent parameter to prepend parent's system
  prompt as a base layer before subagent specialisation
- is_subagent_only: profile flag blocking switch_profile, allowing
  spawn_agent only; shown with [subagent only] tag in list_profiles
- Document both in docs/profiles.md and docs/tools.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Enhance native toolset and add persistent KV store ...

- Add PostgreSQL-backed KvStore (navi/store/) for session-scoped data.
- Migrate todo and scratchpad from in-memory dicts to KvStore.
- Filesystem: add copy, grep, diff actions; compress description.
- CodeExec: remove language param, expose working_dir in schema.
- ImageView: resize to 1024px JPEG + Content-Type guard for URLs.
- Memory list: return distinct categories instead of all facts.
- SSH: add scp action with upload/download support.
- Update CLAUDE.md (Postgres-only), docs/tools.md, add docs/store.md.
- Fix agent/planning/context_builder async signatures for todo helpers.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix(recall): stabilize scheduled callback system and improve UX ...

Backend fixes:
- stop_session now stops headless recall runs via _busy_sessions dict
- _fire_recall sets user ContextVars so tools work correctly
- MaxIterationsReached treated as success, not failure
- skip_next_recall uses GREATEST(trigger_at, now) for overdue recalls
- schedule_recall rejects past trigger times
- timezone offset double-adjustment fixed for aware datetimes
- _fire_recall registers _AgentRun for reconnect/replay support
- session_sync race with stream_start fixed

Frontend improvements:
- Recall banner moved to ChatHeader with live Cancel/Skip buttons
- Recall messages styled with is_recall flag and badge
- Real-time recall updates via WebSocket (recall_update events)
- Recall filter moved to sessions-header as toggle button
- Session list shows clock icon for sessions with pending recall
- Empty state messages for empty/filtered session lists
- Fixed missing api import in ChatHeader.vue

Tests:
- Updated scheduler_loop tests for _busy_sessions dict change

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Persist uploaded files in messages, live file tree updates, and UI polish ...

Backend:
- Add `files` field to `Message` model so uploaded file metadata survives page refresh
- Pass `files` through websocket handler → `agent.run_stream` / `agent.run`
- `list_tools`: make `profile_id` required; return error instead of all-tools fallback

Webclient:
- Call `fetchFiles()` after successful file upload for immediate Files tab update
- Live refresh file tree on filesystem (write/edit/append/mkdir/rm/cp/mv), terminal, and code_exec tool calls
- Add manual refresh button (desktop) and pull-to-refresh (mobile) to Files tab
- Fix live link updates: move regex creation inside per-message loop to avoid lastIndex state leak
- Restore full profile name text next to avatar in ChatHeader; hide avatar in header
- Fix mobile ArtifactsPanel: collapse tab text labels so close button fits with 3 tabs

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix agent.py _tool_list to use colon-delimited MCP names ...

The main registry and API routes were migrated, but Agent._tool_list
still built names with the old underscore format. This meant MCP tools
were silently dropped from the tool list passed to the LLM, causing
"tool not found" when the model correctly called mcp:server:tool names.

Also updates stale docstring in McpTools.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Clarify knowledge persistence prompts

Auto-inject relevant memory facts into LLM context on every user turn. ...

Semantic search against the vector memory store runs in parallel with
context provider collection. Number of injected facts scales with
message length (1–3) to reduce noise on short queries. Guardrails:
min length gate, per-turn deduplication, and structured logging at
info level for observability.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix ollama_backends / FallbackOllamaBackend issues ...

- registry.py: always use FallbackOllamaBackend (unified backend).
  Enables model priority lists in all deployments, not just multi-server.
- agent.py: add missing think=profile.think_enabled to run() (REST endpoint).
- compressor.py: fix model param type (str → list[str] | str | None).
- fallback.py: harden load_servers_from_file against missing/bad JSON files
  and entries without host. Add clear_blacklists() for manual reset.
- admin.py: add POST /admin/ollama/clear-blacklists endpoint.
- tech_debt_review: document dead stream() methods.
- tests: add tests for single-server fallback, bad file handling,
  missing host skipping, and blacklist clearing.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add deterministic line-based file editing (edit_lines), rating UI fix, and session refresh ...

- filesystem.py: add edit_lines action (deterministic line ops via operations array)
           + numbered param for read (1-based line numbers in output)
           + clarify four editing modes in tool description
- chat.js: fix rating IDs for streaming messages (assign h_ ID on stream_end)
- SessionList.vue: mobile pull-to-refresh with PTR_THRESHOLD=80
- AppSidebar.vue: desktop refresh button next to Conversations header
- planning.py: knowledge source assessment in Phase 1
- debug panel: MCP servers tab + resolved tools per profile
- NAVI.md: reposition as neutral quick-reference

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Inject MCP server instructions into system prompt ...

- McpClient: collect instructions from MCP initialize handshake
- McpServerConfig: add 'instructions' field for Navi-side overlay
- McpManager.get_instructions(): merge server + config instructions
- ContextBuilder: new _mcp_context_msg() injects MCP server
  descriptions into every LLM context as a system message
- Agent passes mcp_manager to ContextBuilder
- mcp_servers.json: add overlay instructions for gnexus-book

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Refactor MCP integration: server groups in profiles ...

- mcp_servers.json: add 'groups' (read/write/admin) for gnexus-book
- AgentProfile: new 'mcp_servers' field (server_name -> group list)
- Profile loader: parse and persist 'mcp_servers' in config.json
- Agent._tool_list(): expands mcp_servers into concrete tool names
  via McpManager.resolve_group(), wildcard '*' supported
- /agents/profiles API: includes 'mcp_servers' in response
- Profiles no longer list individual mcp_ tools in 'enabled_tools'
- discuss: gnexus-book read group
- server_admin: gnexus-book read+write+admin groups

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Propagate user profile to LLM context via current_user_info ContextVar ...

- Extend User model: username, first_name, last_name, phone, birth_date,
  country, city, locale (all from gnexus-auth profile)
- navi_users DDL: add new profile columns
- auth/deps + auth/callback: populate new fields on upsert
- /auth/me: return all profile fields
- Add current_user_info ContextVar for full user profile propagation
- websocket + messages: set current_user_info before agent.run()
- run_ephemeral: inherit and restore current_user_info
- ContextBuilder: _user_context_msg() injects [User context] with name,
  email, location, locale, role into LLM system messages
- _security_policy_msg: reads user_id/role from ContextVar directly

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>