Clarify share file publishing boundaries

Docs audit: fix outdated references (PostgreSQL, Vue client, profiles, tools) ...

- architecture.md: SQLite → PostgreSQL + pgvector
- README.md: vanilla JS → Vue 3 + Pinia, remove smart_home, update temps
- profiles.md: remove smart_home, add discuss, update temperatures
- tools.md: add missing built-in tools, update user tools list
- sessions.md: fix outdated references
- NAVI.md: sync with current stack
- persona.txt: update self-extension and content_publish guidance
- content_publish.md manual: reflect session_dir-based no-copy design
- content_store.py & content_publish.py: inline with docs

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add content_publish manual and improve tool description ...

- Create manuals/content_publish.md with full usage guide,
  internal mechanics, supported types, and best practices
- Update ContentPublishTool description with usage guidelines
  and examples in the tool schema
- All tests pass (150 backend + 47 webclient)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Rework content_publish: files live in session_dir, no copying ...

content_publish now registers existing session files for inline viewing
instead of copying them to navi/content/<uuid>/. This makes files
editable by the agent and immediately visible to the user via the
stable URL /sessions/{id}/files/{filename}.

- content_store.publish() → registers metadata, no shutil.copy2
- content_store.delete_content() → removes DB record only (file stays
  in session_dir; cleaned up by session lifecycle)
- content_store.cleanup_old() → deletes stale DB records
- ContentPublishTool:
  - accepts 'filename' instead of 'path'
  - looks up file in uploads/sessions/<id>/
  - returns clear error if file not found with instructions
  - includes full local path in output for future editing
- Added unique index on (session_id, filename) for idempotent re-publish
- Removed /content static mount from main.py (endpoint still works
  for legacy files; new files use /sessions/{id}/files/)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Architecture extensibility — event bus, middleware, auto-discovery, Pydantic profiles ...

- EventBus: async pub/sub for AgentEvents, WebSocket subscribes instead of direct yield
- Declarative serialization: AgentEvent.to_wire() on all event types
- Auto-discovery for LLM backends (_discover_backends) and workers (scan navi/workers/*.py)
- AgentProfile: Pydantic BaseModel with extra='allow', @field_validator for model coercion
- Tool middleware chain: pre/post execute hooks via ToolRegistry.add_middleware()
- LoggingMiddleware: built-in, logs every tool call
- Fix pg_trgm DDL: conditional GIN indexes via DO $$ block, no CREATE EXTENSION
- New files: event_bus.py, middleware.py, logging_middleware.py

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Architecture cleanup: old_webclient, SSH shutdown, todo decoupling, ContextVar reset ...

- Remove old_webclient directory and /static mount from main.py
- Add shutdown handler in main.py to close all SSH pooled connections
- Decouple agent.py from todo module internals:
  - Add public API to navi/tools/todo.py: get_task_snapshot, get_failed_steps,
    get_progress_message, set_tasks, render_todo_lines
  - Replace all direct _plans/_STATUS_ICON/_Task imports in agent.py with API calls
- Wrap run_ephemeral in try/finally to restore _sid_var and _model_var after subagent

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add content hosting system with inline viewers ...

Backend:
- Add navi/content/ directory for published files
- Add content_store.py with publish/list/delete/cleanup functions
- Add content_publish tool for publishing files as viewable content
- Add /content static file mount in main.py
- Add /content-viewers mount for viewer pages
- Extend ToolEvent with metadata field
- Forward metadata through websocket tool_call events
- Update Agent to include metadata in ToolEvent

Frontend:
- Add ContentCard.vue component for displaying published content
- Add viewer pages: stl.html (Three.js), svg.html, html.html, pdf.html
- Update AssistantMessage.vue to render ContentCard for content_publish
- Update chat store to preserve metadata in tool cards
- Update websocket protocol docs with metadata field

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Wire pgvector semantic search into memory system ...

- Add vector(768) column + HNSW index to memory_facts

- Add LLMBackend.embed() with Ollama + fallback implementation

- MemoryStore: cosine-distance search with ILIKE fallback

- New memory tool params: source, confidence, expires_days, source_context

- Update extractor, sqlite_store, deps wiring

- Add pgvector to requirements

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Stop image_view hallucinations on inline-attached images ...

The model was inventing fake paths/URLs (e.g. files.oaiusercontent.com,
/home/ubuntu/navi-1/input_file_0.png) and calling image_view on them
when the user attached an image directly in chat — the image was
already in the multimodal context, but the tool description and lack
of a signal pushed the model to "load" it anyway.

- websocket.py: when a user message has inline images, append a brief
  note that they are already in context.
- image_view.py: soften the description — keep proactive use for paths
  and URLs the model genuinely cannot see, but tell it inline images
  don't need this tool.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Strengthen todo progress discipline

Tune reflect autonomy guidance

Fix profile prompt inconsistencies

filesystem mkdir: parents=False to prevent accidental path chain creation ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add mkdir action to filesystem tool (parents=True, exist_ok=True) ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add context providers: dynamic system message injection per LLM call ...

- navi/context_providers/ registry + built-in public_url provider (global, always injected)
- context_providers/ user directory, hot-reloaded via reload_tools
- AgentProfile.context_providers field for per-profile opt-in providers
- Agent._collect_context_injections() called before every tool-calling loop
- reload_tools now reloads both user tools and user context providers
- manuals/write_context_provider.md for Navi, docs/context_providers.md reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add Ollama multi-server fallback with in-memory blacklisting ...

- New FallbackOllamaBackend (navi/llm/fallback.py): tries servers and
  models in priority order; on LLMConnectionError blacklists the server
  for the process lifetime, on LLMModelNotFoundError blacklists the
  (server, model) pair — eliminates latency from repeated failed probes
- OllamaBackend now raises typed LLMConnectionError / LLMModelNotFoundError
  instead of bare LLMBackendError; accepts list[str] | str | None for model
- AgentProfile.model changed from str to list[str] (str auto-normalised);
  all profiles updated to ["gemma4:31b-cloud", "gemma4:26b-a4b-it-q4_K_M"]
- New config field OLLAMA_BACKENDS_FILE: path to [{host, api_key?}] JSON;
  when set, registry creates FallbackOllamaBackend instead of OllamaBackend
- ollama_backends.json template added (gitignored — contains API key)
- current_model ContextVar type widened to list[str] | str | None

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Make server I/O non-blocking; update docs ...

- Wrap all heavy filesystem ops in asyncio.to_thread: filesystem tool
  (read/write/append/list/find/info/move/delete/query/smart_edit),
  image_view (read_bytes), share_file (shutil.copy2), write_tool
  (write_text, _register_user_tool), session_files (shutil.rmtree,
  iterdir), sessions upload endpoint (sync open/write → to_thread)
- Make delete_session_dir async; update its caller in sessions.py
- docs/config.md: fix wrong defaults (threshold 0.70, keep_recent 8),
  remove phantom SESSION_FILES_TTL_HOURS, add LLM timeouts, DATABASE_URL,
  PUBLIC_URL, Gmail, CONTEXT_SUMMARY_MAX_TOKENS sections
- docs/profiles.md: add missing tool_developer profile to table
- android-client: add WebView remote debugging; remove unused toolbar menu
- Remove stale helper scripts and test files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Agent improvements: mandatory planning, tool cleanup, smart_edit fixes ...

- Planning now mandatory on first message of every session (force_plan)
- RESOURCES, COMMITMENTS, ATOMICITY fields added to planning phase 1
- Todo auto-injected at iteration 0 so model tracks steps immediately
- Execution trigger injected after plan to prevent model treating plan as response
- Split developer profile: tool_developer (Navi tools) vs developer (general code)
- Simplified persona.txt: trimmed redundant content now handled by mechanics
- AIHelper.ask(): 120s timeout via asyncio.wait_for to prevent smart_edit hangs
- filesystem._smart_edit(): atomic write via temp file + os.replace()
- Removed 5 junk user tools (game project artifacts, trivial utilities)
- Removed instagram tools (to be rewritten); cleaned enabled.json

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Restructure persona: concrete information gathering protocol ...

- Replace abstract CONTEXT FIRST principle with explicit numbered protocol
  (check summary → memory_search → NAVI.md → ask user)
- Move protocol to top of persona (after identity) so it has maximum weight
- NAVI.md section: remove READ instructions (now in protocol), keep only WRITE
- LONG-TERM MEMORY: remove "search proactively" line (now in protocol),
  keep only save/forget instructions
- memory_search description: remove misleading "call at start of each session"
  (summary is already auto-injected); clarify when to actually call it

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Autonomous reasoning improvements: budget, anchoring, anti-stall, validation ...

- AgentProfile: per-profile thinking mechanics flags (think_enabled,
  iteration_budget_enabled, goal_anchoring, anti_stall, step_validation,
  planning_reflect, adaptive_replan) — all profiles updated in config.json
- Iteration budget: inject remaining iterations into context so model knows
  when to wrap up; urgency levels at ≤7 and ≤3 remaining
- Goal anchoring: inject original goal + todo state every N iterations to
  prevent drift on long tasks
- Anti-stall: two signals — no todo progress for N iterations, or identical
  tool calls repeated N times; warning injected into context
- Todo step validation: marking done requires a validation field describing
  how result was verified; failed gets a soft nudge with tip for re-planning
- stream_complete: add think param to base class, ollama and openai backends
- Summarizer: raise max_tokens 1024→3000, expand system prompt with
  user-preferences section and verbatim-value instructions
- Compression card: persist to session.messages (is_compression flag on
  Message), show expandable summary in webclient with markdown body
- ToolResult.to_message_content: always include output on failure so
  tracebacks and error details reach the model (fixes silent Error: None)
- Developer profile: fix subagent profile secretary→developer, add write_tool
  to subagent_tools, clarify write_tool vs filesystem in system prompt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Planning debug panel, todo auto-populate, scratchpad/persona improvements ...

- Planning debug panel: new Planning tab in debug/index.html shows raw
  phase 1/2 outputs and token counts per planning run, stored in
  session.planning_logs (new column in both SQLite and PostgreSQL)
- New GET /sessions/{id}/planning API endpoint
- PlanningDebugData internal event wires _run_planning() output into
  session storage; never forwarded to WebSocket clients
- Phase 3 (plan critic) disabled — to be reworked with reflect integration
- Todo tool: auto-populated from plan steps after phase 2; model only
  needs to call update/view, not set
- Scratchpad: clarified description and persona instructions; removed
  context_transfer from user-facing docs (internal mechanism only)
- web_search: switched to ddgs package, SearXNG as primary backend,
  DDG html-only fallback; added find_up action to filesystem tool
- Persona: added SCRATCHPAD and TODO sections with clear usage rules;
  added NAVI.md project context instructions
- chat.js: fixed subagent planning event fallthrough into parent UI;
  statusLabel cleared on first stream delta

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Route subagent planning events into spawn_agent card in the UI ...

Previously PlanningStatus/PlanReady had no is_subagent flag, so subagent
planning spinners and plan cards rendered as top-level Navi planning UI.

Backend:
- Add is_subagent field to PlanningStatus and PlanReady events
- _run_planning accepts is_subagent param, passes it through all yields
- run_ephemeral calls _run_planning with is_subagent=True
- websocket.py forwards is_subagent in planning_status and plan_ready messages

Frontend (chat.js):
- onPlanningStatus: if is_subagent, set planningLabel on the last spawn_agent
  card instead of msg.statusLabel
- onPlanReady: if is_subagent, push plan into spawn card steps and clear
  planningLabel; otherwise behave as before

Frontend (ToolCard.vue):
- Render subagent-planning-indicator (spinner + label) when planningLabel set
- Render plan cards inside subagent steps using the same plan-card pattern

Also includes leftover session changes: spawn_agent default 40 in description
and manual, updated manual content.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove context_transfer from all user-facing prompts — internal mechanism only ...

context_transfer is the scratchpad section name used internally by spawn_agent
to auto-inject parent state. Navi doesn't control it and doesn't need to know
about it. Removed from: persona, secretary, server_admin, spawn_agent description,
manual. Internal code (spawn_agent.py) still reads the section transparently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix core subagent misuse: enforce 1 plan step = 1 spawn_agent call ...

Root cause: nowhere was it stated that each AGENT step in the plan
maps to a separate spawn_agent call. Navi was bundling all AGENT steps
into a single call, dumping the full plan on one subagent.

spawn_agent description:
- Lead with: "Delegate EXACTLY ONE step of your plan"
- Explicit: "3 AGENT steps = 3 spawn_agent calls"
- Remove "multi-step sub-task" wording that invited bundling
- briefing: clarify as static context only (credentials, paths, instructions)
  Dynamic findings from prior steps → context_transfer, not briefing

Planning Phase 2 prompt:
- Add AGENT scoping rules: each step = one focused unit, not "do everything"
- Add good/bad examples of AGENT step granularity
- Show multiple AGENT steps in the format example

Secretary & server_admin system prompts:
- Add explicit 1:1 rule with counter-example
- Show correct multi-agent execution pattern with code example
- Clarify briefing vs context_transfer boundary everywhere

Persona:
- "ONE PLAN STEP = ONE spawn_agent CALL" as first sentence in SUB-AGENTS
- Field descriptions tightened: briefing = static, context_transfer = dynamic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Subagent system prompt rework: separate from parent, briefing as system context ...

run_ephemeral:
- Add briefing param (passed from spawn_agent, injected into system prompt)
- Subagent system prompt is now completely separate from parent's system_prompt:
    1. profile.subagent_system_prompt  (executor persona)
    2. custom_system_prompt            (role specialisation for this task)
    3. briefing                        (task context as system-level instruction)
  Fallback to profile.system_prompt only if subagent_system_prompt is not defined

spawn_agent:
- task → user message only (the goal)
- briefing → system prompt (credentials, context, instructions)
- system_prompt → role specialisation injected alongside briefing
- Removed old user-message composition (## Context / ## Task split)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix spawn_agent: restore briefing, fix status leakage, enable subagent planning ...

spawn_agent:
- Restore briefing param (task = goal, briefing = context — good separation)
- Add system_prompt as third param for role specialisation per task
- Remove [STATUS: ...] prefix that was leaking into Navi's responses and
  causing hallucination — replaced with natural-language headers that are
  less likely to be regurgitated verbatim
- completed → neutral header; limit_reached → explicit warning about incompleteness

Profiles:
- subagent_planning_enabled: false → true in all three profiles
  (planning is on by default, disable per-profile if needed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve subagent system: isolated tools, custom prompts, context transfer, timeout ...

AgentProfile:
- New fields: subagent_tools, subagent_planning_enabled, subagent_system_prompt
- loader.py: loads subagent_tools/subagent_planning_enabled from config.json,
  reads optional subagent_system_prompt.txt per profile

Profiles:
- Each profile now has a dedicated subagent_tools list (focused subset, no admin tools)
- subagent_planning_enabled: false (configurable per profile)
- New subagent_system_prompt.txt per profile with executor-focused instructions

run_ephemeral:
- Uses profile.subagent_tools instead of enabled_tools
- Builds subagent context without persona or profiles block (focused executor)
- Injects subagent_system_prompt after profile.system_prompt
- Accepts context_transfer: priming exchange injected before task message
- Wall-clock timeout (default 5 min) checked per iteration
- Returns (result_text, completed: bool) instead of bare string
- Optionally runs planning phase if profile.subagent_planning_enabled

spawn_agent:
- Removed briefing param; task is now fully self-contained
- Added system_prompt param: custom injected prompt for this specific task
- Auto-reads parent scratchpad context_transfer section via get_section()
- Result prefixed with [STATUS: completed|limit_reached]
- Timeout 300s

scratchpad:
- Added get_section(session_id, section) helper for cross-session reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Planning phases, context compression, and tool improvements ...

Agent:
- Planning now a 3-phase async generator: Analysis → Execution plan → AIHelper critic
- Yield PlanningStatus events before each phase (UI progress labels)
- Phase 1 runs with think=True for deeper analysis
- Phase 2 includes available tool list so executor assignments are accurate
- Phase 3: independent critic pass validates and corrects TOOL: names against real tool list
- Planning converted from list return to async generator (fixes token accounting)

Backend:
- Context compression threshold: 80% → 70% to trigger earlier
- Compressor summary prompt: structured sections (goal, work state, key facts, outputs, errors)
- Terminal output capped at 5000 chars to prevent context flooding
- Web search: region=wt-wt for DDG, country=ALL for Brave, language=all for SearxNG
- Scratchpad: mandate writing a 'goal' section at start of multi-step tasks
- secretary max_iterations: 40→25, temperature: 0.7→0.5
- server_admin max_iterations: 40→20

Webclient:
- ThinkingCard strips <thought> XML tags leaked by Ollama
- planning_status WS event wired to chat.onPlanningStatus()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Webclient UI improvements + backend fixes ...

Webclient:
- Draft persistence across page refreshes (localStorage, reactive watch)
- Image lightbox modal using UI kit classes on thumbnail click
- Copy button on user and assistant messages
- Selection reply toolbar: select assistant text → quote inserted into input
- User message rendering: proper HTML escaping, styled blockquote for > replies
- Markdown table fix: preprocessor to inject missing separator rows
- Planning status labels (rebuild dist)

Backend:
- Developer profile: enable subagent delegation, increase max_iterations to 35
- share_file: updated description + manual with absolute path requirement and URL sharing
- persona.txt: instructions for quote replies and GFM table format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

reflect: force clarification when Detailer finds strategic ambiguity ...

Closing instruction now explicitly requires stopping to ask the user
if the Detailer identified ambiguities about core direction (what to
build, which approach, what the user actually wants). Prevents Navi
from using the Pragmatist's simplifications as an escape hatch when
the real problem is underspecified requirements.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>