Remove context_transfer from all user-facing prompts — internal mechanism only ...

context_transfer is the scratchpad section name used internally by spawn_agent
to auto-inject parent state. Navi doesn't control it and doesn't need to know
about it. Removed from: persona, secretary, server_admin, spawn_agent description,
manual. Internal code (spawn_agent.py) still reads the section transparently.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix core subagent misuse: enforce 1 plan step = 1 spawn_agent call ...

Root cause: nowhere was it stated that each AGENT step in the plan
maps to a separate spawn_agent call. Navi was bundling all AGENT steps
into a single call, dumping the full plan on one subagent.

spawn_agent description:
- Lead with: "Delegate EXACTLY ONE step of your plan"
- Explicit: "3 AGENT steps = 3 spawn_agent calls"
- Remove "multi-step sub-task" wording that invited bundling
- briefing: clarify as static context only (credentials, paths, instructions)
  Dynamic findings from prior steps → context_transfer, not briefing

Planning Phase 2 prompt:
- Add AGENT scoping rules: each step = one focused unit, not "do everything"
- Add good/bad examples of AGENT step granularity
- Show multiple AGENT steps in the format example

Secretary & server_admin system prompts:
- Add explicit 1:1 rule with counter-example
- Show correct multi-agent execution pattern with code example
- Clarify briefing vs context_transfer boundary everywhere

Persona:
- "ONE PLAN STEP = ONE spawn_agent CALL" as first sentence in SUB-AGENTS
- Field descriptions tightened: briefing = static, context_transfer = dynamic

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix subagent instruction conflicts across persona and profiles ...

Persona:
- Fix [STATUS: completed|limit_reached] reference (format was removed)
- Clarify three fields: task / briefing / system_prompt with distinct roles
- Clarify context_transfer vs briefing: transfer = working state, briefing = credentials

Secretary system_prompt:
- Replace vague "write all context to context_transfer" with explicit field breakdown
- task / briefing / system_prompt each described with their purpose
- context_transfer correctly limited to intermediate findings, not credentials

Server admin system_prompt:
- Same fix: explicit field breakdown for spawn_agent
- Remove dangling "see persona" reference for briefing ending

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix spawn_agent: restore briefing, fix status leakage, enable subagent planning ...

spawn_agent:
- Restore briefing param (task = goal, briefing = context — good separation)
- Add system_prompt as third param for role specialisation per task
- Remove [STATUS: ...] prefix that was leaking into Navi's responses and
  causing hallucination — replaced with natural-language headers that are
  less likely to be regurgitated verbatim
- completed → neutral header; limit_reached → explicit warning about incompleteness

Profiles:
- subagent_planning_enabled: false → true in all three profiles
  (planning is on by default, disable per-profile if needed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Strengthen orchestration mandate: spawn first, inline last ...

secretary/server_admin system prompts:
- Explicit spawning rule: MUST spawn for any sub-task requiring 3+ tool calls
- Additional mandatory triggers listed (research, file processing, remote ops, large output)
- "If in doubt — spawn" as explicit fallback
- AGENT steps: "MANDATORY, never execute inline — defeats the orchestrator model"
- context_transfer pattern: write to scratchpad before spawning, injected automatically

persona.txt:
- Updated SUB-AGENT BRIEFING section: renamed to SUB-AGENTS
- Reflects new context_transfer automatic injection (no longer needs to be in task)
- Added: check [STATUS: ...] in result before deciding next action

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve subagent system: isolated tools, custom prompts, context transfer, timeout ...

AgentProfile:
- New fields: subagent_tools, subagent_planning_enabled, subagent_system_prompt
- loader.py: loads subagent_tools/subagent_planning_enabled from config.json,
  reads optional subagent_system_prompt.txt per profile

Profiles:
- Each profile now has a dedicated subagent_tools list (focused subset, no admin tools)
- subagent_planning_enabled: false (configurable per profile)
- New subagent_system_prompt.txt per profile with executor-focused instructions

run_ephemeral:
- Uses profile.subagent_tools instead of enabled_tools
- Builds subagent context without persona or profiles block (focused executor)
- Injects subagent_system_prompt after profile.system_prompt
- Accepts context_transfer: priming exchange injected before task message
- Wall-clock timeout (default 5 min) checked per iteration
- Returns (result_text, completed: bool) instead of bare string
- Optionally runs planning phase if profile.subagent_planning_enabled

spawn_agent:
- Removed briefing param; task is now fully self-contained
- Added system_prompt param: custom injected prompt for this specific task
- Auto-reads parent scratchpad context_transfer section via get_section()
- Result prefixed with [STATUS: completed|limit_reached]
- Timeout 300s

scratchpad:
- Added get_section(session_id, section) helper for cross-session reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Planning phases, context compression, and tool improvements ...

Agent:
- Planning now a 3-phase async generator: Analysis → Execution plan → AIHelper critic
- Yield PlanningStatus events before each phase (UI progress labels)
- Phase 1 runs with think=True for deeper analysis
- Phase 2 includes available tool list so executor assignments are accurate
- Phase 3: independent critic pass validates and corrects TOOL: names against real tool list
- Planning converted from list return to async generator (fixes token accounting)

Backend:
- Context compression threshold: 80% → 70% to trigger earlier
- Compressor summary prompt: structured sections (goal, work state, key facts, outputs, errors)
- Terminal output capped at 5000 chars to prevent context flooding
- Web search: region=wt-wt for DDG, country=ALL for Brave, language=all for SearxNG
- Scratchpad: mandate writing a 'goal' section at start of multi-step tasks
- secretary max_iterations: 40→25, temperature: 0.7→0.5
- server_admin max_iterations: 40→20

Webclient:
- ThinkingCard strips <thought> XML tags leaked by Ollama
- planning_status WS event wired to chat.onPlanningStatus()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Webclient UI improvements + backend fixes ...

Webclient:
- Draft persistence across page refreshes (localStorage, reactive watch)
- Image lightbox modal using UI kit classes on thumbnail click
- Copy button on user and assistant messages
- Selection reply toolbar: select assistant text → quote inserted into input
- User message rendering: proper HTML escaping, styled blockquote for > replies
- Markdown table fix: preprocessor to inject missing separator rows
- Planning status labels (rebuild dist)

Backend:
- Developer profile: enable subagent delegation, increase max_iterations to 35
- share_file: updated description + manual with absolute path requirement and URL sharing
- persona.txt: instructions for quote replies and GFM table format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Audit and trim system prompts (~470 tokens saved) ...

persona.txt:
- Shortened personality paragraph (~30% cuts, no content loss)
- Removed duplicate list_tools instruction
- Removed hardcoded 'developer' profile rule (handled by dynamic profiles block)
- Condensed EXECUTION MODES fundamental blockers to one sentence
- Moved sub-agent briefing boilerplate here (single source of truth)
- Trimmed REFLECTION section (tool description handles the how)
- Removed redundant RESPONSE HYGIENE explanation sentence
- Moved 'never assume file exists' into EXECUTION DISCIPLINE
- Removed DOCUMENTATION section

profiles (all three):
- Replaced ~100-token sub-agent briefing boilerplate with pointer to persona
- developer: removed data persistence code block (covered by _template.py)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add reflect tool: three parallel expert perspectives ...

ReflectTool runs Critic / Pragmatist / Detailer advisors concurrently
via asyncio.gather() + AIHelper.ask(). Each role has a distinct system
prompt designed to produce genuinely different analysis:
- Critic: challenges assumptions, surfaces risks and logical gaps
- Pragmatist: finds the simplest path, cuts unnecessary complexity
- Detailer: spots missing requirements, edge cases, ambiguities

Parameters: situation (required), assumptions (required list — the key
input that forces Navi to surface implicit beliefs), tried (optional).

Registered as a builtin with AIHelper injection. Added to all three
profiles. Persona updated with guidance on when to use it (complex or
ambiguous tasks before planning, or when stuck mid-execution).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add profile discoverability: list_profiles tool + system prompt injection ...

- AgentProfile: new short_description (1-line) and full_description (dict
  with specialization / when_to_use / key_tools) fields
- All 3 profile configs: structured descriptions added; list_profiles added
  to enabled_tools
- _build_system_prompt: now accepts full AgentProfile; injects compact
  "Available profiles" block into every system prompt so Navi always knows
  what other profiles exist and when to switch — dynamically, no hardcoding
- ListProfilesTool: new built-in; returns structured per-profile details
  (specialization, when_to_use, key_tools); accepts optional profile_id
  for single-profile lookup
- registry: register list_profiles_tool after profiles registry is built

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix Ollama connection leak and empty message bug in agent ...

- _iter_stream_guarded: track chunk_task as nullable, cancel in finally
  block to prevent zombie HTTP connections accumulating under load
- Final turn: use `content or None` so empty text isn't saved to DB
- client/index.html: point to new Vue webclient build
- profiles: add email_manager tool

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add autonomous execution mode; clarify code_exec runs locally ...

persona.txt: EXECUTION MODES section — autonomous mode triggered by user phrase,
handles obstacles independently, only stops on fundamental blockers.

server_admin, developer profiles: explicit note that code_exec / terminal /
filesystem run on the LOCAL machine, never on remote hosts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move orchestration from persona to profiles; tune per-profile delegation strategy ...

persona.txt now contains only: identity, profile switching, workspace,
response hygiene, memory, and documentation. All orchestration instructions
removed from the global scope.

Each profile gets its own orchestration model:
- secretary: full orchestrator — delegate any 2+ tool-call sub-task to agents,
  scratchpad as blackboard, todo for milestone tracking
- server_admin: heavy orchestrator — one agent per host / per concern,
  parallel delegation, diagnose-before-act discipline
- developer: builder + research delegation — implementation always inline,
  spawn only for large API/codebase research tasks

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove smart_home profile — to be re-added properly later ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Restructure profiles: directory-based format with config.json + system_prompt.txt ...

Each profile is now a subdirectory under navi/profiles/ containing:
  config.json       — model, temperature, enabled_tools, and other settings
  system_prompt.txt — raw system prompt, editable without touching Python

Added navi/profiles/loader.py for auto-discovery of profile directories.
Removed individual profile .py files (secretary, server_admin, smart_home, developer).
profiles/__init__.py now simply calls load_profiles_from_dir() at import time.

New profiles can be added by creating a directory with the two required files —
no Python changes needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Migrate storage to PostgreSQL with SQLite fallback; misc fixes ...

- Add PgSessionStore (asyncpg pool) and PgMemoryStore replacing aiosqlite
- Keep SqliteSessionStore + SqliteMemoryStore for zero-dependency quick start
- Selection logic in deps.py: DATABASE_URL set → PG, else → SQLite
- Add asyncpg>=0.29 to dependencies; add DATABASE_URL / DB_PATH to config
- Add RESPONSE HYGIENE rule to persona: never echo tool output or plan state
- Add developer profile user tools: weather, internal_monitor
- Update README: developer profile, DB section, current tool/profile state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add delete_tool: trash-based tool removal with restore support ...

Moves tool files to tools/.trash/ instead of deleting permanently.
Actions: remove (trash + unregister), restore (recover + re-register), list.
Data files are intentionally left in place on both remove and restore.
Available only in the developer profile.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix tool_manual leak: restore to server_admin and smart_home ...

tool_manual was accidentally removed alongside write_tool, but they are
unrelated. persona.txt references tool_manual globally — all profiles
must have it to avoid prompt/toolset mismatch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add developer profile; replace write_tool pattern with direct filesystem approach ...

- New TestToolTool: runs a user tool's execute() from disk in isolation,
  returns result or full traceback. No stale module cache — always fresh import.
- New developer profile: full architecture knowledge in system prompt
  (format rules, file locations, workflow, data persistence, common mistakes),
  test_tool + reload_tools + filesystem/terminal/code_exec toolset, spawn_agent
  for API research only.
- Remove write_tool and reload_tools from server_admin and smart_home profiles.
- persona.txt: drop SELF-EXTENSION block; add one-liner to switch to developer
  profile when the user asks to create/edit a tool.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Consolidate memory_search/save/forget into single memory tool ...

Three separate tools → one tool with action enum (save/search/forget/list).
Reduces tool-slot pressure; same functionality, same MemoryStore backend.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add memory_save tool for proactive fact persistence ...

Navi previously had no way to write to memory mid-conversation — she
could only search and forget. Facts were extracted automatically after
sessions went idle for 30+ min, so important context shared by the user
could be lost or delayed.

- New MemorySaveTool (navi/tools/memory_save.py): upsert a fact by
  category/key/value; overwrites existing key so no separate forget needed
- Registered as builtin alongside memory_search/memory_forget
- Added to all three profiles (secretary, server_admin, smart_home)
- persona.txt: explicit "call memory_save immediately when..." guidance
  so Navi saves stable facts as they arrive, not only post-session

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add share_file tool and session-lifetime file storage ...

Session file directories now live until the session is deleted, not
24h TTL. Cleanup loop only removes orphaned dirs (session gone from DB).

New share_file tool: copies any file to the session directory and returns
a clickable download URL. Navi can call this after generating any file
the user will want to keep.

New GET /sessions/{id}/files/{filename} endpoint serves files with
correct Content-Disposition (inline for images/HTML/PDF, attachment
for everything else).

Added PUBLIC_URL config key for building correct download links behind
reverse proxies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

self edited

Review and tighten all system prompts ...

persona.txt:
- PLANNING: threshold now 'plan has 2+ steps' instead of '2+ tool calls'
- MEMORY: remove mandatory session-start memory_search (was conflicting with
  planning order); replace with contextual trigger rules
- SCRATCHPAD: add 'if you've written anything to it' qualifier before read
- DELEGATION: clarify sequential spawning is fine; tighten when-not-to-spawn

secretary: trim redundant execution discipline, add 'test in code_exec before
writing to disk' rule, profile-specific scratchpad sections

server_admin: add explicit diagnostic workflow (gather → diagnose → act),
profile-specific scratchpad sections, expanded safety and delegation guidance

smart_home: add 'read-before-act' rule (check entity state before modifying),
profile-specific scratchpad sections, tighten safety rules

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Strengthen Navi planning/delegation, unify toolsets, isolate subagent scratchpad ...

persona.txt:
- DELEGATION: 'default to spawning, not to doing inline' — stronger default,
  clearer triggers, explicit when-not-to-spawn rules
- PLANNING: ties automatic planning phase to mandatory todo(op='set') as first
  tool call; reconciles pre-loop plan with in-loop execution discipline
- SCRATCHPAD: new section — when to write, section naming conventions,
  mandatory read before final answer

Profiles (secretary, server_admin, smart_home):
- All three now share the same 18-tool set (each file independent)
- planning_enabled=True on all three
- scratchpad and web_search added to smart_home
- System prompts updated with scratchpad/todo execution discipline sections

agent.py run_ephemeral:
- Each subagent gets a unique session ID (subagent_<uuid>) for scratchpad
  isolation — parallel or sequential subagents no longer share working notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add planning phase and scratchpad tool for smarter task execution ...

- ScratchpadTool: session-scoped working notepad with named sections
  (write/append/read/clear). Lets Navi capture intermediate findings
  between tool calls instead of losing track of them.

- Planning phase: when profile.planning_enabled=True, a fast pre-loop
  LLM call (think=False, no tools) outlines a numbered plan before
  any actions are taken. The plan is injected into session context as
  an assistant message so the model naturally continues from it.

- PlanReady event + plan_ready WebSocket message + plan card in UI
  (green-tinted, collapsible, mirroring thinking card design).

- secretary and server_admin profiles: planning_enabled=True,
  scratchpad added to enabled_tools, system prompts updated with
  explicit execution discipline instructions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add stop button and fix context compression hang ...

Stop generation:
- Client: send button toggles to red ■ during streaming; sends {type:stop} via WS
- Server: _stream_recv concurrently reads incoming messages during streaming using
  asyncio.wait — stop signal is handled immediately without polling
- Cooperative stop via asyncio.Event (current_stop_event ContextVar): agent breaks
  out of LLM async-for cleanly so aclose() fires → Ollama stream closes gracefully,
  model stays in VRAM. No task.cancel() which would eject the model.
- StreamStopped event propagates through run_stream/run_ephemeral; sub-agents stop
  via the same shared stop_event inherited through task context

Context compression fix:
- compress_context passes think=False to llm.complete() — no extended reasoning
  during summarization which caused GPU hang
- Input truncated to 12k chars before sending to summarizer
- LLMBackend.complete() / OllamaBackend.complete() accept think: bool | None override

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add ssh_exec to secretary profile ...

Was missing, causing 'tool not found' when Navi tried to SSH from secretary mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add switch_profile tool for automatic profile switching ...

Navi can now switch her own profile mid-session when the task domain
changes. The new profile (tools + system prompt) takes effect from the
next user message. Injected with session_store + profile_registry like
SpawnAgentTool. Added to all profiles' enabled_tools and persona.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>