Fix Ollama connection leak and empty message bug in agent ...

- _iter_stream_guarded: track chunk_task as nullable, cancel in finally
  block to prevent zombie HTTP connections accumulating under load
- Final turn: use `content or None` so empty text isn't saved to DB
- client/index.html: point to new Vue webclient build
- profiles: add email_manager tool

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add autonomous execution mode; clarify code_exec runs locally ...

persona.txt: EXECUTION MODES section — autonomous mode triggered by user phrase,
handles obstacles independently, only stops on fundamental blockers.

server_admin, developer profiles: explicit note that code_exec / terminal /
filesystem run on the LOCAL machine, never on remote hosts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Move orchestration from persona to profiles; tune per-profile delegation strategy ...

persona.txt now contains only: identity, profile switching, workspace,
response hygiene, memory, and documentation. All orchestration instructions
removed from the global scope.

Each profile gets its own orchestration model:
- secretary: full orchestrator — delegate any 2+ tool-call sub-task to agents,
  scratchpad as blackboard, todo for milestone tracking
- server_admin: heavy orchestrator — one agent per host / per concern,
  parallel delegation, diagnose-before-act discipline
- developer: builder + research delegation — implementation always inline,
  spawn only for large API/codebase research tasks

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Remove smart_home profile — to be re-added properly later ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Restructure profiles: directory-based format with config.json + system_prompt.txt ...

Each profile is now a subdirectory under navi/profiles/ containing:
  config.json       — model, temperature, enabled_tools, and other settings
  system_prompt.txt — raw system prompt, editable without touching Python

Added navi/profiles/loader.py for auto-discovery of profile directories.
Removed individual profile .py files (secretary, server_admin, smart_home, developer).
profiles/__init__.py now simply calls load_profiles_from_dir() at import time.

New profiles can be added by creating a directory with the two required files —
no Python changes needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Migrate storage to PostgreSQL with SQLite fallback; misc fixes ...

- Add PgSessionStore (asyncpg pool) and PgMemoryStore replacing aiosqlite
- Keep SqliteSessionStore + SqliteMemoryStore for zero-dependency quick start
- Selection logic in deps.py: DATABASE_URL set → PG, else → SQLite
- Add asyncpg>=0.29 to dependencies; add DATABASE_URL / DB_PATH to config
- Add RESPONSE HYGIENE rule to persona: never echo tool output or plan state
- Add developer profile user tools: weather, internal_monitor
- Update README: developer profile, DB section, current tool/profile state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add delete_tool: trash-based tool removal with restore support ...

Moves tool files to tools/.trash/ instead of deleting permanently.
Actions: remove (trash + unregister), restore (recover + re-register), list.
Data files are intentionally left in place on both remove and restore.
Available only in the developer profile.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix tool_manual leak: restore to server_admin and smart_home ...

tool_manual was accidentally removed alongside write_tool, but they are
unrelated. persona.txt references tool_manual globally — all profiles
must have it to avoid prompt/toolset mismatch.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add developer profile; replace write_tool pattern with direct filesystem approach ...

- New TestToolTool: runs a user tool's execute() from disk in isolation,
  returns result or full traceback. No stale module cache — always fresh import.
- New developer profile: full architecture knowledge in system prompt
  (format rules, file locations, workflow, data persistence, common mistakes),
  test_tool + reload_tools + filesystem/terminal/code_exec toolset, spawn_agent
  for API research only.
- Remove write_tool and reload_tools from server_admin and smart_home profiles.
- persona.txt: drop SELF-EXTENSION block; add one-liner to switch to developer
  profile when the user asks to create/edit a tool.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add explicit output token budget for summarizer (context_summary_max_tokens) ...

Previously there was no num_predict set for the summarization LLM call,
so Ollama used its server default (often 128 tokens — very short summaries).

- Add max_tokens param to LLMBackend.complete() and OllamaBackend (→ num_predict)
- Add context_summary_max_tokens: int = 1024 to config
- Thread it through compress_context() and CompressionWorker

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Expand summarization budget for better context quality ...

- _MAX_SUMMARY_INPUT_CHARS: 12k → 24k chars (2x input fed to summarizer)
- context_keep_recent: 10 → 8 turns (2 more turns go into each summary batch)
- Summarizer prompt: replace "Be brief" with "Be thorough" — capture code/config
  snippets and enough detail to continue the conversation without original messages

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Consolidate memory_search/save/forget into single memory tool ...

Three separate tools → one tool with action enum (save/search/forget/list).
Reduces tool-slot pressure; same functionality, same MemoryStore backend.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add memory_save tool for proactive fact persistence ...

Navi previously had no way to write to memory mid-conversation — she
could only search and forget. Facts were extracted automatically after
sessions went idle for 30+ min, so important context shared by the user
could be lost or delayed.

- New MemorySaveTool (navi/tools/memory_save.py): upsert a fact by
  category/key/value; overwrites existing key so no separate forget needed
- Registered as builtin alongside memory_search/memory_forget
- Added to all three profiles (secretary, server_admin, smart_home)
- persona.txt: explicit "call memory_save immediately when..." guidance
  so Navi saves stable facts as they arrive, not only post-session

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Expose compression summary as collapsible debug card in chat UI ...

ContextCompressed event now carries the full summary text produced by the
LLM. Compression notice in chat becomes a <details> element showing
message count (before→after) with the summary expandable on click.
Rendered as markdown via marked.js.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add share_file tool and session-lifetime file storage ...

Session file directories now live until the session is deleted, not
24h TTL. Cleanup loop only removes orphaned dirs (session gone from DB).

New share_file tool: copies any file to the session directory and returns
a clickable download URL. Navi can call this after generating any file
the user will want to keep.

New GET /sessions/{id}/files/{filename} endpoint serves files with
correct Content-Disposition (inline for images/HTML/PDF, attachment
for everything else).

Added PUBLIC_URL config key for building correct download links behind
reverse proxies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve filesystem, web search, context guard, and subagent narration ...

filesystem: add find (glob), info (stat), move, append actions; read now
supports offset/limit with hard 1MB guard; list shows sizes, dates,
optional recursion.

web_search: retry DDG across auto/html/lite backends; add optional Brave
Search API and SearXNG fallbacks configured via .env.

agent: fix ContextTooLargeError to surface as Navi response instead of
raw system error; fix _check_context_size to calculate from remaining
budget (window - output_reserve) rather than a fixed 92% threshold.

persona: add ReAct narration instruction to subagent briefing template.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix LLM hang: stop button during prefill, context guard, timeouts ...

Root cause: during prefill (processing input tokens), Ollama emits no
HTTP chunks. The `async for chunk in stream_complete()` loop body never
executes, so stop_event is never checked — Stop button has no effect.
Same issue with complete() calls (planning, compression): blocking await
with no cancellation path.

Fixes:

_iter_stream_guarded() (agent.py, module-level):
  Wraps any stream_complete() generator. Polls stop_event every 1s while
  waiting for the next chunk using asyncio.wait() — so Stop works even
  during multi-minute prefill. On stop or timeout, calls aclose() on the
  generator which closes the HTTP connection to Ollama → generation halts
  → GPU drops to idle. Applied to both run_stream() and run_ephemeral().

_check_context_size() (Agent method):
  Estimates context tokens (chars/4 + 500 per image) before every LLM
  call. Raises ContextTooLargeError (new NaviError subclass) at 92% of
  ollama_num_ctx — before Ollama ever receives the request.

_run_planning() timeouts:
  Both complete() calls (phase 1 and 2) wrapped with asyncio.wait_for().
  Timeout logged and planning skipped gracefully — execution continues.

New config (config.py):
  llm_complete_timeout = 120s
  llm_stream_first_chunk_timeout = 180s  (prefill budget)
  llm_stream_chunk_timeout = 60s         (inter-token budget)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve planning: two-phase pipeline and orchestrator discipline ...

agent.py:
- _run_planning() now runs two sequential LLM calls:
  Phase 1 (analysis): reformulate task, identify subtasks and unknowns;
  skip immediately if DIRECT.
  Phase 2 (execution plan): assign each subtask an executor —
  TOOL/AGENT/SELF — using a structured ## Plan format.
  Phase 2 context = analysis (embedded in system prompt) + last user
  message only; full history excluded to keep focus on plan structure.
- Warn in logs when plan lacks TOOL/AGENT/SELF executor assignments.

persona.txt:
- MANDATORY sequence: step 0 = scratchpad init before anything else;
  todo tasks must mirror plan steps exactly (same order, same executors).
- PLAN → EXECUTION BINDING: explicit rule — never switch an AGENT step
  to inline execution silently.
- SCRATCHPAD: initialize sections at task start, not after first tool call;
  write context to scratchpad before briefing subagents.
- Fix typo in BRIEFING ("sub-lagent" → "sub-agent").
- Replace stale Knowledge Retrieval Protocol with accurate one-liner.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

self edited

Review and tighten all system prompts ...

persona.txt:
- PLANNING: threshold now 'plan has 2+ steps' instead of '2+ tool calls'
- MEMORY: remove mandatory session-start memory_search (was conflicting with
  planning order); replace with contextual trigger rules
- SCRATCHPAD: add 'if you've written anything to it' qualifier before read
- DELEGATION: clarify sequential spawning is fine; tighten when-not-to-spawn

secretary: trim redundant execution discipline, add 'test in code_exec before
writing to disk' rule, profile-specific scratchpad sections

server_admin: add explicit diagnostic workflow (gather → diagnose → act),
profile-specific scratchpad sections, expanded safety and delegation guidance

smart_home: add 'read-before-act' rule (check entity state before modifying),
profile-specific scratchpad sections, tighten safety rules

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Strengthen Navi planning/delegation, unify toolsets, isolate subagent scratchpad ...

persona.txt:
- DELEGATION: 'default to spawning, not to doing inline' — stronger default,
  clearer triggers, explicit when-not-to-spawn rules
- PLANNING: ties automatic planning phase to mandatory todo(op='set') as first
  tool call; reconciles pre-loop plan with in-loop execution discipline
- SCRATCHPAD: new section — when to write, section naming conventions,
  mandatory read before final answer

Profiles (secretary, server_admin, smart_home):
- All three now share the same 18-tool set (each file independent)
- planning_enabled=True on all three
- scratchpad and web_search added to smart_home
- System prompts updated with scratchpad/todo execution discipline sections

agent.py run_ephemeral:
- Each subagent gets a unique session ID (subagent_<uuid>) for scratchpad
  isolation — parallel or sequential subagents no longer share working notes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Skip planning phase for simple/direct requests ...

The planning prompt now asks the model to respond with "DIRECT" if the
request doesn't need multiple steps. Added a regex fallback: if the
response has no numbered steps it's also discarded. This prevents plan
cards appearing for conversational replies that would just duplicate
the final message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add planning phase and scratchpad tool for smarter task execution ...

- ScratchpadTool: session-scoped working notepad with named sections
  (write/append/read/clear). Lets Navi capture intermediate findings
  between tool calls instead of losing track of them.

- Planning phase: when profile.planning_enabled=True, a fast pre-loop
  LLM call (think=False, no tools) outlines a numbered plan before
  any actions are taken. The plan is injected into session context as
  an assistant message so the model naturally continues from it.

- PlanReady event + plan_ready WebSocket message + plan card in UI
  (green-tinted, collapsible, mirroring thinking card design).

- secretary and server_admin profiles: planning_enabled=True,
  scratchpad added to enabled_tools, system prompts updated with
  explicit execution discipline instructions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix WebSocket state corruption preventing messages after first reply ...

Replace concurrent WS reads (_stream_recv + recv_task.cancel()) with
HTTP stop endpoint (POST /sessions/{id}/stop). Cancelling a background
receive_text() task corrupted Starlette's WS state, breaking all
subsequent receives. Now the WS has a single reader at all times.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add stop button and fix context compression hang ...

Stop generation:
- Client: send button toggles to red ■ during streaming; sends {type:stop} via WS
- Server: _stream_recv concurrently reads incoming messages during streaming using
  asyncio.wait — stop signal is handled immediately without polling
- Cooperative stop via asyncio.Event (current_stop_event ContextVar): agent breaks
  out of LLM async-for cleanly so aclose() fires → Ollama stream closes gracefully,
  model stays in VRAM. No task.cancel() which would eject the model.
- StreamStopped event propagates through run_stream/run_ephemeral; sub-agents stop
  via the same shared stop_event inherited through task context

Context compression fix:
- compress_context passes think=False to llm.complete() — no extended reasoning
  during summarization which caused GPU hang
- Input truncated to 12k chars before sending to summarizer
- LLMBackend.complete() / OllamaBackend.complete() accept think: bool | None override

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Terminal icon ⚡, subagent max_iterations 20→100 ...

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix ssh_exec: is_closing() → is_closed() (asyncssh API) ...

SSHClientConnection has no is_closing() — that's asyncio Transport.
The correct public method is is_closed().

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix save(): persist profile_id to DB ...

profile_id was never included in the UPDATE statement — only set on
initial INSERT. Profile switches appeared to work in-memory but reverted
on page reload or server restart.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix profile switch: reload tools/schema after switch_profile tool call ...

switch_profile updates profile_id in DB, but run_stream() held a stale
local session object — the final save would overwrite the change, and
subsequent LLM calls in the same turn still used the old tool schemas.

After each tool-call iteration, compare DB profile_id with the local
session object. On mismatch: update session.profile_id, reload profile,
tools, tool_schemas, and llm backend so the next LLM call gets the
correct schema and the final save preserves the new profile.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add ssh_exec to secretary profile ...

Was missing, causing 'tool not found' when Navi tried to SSH from secretary mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>