Remove SQLite legacy support ...

SQLite is no longer supported; PostgreSQL is now required.

- Delete navi/core/sqlite_session_store.py
- Delete navi/memory/sqlite_store.py
- Remove SqliteSessionStore from navi/core/__init__.py exports
- deps.py: drop SQLite fallback, raise RuntimeError if DATABASE_URL missing
- config.py: remove db_path setting
- pyproject.toml & requirements.txt: drop aiosqlite dependency
- .gitignore: remove navi.db entry
- tech_debt_review_2026-04-29.md: mark #8 as REMOVED

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Stability fixes batch — tech debt review 2026-04-29 ...

Critical:
- Concurrent WS run race guard (#1)
- Tool task cancellation on generator teardown (#2)
- StopAsyncIteration kills fallback chain (#3)
- Session loading race with _lastLoadId guard (#4)
- ContentCard .match() crash on non-string result (#5)
- Image data type guard in buildMessageList (#6)

High:
- Cap WS replay buffer at 500 events (#7)
- Deduplicate memory extraction task with asyncio.Lock (#9)
- TTL-based fallback blacklisting (5 min) (#10)
- Subagent tool exception isolation (#11)
- Inline image size/count validation on WS (#12)
- Clean up orphaned file on DB insert failure (#13)
- Deep watch streamingMsg for auto-scroll (#14)
- WS_SCHEME wss:// support for HTTPS (#15)
- Sending guard against duplicate message sends (#16)
- Global unhandledrejection listener in API layer (#17)

Medium:
- Cap planning_logs at 20 entries (#22)
- Store cleanup_loop task reference (#23)
- BaseException → Exception in _run_with_sentinel (#24)
- Propagate SystemExit in agent loop (#25)
- Configurable output_reserve_tokens (#26)
- Always reloadSession on session_sync (#30)
- FIFO queue for confirm dialogs (#31)
- Reset body.overflow on ImageLightbox unmount (#32)
- try/finally in fallback copy (#33)
- _isConnecting guard in WS send() (#34)

Low:
- Lazy-init deps.py singletons (#36)
- Replace __import__ with direct imports (#38)
- Preserve token count 0 in ollama.py (#39)
- Clear orphaned streamingMsg on reconnect reload (#43)
- Escape single quote in UserMessage (#44)
- Polyfill-free findLast replacement (#48)
- Match <table> tags with attributes in markdown (#49)
- Attach copy buttons only when msg.done (#50)
- Fix hasMeta falsy-0 bug (#53)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add dedicated CPU embedding server for memory backfill ...

- Install Ollama CPU-only on 192.168.1.168 server
- Pull nomic-embed-text:latest on server
- Create systemd service ollama-embed.service (0.0.0.0:11434)
- Add embedding_ollama_host / embedding_ollama_api_key to config.py
- Update deps.py to build separate embedding backend when host configured
- Update backfill_embeddings.py to use dedicated embedding backend
- Add _generate_embeddings batch helper and backfill_embeddings to store.py
- Backfilled 119 existing facts with embeddings

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Wire pgvector semantic search into memory system ...

- Add vector(768) column + HNSW index to memory_facts

- Add LLMBackend.embed() with Ollama + fallback implementation

- MemoryStore: cosine-distance search with ILIKE fallback

- New memory tool params: source, confidence, expires_days, source_context

- Update extractor, sqlite_store, deps wiring

- Add pgvector to requirements

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add context providers: dynamic system message injection per LLM call ...

- navi/context_providers/ registry + built-in public_url provider (global, always injected)
- context_providers/ user directory, hot-reloaded via reload_tools
- AgentProfile.context_providers field for per-profile opt-in providers
- Agent._collect_context_injections() called before every tool-calling loop
- reload_tools now reloads both user tools and user context providers
- manuals/write_context_provider.md for Navi, docs/context_providers.md reference

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add Ollama multi-server fallback with in-memory blacklisting ...

- New FallbackOllamaBackend (navi/llm/fallback.py): tries servers and
  models in priority order; on LLMConnectionError blacklists the server
  for the process lifetime, on LLMModelNotFoundError blacklists the
  (server, model) pair — eliminates latency from repeated failed probes
- OllamaBackend now raises typed LLMConnectionError / LLMModelNotFoundError
  instead of bare LLMBackendError; accepts list[str] | str | None for model
- AgentProfile.model changed from str to list[str] (str auto-normalised);
  all profiles updated to ["gemma4:31b-cloud", "gemma4:26b-a4b-it-q4_K_M"]
- New config field OLLAMA_BACKENDS_FILE: path to [{host, api_key?}] JSON;
  when set, registry creates FallbackOllamaBackend instead of OllamaBackend
- ollama_backends.json template added (gitignored — contains API key)
- current_model ContextVar type widened to list[str] | str | None

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Use gemma4 cloud model by default

Support Ollama Cloud API key

Autonomous reasoning improvements: budget, anchoring, anti-stall, validation ...

- AgentProfile: per-profile thinking mechanics flags (think_enabled,
  iteration_budget_enabled, goal_anchoring, anti_stall, step_validation,
  planning_reflect, adaptive_replan) — all profiles updated in config.json
- Iteration budget: inject remaining iterations into context so model knows
  when to wrap up; urgency levels at ≤7 and ≤3 remaining
- Goal anchoring: inject original goal + todo state every N iterations to
  prevent drift on long tasks
- Anti-stall: two signals — no todo progress for N iterations, or identical
  tool calls repeated N times; warning injected into context
- Todo step validation: marking done requires a validation field describing
  how result was verified; failed gets a soft nudge with tip for re-planning
- stream_complete: add think param to base class, ollama and openai backends
- Summarizer: raise max_tokens 1024→3000, expand system prompt with
  user-preferences section and verbatim-value instructions
- Compression card: persist to session.messages (is_compression flag on
  Message), show expandable summary in webclient with markdown body
- ToolResult.to_message_content: always include output on failure so
  tracebacks and error details reach the model (fixes silent Error: None)
- Developer profile: fix subagent profile secretary→developer, add write_tool
  to subagent_tools, clarify write_tool vs filesystem in system prompt

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Planning phases, context compression, and tool improvements ...

Agent:
- Planning now a 3-phase async generator: Analysis → Execution plan → AIHelper critic
- Yield PlanningStatus events before each phase (UI progress labels)
- Phase 1 runs with think=True for deeper analysis
- Phase 2 includes available tool list so executor assignments are accurate
- Phase 3: independent critic pass validates and corrects TOOL: names against real tool list
- Planning converted from list return to async generator (fixes token accounting)

Backend:
- Context compression threshold: 80% → 70% to trigger earlier
- Compressor summary prompt: structured sections (goal, work state, key facts, outputs, errors)
- Terminal output capped at 5000 chars to prevent context flooding
- Web search: region=wt-wt for DDG, country=ALL for Brave, language=all for SearxNG
- Scratchpad: mandate writing a 'goal' section at start of multi-step tasks
- secretary max_iterations: 40→25, temperature: 0.7→0.5
- server_admin max_iterations: 40→20

Webclient:
- ThinkingCard strips <thought> XML tags leaked by Ollama
- planning_status WS event wired to chat.onPlanningStatus()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix gmail auth: read credentials from settings, not os.environ ...

pydantic-settings loads .env only into the Settings object — it does not
populate os.environ. Added gmail_address and gmail_app_password fields to
Settings; gmail tool now reads from settings instead of os.environ.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Migrate storage to PostgreSQL with SQLite fallback; misc fixes ...

- Add PgSessionStore (asyncpg pool) and PgMemoryStore replacing aiosqlite
- Keep SqliteSessionStore + SqliteMemoryStore for zero-dependency quick start
- Selection logic in deps.py: DATABASE_URL set → PG, else → SQLite
- Add asyncpg>=0.29 to dependencies; add DATABASE_URL / DB_PATH to config
- Add RESPONSE HYGIENE rule to persona: never echo tool output or plan state
- Add developer profile user tools: weather, internal_monitor
- Update README: developer profile, DB section, current tool/profile state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add explicit output token budget for summarizer (context_summary_max_tokens) ...

Previously there was no num_predict set for the summarization LLM call,
so Ollama used its server default (often 128 tokens — very short summaries).

- Add max_tokens param to LLMBackend.complete() and OllamaBackend (→ num_predict)
- Add context_summary_max_tokens: int = 1024 to config
- Thread it through compress_context() and CompressionWorker

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Expand summarization budget for better context quality ...

- _MAX_SUMMARY_INPUT_CHARS: 12k → 24k chars (2x input fed to summarizer)
- context_keep_recent: 10 → 8 turns (2 more turns go into each summary batch)
- Summarizer prompt: replace "Be brief" with "Be thorough" — capture code/config
  snippets and enough detail to continue the conversation without original messages

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add share_file tool and session-lifetime file storage ...

Session file directories now live until the session is deleted, not
24h TTL. Cleanup loop only removes orphaned dirs (session gone from DB).

New share_file tool: copies any file to the session directory and returns
a clickable download URL. Navi can call this after generating any file
the user will want to keep.

New GET /sessions/{id}/files/{filename} endpoint serves files with
correct Content-Disposition (inline for images/HTML/PDF, attachment
for everything else).

Added PUBLIC_URL config key for building correct download links behind
reverse proxies.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Improve filesystem, web search, context guard, and subagent narration ...

filesystem: add find (glob), info (stat), move, append actions; read now
supports offset/limit with hard 1MB guard; list shows sizes, dates,
optional recursion.

web_search: retry DDG across auto/html/lite backends; add optional Brave
Search API and SearXNG fallbacks configured via .env.

agent: fix ContextTooLargeError to surface as Navi response instead of
raw system error; fix _check_context_size to calculate from remaining
budget (window - output_reserve) rather than a fixed 92% threshold.

persona: add ReAct narration instruction to subagent briefing template.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix LLM hang: stop button during prefill, context guard, timeouts ...

Root cause: during prefill (processing input tokens), Ollama emits no
HTTP chunks. The `async for chunk in stream_complete()` loop body never
executes, so stop_event is never checked — Stop button has no effect.
Same issue with complete() calls (planning, compression): blocking await
with no cancellation path.

Fixes:

_iter_stream_guarded() (agent.py, module-level):
  Wraps any stream_complete() generator. Polls stop_event every 1s while
  waiting for the next chunk using asyncio.wait() — so Stop works even
  during multi-minute prefill. On stop or timeout, calls aclose() on the
  generator which closes the HTTP connection to Ollama → generation halts
  → GPU drops to idle. Applied to both run_stream() and run_ephemeral().

_check_context_size() (Agent method):
  Estimates context tokens (chars/4 + 500 per image) before every LLM
  call. Raises ContextTooLargeError (new NaviError subclass) at 92% of
  ollama_num_ctx — before Ollama ever receives the request.

_run_planning() timeouts:
  Both complete() calls (phase 1 and 2) wrapped with asyncio.wait_for().
  Timeout logged and planning skipped gracefully — execution continues.

New config (config.py):
  llm_complete_timeout = 120s
  llm_stream_first_chunk_timeout = 180s  (prefill budget)
  llm_stream_chunk_timeout = 60s         (inter-token budget)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Major feature batch: visibility, planning, file uploads, streaming ...

- stream_complete(): streaming with tools for all LLM turns — thinking
  now streams as ThinkingDelta/ThinkingEnd in real-time during tool-
  selection turns, not just on the final response
- todo built-in tool: session-scoped plan manager (set/view/update/clear);
  persona + all profiles updated with mandatory planning instructions
- TurnThinking event: sub-agent thinking forwarded to parent sink as a
  collapsible block in the spawn_agent card
- File uploads: non-image files uploaded via XHR, shown as badges in
  message bubble; SVG treated as regular file (not base64 image)
- session_files: POST /sessions/{id}/files, TTL cleanup, forbidden exts
- WebSocket reconnect: _AgentRun broadcast pattern, re-attach mid-stream
- UI: favicon, sidebar logo, turn-thinking cards, subagent thinking blocks,
  token counter, draft persistence, file progress bar
- Removed AgentNote (content is always None alongside tool_calls)
- Ollama stream_complete: tool_calls captured from non-final chunk (done=False)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Fix NAVI_PERSONA not loading: move persona to persona.txt ...

python-dotenv cannot parse multi-line env values that contain blank lines or
lines resembling key=value assignments (like the write_tool code examples).
The persona was silently dropped, leaving Navi with no identity.

Solution: navi_persona_file setting (default empty) — if set, the persona is
read from the specified file at startup via a model_validator. The .env now
points to persona.txt instead of embedding the multi-line string.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Separate display history from LLM context; formalize worker system ...

Architecture change:
- session.messages: full display history, never modified by compression
- session.context: what the LLM sees, may be compressed by workers
- System messages go only into context (not display history)
- Image injections (synthetic) go only into context
- User/assistant/tool messages go into both

SQLite: add context column with backward-compat migration
(empty context → initialized from messages on load)

Workers (navi/workers/):
- Worker ABC + WorkerContext + WorkerResult (base.py)
- CompressionWorker: compresses session.context when above threshold
- build_default_workers() returns [CompressionWorker()]
- Agent accepts workers list, runs them after StreamEnd
- Workers injected via deps.py get_workers() (lru_cached singleton)
- WebSocket agent construction also receives workers

Compressor: compress_context() now takes context[], not messages[]
Config: context_keep_recent 6 → 10
Agent: _run_workers() collects events from all workers and yields them

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add context compression: rolling summarization when context fills up ...

Mechanism:
- After streaming ends, if context_tokens >= threshold (80% of num_ctx),
  compress old turns into a summary message using the same LLM
- Partition: keep system msg + last N turns verbatim (default 6);
  everything older goes to the summarizer
- Tool call groups (assistant + tool results) never split across boundary
- Existing summary messages folded into new compression pass — no stack growth
- Summary stored as Message(role=user, is_summary=True) after system msg
- On failure: logged, session left unchanged (non-fatal)

New files:
- navi/core/compressor.py: should_compress, partition_messages,
  compress_session (pure logic, testable without agent)

New config (navi/config.py):
- context_compression_enabled: bool = True
- context_compression_threshold: float = 0.80
- context_keep_recent: int = 6
- context_summary_temperature: float = 0.3

New agent event: ContextCompressed(messages_before, messages_after)
Message.is_summary: bool field marks compressed history blocks

Client:
- context_compressed WS event → subtle inline notice in message list
- loadHistory: is_summary messages rendered as collapsible summary cards
- style.css: .summary-card, .compression-notice

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add context token counter: 64k default, live UI display ...

- config: ollama_num_ctx default 8192 → 65536
- LLMChunk: add prompt_tokens / completion_tokens fields
- OllamaBackend.stream: populate token counts from final chunk
  (prompt_eval_count + eval_count when chunk.done)
- StreamEnd: add context_tokens and max_context_tokens
- Agent.run_stream: capture token counts, pass to StreamEnd
- websocket: include context_tokens / max_context_tokens in stream_end
- index.html: split chat-header into title span + token-counter span
- sidebar.js: updateChatHeader targets #chat-header-title, not innerHTML
- app.js: updateTokenCounter() shows "X/Y (Z%) tokens", colors:
  gray <50%, amber 50–79%, red ≥80%
- style.css: .token-counter, .warn, .danger styles

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add thinking/reasoning streaming support ...

Enable Ollama think param and stream reasoning chunks to client.
New agent events: ThinkingDelta, ThinkingEnd. Config gains ollama_think
and ollama_num_ctx settings. WebSocket protocol updated accordingly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Persistent sessions and client module refactor ...

Server:
- SqliteSessionStore replaces InMemorySessionStore as default backend
- Sessions survive server restarts (stored in navi.db)
- DB_PATH configurable via .env

Client:
- Split monolithic app.js into ES modules:
  js/api.js    — REST calls
  js/ws.js     — WebSocket wrapper (WsClient class)
  js/chat.js   — message area DOM helpers
  js/sidebar.js — session list and header helpers
  js/app.js    — state, wiring, boot
- Active session persisted in localStorage — restored on page reload

Filesystem tool: unrestricted mode by default ...

FS_ALLOWED_PATHS=* (default) allows any path, consistent with terminal.
Path check is now lazy (per-request) instead of module-level,
so config is always fresh after restart.
~ expansion added.

Unrestricted terminal mode and SSH tool ...

- Terminal: TERMINAL_ALLOWED_COMMANDS=* (default) runs via shell,
  supports pipes, redirects, subshells; allowlist mode still available
- FS_ALLOWED_PATHS expanded to cover /home /etc /var /opt
- New ssh_exec tool: execute commands on remote hosts via asyncssh,
  supports named connections (ssh_hosts.json) and inline user@host
- ssh_hosts.json gitignored, ssh_hosts.json.example added as reference
- ssh_exec added to server_admin and smart_home profiles

Initial implementation of the agent system core ...

- FastAPI server with REST API and WebSocket streaming
- Modular LLM backend abstraction (Ollama implemented, OpenAI stub)
- Tool system: web_search (ddgs), filesystem, http_request, code_exec, terminal
- Agent profiles: smart_home, server_admin, secretary
- Tool-calling loop with concurrent tool execution
- In-memory session store with SessionStore ABC for future persistence
- Registry pattern for tools, profiles, and backends
- Orchestrator stub as foundation for multi-agent scenarios