navi-web: search backends, three-at-a-time batches, and pages read as markdown ...

web_search runs SearXNG first, DuckDuckGo with one retry after a real failure,
and Brave last; measured 0.2-1.4s through SearXNG against 1.7-11.8s on
scraping. Both tools take one item or up to three and run them concurrently —
three pages went from 17.3s to 1.9s, three searches from 3.5s to 0.6s — each
under its own deadline, so one hung backend or page is named in the output
instead of erasing what the others already returned. Text is capped at 20k for
one page and 40k for a batch. A call that opened nothing now raises: a green
journal line carrying "success": false is how a server whose browser could not
launch at all looked like a working one, and how 604 search calls in two weeks
reported no errors.

web_view returns the main content as markdown, not innerText after a tag
kill-list. The old path could not tell navigation from prose (462 of the asyncio
page's 1018 lines were shorter than 40 characters), collapsed tables into one
line per thousand characters — news.ycombinator.com came back as four lines —
marked no headings and dropped every link. trafilatura now picks the content by
density, and its html output keeps the whitespace inside <pre> that its markdown
output throws away, so the fragment is rendered here: # headings, fenced code
with its indentation, table rows joined by |, and up to 40 links lifted into a
"Links on this page" appendix rather than sprinkled through the sentences.
Threads are read again with comments when the extractor keeps less than a
quarter of the visible text; if it still finds nothing, the old visible-text
read answers and metadata.extractor says which one did.

trafilatura goes into the server's own venv, which deploy/install.sh does not
build — docs/mcp.md and deploy/UPDATE.md carry the install line. The tests load
the server by path, so it also joins Navi's dev extra.

deps: declare html2text, which tools/gmail.py imports ...

The import worked on the server only because the package had been
installed into the venv by hand; a fresh sync would have pruned it and
tools/gmail.py would have stopped loading while tools/enabled.json kept
naming gmail. Reload now reports that drift, but the fix is to declare
the dependency. Lock gains html2text and nothing else.

handbook alignment: profile.updated mirror, health contract, Synapse gateway ...

Handbook gaps closed (10-platform auth/health/notifications):

- webhooks: handle user.profile_updated — mirror the gnexus-auth profile
  into navi_users (name, contact fields, avatar_url — new column, boot
  migration); role/permissions keep their dedicated events
- auth: avatar_url now flows through the login upsert and the API-token
  resolution path
- /health: status is now the aggregate of the sub-checks (degraded embed
  or hive no longer reports plain ok); embed probe cached for 10 s; body
  grew the machine-readable checks{} map, legacy embed/hive payloads kept
- Synapse: s2s delivery gateway — POST /webhooks/synapse (both spellings),
  per-user delivery secrets (synapse_targets, Fernet-encrypted, shown
  once), verification via gnexus-synapse v0.1.2 client lib, idempotent
  event ids; deliveries are verified and logged for now — navi generates
  no outbound events by decision; web-push stays as the local browser
  channel
- webclient settings: Synapse deliveries panel (add/revoke targets)

agent parallelism: background tools, bg spawn_agent, parallel tool batches, message queue ...

- backgroundable tools (terminal/ssh_exec/peer/spawn_agent/code_exec) detach
  via TaskManager with per-session/global/spawn caps, rate limit and TTL
- completion delivery both ways: task_update event (out-of-band) + pending
  notes injected into the next turn; tasks tool (list/check/wait/cancel)
- parallel tool-call batches (profile-level gate, off by default) with
  ToolStarted up-front, tagged event mux, call-order results, one save,
  plus dangling tool_call repair on session load
- user message queue instead of busy error: message_queued frame,
  back-to-back drain on the same socket, headless fallback on disconnect
- pin mcp<2 (v2 renames FastMCP with breaking API changes)
- docs: tasks.md (new), websocket/api/agent/config updates, tasks manual,
  spawn_agent background param, persona contract section

PWA: installable webclient, offline shell, web push ...

Installability:
- public/manifest.webmanifest (standalone, theme #16161E) + PNG icons
  generated from logo.svg (regular + maskable, served via /images mount)
- index.html: manifest link, theme-color, apple-touch-icon

Offline shell:
- hand-rolled sw.js (no workbox): navigation = network-first (3s race)
  with cached-shell fallback + background refresh (a stale cached shell
  would 404 on entry chunks after a deploy); /assets/* cache-first
  (content-hashed); /images/* cache-first capped; /api,/ws,/auth,/push,
  /content pass-through
- vite closeBundle plugin stamps __NAVI_BUILD_VERSION__ (digest of
  index.html + asset names) into dist/sw.js; sw.js served no-store so
  every deploy reactivates the SW and activation evicts old caches
- SW registration in main.js, PROD only (dev HMR untouched)
- OfflineBanner (useOnline composable) over the app shell

Web push (VAPID, pywebpush):
- navi/push/ package: push_subscriptions table (postgres, boot-time DDL),
  PushSubscriptionStore, PushService (async fan-out, to_thread sends,
  404/410 prunes dead endpoints, per-session cooldown)
- routes: GET /push/vapid-key, POST/DELETE /push/subscribe (auth-gated)
- trigger in orchestrator run_agent + run_recall: push on StreamEnd when
  no WebSocket client watches the session; fire-and-forget, never
  disturbs the run; anonymous fallback only when auth is off
- client: usePush composable + Notifications settings panel (enable/
  disable via PushManager.subscribe with the server VAPID key)
- notification click focuses the app at /#<session_id> (hash routing
  opens the right chat); payload body is a markdown-stripped <=140-char
  preview

NAVIVAPID keys empty = push fully disabled (graceful, like other optional
integrations). dist/ artifacts committed per repo convention.

Tests: pytest push store/service/routes/trigger (+23), vitest usePush
(83 webclient tests green). Full suite 1143 passed.

fix: declare rich and python-dateutil — imported directly by the TUI and time_parser ...

Same masking pattern as pillow: the fat dev venv had both (rich via
textual, dateutil via matplotlib), fresh installs did not. dateutil is
a guarded lazy import so it only degraded time parsing silently; rich
is imported unguarded across clients/terminal/.

fix: declare pillow — PIL is imported directly (agent.py, image_view.py) ...

Pillow reached local dev venvs transitively via unrelated dev packages
(CairoSVG, matplotlib, reportlab), masking a missing direct dependency.
On a fresh install the server crashed at import: ModuleNotFoundError:
No module named 'PIL'.

fix: require gnexus-gauth under its real distribution name ...

pip (recent versions) validates that a direct-reference dependency name
matches the built distribution metadata. The gnexus-auth-client-py repo
produces a distribution named gnexus-gauth (import: gnexus_gauth), so
declaring it as gnexus-auth-client-py made pip discard the source and
fail install on the deployment server. The URL is unchanged; only the
requirement name now matches the metadata.

server: NAVI_WEBCLIENT_ENABLED gate + navi-server launcher ...

NAVI_WEBCLIENT_ENABLED=false removes every web-facing route (/, /assets,
/images, /content-viewers, /content, /admin panel, /debug*) and skips
the navi_ui MCP server — its only consumer is the webclient. The REST
API and WS the terminal client uses stay fully intact, and so does the
/admin/* JSON API (still gated by require_admin). Static mounts use
check_dir=False so a tree without webclient/dist imports fine.

New navi-server console script (navi/server.py): uvicorn launcher that
reads NAVI_HOST/NAVI_PORT from settings — the systemd unit needs no
hardcoded values.

Default stays true (full web UI), so nothing changes for dev setups;
subprocess route tests cover both modes.

Add MIT license ...

Project is going open source under a permissive MIT license.

- LICENSE: MIT, Copyright (c) 2026 Eugene Sukhodolskiy
- pyproject.toml: license field, OSI classifier, allow-direct-references for the git dependency
- README.md: license pointer
- .gitignore: ignore .env, venv, build artifacts, local db/workspace files

Navi Code: Phase 3 — Textual TUI skeleton (OpenCode-style) ...

- Add clients/terminal/tui package with micro-architecture:
  - events, context, chat_model, ws_bridge, permissions engine.
  - widgets: ChatPanel, StatusPanel, InputBox.
  - renderers: user/assistant messages, thinking, tool calls, errors,
    markdown, plain — registry-based and extensible.
  - slash commands: /help, /new, /sessions, /switch, /profile, /thinking,
    /compact, /quit — registry-based and extensible.
- Wire navi-code to launch TUI by default; keep click-CLI via --raw.
- Add textual>=0.70 dependency.
- Add TUI smoke tests via Textual Pilot.
- Add docs/plan_navi_code_tui.md with full Phase 4/5 roadmap.

Tests: 463 passed, 1 skipped (excluded unrelated websocket test).

Co-Authored-By: Claude <noreply@anthropic.com>

Navi Code: Phase 2 — CLI terminal client, tests, docs ...

- Add clients/terminal package: config, state, REST API wrappers, WebSocket
  client, renderer, and click-based interactive CLI.
- Wire navi-code console script via pyproject.toml.
- Add unit and WebSocket integration tests for the terminal client.
- Update docs/profiles.md, docs/config.md, README.md with navi_code profile
  and default-profile instructions.
- Add docs/navi_code.md setup guide and docs/navi_code_cli.md usage reference.
- Fix lint in new test files and test_auth_disabled.py.

Tested: 459 passed, 1 skipped (excluded unrelated websocket test).

Co-Authored-By: Claude <noreply@anthropic.com>

Migrate 3D modeling tools to standalone MCP server navi-3d ...

Phase 0 — Fix MCP error propagation:
- McpClient.call_tool now returns (output, is_error) tuple
- McpTool.execute sets success=False when MCP reports isError=True
- Fixes silent failures where MCP errors looked like success

Phase 1 — Create mcp-server-navi-3d:
- New standalone MCP server in mcp-servers/navi-3d/
- Tools: compile_scad, render_stl, lint_scad
- Session-scoped path resolution via SESSION_FILES_DIR env
- Anti-escape validation for security
- Includes tests/unit/test_scad_analyze.py

Phase 2 — Remove from Navi core:
- Deleted navi/tools/{model_3d,render_3d,scad_lint}.py
- Removed from registry.py builtins and navi/tools/__init__.py
- Updated pyproject.toml testpaths to exclude mcp-servers/

Phase 3 — Wire into Navi config:
- Added navi-3d block to mcp_servers.json (SSE on :8002)
- Updated modeler_3d profile: mcp_servers + updated system_prompt
- All old tool names replaced with mcp_navi-3d_* equivalents

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add MCP server support and fix memory tools user isolation ...

MCP integration:
- New navi/mcp/ package: client, manager, config, tools
- ToolRegistry learns register_external() for MCP tools
- reload_tools reconnects MCP servers on hot reload
- New built-in mcp_status tool
- Startup/shutdown wiring for MCP connections
- 12 new tests (unit + integration with real stdio server)

Memory tools fix:
- memory, memory_save, memory_search, memory_forget now read
  current_user_id from tool context and pass it to MemoryStore
- Fixes invisible facts for authenticated users

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add missing playwright dependency to pyproject.toml ...

playwright is used by navi/tools/web_view.py but was not declared.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add gnexus-auth-client-py and cryptography to pyproject.toml dependencies ...

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Remove SQLite legacy support ...

SQLite is no longer supported; PostgreSQL is now required.

- Delete navi/core/sqlite_session_store.py
- Delete navi/memory/sqlite_store.py
- Remove SqliteSessionStore from navi/core/__init__.py exports
- deps.py: drop SQLite fallback, raise RuntimeError if DATABASE_URL missing
- config.py: remove db_path setting
- pyproject.toml & requirements.txt: drop aiosqlite dependency
- .gitignore: remove navi.db entry
- tech_debt_review_2026-04-29.md: mark #8 as REMOVED

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Migrate storage to PostgreSQL with SQLite fallback; misc fixes ...

- Add PgSessionStore (asyncpg pool) and PgMemoryStore replacing aiosqlite
- Keep SqliteSessionStore + SqliteMemoryStore for zero-dependency quick start
- Selection logic in deps.py: DATABASE_URL set → PG, else → SQLite
- Add asyncpg>=0.29 to dependencies; add DATABASE_URL / DB_PATH to config
- Add RESPONSE HYGIENE rule to persona: never echo tool output or plan state
- Add developer profile user tools: weather, internal_monitor
- Update README: developer profile, DB section, current tool/profile state

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Major feature batch: visibility, planning, file uploads, streaming ...

- stream_complete(): streaming with tools for all LLM turns — thinking
  now streams as ThinkingDelta/ThinkingEnd in real-time during tool-
  selection turns, not just on the final response
- todo built-in tool: session-scoped plan manager (set/view/update/clear);
  persona + all profiles updated with mandatory planning instructions
- TurnThinking event: sub-agent thinking forwarded to parent sink as a
  collapsible block in the spawn_agent card
- File uploads: non-image files uploaded via XHR, shown as badges in
  message bubble; SVG treated as regular file (not base64 image)
- session_files: POST /sessions/{id}/files, TTL cleanup, forbidden exts
- WebSocket reconnect: _AgentRun broadcast pattern, re-attach mid-stream
- UI: favicon, sidebar logo, turn-thinking cards, subagent thinking blocks,
  token counter, draft persistence, file progress bar
- Removed AgentNote (content is always None alongside tool_calls)
- Ollama stream_complete: tool_calls captured from non-final chunk (done=False)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Persistent sessions and client module refactor ...

Server:
- SqliteSessionStore replaces InMemorySessionStore as default backend
- Sessions survive server restarts (stored in navi.db)
- DB_PATH configurable via .env

Client:
- Split monolithic app.js into ES modules:
  js/api.js    — REST calls
  js/ws.js     — WebSocket wrapper (WsClient class)
  js/chat.js   — message area DOM helpers
  js/sidebar.js — session list and header helpers
  js/app.js    — state, wiring, boot
- Active session persisted in localStorage — restored on page reload

Unrestricted terminal mode and SSH tool ...

- Terminal: TERMINAL_ALLOWED_COMMANDS=* (default) runs via shell,
  supports pipes, redirects, subshells; allowlist mode still available
- FS_ALLOWED_PATHS expanded to cover /home /etc /var /opt
- New ssh_exec tool: execute commands on remote hosts via asyncssh,
  supports named connections (ssh_hosts.json) and inline user@host
- ssh_hosts.json gitignored, ssh_hosts.json.example added as reference
- ssh_exec added to server_admin and smart_home profiles

Add web chat client ...

- Single-page chat UI served as static files from FastAPI
- Sidebar with profile selector, new chat button, session list
- WebSocket streaming: shows tool calls and text deltas in real time
- Typing indicator, streaming cursor, auto-resize textarea
- Session history loads on switch

Initial implementation of the agent system core ...

- FastAPI server with REST API and WebSocket streaming
- Modular LLM backend abstraction (Ollama implemented, OpenAI stub)
- Tool system: web_search (ddgs), filesystem, http_request, code_exec, terminal
- Agent profiles: smart_home, server_admin, secretary
- Tool-calling loop with concurrent tool execution
- In-memory session store with SessionStore ABC for future persistence
- Registry pattern for tools, profiles, and backends
- Orchestrator stub as foundation for multi-agent scenarios