swarm stage 2: peer-to-peer channel — /peer endpoints + peer tool ...

Server side (navi/api/routes/peer.py, port 8099):
- GET /peer/hello — open discovery ping (name, uuid prefix, version)
- GET /peer/status — PSK; identity, uptime, machine facts, hive view
- POST /peer/ask — PSK; one-shot agent run under PEER_ASK_PROFILE
  (default server_admin) answers, one concurrent ask at a time
- loop guard: answering agent runs without the peer tool (deterministic
  recursion cut) + own-uuid asks refused with 409
- verify_swarm_key in navi/swarm.py accepts .swarm-key.previous
  (rotation window, constant-time)

Client side (navi/tools/peer.py):
- peer tool: list (hive book, stale cache fallback marked), status,
  ask by swarm name; asks go direct peer-to-peer, hive never on the
  message path
- audit events on both sides (peer.ask_sent/received/answered/failed)

peer tool added to server_admin, navi_code, developer profiles.

swarm stage 1: instance identity, PSK, hive registry, announce loop ...

- navi/identity.py: adjective-animal names + uuid in instance.json
  (generated at install, renameable by hand)
- navi/swarm.py: HiveAnnouncer - periodic POST /announce to the hive
  with non-blocking reachability tracking (transitional logs, /health
  export, hive_status context provider reports outages to the agent)
- hive/: standalone FastAPI address book (SQLite, port 8087, PSK via
  X-Swarm-Key with .swarm-key.previous rotation window, TTL online
  status, host from client IP, port from payload). Not installed or
  started by default - run manually on the main server.
- ports moved to 8099 (API) / 8098 (UI MCP)
- install.sh: PYTHONIOENCODING=utf-8 in the systemd unit, generates
  instance.json and .swarm-key on fresh installs

fix: ASCII-safe startup log messages + force UTF-8 stdout in the systemd unit ...

A unit on an old distro without a UTF-8 locale gets latin-1/ascii
stdout; structlog's print of the em-dash in the webclient-disabled
message raised UnicodeEncodeError and killed the app at startup
(Application startup failed, crash-loop restart counter 28). The unit
now sets PYTHONIOENCODING=utf-8 so any future non-ASCII log line is
safe too.

server: NAVI_WEBCLIENT_ENABLED gate + navi-server launcher ...

NAVI_WEBCLIENT_ENABLED=false removes every web-facing route (/, /assets,
/images, /content-viewers, /content, /admin panel, /debug*) and skips
the navi_ui MCP server — its only consumer is the webclient. The REST
API and WS the terminal client uses stay fully intact, and so does the
/admin/* JSON API (still gated by require_admin). Static mounts use
check_dir=False so a tree without webclient/dist imports fine.

New navi-server console script (navi/server.py): uvicorn launcher that
reads NAVI_HOST/NAVI_PORT from settings — the systemd unit needs no
hardcoded values.

Default stays true (full web UI), so nothing changes for dev setups;
subprocess route tests cover both modes.

Merge branch 'master' into feature/navi-code ...

# Conflicts:
#	navi/api/websocket.py
#	navi/config.py
#	navi/core/agent.py
#	navi/core/orchestrator.py
#	navi/main.py

security: critical batch 1 — RCE/XSS/CORS/webhook hardening ...

Backend:
- auth/deps: fix refresh-lock clock race (cleanup now monotonic like the cache)
- core/registry: add missing structlog logger (NameError on fallback path)
- config: GNAUTH_WEBHOOK_SECRET, NAVI_ALLOWED_ORIGINS (+list property)
- webhooks: HMAC-SHA256 signature verification (503 unconfigured in auth mode,
  unsigned+warning in no-auth mode, 403 bad/stale signature, 400 bad JSON)
- main: CORS from NAVI_ALLOWED_ORIGINS in auth mode (fail-fast on empty),
  eval router behind require_admin, /debug only registered in no-auth mode
- auth routes: mobile-done sid validation (32 hex) + CSP header + safe JS
  escaping (reflected XSS); Secure cookie flag via helper (https base URL)
- websocket: anonymous WS rejected in auth mode (closes legacy-session RCE);
  socket registered only after access checks; stop_session auth gate
- messages: REST agent start now takes session lock + busy flag (409 on
  active run), contextvars reset in finally

Webclient:
- useMarkdown: DOMPurify.sanitize on all rendered markdown (stored XSS),
  image-URL scheme whitelist, delegated error listener (no inline onerror)
- html.html artifact viewer: sandbox without allow-same-origin (opaque origin)
- tests: DOMPurify runs under jsdom (happy-dom Node.prototype.nodeName getter
  breaks DOMPurify); useWebSocket tests get localStorage stub + ui-kit alias

Tests: pytest 1049 passed, 1 skipped; vitest 61 passed

session store: lazy persistence — no empty sessions in DB ...

POST /sessions no longer inserts a DB row. The Session is registered in an
in-memory _pending registry on PgSessionStore and only upserted on the first
save() (first user message / meaningful state change). Empty sessions that
never receive a message never reach the DB and vanish on server restart or
via the hourly pending sweep.

- pg_session_store: _pending dict + lock; create() registers, get() checks
  _pending first, save() upserts the sessions row (INSERT ... ON CONFLICT)
  and pops _pending so the session_messages FK is satisfied; sweep_pending()
  drops abandoned entries; pending_sweep_loop() background task.
- main.py: start/cancel pending_sweep_loop in lifespan.
- tests: 7 new tests for create/get/save/list/sweep semantics; updated
  existing save() test comments for the upsert.

feat: integrate navi_ui MCP server (card_grid + form) into master ...

Port the internal navi_ui MCP server from the vmkdemo branch (it never
landed on master). The server exposes render_component, which returns a
structured JSON envelope; navi/mcp/tools.py extracts metadata.ui_component
onto the role="tool" message, and the webclient renders the component
(card_grid, form) inline inside the assistant turn.

Backend
- navi/mcp/ui_server/: FastMCP server + component registry with card_grid
  and form components (pydantic-validated payloads, LLM-friendly schema docs)
- mcp_servers.d/navi_ui.json: streamable_http config, group "ui"
- config.py: navi_ui_mcp_enabled/host/port flags
- main.py: start UI server in lifespan (task + wait-for-ready + cancel)
- mcp/tools.py: navi_ui envelope parsing; "Error:" results surface as
  failed tool calls so the UI card is not green
- orchestrator.py + agent.py: run_stream(hidden=) for form submissions
  (single is_display=False, is_context=True user message)
- api/websocket.py: extract _start_agent_run helper, add form_submit
  branch that delivers submitted form values as a hidden user message
- profiles/secretary: enable navi_ui "ui" group (agent + subagent)
- .env.example: NAVI_UI_MCP_* flags
- tests/unit/mcp/test_ui_server.py

Webclient
- components/ui/{registry.js,CardGrid.vue,Form.vue}: auto-discovered
  renderers (snake_case <-> PascalCase aliasing)
- components/messages/UiComponentCard.vue: wrapper rendered in
  AssistantMessage when entry.kind === 'ui_component'
- stores/chat.js: extract ui_component from tool_call metadata in both
  live stream and history-replay paths
- composables/useWebSocket.js: note that ui_component rides tool_call
- tests/unit/components/ui/* (17 tests)
- dist rebuilt

Excluded from the vmkdemo port: the stale single-file navi/mcp/ui_server.py
duplicate, the realtor profile, and the vmk_data server (unrelated real
estate work that was interleaved with navi_ui on vmkdemo).

Co-Authored-By: Claude <noreply@anthropic.com>

Add NAVI_AUTH_ENABLED switch for optional auth ...

- Add navi_auth_enabled setting (default true) to navi/config.py and .env.example
- When disabled, treat every request as anonymous admin user (id='anonymous')
- Create/update fixed anonymous navi_users row on startup
- Bypass OAuth/cookie/API-token resolution in navi/auth/deps.py
- Update /auth/status to return {enabled, configured}
- Log security warning on startup when auth is disabled
- Update webclient: skip fetchMe/login screen, show Local mode footer,
  expose /admin link, warn in API keys panel
- Rebuild webclient production bundle
- Add unit and integration tests for no-auth mode
- Update docs: auth.md, config.md, api.md, api_tokens.md, sessions.md,
  websocket.md, mechanics.md, index.md

Co-Authored-By: Claude <noreply@anthropic.com>

Suppress noisy MCP SDK health-check logs ...

mcp.server.lowlevel.server logs "Processing request of type ListToolsRequest"
on every health-check poll. Raise its threshold to WARNING so we only
see real errors.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add API token auth system for headless/micro clients ...

Backend:
- navi/auth/_ddl.py: add api_tokens table with boot-time migration
- navi/auth/deps.py: _resolve_user now falls back to X-Api-Token header
  and ?api_token query param for WebSocket auth
- navi/auth/__init__.py: add ApiToken pydantic model
- navi/api/routes/api_tokens.py: CRUD endpoints (POST/GET/DELETE)
- navi/main.py: wire api_tokens router

Frontend:
- webclient/src/App.vue: add #settings hash routing
- webclient/src/components/settings/: SettingsView, ApiKeysPanel,
  CreateKeyModal with copy-to-clipboard flow
- webclient/src/api/index.js: token CRUD API functions
- webclient/src/stores/apiTokens.js: Pinia store
- webclient/src/components/sidebar/AppSidebar.vue: settings link
- webclient/src/composables/useWebSocket.js: append ?api_token= when
  localStorage token is present

Tests:
- tests/unit/auth/test_api_tokens.py: 10 unit tests covering token
  resolution (header + query param), revoke, missing/revoked tokens,
  orphan users, and CRUD endpoints

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix session file download URL — reverse legacy redirect ...

The webclient requests files at /api/sessions/{id}/files/{name},
but the actual endpoint lives at /sessions/{id}/files/{name}.
The old legacy redirect pointed the wrong way (/sessions → /api),
which always 404'd because /api/sessions/... was never registered.

- Replace legacy redirect with /api/sessions/... → /sessions/... (307)
- 307 preserves the request method and cookies during the redirect

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract single shared Database pool, eliminate 4 duplicated pool creations ...

- Create navi/db.py::Database managing one asyncpg pool
- KvStore, PgSessionStore, MemoryStore, RecallScheduler now accept pool in constructor
- AppContainer holds Database, shutdown closes one pool instead of 4
- create_container creates one pool and passes it to all stores
- All tests updated to set _initialized=True on fakes to skip DDL

392 passed, 1 skipped

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Extract WebSocket business logic into AgentSessionOrchestrator ...

- Create navi/core/orchestrator.py with AgentSessionOrchestrator and SessionRun
- Orchestrator owns _runs, _busy_sessions, Agent creation, run_agent(), run_recall()
- Transport-agnostic: accepts notify callback from WebSocket handler
- WebSocket handler (websocket.py) now only does serialization/deserialization
- _fire_recall delegates to orchestrator.run_recall() instead of inline logic
- recall_scheduler_loop now accepts orchestrator parameter
- AppContainer gains .orchestrator field, created in create_container()
- deps.py: add get_orchestrator()
- Update integration tests for scheduler_loop and websocket unit tests

All 392 tests pass.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Replace global lazy singletons with explicit AppContainer + lifespan ...

- Create navi/core/container.py with AppContainer dataclass and create_container()
- Rewrite navi/api/deps.py: remove module-level singletons, add _container global
  fallback + set_container(), use _resolve_container() for all getters
- Replace @app.on_event with @asynccontextmanager lifespan in main.py
- Update routes to use Depends(get_scheduler) instead of calling get_scheduler()
- Fix FastAPI body parsing bug: remove dataclass parameters from Depends getters
  (FastAPI was treating AppContainer sub-dependencies as Body params, forcing
  embed=True on all endpoint body params and causing 422 errors)
- Update websocket.py to use _resolve_container() instead of get_registries()
- Update integration test fixtures to build AppContainer and call set_container()
- Remove obsolete tests/unit/test_startup.py (tests removed _on_startup)
- Fix test_scheduler_loop.py fixture (get_registries no longer exists)

All 387 tests pass (excluding websocket hang tests).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add legacy redirect for session files and fix STL viewer error handling ...

- Add /sessions/{id}/files/{filename} → /api/sessions/{id}/files/{filename}
  redirect in main.py for backward compatibility with old persisted URLs
- Fix STL viewer: wrap all renderer-dependent code in `if (renderer)`
  to prevent cascading errors when WebGL is unavailable
- Rebuild dist

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add self-recall (scheduled callback) system ...

Core features:
- schedule_recall tool: once/recurring/immediate callbacks
- manage_recall tool: cancel/skip/list scheduled recalls
- Natural-language time parser (ISO, relative, "tomorrow at 09:00")
- PostgreSQL-backed RecallScheduler with lazy pool init
- Background recall_scheduler_loop with asyncio.Semaphore(3)
- _busy_sessions guard prevents user messages during headless runs
- Agent.run() preserves thinking field for session history visibility
- API endpoints: GET/DELETE/POST for session recall, admin list
- Frontend: recall badge, filter, cancel/skip in sidebar and chat header
- Tests: parser, scheduler CRUD, tools, API, scheduler loop (53 tests)
- Manuals: schedule_recall.md and manage_recall.md

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

fix: wire test_mcp_tool into MCP manager startup injection ...

The startup loop in main.py assigns _mcp_manager only to tools that
have the attribute, but test_mcp_tool never declared it and mcp_status
used _manager instead of _mcp_manager — so both received None forever.

Changes:
- test_mcp_tool: add __init__(mcp_manager) with _mcp_manager, fallback
  to module-level import if startup wiring skipped
- mcp_status: rename _manager → _mcp_manager, same fallback
- registry.py: register create_mcp_server and test_mcp_tool builtins
- main.py: include test_mcp_tool in startup wiring loop
- client.py: add 5s timeout to disconnect cleanup

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add profile_id support to list_tools ...

ListToolsTool now accepts an optional `profile_id` parameter. When
provided, it returns only the tools enabled for that specific profile
(enabled_tools + user tools + resolved MCP tools). This helps the agent
preview what a profile can do before switching.

- Updated ListToolsTool.execute() with profile-aware filtering
- Patched _profile_registry in build_default_registries
- Patched _mcp_manager in main.py startup loop

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Wire MCP manager into sub-agent (spawn_agent) so MCP tools are available to sub-agents ...

- SpawnAgentTool: add mcp_manager parameter and pass it to Agent
- main.py: patch _mcp_manager into spawn_agent after MCP connect

Fixes: tool 'mcp_gnexus-book_*' not found in sub-agent sessions.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix graceful MCP disconnect during server shutdown ...

- McpClient.disconnect(): catch CancelledError / RuntimeError from
  anyio SSE transport teardown
- McpManager.disconnect_all(): shield gather from CancelledError
- main.py _on_shutdown(): catch CancelledError and RuntimeError
  explicitly (BaseException, not Exception)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add MCP server support and fix memory tools user isolation ...

MCP integration:
- New navi/mcp/ package: client, manager, config, tools
- ToolRegistry learns register_external() for MCP tools
- reload_tools reconnects MCP servers on hot reload
- New built-in mcp_status tool
- Startup/shutdown wiring for MCP connections
- 12 new tests (unit + integration with real stdio server)

Memory tools fix:
- memory, memory_save, memory_search, memory_forget now read
  current_user_id from tool context and pass it to MemoryStore
- Fixes invisible facts for authenticated users

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Add pagination, search, and sorting to admin sessions ...

Backend:
- Add count_all and search_list abstract methods to SessionStore
- Implement count_all and search_list in PgSessionStore (SQL with ILIKE)
- Implement count_all and search_list in InMemorySessionStore
- Update /admin/sessions to accept limit, offset, search, sort_by, sort_order
- Return {total, limit, offset, items} from /admin/sessions

Frontend:
- Add search input for sessions in admin panel
- Add clickable sortable column headers with asc/desc toggle
- Add pagination controls (prev/next, page size selector, item count)
- Debounce search input (300ms)

Tests:
- Add integration tests for pagination, offset, search, and sorting
- All 217 tests pass

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix DDL race condition and PostgreSQL syntax error on startup ...

- _ensure_auth_tables now uses raw asyncpg.connect instead of get_session_store,
  avoiding PgSessionStore._MIGRATE running before navi_users exists
- Move _ensure_auth_tables before ensure_tables in startup so navi_users is
  created before any foreign-key references
- Replace invalid "ADD CONSTRAINT IF NOT EXISTS" with DO $$ block for
  memory_facts_user_cat_key

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Multi-user auth via gnexus-auth OAuth + hybrid role/permission model ...

- Integrate gnexus-auth-client-py (GAuthClient) for OAuth flow, token refresh,
  and webhook parsing
- Add navi/auth/ package: User model, Fernet encryptor, client singleton,
  deps (get_current_user, require_admin, require_permission)
- New tables: navi_users, user_auth_sessions (auto-created on startup)
- Session/memory isolation by user_id with legacy NULL support
- Cookie-based auth proxy: /auth/login, /callback, /logout, /me
- Webhook receiver /webhooks/gnexus-auth handling user events, global logout,
  session revocation, role/permission changes
- Admin endpoints (/admin/*) gated by role + permissions
- Webclient auth store with isAdmin/hasPermission guards
- Admin-only profile filtering in /agents/profiles
- 200/200 tests passing

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix 3D modeling feedback: session context, CORS, and PNG workflow ...

- Embed session_id into the main system prompt so the model reliably
  sees the correct session files directory instead of hallucinating one.
- Add global CORS middleware to fix STL viewer iframe fetch errors.
- Switch modeler_3d PNG previews from content_publish to image_view
  so renders are debug-only for Navi, not user-facing cards.
- Remove literal {session_id} template from persona.txt.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Initialize embedding backend before health check

Architecture cleanup: old_webclient, SSH shutdown, todo decoupling, ContextVar reset ...

- Remove old_webclient directory and /static mount from main.py
- Add shutdown handler in main.py to close all SSH pooled connections
- Decouple agent.py from todo module internals:
  - Add public API to navi/tools/todo.py: get_task_snapshot, get_failed_steps,
    get_progress_message, set_tasks, render_todo_lines
  - Replace all direct _plans/_STATUS_ICON/_Task imports in agent.py with API calls
- Wrap run_ephemeral in try/finally to restore _sid_var and _model_var after subagent

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Stability fixes batch — tech debt review 2026-04-29 ...

Critical:
- Concurrent WS run race guard (#1)
- Tool task cancellation on generator teardown (#2)
- StopAsyncIteration kills fallback chain (#3)
- Session loading race with _lastLoadId guard (#4)
- ContentCard .match() crash on non-string result (#5)
- Image data type guard in buildMessageList (#6)

High:
- Cap WS replay buffer at 500 events (#7)
- Deduplicate memory extraction task with asyncio.Lock (#9)
- TTL-based fallback blacklisting (5 min) (#10)
- Subagent tool exception isolation (#11)
- Inline image size/count validation on WS (#12)
- Clean up orphaned file on DB insert failure (#13)
- Deep watch streamingMsg for auto-scroll (#14)
- WS_SCHEME wss:// support for HTTPS (#15)
- Sending guard against duplicate message sends (#16)
- Global unhandledrejection listener in API layer (#17)

Medium:
- Cap planning_logs at 20 entries (#22)
- Store cleanup_loop task reference (#23)
- BaseException → Exception in _run_with_sentinel (#24)
- Propagate SystemExit in agent loop (#25)
- Configurable output_reserve_tokens (#26)
- Always reloadSession on session_sync (#30)
- FIFO queue for confirm dialogs (#31)
- Reset body.overflow on ImageLightbox unmount (#32)
- try/finally in fallback copy (#33)
- _isConnecting guard in WS send() (#34)

Low:
- Lazy-init deps.py singletons (#36)
- Replace __import__ with direct imports (#38)
- Preserve token count 0 in ollama.py (#39)
- Clear orphaned streamingMsg on reconnect reload (#43)
- Escape single quote in UserMessage (#44)
- Polyfill-free findLast replacement (#48)
- Match <table> tags with attributes in markdown (#49)
- Attach copy buttons only when msg.done (#50)
- Fix hasMeta falsy-0 bug (#53)

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Fix system prompt leakage into chat history; polish content cards ...

Backend:
- websocket.py + agent.py: separate user-visible display_message from
  LLM user_message. System hints (image/file attachments) no longer leak
  into session.messages and appear after page reload.
- main.py: add ensure_tables() on startup so session_content table is
  created before first publish.
- profiles: add kimi-k2.6:cloud to all model lists as fallback.

Frontend:
- ContentCard.vue: remove border-radius, add scrollbar styles, fix
  metadata fallback parsing so cards survive page reload.
- content-viewers/*.html: add matching scrollbar styles.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>