protocol, clients: acks, queue notices and reconnect without duplicates ...

- client_id is echoed in message_accepted / message_queued; queue overflow
  broadcasts message_dropped to every socket watching the session
- stream_end / stream_stopped carry message_index, the global display rank
  that also stamps history rows
- webclient: cheap session_sync via /sessions/{id}/meta, the ack deadline is
  armed at the write and re-armed while a run is active, fatal 4003/4004 stop
  the reconnect loop instead of retrying forever
- TUI: WsBridge posts WsResumed / MessageWritten / SessionUnavailable, the app
  drops the half-written turn on reconnect, shows queued and not-delivered
  markers on the bubble, and reloads history only when the meta token changes
- dist rebuilt

core, webclient: cheap session lists, changed-row saves, smaller bundle ...

The sidebar asked for a count and a 60-char snippet and got every hot
message of every listed session: 2,097 rows / 4.4 MB for 507 sessions,
plus a full Pydantic hydration. list_page now aggregates counts and the
last displayed line in SQL (4.0 ms, 30 rows); get_meta folds its COUNT
into the same round trip.

save() runs after every streaming step and rewrote the whole hot
history each time — 1,016 UPDATEs, 46.8 ms, per call, growing with the
session. A fingerprint of the mutable columns lets it touch only the
rows that changed, which is normally the tail.

HTTP: content-hashed /assets are immutable for a year, gzip drops to
level 6 (same ratio within a couple of percent on JSON, half the CPU),
and session search gets pg_trgm GIN indexes when the extension exists.

Client: one shared highlight.js on lib/common plus the languages we
actually render (openscad included) instead of the full ~190-language
entry point, and the unused @tanstack/vue-virtual is gone.
Bundle 1566 -> 760 kB (gzip 495 -> 238 kB).

webclient: make session loads cheap, cancellable and honest ...

Opening a session moved ~1 MB of history JSON and a 1.5 MB bundle
uncompressed, then reloaded the whole transcript on every WebSocket
reconnect; a failed request wiped what was on screen.

Server: GZipMiddleware(1 KB) - one history page shrinks 5.1x, the SPA
bundle 3.2x.

Client:
- session_sync asks /meta instead of reloading history, and only when
  the view is idle;
- session loading keeps the previous transcript until the new data
  arrives, aborts the request it replaced, ignores stale responses and
  surfaces a retryable error banner;
- search-jump pagination no longer pages through the previous session's
  history;
- REST calls get a 30 s timeout and caller abort support;
- WS queue overflow rejects the message and marks the bubble "not sent"
  instead of silently dropping the oldest;
- tool card bodies mount on first open (a 38-card session drops ~1 MB of
  markup and ~25k elements);
- identical /sessions requests coalesce, the streaming watch is a cheap
  signature, the content-tool key is one store computed, useTime shares
  one 30 s ticker, connect() clears the retry timer.

e2e opens result cards explicitly now that bodies are lazy.

tools: split filesystem into read, write and file_ai ...

- filesystem_read: read, list, find, find_up, grep, info, exists, diff
- filesystem_write: write, append, edit, edit_lines, insert, mkdir, move, copy, delete
- file_ai: ask (was query), edit (was smart_edit)
- paged numbered reads (500 lines / 50 KB), raw copy mode, binary refusal
- nearest-match and numbered-copy hints on failed edits, context after edit
- guards: empty overwrite, non-empty delete, move/copy overwrite
- cross-tool redirects instead of silent wrong-action errors
- legacy 'filesystem' stays for archived sessions (compression, renderers, webclient)
- profiles, prompts, docs and manuals updated; TUI and webclient render the new tools

core: anchor restricted-profile boundaries in subagent prompts ...

Sub-agents of non-admin profiles replaced the parent system prompt with
their own worker prompt, so the "one directory, no network, no secrets"
rules never reached them. build_subagent_system_prompt now appends a
[Boundaries] block for every non-admin profile, whatever prompt was
chosen.

Align the profile prompts with the actual tool surface alongside it:
allow 3D sub-agents to draft geometry but not publish or decide, publish
a preview PNG with the STL in both 3D profiles, document the assistant's
remaining tools, canonicalise MCP names, and fix the developer/navi_code
spawn ending, the server_admin field list, the discuss memory_search
guidance, and the navi-3d/navi-web/gnexus-book instructions.

core: split the agent and context builder into packages ...

Agent becomes one class assembled from mixins: runtime (turn lifecycle),
streaming (the turn loop), tools (tool execution), compression (context
gates) and context (tool lists, backend lookup, window). The module-level
helpers and sentinels move to helpers.py.

ContextBuilder splits into helpers (pure truncation/freshness/prompt
blocks) and builder (the class). Both old modules become packages with
the same import surface, so navi.core.agent.Agent and
navi.core.context_builder.ContextBuilder keep working.

Method bodies are unchanged; tests now patch settings/log/should_compress
on the mixin that actually reads them.

api: split the sessions routes into a package ...

The 1064-line module becomes one /sessions router plus area modules:
common (models and helpers), listing, detail, files, messages, naming,
manage, recall. The package re-exports every view so existing imports
keep working.

Route set, order and status codes are unchanged; tests now patch
settings/asyncio where they are actually read (the submodules).

tools: split the filesystem tool into a package ...

`tools/filesystem.py` was 1145 lines: path/format/diff/AI-text helpers at the
top, then a `FilesystemTool` class with every action handler. Move the pure
layer out into `navi/tools/_filesystem/`:

* `helpers.py` — path checks, formatting, line numbering, unified-diff
  annotation, chunking, edit-op validation/application, AI prompts;
* `tool.py` — the dispatcher and its read/write/AI handlers.

The package is underscore-prefixed because `navi/tools/filesystem` also has
to stay available as the compatibility shim at the same import path. No
behavior change: the shim re-exports every top-level name the module had, and
the tests patch `settings` on the modules that now read it.

core: split the compressor into a package ...

`navi/core/compressor.py` was 1355 lines covering three different jobs:
deciding what a pass summarizes, building and calling the summarizer, and
the retry/truncate/persist wrapper around it. Move each to its own module:

* `compression/partition.py` — pure partitioning and the single
  `_plan_compression` decision shared with `would_compress`;
* `compression/summarize.py` — prompt formatting, the KEYS line, the
  meta-summary and `compress_context`;
* `compression/compressor.py` — `ContextCompressor` and its fallbacks.

`navi/core/compressor.py` stays as a compatibility shim that re-exports the
public surface, so imports elsewhere are untouched. Because each module now
binds its own `settings`, the tests that override settings patch all of them
(and `_meta_summarize`/`compress_context` are patched where they are read,
not on the shim). Adds a regression test for the hard-truncate trim inside a
single oversized turn, which the moved `_split_turn_tail` import silently
broke (ruff F821, no test hit that branch).

memory: fix review findings in unified search ...

- index each compression-dropped turn once: messages and context share
  objects, and earlier drops stay in the display history
- label raw excerpts against facts even when recall asks for scope=raw
- top up short vector results from the lexical index (NULL embeddings)
- count whole content words only and cap query terms at 12
- carry source_session_id on fact hits
- cap injected_chunk_ids at 100 entries
- keep a session delete successful when chunk cleanup fails
- stop logging query text in the memory paths
- stop fabricating similarity scores for recent chunks
- consult session_messages_archive in latest_summary_at

memory: one search across facts, raw history and sketches ...

Compression removes messages from the model's window, not from the
conversation: every compression path now writes the turns it demotes into
`memory_chunks` — verbatim, append-only, with the turn range, timestamps and
breadcrumb keys (the summarizer's `KEYS:` line plus a deterministic extractor,
which also covers the hard-truncate fallback that never sees a summarizer).

`memory_search` is the single read path over the three stores: curated facts
first, then raw excerpts by relevance, then the current session's sketches.
Every hit carries provenance (source, session, turn range, timestamp), and a
raw excerpt that a newer curated fact overrides is labelled outdated rather
than hidden — "we changed our mind, the old value was X" is often the
question. The agent also gets up to two relevant excerpts injected per turn
once the context holds a summary, deduped across turns.

Fixes found by running the pilot against a real database:
- `search_facts` returns ISO-string timestamps while chunks return datetimes;
  the freshness comparison raised TypeError and took the whole search down.
- an existing `memory_chunks` table created without pgvector never gained the
  `embedding` column (CREATE TABLE IF NOT EXISTS does not alter), and the
  INSERT failed on it; the DDL now adds the column, and a database without
  pgvector writes text-only rows found through FTS.

context: price images by area and detect backend-truncated prompts ...

Every window decision priced an inline image at a flat 500 tokens, while a
1024x1024 screenshot costs ~1026 on qwen3.6 and a 2048x2048 one ~4098. The
undercount kept sessions below the 90% trigger; Ollama then silently truncated
the prompt to about half the window (dropping the system prompt), and the
truncated prompt_eval_count was cached as the session baseline, pinning the
estimate at ~50% for good - the session never compressed while the model was
already told the context was full.

- image_tokens.py prices each image from its header (one token per 32x32 px,
  downscaled to <=2048^2, clamped to 256..4096); unreadable headers fall back
  to CONTEXT_IMAGE_TOKENS. Used by both estimators, the delta and
  check_context_size; the per-turn image budget fits images by their own cost.
- A prompt reported far below what was sent is a truncation: it is not cached
  as the baseline, and the post-turn worker compresses regardless of the
  threshold. The session keeps the honest token count.
- The pre-turn gate prices the incoming message and its images, which are not
  in session.context yet.
- The summarizer receives images under a window-scaled budget, and prompts
  carrying images stay out of the calibration fit (their bytes are not in
  chars and would skew it).
- Oversized user messages are capped for the LLM view (at least num_ctx/3),
  so a paste bigger than the window no longer dead-ends a session.
- The in-flight partition also trims by token budget: a gate that fires can
  now actually shrink a turn made of a few huge messages.

sessions: share a chat as a public read-only link ...

A session's owner can hand out one capability link that opens the display
history without an account: no input, no chat list, no artifacts panel.
The token lives in the URL hash, so it never reaches the server in a request
line or a Referer header.

The link is permanent by decision - no expiry, no revoke. Deleting the chat
removes the share row (FK cascade) and closes it, which is the only switch.

Backend: session_shares table (unique token, FK cascade), POST
/sessions/{id}/share issuing the session's one link (owner or admin; a
read-all permission may read another user's chat but not publish it), and
public GET /shared/{token}[/messages] serving display history only, with
session and owner ids left out of the payload.

Webclient: #share=<token> renders a standalone page with the owner's cards,
rating off, interactive UI components filtered out and scroll-up paging; a
share button in the chat header opens a dialog that creates the link.

Also fixes ChatHeader's rename, which called an undefined `api` and failed
silently.

llm: rotate the routing config under a running process ...

A key that lives in the central config is only worth centralizing if
publishing it does not cost a fleet restart. The backend bakes host and
Authorization header into its clients, so it now accepts a fresh
LLMConfig: clients are dropped and rebuilt on the next call, health
stays keyed by target name, and the probe follows the new hosts too.

The overlay's llm section is no longer always staged. It is re-parsed
against the local document and swapped live; only a backend without the
swap method or a document that fails to parse falls back to
restart_required.

overlay: ack the config again after a restart ...

A document that needed a restart was acked as 'staged, restart required'
and never updated afterwards: the machine restarted into the new config but
the hive kept showing the old ack. Startup now re-reports what it is running,
so the status view reflects reality instead of the last live refresh.

overlay: report llm and mcp changes as restart_required ...

A live refresh cannot swap LLM backends or the MCP pool — both are built
with the container — so a document that changes those sections was reported
as applied while the old wiring kept running. Compare the sections against
the previous document and stage them by name.

navi: overlay the hive config onto the instance's own ...

An instance that knows a hive with the swarm key starts from the hive
document: layers that apply to it are merged over its local config, fields
the hive does not mention stay local, and an empty answer means 'run exactly
your own config'. Settings, profiles, LLM routes and MCP servers are covered;
database URL, ports, auth switches and the channel's own settings can never be
overridden.

Startup fetches with a short timeout, falls back to the last validated overlay
on disk and refuses to crash-loop on a document that killed the previous start.
A live refresh applies secrets and timeouts in place (the frozen singleton is
written through one deliberate escape hatch), rebuilds profiles from their
local base and reports everything else as restart_required. The hive reaches
the instance by push, by the announce reply and by a retry sweep; every apply
is acked back, which is what the hive's status view reads.

hive: versioned central config with layered targeting ...

The hive now stores one config document for the fleet: a list of layers,
each with an optional match (names/instance_ids/tags), appended as immutable
versions. Publish, history, rollback and status are admin-only (admin key or
loopback); the swarm key reads config, so a compromised instance cannot
rewrite what the fleet runs.

Delivery has three paths that degrade into each other: a push notification,
a 'you are behind' hint in the announce reply, and a periodic sweep over
instances that still report an older hash. Acks are tracked per instance and
version, so the status view answers who took the config and who needs a nudge.

A layer list is validated as a whole; an empty list is a legal document that
means 'no overlay'.

profiles: teach the agent to scout profiles and tools before switching ...

Persona gains a reconnaissance order (list_profiles, then list_tools for the
real toolset, then tool_manual) and the switch-vs-delegate rule: switch when
the task itself is another profile's work, delegate a side piece with
spawn_agent. The profiles block is built in one place and shared with the
admin prompt debug view, and the stale 'next user message' claim is fixed.

list_tools applies the admin-only role gate to an explicitly named profile,
the same one list_profiles uses, so it cannot enumerate hidden profiles.

skills: re-authorize a replacement when the proposal is accepted ...

A proposal proved ownership when it was written, but the protocol can be
promoted to shared while it sits pending. Check again at the click: a shared
protocol is now admins only, a personal one belongs to its owner.

webclient: read the skills catalog as documents, split it into two tabs ...

Settings had one long panel mixing skills and protocols as raw markdown,
which was hard to read and wasted the width. Skills and Protocols are now
two tabs over one panel keyed by kind.

Selecting an entry shows the parsed document — triggers, preconditions,
steps, post-checks, escape hatches, body — and an Edit button switches to
the source, so the catalog is read the way the agent loads it. The
single-entry payload carries the parsed document; the list still does not,
so it stays cheap. SkillDocument is shared with the chat proposal card,
which now only draws its own frame and header.

skills: name the kind when a document already had one implied ...

A proposal is parsed as a protocol when it is stored, but accepting it
re-parsed the same text with no override, so a front matter that left
'kind' out made the catalog refuse a document the user had just agreed
to. The create path and the agent's own skill writes had the same gap.

prompt: give the methodologies catalog its rules in the system prompt

protocol: run a protocol as a committed plan, with a log of how it ended ...

A protocol was only text until now. The new `protocol` tool starts one: its
steps are spliced into the session's todo plan after the agent's current
position (nothing already done is reordered), a row appears in `protocol_runs`
with a snapshot of the version, steps and post-checks, and the run stays
`running` until it is closed.

`finish` refuses while any step of the run is open, refuses outright when one
failed, and requires a report — it is the run's record for the user. `cancel`
and `fail` require a reason and mark the rest of the run's steps `skipped`, so
the plan does not keep work that was never done. One run per session is active
(a partial unique index enforces it).

The commitment also has to survive the context: every model turn while a run is
open carries a system reminder with the protocol, its steps and the way to close
it, and the final-turn intercept now fires for an open run even when the todo
shows no open steps, so a bare-text turn cannot leave one running.

`protocol` is wired into assistant, developer, discuss and server_admin next to
`skills`; sub-agents get neither, because a protocol belongs to the session's
plan. The run log has no UI yet — `protocol(op='status')` is the current view.

skills: agent-written skills and protocol proposals with chat buttons ...

A skill the agent writes now lands in the catalog on its own; a protocol
never does. `skills(op="propose")` stores the document in `skill_proposals`
and returns a card through the tool result's `ui_component` metadata — no
`render_component` call, so profiles without `navi_ui` get the card too.

The decision is the server's: the browser sends `skill_proposal` over the
WebSocket, `SkillStore.decide_proposal` writes the catalog row from the
*stored* document, names the clicking user as the author, and answers with
`skill_proposal_decided`. The agent learns the outcome as a hidden user
message, the same path a `navi_ui` form submission takes.

`create`/`update` refuse protocol documents and point at `propose` with
`replaces_id`, so the version history of a protocol always starts from a
user's yes. The chat store now renders `ui_component` from any tool whose
result carries one, not only from `mcp__navi_ui__render_component`.

skills: catalog of skills and protocols with a settings tab ...

One registry for reusable methodologies: markdown with YAML front matter,
stored in PostgreSQL alongside a full edit history. A row is either a
`skill` (advice, adapt as needed) or a `protocol` (a committed order of
actions), and a protocol is refused on save unless it declares its steps,
post-checks and the three escape hatches — on_deviation, on_blocked,
on_already_done — because an honest run needs a way out when reality
differs from the plan.

- `SkillStore`: lazy DDL, uniqueness on (owner_id, kind, name), version
  bump with an appended `skill_versions` row, `content_hash` as the only
  hive-sync hook. Personal rows carry an owner; shared rows have an empty
  one and are writable by an admin.
- REST `/skills` (list/get/create/update/delete/versions) and the agent
  tool `skills` (search/load) that renders a compact document — numbered
  steps and escape hatches for a protocol, the instructions for a skill.
- Settings → Скилы и протоколы: catalog with search and kind/area
  filters, a document editor that shows the server's validation error
  verbatim, create from a template, delete, edit history, and the
  personal/shared switch for an admin.

Manuals stay where they are: tool knowledge, separate from this catalog.

todo: give plans their own table and recover them after a crash ...

Todo already survived restarts inside the generic session_store KV row, but
the plan there was opaque JSON with no timestamp: after a server crash the
agent resumed a step marked in_progress with no way to tell whether it had
ever been done.

session_todos (user_id, session_id, tasks JSONB, revision, updated_at) now
holds the plan. Existing rows are migrated on first use and the KV source is
deleted in the same step, so clearing a plan is not undone by the next
process start.

Each step gains provenance fields (source, step_id, protocol_run_id) for the
protocol work, and insert_steps() splices a block after the step the agent
has already reached instead of making it re-issue the whole plan.

get_progress_message() now flags a plan whose in_progress step predates this
process once per (session, revision): re-check the real result before
marking it done.

The todo op "clear" now uses the same user scope as load/save; it used to
prefer ctx.user_id and could delete a different row than the one written.

export: token-served age snapshot of dialogue for an n-gram consumer ...

Navi had no way to hand its history to another machine. A database dump
was the obvious idea and the wrong one: prod's 597 MB is mostly things
that must never leave (Fernet-encrypted MCP keys, user rows, attachments),
and the dialogue an n-gram table wants is 14 MB of it.

So the export is a fixed, versioned SELECT — role, content, session, seq,
ts — over both message tables. Default corpus is user+assistant only:
tool output is 52 MB of scraped web text and command logs, and system
messages are near-duplicate boilerplate; either would dominate the counts.
A rolling window takes whole sessions, newest first, up to 150 MB, so the
snapshot stays bounded as history grows.

The snapshot is age-encrypted to a recipient whose private half never
reaches this server, so Navi cannot decrypt what it exports. The plaintext
gzip is deterministic, and its digest becomes content_id: an unchanged
corpus keeps the ciphertext and its ETag, so a consumer that polls every
few minutes downloads nothing until something actually changed.

Endpoints sit behind a bearer token (machine credential, not user auth):
/export/v1/manifest describes, /export/v1/messages streams and answers 304
on a matching If-None-Match. /health carries an informational freshness
object that never moves the aggregate status.

scripts/navi_export.py is the consumer side: keygen, then pull — manifest
first, download, checksum, decrypt, gzip-verify, atomic rename; exit 3
when nothing changed.

navi-web: make the batch shape the schema, drop Brave ...

Every recorded web_search and web_view call in production carried a scalar
(20 of 20, 8 of 8), including sessions where three searches on one topic went
out one after another. The recommendation was already written in three
places — the parameter description, the tool docstring and the server's
summary line — so prose was not the problem: `str | list[str]` renders as
`anyOf: [string, array]`, and the model took the first branch.

The parameter is now a pure array with minItems/maxItems and an examples
entry. A BeforeValidator folds a bare "query": "…" into a one-item list, so a
laxer caller still works; the cap rides in json_schema_extra rather than
pydantic's max_length, so four queries reach the code that runs three and
names the rest instead of failing validation.

Brave is gone on both paths: the keyed Search API (signup answers 403, so a
third backend that can never be enabled) and the ddgs engine of the same name,
which answers nothing from this host. The chain is SearXNG -> ddgs.

The persona's parallelism rule now separates "several items in one call" from
"several calls in one message" — it used to read "when the profile allows
parallel tool calls", which the model could only read as "not here".

navi-web: pin the ddgs engines, drop bing, and give the fallback room ...

SearXNG answered the session's searches with noise. Bing, asked explicitly,
returns trending/homepage blocks for queries it cannot match (Block Blast for
a motherboard query); because the list is non-empty the chain accepted it as an
answer and never reached the working fallback. The request now pins
google,duckduckgo and asks for language=all — the pinned uk-UA was what made the
instance's DDG engine return CAPTCHA (kl=ua-uk) on every request.

The ddgs leg had a quieter bug. ddgs v9 has no "html" backend, and an unknown
name does not fail: it silently means "auto" — all engines shuffled together,
yandex and grokipedia included. That is where the slow, flaky, mixed-provenance
results came from. The engine set is now explicit and yandex-free. The 4s client
timeout and 12s chain budget also could not fit a real 1-13s multi-engine scrape,
so they become 10s and 20s.

Wire the Brave key placeholder the docs already described: optional, mapped
with a ":-" fallback so an empty value means off.