History export

A periodically rebuilt, age-encrypted snapshot of message text, served over HTTP to one machine on the local network that holds the decryption key. Built for n-gram tables: the consumer needs a corpus of real dialogue, not a copy of the database.

Why not a dump

The database is ~600 MB on prod, but almost none of it is dialogue:

The default corpus — user + assistant content across both message tables — is 14 MB of UTF-8, 23,672 rows on prod. That gzips to a few megabytes.

A dump would also carry mcp_user_keys (Fernet-encrypted MCP credentials), users (PII, password hashes) and api_tokens (token hashes). Those must not leave the server in any form, under any key — so the export is a fixed, versioned SELECT, not pg_dump.

EXPORT_VERSION (navi/export/builder.py) plus the fields list in the manifest is the contract. Adding a field means bumping the version, because the consumer validates the list.

What is in the file

gzip-JSONL, one object per line:

{"v":1,"session":"…","seq":42,"role":"assistant","content":"…","ts":"2026-11-04T10:22:31+00:00"}

Default corpus: role = user | assistant, content only. NAVI_EXPORT_ROLES adds or removes roles:

Summary and compression artefacts (is_summary, is_compression) are always excluded — they duplicate dialogue in a summarised form.

The rolling window

Whole sessions, newest activity first, until the cumulative text reaches NAVI_EXPORT_MAX_BYTES (150 MB). Today the entire corpus is 14 MB, so nothing is cut (window.truncated: false); the cap exists so the snapshot stays bounded as history grows. The cut is always on a session boundary — a conversation never arrives half a story.

Both session_messages and session_messages_archive feed the window, so compressed-away history stays in the corpus.

Build

A background loop (navi/export/loop.py) starts at boot and repeats every NAVI_EXPORT_INTERVAL_HOURS. Inside:

1. one read-only REPEATABLE READ transaction, so window and rows agree;
2. rows stream through an asyncpg cursor;
3. gzip is written deterministically (fixed mtime, sorted keys) — the same corpus always yields the same digest;
4. the plaintext is encrypted with age to NAVI_EXPORT_AGE_RECIPIENT and the plaintext is deleted; only messages-v1.jsonl.gz.age remains.

content_id is the SHA-256 of the plaintext JSONL. When a rebuild produces the same content_id, the ciphertext is left untouched and only built_at moves — so an idle server does not invalidate a consumer's cache.

If a build fails, the previous file and manifest stay in place. built_at keeps telling the truth, so a consumer sees the snapshot ageing instead of silently reading stale data as fresh.

/health carries an informational export object when this feature is configured — built, built_at, rows, age_seconds, stale (> two missed rebuilds). It never moves the aggregate status: a slow snapshot is not the service being unhealthy.

Endpoints

Both require Authorization: Bearer <NAVI_EXPORT_TOKEN>. This is a machine credential: no cookies, no user, no role. A missing token and a wrong token get the same 401.

GET /export/v1/manifest

{
  "export_version": 1,
  "built_at": "2026-11-04T12:00:00+00:00",
  "content_id": "9f2c…",
  "file": {"name": "messages-v1.jsonl.gz.age", "bytes": 5242880, "sha256": "…"},
  "rows": 23672,
  "sessions": 1177,
  "sessions_total": 1177,
  "text_bytes": 14680064,
  "fields": ["v", "session", "seq", "role", "content", "ts"],
  "roles": ["user", "assistant"],
  "window": {"from": "…", "to": "…", "truncated": false, "max_bytes": 157286400},
  "age_recipient": "age1…"
}

503 when nothing has been built yet, or when the export is not configured.

GET /export/v1/messages

The ciphertext, streamed from disk. Headers: ETag (derived from content_id), X-Export-Content-Id, X-Export-Built-At. A conditional request with the matching If-None-Match answers 304 with an empty body — the normal case on a 4-hour schedule.

Consumer

scripts/navi_export.py — one file, one dependency (pip install pyrage), Python 3.11+.

# once: make a keypair here, put the public half into Navi's .env
./navi_export.py keygen --out ~/.config/navi-export
#   -> ~/.config/navi-export/key.txt   (secret, 0600; never leaves this machine)
#   -> age1…                            (NAVI_EXPORT_AGE_RECIPIENT on Navi)

# every run
./navi_export.py pull \
    --url https://navi.example \
    --token-file ~/.config/navi-export/token \
    --identity ~/.config/navi-export/key.txt \
    --out ~/ngram/messages.jsonl.gz

The pull reads the manifest first and exits 3 with no download when content_id matches the local state file. Otherwise it downloads to .part, verifies the manifest's SHA-256, decrypts, fully decompresses once to prove the gzip is intact, and atomically renames into place.

Exit codes: 0 updated, 3 unchanged, 1 error.

Cron: polling every 30 minutes is cheap — the steady state is one manifest GET.

Threat model

• Navi cannot read its own export. Only the public half of the age key is on the server. A full compromise of Navi yields the ciphertext, not the corpus.
• The token is transport control, not confidentiality. It gates who may fetch. Confidentiality is the age key. Rotate the token by editing .env and restarting; rotate the key by generating a new pair and updating NAVI_EXPORT_AGE_RECIPIENT (the old ciphertext becomes unreadable, which is the point).
• What a token holder still learns: metadata in the manifest (row counts, date range) and the size of the corpus.
• Nothing about attachment bodies, user accounts, MCP keys or API tokens is reachable through these endpoints, by construction of the SELECT.