export: token-served age snapshot of dialogue for an n-gram consumer
Navi had no way to hand its history to another machine. A database dump
was the obvious idea and the wrong one: prod's 597 MB is mostly things
that must never leave (Fernet-encrypted MCP keys, user rows, attachments),
and the dialogue an n-gram table wants is 14 MB of it.

So the export is a fixed, versioned SELECT — role, content, session, seq,
ts — over both message tables. Default corpus is user+assistant only:
tool output is 52 MB of scraped web text and command logs, and system
messages are near-duplicate boilerplate; either would dominate the counts.
A rolling window takes whole sessions, newest first, up to 150 MB, so the
snapshot stays bounded as history grows.

The snapshot is age-encrypted to a recipient whose private half never
reaches this server, so Navi cannot decrypt what it exports. The plaintext
gzip is deterministic, and its digest becomes content_id: an unchanged
corpus keeps the ciphertext and its ETag, so a consumer that polls every
few minutes downloads nothing until something actually changed.

Endpoints sit behind a bearer token (machine credential, not user auth):
/export/v1/manifest describes, /export/v1/messages streams and answers 304
on a matching If-None-Match. /health carries an informational freshness
object that never moves the aggregate status.

scripts/navi_export.py is the consumer side: keygen, then pull — manifest
first, download, checksum, decrypt, gzip-verify, atomic rename; exit 3
when nothing changed.
1 parent 32efc72 commit 52066fcacedc31c2fc53eb5855de662f5e912541
@Eugene Sukhodolskiy Eugene Sukhodolskiy authored 1 day ago
Showing 19 changed files
View
.gitignore
View
deploy/env.template
View
docs/api.md
View
docs/config.md
View
docs/export.md 0 → 100644
View
docs/index.md
View
navi/api/routes/export.py 0 → 100644
View
navi/api/routes/health.py
View
navi/config.py
View
navi/export/__init__.py 0 → 100644
View
navi/export/builder.py 0 → 100644
View
navi/export/loop.py 0 → 100644
View
navi/main.py
View
pyproject.toml
View
scripts/navi_export.py 0 → 100755
View
tests/unit/api/test_export.py 0 → 100644
View
tests/unit/export/test_builder.py 0 → 100644
View
tests/unit/export/test_cli.py 0 → 100644
View
uv.lock