# History export

A periodically rebuilt, age-encrypted snapshot of message text, served over
HTTP to one machine on the local network that holds the decryption key. Built
for n-gram tables: the consumer needs a corpus of real dialogue, not a copy of
the database.

## Why not a dump

The database is ~600 MB on prod, but almost none of it is dialogue:

| Part | Size |
|---|---|
| `session_messages.content`, all roles | 68 MB |
| `session_messages.thinking` | 40 MB |
| `session_messages.tool_calls` | 16 MB |
| attachments (images/files/metadata) | 95 MB |
| everything else (`sessions` JSON, store, vectors) | ~370 MB |

The default corpus — `user` + `assistant` content across both message tables —
is **14 MB of UTF-8, 23,672 rows** on prod. That gzips to a few megabytes.

A dump would also carry `mcp_user_keys` (Fernet-encrypted MCP credentials),
`users` (PII, password hashes) and `api_tokens` (token hashes). Those must not
leave the server in any form, under any key — so the export is a fixed,
versioned `SELECT`, not `pg_dump`.

`EXPORT_VERSION` (`navi/export/builder.py`) plus the `fields` list in the
manifest is the contract. Adding a field means bumping the version, because the
consumer validates the list.

## What is in the file

gzip-JSONL, one object per line:

```json
{"v":1,"session":"…","seq":42,"role":"assistant","content":"…","ts":"2026-11-04T10:22:31+00:00"}
```

Default corpus: `role = user | assistant`, `content` only. `NAVI_EXPORT_ROLES`
adds or removes roles:

| Role | Default | Why |
|---|---|---|
| `user`, `assistant` | on | the dialogue itself — what an n-gram table should predict |
| `tool` | off | 52 MB of scraped pages, command output and file contents; a different distribution that would dominate the counts |
| `system` | off | near-identical boilerplate (867 copies of the same prompts) |
| `thinking` | never | model reasoning is a separate style; not part of the schema |

Summary and compression artefacts (`is_summary`, `is_compression`) are always
excluded — they duplicate dialogue in a summarised form.

## The rolling window

Whole sessions, newest activity first, until the cumulative text reaches
`NAVI_EXPORT_MAX_BYTES` (150 MB). Today the entire corpus is 14 MB, so nothing
is cut (`window.truncated: false`); the cap exists so the snapshot stays
bounded as history grows. The cut is always on a session boundary — a
conversation never arrives half a story.

Both `session_messages` and `session_messages_archive` feed the window, so
compressed-away history stays in the corpus.

## Build

A background loop (`navi/export/loop.py`) starts at boot and repeats every
`NAVI_EXPORT_INTERVAL_HOURS`. Inside:

1. one read-only `REPEATABLE READ` transaction, so window and rows agree;
2. rows stream through an asyncpg cursor;
3. gzip is written deterministically (fixed `mtime`, sorted keys) — the same
   corpus always yields the same digest;
4. the plaintext is encrypted with `age` to `NAVI_EXPORT_AGE_RECIPIENT` and the
   plaintext is deleted; only `messages-v1.jsonl.gz.age` remains.

`content_id` is the SHA-256 of the plaintext JSONL. When a rebuild produces the
same `content_id`, the ciphertext is left untouched and only `built_at` moves —
so an idle server does not invalidate a consumer's cache.

If a build fails, the previous file and manifest stay in place. `built_at` keeps
telling the truth, so a consumer sees the snapshot ageing instead of silently
reading stale data as fresh.

`/health` carries an informational `export` object when this feature is
configured — `built`, `built_at`, `rows`, `age_seconds`, `stale` (> two missed
rebuilds). It never moves the aggregate `status`: a slow snapshot is not the
service being unhealthy.

## Endpoints

Both require `Authorization: Bearer <NAVI_EXPORT_TOKEN>`. This is a machine
credential: no cookies, no user, no role. A missing token and a wrong token get
the same `401`.

### `GET /export/v1/manifest`

```json
{
  "export_version": 1,
  "built_at": "2026-11-04T12:00:00+00:00",
  "content_id": "9f2c…",
  "file": {"name": "messages-v1.jsonl.gz.age", "bytes": 5242880, "sha256": "…"},
  "rows": 23672,
  "sessions": 1177,
  "sessions_total": 1177,
  "text_bytes": 14680064,
  "fields": ["v", "session", "seq", "role", "content", "ts"],
  "roles": ["user", "assistant"],
  "window": {"from": "…", "to": "…", "truncated": false, "max_bytes": 157286400},
  "age_recipient": "age1…"
}
```

`503` when nothing has been built yet, or when the export is not configured.

### `GET /export/v1/messages`

The ciphertext, streamed from disk. Headers: `ETag` (derived from
`content_id`), `X-Export-Content-Id`, `X-Export-Built-At`. A conditional
request with the matching `If-None-Match` answers `304` with an empty body —
the normal case on a 4-hour schedule.

## Consumer

[`scripts/navi_export.py`](../scripts/navi_export.py) — one file, one
dependency (`pip install pyrage`), Python 3.11+.

```bash
# once: make a keypair here, put the public half into Navi's .env
./navi_export.py keygen --out ~/.config/navi-export
#   -> ~/.config/navi-export/key.txt   (secret, 0600; never leaves this machine)
#   -> age1…                            (NAVI_EXPORT_AGE_RECIPIENT on Navi)

# every run
./navi_export.py pull \
    --url https://navi.example \
    --token-file ~/.config/navi-export/token \
    --identity ~/.config/navi-export/key.txt \
    --out ~/ngram/messages.jsonl.gz
```

The pull reads the manifest first and exits `3` with **no download** when
`content_id` matches the local state file. Otherwise it downloads to `.part`,
verifies the manifest's SHA-256, decrypts, fully decompresses once to prove the
gzip is intact, and atomically renames into place.

Exit codes: `0` updated, `3` unchanged, `1` error.

Cron: polling every 30 minutes is cheap — the steady state is one manifest GET.

## Threat model

- **Navi cannot read its own export.** Only the public half of the age key is on
  the server. A full compromise of Navi yields the ciphertext, not the corpus.
- **The token is transport control, not confidentiality.** It gates who may
  fetch. Confidentiality is the age key. Rotate the token by editing `.env` and
  restarting; rotate the key by generating a new pair and updating
  `NAVI_EXPORT_AGE_RECIPIENT` (the old ciphertext becomes unreadable, which is
  the point).
- **What a token holder still learns:** metadata in the manifest (row counts,
  date range) and the size of the corpus.
- **Nothing about attachment bodies, user accounts, MCP keys or API tokens is
  reachable through these endpoints**, by construction of the `SELECT`.