# Memory System

Long-term user memory: facts extracted from conversations and tool executions, stored in PostgreSQL, injected into every session.

## PostgreSQL + pgvector + pg_trgm (semantic + text search)

The memory system requires **PostgreSQL** with two extensions:

| Extension | Purpose | Auto-created by app? |
|---|---|---|
| `vector` (pgvector) | Semantic search via cosine distance on `embedding vector(768)` | Yes (`CREATE EXTENSION IF NOT EXISTS`) |
| `pg_trgm` | Fast ILIKE fallback via GIN trigram indexes on `category`, `key`, `value` | **No — must be installed by DBA** |

**pgvector** is created automatically because the app typically runs with sufficient privileges on its own database. **pg_trgm** is a core PostgreSQL extension that may require superuser privileges to install. If it is already installed, GIN trigram indexes are created automatically; if not, the app falls back to plain ILIKE without indexes (functional but slower on large tables).

To install pg_trgm manually:

```sql
CREATE EXTENSION IF NOT EXISTS pg_trgm;
```

| Feature | SQLite | PostgreSQL |
|---|---|---|
| Storage | File-based | Server |
| Semantic search | No | Yes (cosine distance on `vector(768)`) |
| Embeddings | None | Generated via Ollama (`nomic-embed-text:latest`) |
| Metadata | `category, key, value` | `+ embedding, source, confidence, expires_at, source_context, merged_into` |

### Dedicated embedding backend

The chat LLM backend (Ollama Cloud, OpenAI, etc.) and the embedding backend are **separate**.

```
.env
EMBEDDING_OLLAMA_HOST=http://192.168.1.168:11434  # local CPU server
EMBEDDING_OLLAMA_API_KEY=
```

If `EMBEDDING_OLLAMA_HOST` is empty, memory falls back to the main `OLLAMA_HOST`.

---

## Schema migration

When upgrading the memory system to a new schema (e.g. adding pgvector columns), run:

```bash
.venv/bin/python navi/memory/migrate_pgvector.py
```

This script:
1. Verifies the `vector` extension is installed in PostgreSQL
2. Adds missing columns: `embedding`, `source`, `confidence`, `expires_at`, `last_verified_at`, `source_context`, `merged_into`
3. Creates indexes: `hnsw(embedding)`, `expires`, `source+category`, `pg_trgm` GIN indexes for ILIKE fallback

Safe to run multiple times — all operations use `IF NOT EXISTS`.

---

## Backfill embeddings

After enabling pgvector, existing facts have `embedding IS NULL`. Generate embeddings for them:

```bash
.venv/bin/python navi/memory/backfill_embeddings.py
```

- Batches of 8 facts at a time
- 2-second sleep between batches (rate-limit safety)
- Safe to run multiple times — only touches rows without embeddings

---

## Storage (`navi/memory/store.py`)

Four tables in the database:

| Table | Purpose |
|---|---|
| `memory_facts` | Individual facts: `(user_id, category, key, value)` — unique on `(user_id, category, key)` |
| `memory_summary` | Per-user narrative summary (`user_id` scoped) |
| `session_memory_state` | Tracks which sessions have been processed (by `extracted_at`) |
| `memory_chunks` | Verbatim messages dropped by context compression — append-only, unique on `(session_id, content_hash)` |

`user_id` references `navi_users(id)` with `ON DELETE CASCADE`. Facts and summaries are scoped per user. Admin with `navi.memory.read_all` can pass `user_id=None` for global search.

`memory_chunks` deliberately has **no foreign key** on `session_id`: the memory DDL may run before `sessions` exists on a fresh database. Rows are deleted by `PgSessionStore.delete()` instead.

`merged_into` is a nullable pointer to the row that superseded this one — the extractor writes the same fact under a new spelling of the key, and the older row is hidden rather than deleted, so the merge can be undone with `UPDATE memory_facts SET merged_into = NULL`. Every read path filters `merged_into IS NULL`; `get_all_facts(include_merged=True)` and `fact_count(include_merged=True)` are the only ways to see a hidden row, and `GET /admin/memory?include_merged=true` exposes them.

`MemoryStore` lazily creates tables on first async operation via `_get_pool()`. All operations are async via asyncpg (PostgreSQL).

### Key operations

| Method | Description |
|---|---|
| `upsert_fact(..., user_id=None, dedupe=False)` | Insert or update a fact scoped to user. With `dedupe=True` (the extractor) a row already holding the same key under a different spelling is hidden behind the surviving one. |
| `search_facts(query, user_id=None, limit=15)` | **Vector search first** (cosine distance, cutoff 0.3), then ILIKE fallback. `user_id=None` requires admin permission for global search. |
| `delete_fact(key, category=None, user_id=None)` | Delete by key, optionally filtered by category and user. Drops the hidden rows pointing at it first, so none is left orphaned. |
| `get_all_facts(user_id=None, all_users=False, limit=None, offset=0, search=None, sort_by="category", sort_order="desc", include_merged=False)` | All facts ordered by `sort_by`. Pass `all_users=True` for admin global view. |
| `get_summary(user_id=None)` | Current narrative summary text for user |
| `set_summary(content, user_id=None)` | Replace the summary for user |
| `mark_session_extracted(session_id)` | Record extraction timestamp |
| `get_extracted_at(session_id)` | Check if/when a session was processed |
| `backfill_embeddings(batch_size=8)` | Generate embeddings for facts with `embedding IS NULL` (hidden rows skipped) |
| `add_chunks(chunks, user_id=None, session_id=None)` | Write raw chunks (`memory_chunks`, `ChunkMixin`). Batched embeddings, one `INSERT … ON CONFLICT (session_id, content_hash) DO NOTHING` per row, returns the number actually written |
| `search_chunks(query, user_id=None, session_id=None, limit=8)` | **Vector search first** (cosine distance < 0.45), then Postgres FTS (`plainto_tsquery` on `content`, plus `ILIKE ANY` on content and `breadcrumbs`) with Python `terms_hit` scoring. An empty normalized query returns the most recent chunks |
| `latest_summary_at(session_id)` | Timestamp of the session's last summary — the freshness boundary for raw excerpts. Failures return `None` |

`KvStore.search_values(user_id, scope, terms, limit=10, session_id=None, prefer_session_id=None)` (`navi/store/__init__.py`) is the read path for scratchpad sketches: ILIKE against `key` and `value` inside a scope, current session ranked first.

---

## Automatic extraction (`navi/memory/extractor.py`)

Facts are extracted from stale sessions automatically.

**Trigger:** `POST /sessions` (create new session) fires `_process_stale_sessions()` as a background task.

**Stale criterion:** `session.last_active < now - 30 minutes` AND not yet extracted (or extracted before last activity).

**Extraction process:**
1. Render conversation as plain text, including:
   - User messages
   - Assistant messages
   - `[Tool call] tool_name(args)` lines
   - `[Tool result] tool_name: output` lines (truncated to 500 chars)
2. Truncate overall transcript to 12 000 chars (keep head + tail, drop middle).
3. Call LLM with extraction prompt: "extract stable facts about the user."
4. Parse JSON array with fields: `category`, `key`, `value`, `source`, `source_context`.
5. Map confidence: `tool_call`/`auto_discovery` → 95, `user_explicit` → 90, default → 70.
6. Upsert each fact into `memory_facts` with `dedupe=True`: a fact already stored under a differently spelled key (`language` vs `preferred_language`) is merged instead of duplicated.
7. Regenerate `memory_summary` from all current facts.
8. Mark session as extracted.

---

## Memory injection into agent context

At the start of each `run_stream()` / `run()` / `run_ephemeral()` call, `_memory_msg()` is called:

```python
async def _memory_msg(self) -> Message | None:
    summary = await self._memory.get_summary()
    if not summary:
        return None
    return Message(role="system", content=f"## What I remember about the user\n\n{summary}")
```

This message is inserted after the main system message but before conversation history. The agent reads it on every turn.

Individual facts reach the prompt as their own list (`navi/core/context_builder.py`), one line each, carrying both the confidence and the age of the fact — a stale fact then reads as stale instead of as current:

```
• [preferences] editor → neovim (confidence: 90%, updated 34d ago)
```

---

## Raw chunks and breadcrumbs (`navi/memory/_chunks.py`, `navi/memory/breadcrumbs.py`)

Every context compression is a lossy step: the messages it drops are gone from the model's window forever. `memory_chunks` keeps them verbatim.

### What is written

Compression collects the messages it demotes (`demoted` — exactly the rows it flips to `is_context=False`) and hands them to `build_chunks`:

- messages are packed into chunks of up to `CHUNK_MAX_CHARS` (4000) chars; a single message is capped at `CHUNK_MESSAGE_MAX_CHARS` (12 000) with its middle elided;
- `user`, `assistant` and `tool` messages are included, each prefixed (`User:`, `Assistant:`, `Tool result (name):`); `system` messages, old summaries and compression markers are skipped;
- each chunk records `start_seq`/`end_seq`, `started_at`/`ended_at`, and a `sha256` of its content.

The table is **append-only and immutable**: a correction is a new chunk, never an edit. `(session_id, content_hash)` is unique, so indexing the same demoted range twice is a no-op.

### Breadcrumbs

The breadcrumbs are the entities a later question will be phrased with — a file, a command, a port, an amount, a ticket, a link — and they are searched alongside the chunk text. Two sources, merged:

1. **The summarizer's `KEYS:` line.** The summary prompt ends with *"End with ONE line `KEYS: item, item, …` — the breadcrumbs of this history … at most 30, most important first. … write `KEYS: NONE` when nothing is worth indexing."* `split_keys_line` removes that line from the summary text and returns the keys; the summary message keeps the rest.
2. **A deterministic extractor** (`extract_breadcrumbs`): URLs, inline code, paths, `SNAKE_CASE`, `CamelCase` and `CONSTANT` identifiers, amounts (`4,5 GB`, `12 ms`, `80%`), `#ticket` numbers and quoted phrases. It runs on the dropped messages themselves, so it also covers the hard-truncate fallback — the path that never calls a summarizer and therefore never produces a `KEYS:` line.

The union (`MAX_BREADCRUMBS` = 40) is attached to every chunk produced by that compression. Indexing failures are logged (`compressor.chunk_index_failed`) and never fail the compression.

Indexes: `(user_id, session_id, ended_at DESC)`, an expression GIN on `to_tsvector('simple', content)`, a GIN on `breadcrumbs`, and an HNSW index on `embedding` when pgvector is available.

---

## Unified search (`navi/memory/search.py`)

Three stores, one read path. Writing is deliberately separate; reading is not — an agent asked "what did we decide about the port" should not have to know which store holds the answer.

| Store | Shape | Written by |
|---|---|---|
| `facts` | curated, rewritten by key, user lifetime | extractor, `memory` tool |
| `raw` | verbatim messages dropped by compression | `compress_and_save_session` |
| `sketches` | a session's working notes, updated in place | `ScratchpadTool` (`session_store` scope `scratchpad`) |

`UnifiedMemorySearch.search(query, scope="all", k=5, user_id=None, session_id=None, current_session_id=None, summary_at=None)` returns `MemoryHit` objects. The merge is tiered, not a blind mix:

1. **facts** — curated truth first;
2. **raw** — by relevance (vector similarity, else matched terms);
3. **sketches** — the current session's topic pad last.

`scope` narrows to one store (`facts` / `raw` / `sketches`); `all` is the default and an unknown value falls back to it. `k` is per store, clamped to 1–20.

Every hit carries provenance: `source_type`, `session_id`, `timestamp`, `key`/`category` for a fact, `start_seq`/`end_seq` for a raw excerpt, section for a sketch.

**Freshness.** A raw excerpt is *reference*, a fact is *curated*. `_mark_freshness` compares them: a raw chunk that shares at least two content words with a **newer** fact is marked `superseded_by=<key>` and rendered with a warning; a raw chunk older than the same session's last summary is marked `before_summary` and rendered as a verbatim record from before the current summary. The outdated item is still returned — "we changed our mind, the old value was X" is often exactly the question.

`format_hits` renders the tool output, e.g.:

```
memory_search "server port" (scope=all): 2 hit(s)

1. fact [technical] · session s1 · updated 2026-11-09 12:00
   server_port: 8099
2. raw · session s1 · turn 12–14 · 2026-11-06 18:41
   ⚠ outdated: a newer curated fact overrides this
   keys: server_port, 8098
   User: ...verbatim...
```

### Automatic recall

`ContextBuilder._memory_recall_msg` injects up to `memory_recall_k` (2) raw excerpts **once per turn, only when the context already holds a summary** — that is the moment when dropped history matters. The query is built from the last four user/assistant messages plus the incoming one, so the search follows the current topic. Hits below `memory_recall_min_similarity` (0.55 vector, or 2 matched terms on the FTS path) and hits marked `superseded_by` are dropped. Delivered chunk ids are remembered in `session.session_metadata["injected_chunk_ids"]`, so the same excerpt does not ride in every prompt; the model can still call `memory_search` for anything wider.

Both behaviours are switchable: `MEMORY_SEARCH_ENABLED` (indexing demoted messages) and `MEMORY_RECALL_ENABLED` (the automatic injection), see `docs/config.md`.

---

## Memory tool (`navi/tools/memory.py`)

Single unified `memory` tool with `action` parameter:

| Action | Description |
|---|---|
| `save` | Upsert a fact with `category`, `key`, `value`, `source`, `confidence`, `expires_days`, `source_context` |
| `search` | Find facts by keyword query (vector search → ILIKE fallback) |
| `forget` | Delete a fact by `key` (optionally filtered by `category`) |
| `list` | Return the categories that hold facts, not the facts themselves |

**Sources:** `conversation`, `tool_call`, `auto_discovery`, `user_explicit`

**Confidence:** 0-100. Tool output = 95, user statement = 80, web = 50, guess = 30.

---

## Search tool (`navi/tools/memory_search.py`)

`memory_search` is the read side of memory and the only tool that reads all three stores.

| Parameter | Description |
|---|---|
| `query` (required) | What to look for — a name, a file, a command, a value, a topic |
| `scope` | `all` (default), `facts`, `raw`, `sketches` |
| `k` | Results per store, 1–20, default 5 |

There is deliberately **no separate session-search tool**: one query over everything, tiered merge, provenance on every line. The tool resolves `session_id`/`user_id` from `ToolContext` (falling back to the request context vars) and asks `latest_summary_at(session_id)` for the freshness boundary.

### `memory` vs `memory_search`

- **`memory`** writes. `save` / `forget` / `list`, and `search` for a fact-only lookup when the agent specifically wants curated facts.
- **`memory_search`** reads across everything. Use it when the user refers back to something no longer in view — "as we discussed", "the port we picked", "that file", a name, a number, a link, a decision — before saying you do not remember.

---

## Memory usage guidelines (from persona)

Call `memory_search` when:
- The user refers back to something you can no longer see — "as we discussed", "the port we picked", "that file", a name, a number, a link, a decision — before saying you do not remember.
- The user mentions something personal (location, project, preference, recurring task) and you need the fact behind it.
- About to make an assumption about the user's environment or preferences — verify first.

Prefer a newer curated fact over an older raw excerpt; when they disagree, `memory_search` says which one is outdated.

Do NOT call `memory_search` reflexively at the start of every session — only when context warrants it (automatic recall already covers the compressed-history case).

Call `memory` with `action="forget"` only when the user explicitly asks, or when a stored fact is clearly wrong or outdated.