Memory System

Long-term user memory: facts extracted from conversations and tool executions, stored in PostgreSQL, injected into every session.

PostgreSQL + pgvector + pg_trgm (semantic + text search)

The memory system requires PostgreSQL with two extensions:

pgvector is created automatically because the app typically runs with sufficient privileges on its own database. pg_trgm is a core PostgreSQL extension that may require superuser privileges to install. If it is already installed, GIN trigram indexes are created automatically; if not, the app falls back to plain ILIKE without indexes (functional but slower on large tables).

To install pg_trgm manually:

CREATE EXTENSION IF NOT EXISTS pg_trgm;

Dedicated embedding backend

The chat LLM backend (Ollama Cloud, OpenAI, etc.) and the embedding backend are separate.

.env
EMBEDDING_OLLAMA_HOST=http://192.168.1.168:11434  # local CPU server
EMBEDDING_OLLAMA_API_KEY=

If EMBEDDING_OLLAMA_HOST is empty, memory falls back to the main OLLAMA_HOST.

Schema migration

When upgrading the memory system to a new schema (e.g. adding pgvector columns), run:

.venv/bin/python navi/memory/migrate_pgvector.py

This script:

1. Verifies the vector extension is installed in PostgreSQL
2. Adds missing columns: embedding, source, confidence, expires_at, last_verified_at, source_context, merged_into
3. Creates indexes: hnsw(embedding), expires, source+category, pg_trgm GIN indexes for ILIKE fallback

Safe to run multiple times — all operations use IF NOT EXISTS.

Backfill embeddings

After enabling pgvector, existing facts have embedding IS NULL. Generate embeddings for them:

.venv/bin/python navi/memory/backfill_embeddings.py

• Batches of 8 facts at a time
• 2-second sleep between batches (rate-limit safety)
• Safe to run multiple times — only touches rows without embeddings

Storage (navi/memory/store.py)

Four tables in the database:

user_id references navi_users(id) with ON DELETE CASCADE. Facts and summaries are scoped per user. Admin with navi.memory.read_all can pass user_id=None for global search.

memory_chunks deliberately has no foreign key on session_id: the memory DDL may run before sessions exists on a fresh database. Rows are deleted by PgSessionStore.delete() instead.

merged_into is a nullable pointer to the row that superseded this one — the extractor writes the same fact under a new spelling of the key, and the older row is hidden rather than deleted, so the merge can be undone with UPDATE memory_facts SET merged_into = NULL. Every read path filters merged_into IS NULL; get_all_facts(include_merged=True) and fact_count(include_merged=True) are the only ways to see a hidden row, and GET /admin/memory?include_merged=true exposes them.

MemoryStore lazily creates tables on first async operation via _get_pool(). All operations are async via asyncpg (PostgreSQL).

Key operations

KvStore.search_values(user_id, scope, terms, limit=10, session_id=None, prefer_session_id=None) (navi/store/__init__.py) is the read path for scratchpad sketches: ILIKE against key and value inside a scope, current session ranked first.

Automatic extraction (navi/memory/extractor.py)

Facts are extracted from stale sessions automatically.

Trigger: POST /sessions (create new session) fires _process_stale_sessions() as a background task.

Stale criterion: session.last_active < now - 30 minutes AND not yet extracted (or extracted before last activity).

Extraction process:

1. Render conversation as plain text, including:
   • User messages
   • Assistant messages
   • [Tool call] tool_name(args) lines
   • [Tool result] tool_name: output lines (truncated to 500 chars)
2. Truncate overall transcript to 12 000 chars (keep head + tail, drop middle).
3. Call LLM with extraction prompt: "extract stable facts about the user."
4. Parse JSON array with fields: category, key, value, source, source_context.
5. Map confidence: tool_call/auto_discovery → 95, user_explicit → 90, default → 70.
6. Upsert each fact into memory_facts with dedupe=True: a fact already stored under a differently spelled key (language vs preferred_language) is merged instead of duplicated.
7. Regenerate memory_summary from all current facts.
8. Mark session as extracted.

Memory injection into agent context

At the start of each run_stream() / run() / run_ephemeral() call, _memory_msg() is called:

async def _memory_msg(self) -> Message | None:
    summary = await self._memory.get_summary()
    if not summary:
        return None
    return Message(role="system", content=f"## What I remember about the user\n\n{summary}")

This message is inserted after the main system message but before conversation history. The agent reads it on every turn.

Individual facts reach the prompt as their own list (navi/core/context_builder.py), one line each, carrying both the confidence and the age of the fact — a stale fact then reads as stale instead of as current:

• [preferences] editor → neovim (confidence: 90%, updated 34d ago)

Raw chunks and breadcrumbs (navi/memory/_chunks.py, navi/memory/breadcrumbs.py)

Every context compression is a lossy step: the messages it drops are gone from the model's window forever. memory_chunks keeps them verbatim.

What is written

Compression collects the messages it demotes (demoted — exactly the rows it flips to is_context=False) and hands them to build_chunks:

• messages are packed into chunks of up to CHUNK_MAX_CHARS (4000) chars; a single message is capped at CHUNK_MESSAGE_MAX_CHARS (12 000) with its middle elided;
• user, assistant and tool messages are included, each prefixed (User:, Assistant:, Tool result (name):); system messages, old summaries and compression markers are skipped;
• each chunk records start_seq/end_seq, started_at/ended_at, and a sha256 of its content.

The table is append-only and immutable: a correction is a new chunk, never an edit. (session_id, content_hash) is unique, so indexing the same demoted range twice is a no-op.

Breadcrumbs

The breadcrumbs are the entities a later question will be phrased with — a file, a command, a port, an amount, a ticket, a link — and they are searched alongside the chunk text. Two sources, merged:

1. The summarizer's KEYS: line. The summary prompt ends with "End with ONE line KEYS: item, item, … — the breadcrumbs of this history … at most 30, most important first. … write KEYS: NONE when nothing is worth indexing." split_keys_line removes that line from the summary text and returns the keys; the summary message keeps the rest.
2. A deterministic extractor (extract_breadcrumbs): URLs, inline code, paths, SNAKE_CASE, CamelCase and CONSTANT identifiers, amounts (4,5 GB, 12 ms, 80%), #ticket numbers and quoted phrases. It runs on the dropped messages themselves, so it also covers the hard-truncate fallback — the path that never calls a summarizer and therefore never produces a KEYS: line.

The union (MAX_BREADCRUMBS = 40) is attached to every chunk produced by that compression. Indexing failures are logged (compressor.chunk_index_failed) and never fail the compression.

Indexes: (user_id, session_id, ended_at DESC), an expression GIN on to_tsvector('simple', content), a GIN on breadcrumbs, and an HNSW index on embedding when pgvector is available.

Unified search (navi/memory/search.py)

Three stores, one read path. Writing is deliberately separate; reading is not — an agent asked "what did we decide about the port" should not have to know which store holds the answer.

UnifiedMemorySearch.search(query, scope="all", k=5, user_id=None, session_id=None, current_session_id=None, summary_at=None) returns MemoryHit objects. The merge is tiered, not a blind mix:

1. facts — curated truth first;
2. raw — by relevance (vector similarity, else matched terms);
3. sketches — the current session's topic pad last.

scope narrows to one store (facts / raw / sketches); all is the default and an unknown value falls back to it. k is per store, clamped to 1–20.

Every hit carries provenance: source_type, session_id, timestamp, key/category for a fact, start_seq/end_seq for a raw excerpt, section for a sketch.

Freshness. A raw excerpt is reference, a fact is curated. _mark_freshness compares them: a raw chunk that shares at least two content words with a newer fact is marked superseded_by=<key> and rendered with a warning; a raw chunk older than the same session's last summary is marked before_summary and rendered as a verbatim record from before the current summary. The outdated item is still returned — "we changed our mind, the old value was X" is often exactly the question.

format_hits renders the tool output, e.g.:

memory_search "server port" (scope=all): 2 hit(s)

1. fact [technical] · session s1 · updated 2026-11-09 12:00
   server_port: 8099
2. raw · session s1 · turn 12–14 · 2026-11-06 18:41
   ⚠ outdated: a newer curated fact overrides this
   keys: server_port, 8098
   User: ...verbatim...

Automatic recall

ContextBuilder._memory_recall_msg injects up to memory_recall_k (2) raw excerpts once per turn, only when the context already holds a summary — that is the moment when dropped history matters. The query is built from the last four user/assistant messages plus the incoming one, so the search follows the current topic. Hits below memory_recall_min_similarity (0.55 vector, or 2 matched terms on the FTS path) and hits marked superseded_by are dropped. Delivered chunk ids are remembered in session.session_metadata["injected_chunk_ids"], so the same excerpt does not ride in every prompt; the ledger keeps the last 100 ids (it is stored in the session row and must not grow forever), so a very old excerpt can be recalled again later. The model can still call memory_search for anything wider. The freshness labels are computed even though recall asks for raw only: an excerpt a newer fact has overridden is never injected.

Both behaviours are switchable: MEMORY_SEARCH_ENABLED (indexing demoted messages) and MEMORY_RECALL_ENABLED (the automatic injection), see docs/config.md.

Memory tool (navi/tools/memory.py)

Single unified memory tool with action parameter:

Sources: conversation, tool_call, auto_discovery, user_explicit

Confidence: 0-100. Tool output = 95, user statement = 80, web = 50, guess = 30.

Search tool (navi/tools/memory_search.py)

memory_search is the read side of memory and the only tool that reads all three stores.

There is deliberately no separate session-search tool: one query over everything, tiered merge, provenance on every line. The tool resolves session_id/user_id from ToolContext (falling back to the request context vars) and asks latest_summary_at(session_id) for the freshness boundary.

memory vs memory_search

• memory writes. save / forget / list, and search for a fact-only lookup when the agent specifically wants curated facts.
• memory_search reads across everything. Use it when the user refers back to something no longer in view — "as we discussed", "the port we picked", "that file", a name, a number, a link, a decision — before saying you do not remember.

Memory usage guidelines (from persona)

Call memory_search when:

• The user refers back to something you can no longer see — "as we discussed", "the port we picked", "that file", a name, a number, a link, a decision — before saying you do not remember.
• The user mentions something personal (location, project, preference, recurring task) and you need the fact behind it.
• About to make an assumption about the user's environment or preferences — verify first.

Prefer a newer curated fact over an older raw excerpt; when they disagree, memory_search says which one is outdated.

Do NOT call memory_search reflexively at the start of every session — only when context warrants it (automatic recall already covers the compressed-history case).

Call memory with action="forget" only when the user explicitly asks, or when a stored fact is clearly wrong or outdated.