diff --git a/docs/agent.md b/docs/agent.md index 4e43c50..b9d2048 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -126,7 +126,9 @@ → ThinkingDelta/ThinkingEnd events during reasoning → TextDelta events during text generation → ModelInfo event when the resolved model changes mid-turn - 7. Record real prompt_tokens baseline (record_real_baseline) for the next estimate + 7. Record real prompt_tokens baseline (record_real_baseline) for the next estimate — + a count far below what was sent means the backend truncated the prompt; it is + not recorded and raises the context_truncated flag for the post-turn worker 8a. No tool calls → save session, yield StreamEnd, run workers, return 8b. Tool calls → execute each, yield ToolEvent, append results to context 9. Update anti-stall counters @@ -160,9 +162,9 @@ ## Workers -Run sequentially after `StreamEnd`. Currently: `CompressionWorker`. Workers receive a `WorkerContext` carrying the active `profile`, so `CompressionWorker` applies per-profile compression overrides (`compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file`). The worker gates on the real `context_tokens` from the last LLM call, then delegates to `compress_and_save_session(reason="postturn")` — the same retry / hard-truncate / safety-net / archiving pipeline as the pre- and mid-turn paths (with the mid-turn `keep_recent_messages`). +Run sequentially after `StreamEnd`. Currently: `CompressionWorker`. Workers receive a `WorkerContext` carrying the active `profile`, so `CompressionWorker` applies per-profile compression overrides (`compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file`). The worker gates on the real `context_tokens` from the last LLM call, then delegates to `compress_and_save_session(reason="postturn")` — the same retry / hard-truncate / safety-net / archiving pipeline as the pre- and mid-turn paths (with the mid-turn `keep_recent_messages`). It compresses unconditionally when `WorkerContext.context_truncated` is set: a truncated prompt reports only about half the window, so the threshold would never fire on it. -Pre-turn compression also runs at the start of `run_stream()`: it estimates tokens via `real_baseline_estimate(session.context, session.context)` (real `prompt_eval_count` from the last call + calibrated delta; the calibrated `estimate_tokens` only before the first LLM call of a session) and, when over `window × threshold`, is guarded by `would_compress(current_tokens=…)` before emitting `CompressionStarted`. The window is `profile.context_window` or `settings.ollama_num_ctx` (`Agent._window_for`). See [`sessions.md`](sessions.md). +Pre-turn compression also runs at the start of `run_stream()`: it estimates tokens via `real_baseline_estimate(session.context, session.context)` (real `prompt_eval_count` from the last call + calibrated delta; the calibrated `estimate_tokens` only before the first LLM call of a session) plus the incoming user message — images included, since it is not in `session.context` yet — and, when over `window × threshold`, is guarded by `would_compress(current_tokens=…)` before emitting `CompressionStarted`. The window is `profile.context_window` or `settings.ollama_num_ctx` (`Agent._window_for`). See [`sessions.md`](sessions.md). --- @@ -179,4 +181,4 @@ The built system prompt string is cached per profile ID in `ContextBuilder` to avoid rebuilding on every turn. The cache is invalidated when the profile is reloaded (e.g. after `switch_profile` or hot-reload). This saves ~1–2 ms per turn for profiles with large system prompts. ### Per-message view truncation -`ContextBuilder.build()` head/tail-truncates any single `tool`/`assistant` message whose estimated size exceeds `CONTEXT_MESSAGE_TOKEN_BUDGET` (default `0` → auto, `OLLAMA_NUM_CTX // 6`) in the *built* context only. Stored history is never mutated — a copy with a `[…truncated ~N tokens…]` marker is sent to the LLM. This prevents one huge tool result from alone blowing the window; full context compression still handles the cumulative case. +`ContextBuilder.build()` head/tail-truncates any single `tool`/`assistant` message whose estimated size exceeds `CONTEXT_MESSAGE_TOKEN_BUDGET` (default `0` → auto, `OLLAMA_NUM_CTX // 6`) in the *built* context only. User messages are capped too, but much more generously (at least `OLLAMA_NUM_CTX // 3`), with a `[…user message truncated ~N tokens…]` marker: a paste bigger than the window can never reach the model whole, and leaving it uncapped used to turn it into a dead session. Stored history is never mutated — a copy with a `[…truncated ~N tokens…]` marker is sent to the LLM. This prevents one huge result from alone blowing the window; full context compression still handles the cumulative case. diff --git a/docs/config.md b/docs/config.md index 1ec5ffd..8f4fa3b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -352,7 +352,8 @@ | `CONTEXT_SUMMARY_TEMPERATURE` | float | `0.3` | Temperature for the summarization LLM call | | `CONTEXT_SUMMARY_MAX_TOKENS` | int | `6000` | Max output tokens for the summary LLM call | | `OUTPUT_RESERVE_TOKENS` | int | `2048` | Headroom reserved for model response in context size checks | -| `CONTEXT_MESSAGE_TOKEN_BUDGET` | int | `0` | Per-message token budget for the LLM context view. A single `tool`/`assistant` message whose estimated size exceeds this is head/tail-truncated in the *built* context only (stored history is never mutated) so one huge tool result cannot alone blow the window. `0` = auto (`OLLAMA_NUM_CTX // 6`). | +| `CONTEXT_MESSAGE_TOKEN_BUDGET` | int | `0` | Per-message token budget for the LLM context view. A single `tool`/`assistant` message whose estimated size exceeds this is head/tail-truncated in the *built* context only (stored history is never mutated) so one huge tool result cannot alone blow the window. `user` messages are capped too, at least at `OLLAMA_NUM_CTX // 3`, so an oversized paste cannot dead-end the session. `0` = auto (`OLLAMA_NUM_CTX // 6`). | +| `CONTEXT_IMAGE_TOKENS` | int | `1024` | Tokens charged for one inline image whose dimensions cannot be read from its header. Images with a readable header are priced by area — roughly one token per 32×32 px, measured on qwen3.6 (1024×1024 ≈ 1026 tokens, 512×512 ≈ 258) — so the session total, the compression trigger and the per-turn image budget all see what the model actually gets. See `navi/core/image_tokens.py`. | ## Gmail diff --git a/docs/mechanics.md b/docs/mechanics.md index 8d84b8d..852e0c6 100644 --- a/docs/mechanics.md +++ b/docs/mechanics.md @@ -32,9 +32,11 @@ | **Pre-turn context compression** | Before the assistant reply, estimates tokens via `real_baseline_estimate(session.context, session.context)` (real `prompt_eval_count` bulk + calibrated delta; `estimate_tokens` only before the first LLM call) and compresses when over `window × threshold`, where `window` is the active route entry's `context_window`, else `profile.context_window`, else `OLLAMA_NUM_CTX`. `begin_turn()` has already pinned the entry for this turn, so a recovered primary is used from here on. Guarded by `would_compress(current_tokens=…)` so `CompressionStarted` is only emitted when the partition can actually shrink the stored context. | `CONTEXT_COMPRESSION_ENABLED`, `OLLAMA_NUM_CTX`, `profile.context_window`, `CONTEXT_COMPRESSION_THRESHOLD` | `agent.py` | ✅ | | **Mid-turn context compression** | On iterations > 0, estimates tokens via `real_baseline_estimate(session.context, preflight_ctx)` (real prompt_tokens bulk + calibrated delta) and compresses with `keep_recent_messages=max(12, CONTEXT_KEEP_RECENT*2)`. Guarded by `would_compress(current_tokens=…)`. For long autonomous loops where the entire conversation is one turn. The post-turn worker passes `WorkerContext.max_context_tokens` as the same window. | Same as above + `CONTEXT_KEEP_RECENT` | `agent.py`, `workers/compressor.py` | ✅ | | **Forced `/compact`** | `compact_stream()` — bypasses the token threshold and compresses immediately on client demand (`{"type":"compact"}` WS control message). Emits `CompressionStarted` + `ContextCompressed`; raises `NothingToCompactError` when there is nothing to compress. Bound to `Ctrl+X C` in the TUI. | None | `agent.py`, `websocket.py` | ✅ | -| **Context size check with output reserve** | Raises `ContextTooLargeError` if estimated input tokens exceed `window - OUTPUT_RESERVE_TOKENS` (the reserve is capped at a quarter of the window). When the caller passes `session_context`, the total uses `real_baseline_estimate()` (real bulk + calibrated delta) instead of the local estimate. Images counted at 500 tokens each. | `OLLAMA_NUM_CTX`, `profile.context_window`, `OUTPUT_RESERVE_TOKENS` | `compressor.py` | ✅ | -| **Real-token baseline estimator** | Records the real `prompt_eval_count` returned by each LLM call (`record_real_baseline(len(session.context), …)`, plus the chars of the built context) and reuses it as the bulk of the next estimate, pricing only the messages appended since. Each observation also feeds the per-model token calibration. Used by the pre-turn, mid-turn and post-turn gates and `check_context_size`. Cleared after compression (context shrank). | None | `compressor.py`, `agent.py` | ✅ | -| **Per-model token calibration** | `TokenCalibration` fits `tokens ≈ overhead + chars × tokens_per_char` (least squares) to the last 8 `(chars, real prompt tokens)` pairs for a model; needs ≥2 observations with a ≥1.25× size spread and rejects slopes outside 1 token/char … 1 per 4 chars. `ContextCompressor.estimate_tokens(context, model)` uses it for every keep-region target, the hard-truncate budget and image budgeting, falling back to `chars / 1.8 + imgs * 500`. `chars // 3` undercounted prod prompts ~1.8×. | None | `token_calibration.py`, `compressor.py` | ✅ | +| **Context size check with output reserve** | Raises `ContextTooLargeError` if estimated input tokens exceed `window - OUTPUT_RESERVE_TOKENS` (the reserve is capped at a quarter of the window). When the caller passes `session_context`, the total uses `real_baseline_estimate()` (real bulk + calibrated delta) instead of the local estimate. Images are priced per image by pixel area (see **Image token accounting**). | `OLLAMA_NUM_CTX`, `profile.context_window`, `OUTPUT_RESERVE_TOKENS` | `compressor.py` | ✅ | +| **Real-token baseline estimator** | Records the real `prompt_eval_count` returned by each LLM call (`record_real_baseline(len(session.context), …)`, plus the chars of the built context) and reuses it as the bulk of the next estimate, pricing only the messages appended since. Each observation also feeds the per-model token calibration — except prompts that carried images, whose bytes are not in `chars` and would skew the fit. Used by the pre-turn, mid-turn and post-turn gates and `check_context_size`. Cleared after compression (context shrank). | None | `compressor.py`, `agent.py` | ✅ | +| **Truncated-prompt protection** | A backend that truncates a prompt it cannot fit (Ollama silently keeps about half the window) reports only the evaluated tokens. Recording that as the baseline pins the session at ~50% and the 90% trigger never fires while the model gets a prompt with its head, system prompt included, dropped. `record_real_baseline` compares the reported count with `sent_tokens` and, below 70%, records nothing and raises `context_truncated`; the post-turn worker then compresses regardless of the threshold. | `_TRUNCATION_RATIO` | `compressor.py`, `workers/compressor.py` | ✅ | +| **Per-model token calibration** | `TokenCalibration` fits `tokens ≈ overhead + chars × tokens_per_char` (least squares) to the last 8 `(chars, real prompt tokens)` pairs for a model; needs ≥2 observations with a ≥1.25× size spread and rejects slopes outside 1 token/char … 1 per 4 chars. `ContextCompressor.estimate_tokens(context, model)` uses it for every keep-region target, the hard-truncate budget and image budgeting, falling back to `chars / 1.8` plus the image cost. `chars // 3` undercounted prod prompts ~1.8×. | None | `token_calibration.py`, `compressor.py` | ✅ | +| **Image token accounting** | Every image is priced by its header dimensions: `ceil(w/32) * ceil(h/32) + 2` after downscaling to ≤2048² px, clamped to 256…4096 — measured on qwen3.6 (1024² = 1026 tokens, 512² = 258). Unreadable headers fall back to `CONTEXT_IMAGE_TOKENS`. The flat 500 this code once used undercounted a 1024² screenshot 2× and a 2048² one 8×, so the session never reached the 90% trigger. Used by the pre-turn gate, `check_context_size`, both estimators and the per-turn image budget. | `CONTEXT_IMAGE_TOKENS` | `image_tokens.py`, `compressor.py`, `agent.py` | ✅ | | **Per-profile context window** | Resolved once per turn by `Agent._window_for()`, which asks the route engine for the window of the entry that is serving right now (`active_window`, i.e. the `context_window` of the active route entry) and only then falls back to `AgentProfile.context_window` and `OLLAMA_NUM_CTX`. Drives the compression trigger/target, `check_context_size`, the image budget and `WorkerContext.max_context_tokens`; `num_ctx` is capped at the entry's window too, so a 48k local model is never sent a 115k prompt. | `profile.llm_route`, `profile.context_window`, `llm.json` entry `context_window`, `OLLAMA_NUM_CTX` | `agent.py`, `profiles/base.py`, `llm/fallback.py` | ✅ | | **Context-only demotion persistence** | `compress_and_save_session` collects demoted rows that exist only in `session.context` (`is_display=False`, seq ≥ 0) and flips them through `SessionStore.set_context_flag()` → a narrow `UPDATE session_messages SET is_context=…` by sequence number. `save()` only writes flags for rows in `session.messages`, so without this an old summary came back on the next load (prod: up to 15 live summaries in one session). | None | `compressor.py`, `pg_session_store.py`, `session.py` | ✅ | | **Anti-stall detection** | Tracks two signals: (1) consecutive iterations with no todo status change, (2) identical tool call signatures. When either hits threshold, injects a hard warning system message. | `profile.anti_stall_enabled`, `profile.anti_stall_threshold` | `agent.py` | ✅ | @@ -103,15 +105,15 @@ |---|---|---|---|---| | **Threshold-based trigger** | Returns `True` when `context_tokens >= max_context_tokens * threshold`. | `CONTEXT_COMPRESSION_THRESHOLD` | `compressor.py` | ✅ | | **Turn-based partitioning** | Groups messages into turns. Keeps last `keep_recent` turns verbatim; older go to summarization. Tool call groups never split. Adaptive: `_turn_importance` scores turns and swaps an important old turn into the kept set in place of a filler-recent one. | `CONTEXT_KEEP_RECENT`, `profile.compression_keep_recent` | `compressor.py` | ✅ | -| **Intra-turn fallback partitioning** | `partition_current_turn_messages`: for long autonomous loops where the entire conversation is one turn, keeps the current request + newest N messages verbatim and summarizes older messages from the same turn. | `keep_recent_messages` param | `compressor.py` | ✅ | +| **Intra-turn fallback partitioning** | `partition_current_turn_messages`: for long autonomous loops where the entire conversation is one turn, keeps the current request + newest N messages verbatim and summarizes older messages from the same turn. The kept region is additionally bounded by a token budget (`context_compression_target` minus system messages): a message-count window is meaningless when a handful of messages are huge, so the oldest tail messages move into the summary until it fits (the user head and newest tool group are always kept). | `keep_recent_messages`, `CONTEXT_COMPRESSION_TARGET` | `compressor.py` | ✅ | | **Honest `CompressionStarted` guard** | `would_compress()` runs the same `_plan_compression` decision as the real compression path (no LLM) + the token-budget check, so the agent emits `CompressionStarted` only when the partition can actually shrink the stored context — never a misleading "compression" status with no work done. The partition/hysteresis/fallback decision lives in one function shared with `compress_context`, so prediction and reality cannot drift apart. | None | `compressor.py` | ✅ | -| **Token-budget hard-truncate fallback** | When `compress_session` gets a partition no-op but the token gate fired, drops oldest turns (keeping system + newest whole turns) until under `_HARD_TRUNCATE_TOKEN_FRAC=0.5` of the window, priced with the calibrated estimator. Last-resort, no LLM. | `OLLAMA_NUM_CTX`, `profile.context_window` | `compressor.py` | ✅ | -| **Per-message view truncation** | `ContextBuilder._truncate_oversized` head/tail-truncates a single `tool`/`assistant` message whose estimated size exceeds the per-message budget in the *built* context only — stored history is never mutated. 0 = auto (`OLLAMA_NUM_CTX // 6`). | `CONTEXT_MESSAGE_TOKEN_BUDGET` | `context_builder.py` | ✅ | +| **Token-budget hard-truncate fallback** | When `compress_session` gets a partition no-op but the token gate fired, drops oldest turns (keeping system + newest turns) until under `_HARD_TRUNCATE_TOKEN_FRAC=0.5` of the window, priced with the calibrated estimator. If the newest turn alone is over budget it is trimmed inside (user head + newest messages kept) instead of being kept whole — the old "always keep the newest turn" rule made the fallback a no-op exactly for a first turn of a few huge messages. Last-resort, no LLM. | `OLLAMA_NUM_CTX`, `profile.context_window` | `compressor.py` | ✅ | +| **Per-message view truncation** | `ContextBuilder._truncate_oversized` head/tail-truncates a single `tool`/`assistant` message whose estimated size exceeds the per-message budget in the *built* context only — stored history is never mutated. User messages are capped at least at `OLLAMA_NUM_CTX // 3` (intent gets more room, but an oversized paste cannot dead-end the session). 0 = auto (`OLLAMA_NUM_CTX // 6` for tool/assistant). | `CONTEXT_MESSAGE_TOKEN_BUDGET` | `context_builder.py` | ✅ | | **Profile-aware compression overrides** | `compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file` override the global defaults inside `compress_context` / `compress_session` / the summary system prompt; `context_window` overrides the window for every decision and is sent as `num_ctx`. `Agent.set_profile()` propagates the active profile to the compressor. | `profile.context_window`, `profile.compression_keep_recent`, `profile.compression_max_tokens`, `profile.compression_prompt_file` | `compressor.py`, `profiles/base.py` | ✅ | -| **Real-token baseline estimator** | `record_real_baseline()` caches the last call's real `prompt_eval_count` keyed by `len(session.context)` (and feeds the calibration); `real_baseline_estimate()` returns `real_tokens + calibrated(messages since)` so code-heavy contexts are estimated accurately instead of undercounted by a character heuristic. Used by the pre-turn, mid-turn and `/compact` gates and `check_context_size`. Cleared after compression. | None | `compressor.py`, `agent.py` | ✅ | +| **Real-token baseline estimator** | `record_real_baseline()` caches the last call's real `prompt_eval_count` keyed by `len(session.context)` (and feeds the calibration); `real_baseline_estimate()` returns `real_tokens + calibrated(messages since)`, all priced on the **truncated view** the model is actually sent, so code-heavy contexts are estimated accurately and a raw megabyte-long tool result cannot look like the whole window. Used by the pre-turn, mid-turn and `/compact` gates and `check_context_size`. Cleared after compression. | None | `compressor.py`, `agent.py` | ✅ | | **Per-model token calibration** | `TokenCalibration.tokens_for(chars)` — least-squares line `overhead + chars × tokens_per_char` over the last 8 real observations for a model; fallback `chars / 1.8`. Every keep-region target and budget is measured in these real tokens. | None | `token_calibration.py` | ✅ | | **Context-only demotion persistence** | Demoted rows absent from `session.messages` are flipped in the database through `SessionStore.set_context_flag()` (`UPDATE session_messages SET is_context=…` by sequence number, no `is_display` filter) — `save()` alone would leave them `is_context=True` on the next load. | None | `compressor.py`, `pg_session_store.py` | ✅ | -| **Summary input formatter** | Renders messages as plain text for summarizer: preserves summaries, notes image counts, renders tool calls compactly, collects base64 images for vision models. Critical tool results (`is_compression_critical` or critical tool names) survive verbatim up to 4000 chars, then head+tail halves (2000+2000) instead of collapsing to the non-critical preview; non-critical results capped at an 800-char preview. | None | `compressor.py` | ✅ | +| **Summary input formatter** | Renders messages as plain text for summarizer: preserves summaries, notes image counts, renders tool calls compactly, collects base64 images for vision models up to an image budget (`window × 0.25` tokens, priced per image) — images over it stay out of the call, their `[+ N image(s)]` note still marks them. Critical tool results (`is_compression_critical` or critical tool names) survive verbatim up to 4000 chars, then head+tail halves (2000+2000) instead of collapsing to the non-critical preview; non-critical results capped at an 800-char preview. | `_SUMMARY_IMAGE_BUDGET_FRAC` | `compressor.py` | ✅ | | **Summary input truncate** | Hard cap of 32 000 chars on formatted input sent to summarizer LLM, keeping head (75%, oldest messages) + tail (25%, the messages closest to the kept window) — the newest summarized work is never silently dropped. | Hard-coded `_MAX_SUMMARY_INPUT_CHARS=32000` | `compressor.py` | ✅ | | **Meta-summary consolidation** | When `to_summarize` contains multiple existing summaries totaling > `_META_SUMMARY_THRESHOLD≈10666` chars, consolidates them into one via `_meta_summarize` first so old summaries don't crowd the summarizer input. | None | `compressor.py` | ✅ | | **LLM-based summarization** | Calls LLM with structured summarization prompt (`think=False` for speed) and replaces old messages with an `is_summary=True` user message. Failure is non-fatal. | `CONTEXT_SUMMARY_TEMPERATURE`, `CONTEXT_SUMMARY_MAX_TOKENS`, `profile.compression_max_tokens` | `compressor.py` | ✅ | diff --git a/docs/sessions.md b/docs/sessions.md index 118d0f0..0171fcc 100644 --- a/docs/sessions.md +++ b/docs/sessions.md @@ -104,7 +104,10 @@ - The calibration fits a line `tokens ≈ overhead + chars × tokens_per_char` to the last 8 observations of `(chars of the built context, real prompt_eval_count)`, recorded by `record_real_baseline()` whenever a real count comes back from Ollama. It needs at least two observations whose sizes differ by 1.25× or more, and rejects an implausible slope (outside 1 token/char … 1 token per 4 chars) — which in practice only happens when a prompt was truncated server-side. - Without a usable line it falls back to `chars / 1.8`. The old `chars // 3` was measured ~1.8× below the real prompt size on prod: "65% of the window" was met on paper while the real prompt stayed above the window, so compression re-fired on the next messages (48 passes in a day on one session). -- Images add 500 tokens each. Calibration is per model (`calibration_for(model)`), so an observation from one model never prices another, and it lives in a module-level registry because the post-turn worker builds a fresh compressor per pass. +- Images are priced one by one from their own header (`navi/core/image_tokens.py`): `ceil(w/32) × ceil(h/32) + 2` after a downscale to at most 2048² px, clamped to 256…4096, measured against qwen3.6. A 1024×1024 image costs ~1026 tokens and a 2048² one ~4098; the flat 500 this code used to charge undercounted them 2×–8×, so a session full of screenshots never reached the 90% trigger. An image whose header cannot be read falls back to `context_image_tokens` (1024). Calibration is per model (`calibration_for(model)`), so an observation from one model never prices another, and it lives in a module-level registry because the post-turn worker builds a fresh compressor per pass. +- A prompt that carried images is **not** fed to the calibration fit: image bytes are not in `chars`, so their tokens would land in the fitted intercept/slope and skew every later text estimate for that model. + +**What is counted is the view the model is sent.** `real_baseline_estimate()` prices the delta messages after `truncate_oversized()`, and `compress_session`/`would_compress` wrap their estimator in it too. The raw stored context may hold a megabyte-long tool result that `build()` caps per message; counting it as ~167k tokens made the pre/mid-turn gates fire for work the partition intentionally refuses (a single message cannot be split) and, worse, made `check_context_size` end the run with "context too large" over a prompt the model never received. The reported failure — the *first* operation fills the window, compression does not happen and the session dies — was exactly that mismatch: the check printed the truncated size (~18k) in its own error text while deciding on the raw size (167k). The window is a parameter, not a global. `AgentProfile.context_window` (or `settings.ollama_num_ctx` when unset) is resolved once per turn (`Agent._window_for`) and threaded through `compress_and_save_session(..., window=)` → `compress_context` / `_hard_truncate` / `check_context_size`, and is also sent to Ollama as `num_ctx`. Without this, a model with a larger real window than `ollama_num_ctx` is compressed against the wrong budget. @@ -112,9 +115,9 @@ Three trigger points: -1. **Pre-turn** (in `run_stream()` → `_compression_events_preturn`): before the first LLM call of a turn, estimates tokens via `real_baseline_estimate(session.context, session.context)` (real `prompt_eval_count` from the last call + calibrated delta for the messages appended since it; `estimate_tokens` only when no baseline exists yet) and compresses when `tokens >= window * threshold`. Guarded by `would_compress(..., current_tokens=estimated_tokens, model=...)` so `CompressionStarted` is only emitted when the partition can actually shrink the stored context; the dry run reuses the caller's real-token count instead of re-deriving it with a different heuristic. +1. **Pre-turn** (in `run_stream()` → `_compression_events_preturn`): before the first LLM call of a turn, estimates tokens via `real_baseline_estimate(session.context, session.context)` (real `prompt_eval_count` from the last call + calibrated delta for the messages appended since it; `estimate_tokens` only when no baseline exists yet) and adds the tokens of the incoming message — including its images — which is not yet in `session.context`. Compresses when `tokens >= window * threshold`. Guarded by `would_compress(..., current_tokens=estimated_tokens, model=...)` so `CompressionStarted` is only emitted when the partition can actually shrink the stored context; the dry run reuses the caller's real-token count instead of re-deriving it with a different heuristic. 2. **Mid-turn** (in `run_stream()` → `_compression_events_midturn`, every iteration > 0): estimates tokens via `real_baseline_estimate(session.context, preflight_ctx)` and compresses with `keep_recent_messages=max(12, context_keep_recent*2)`. This is what keeps long autonomous loops (one user message + many tool iterations = one turn) from exhausting the window. -3. **Post-turn** (via `CompressionWorker`): after `StreamEnd`, the worker re-checks (using the real `ctx.context_tokens` from the last call and `WorkerContext.max_context_tokens` as the window) and delegates to `compress_and_save_session(reason="postturn", window=ctx.max_context_tokens)` with the mid-turn `keep_recent_messages` — the same pipeline (retry, hard-truncate, safety net, archiving) as the pre/mid-turn paths. +3. **Post-turn** (via `CompressionWorker`): after `StreamEnd`, the worker re-checks (using the real `ctx.context_tokens` from the last call and `WorkerContext.max_context_tokens` as the window) and delegates to `compress_and_save_session(reason="postturn", window=ctx.max_context_tokens)` with the mid-turn `keep_recent_messages` — the same pipeline (retry, hard-truncate, safety net, archiving) as the pre/mid-turn paths. It also compresses unconditionally when `ctx.context_truncated` is set, because a truncated prompt reports about half the window and the threshold can never see it (see **Truncated prompts** below). A fourth, on-demand path is **forced `/compact`** (`compact_stream()`): the client sends `{"type":"compact"}`, bypassing the threshold entirely; emits `CompressionStarted` + `ContextCompressed`; raises `NothingToCompactError` when there is nothing to compress. @@ -127,12 +130,17 @@ - `context_summary_max_tokens: int = 6000` — max output tokens for the summary LLM call - `output_reserve_tokens: int = 2048` — headroom reserved for the response in `check_context_size` - `context_message_token_budget: int = 0` — per-message view truncation budget (`0` = auto, active window // 6) +- `context_image_tokens: int = 1024` — price of one image whose dimensions cannot be read from its header; readable images are priced by area Per-profile overrides (`AgentProfile`): `context_window` (the window itself), `compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file` — the compression ones are applied inside `compress_context` / `compress_session` / the summary system prompt. navi_code uses `compression_keep_recent=12`. ### Context size guard -Before every LLM call, `check_context_size(built_ctx, session_context=session.context, model=profile.model, window=self._window_for(profile))` raises `ContextTooLargeError` when the estimated input exceeds `window - output_reserve_tokens`. The total uses `real_baseline_estimate()` when a baseline is available (real bulk + calibrated delta), falling back to `estimate_tokens()` (calibrated line, `chars / 1.8` without one, + 500 per image). `output_reserve_tokens` is capped at a quarter of the window, so a deliberately small test/profile window does not fail every check. The error is surfaced to the user as a synthesized assistant response + `StreamEnd` rather than a raw system error. +Before every LLM call, `check_context_size(built_ctx, session_context=session.context, model=profile.model, window=self._window_for(profile))` raises `ContextTooLargeError` when the estimated input exceeds `window - output_reserve_tokens`. The total uses `real_baseline_estimate()` when a baseline is available (real bulk + calibrated delta), falling back to `estimate_tokens()` (calibrated line, `chars / 1.8` without one, plus the per-image cost described above). `output_reserve_tokens` is capped at a quarter of the window, so a deliberately small test/profile window does not fail every check. The error is surfaced to the user as a synthesized assistant response + `StreamEnd` rather than a raw system error. + +### Truncated prompts + +Ollama does not error on a prompt bigger than `num_ctx`: it silently evaluates about half the window, drops the head of the prompt (system prompt included) and reports only what it evaluated in `prompt_eval_count`. Recording that count as the session baseline pinned the session at ~50% — every later estimate started from the truncated number, the 90% trigger never fired, and the model kept receiving a prompt whose beginning was gone («сессия заполнена», while Navi's own meter said 50%). `record_real_baseline(..., sent_tokens=…)` therefore compares the reported count with our estimate of the prompt that was sent; below 70% (`_TRUNCATION_RATIO`) nothing is recorded, the previous honest baseline survives, and the compressor raises a `context_truncated` flag that the post-turn worker consumes to compress regardless of the threshold. The agent also keeps the session's token count at the honest estimate rather than the truncated one, so the UI meter does not drop after a truncated call. ### Compression algorithm @@ -144,7 +152,7 @@ - **Adaptive partitioning:** `_turn_importance` scores each turn; an important old turn can be swapped into the kept set in place of a filler-recent one. - **Intra-turn fallback** (`partition_current_turn_messages`, when `keep_recent_messages` is set): for a single long turn, keeps the current request + newest N messages verbatim and summarizes older messages from the same turn. 2. **Meta-summary:** if `to_summarize` contains multiple existing summaries totaling > `_META_SUMMARY_THRESHOLD ≈ 10_600` chars, consolidate them into one via `_meta_summarize` first so old summaries don't crowd the summarizer input. -3. Format `to_summarize` as plain text. Tool calls are shown as compact previews (max 120 chars for args). **Critical** tool results (`is_compression_critical=True` or critical tool names) survive verbatim up to 4000 chars, then head+tail halves (2000+2000) instead of collapsing to the non-critical preview; non-critical results are capped at an 800-char preview. Base64 images are collected for vision models. +3. Format `to_summarize` as plain text. Tool calls are shown as compact previews (max 120 chars for args). **Critical** tool results (`is_compression_critical=True` or critical tool names) survive verbatim up to 4000 chars, then head+tail halves (2000+2000) instead of collapsing to the non-critical preview; non-critical results are capped at an 800-char preview. Base64 images are collected for vision models up to an image budget (`window × 0.25`, priced per image); images over it are left out of the call, while their `[+ N image(s)]` note still tells the summarizer they existed. Without the cap, a dozen screenshots made a summarization prompt that could not fit its own window, was silently truncated, and produced a summary written as if the history had been read. 4. Truncate formatted input to `_MAX_SUMMARY_INPUT_CHARS = 32_000` chars, keeping head (75%, oldest messages) + tail (25%, the messages closest to the kept window) — a head-only cut used to silently drop the newest summarized work. 5. Call `llm.complete()` with `think=False` to produce a bullet-point summary. 6. Replace `to_summarize` with a single summary message (`role=user`, `is_summary=True`). @@ -206,6 +214,8 @@ open them with image_view: session_files/{id}/image_2.png, session_files/{id}/image_3.png] ``` +The budget is `0.8 × window − estimated context tokens`, and each image is charged its own cost (pixel area, see **Token accounting and the window**) rather than a flat constant — a 1024² screenshot takes four times the room of a 512² thumbnail, and pricing them alike admitted images the model could not actually take. + Before this, a pasted image existed only in the transcript: an agent that had been told `session_files/{id}/` exists went looking there for "the attached images", found nothing, and spent turns on fruitless `find` calls. An image that cannot be stored (no writable directory) is logged as `agent.image_save_failed` and skipped — it never costs the turn. Hidden turns (`hidden=True`, e.g. form submissions) write nothing: they carry machine payloads, not pastes. diff --git a/navi/core/agent.py b/navi/core/agent.py index 3dccbff..c20d9e7 100644 --- a/navi/core/agent.py +++ b/navi/core/agent.py @@ -48,6 +48,7 @@ from .anti_stall import AntiStallMonitor from .compressor import ContextCompressor, should_compress from .context_builder import ContextBuilder +from .image_tokens import context_image_tokens, estimate_image_tokens from .planning import PlanningEngine from navi.tools.plan import PlanRunner from .stream_guard import _iter_stream_guarded @@ -453,7 +454,14 @@ # threshold, compress NOW before calling the LLM. This prevents the # model from seeing an over-full context and generating gibberish # (e.g. a "summary of the conversation" instead of a real answer). - async for _ev in self._compression_events_preturn(session, llm, profile, session_id): + # The incoming message is priced here because the gate runs before it + # is appended — an attachment is the one part of the next prompt this + # context cannot yet account for. + pending = Message(role="user", content=user_message, images=images or None) + pending_tokens = self._compressor.estimate_tokens_delta([pending], profile.model) + async for _ev in self._compression_events_preturn( + session, llm, profile, session_id, pending_tokens=pending_tokens + ): yield _ev if hidden: @@ -498,7 +506,9 @@ # Image token budgeting: fit as many images as possible into the LLM context. # Images that do not fit stay reachable through the session directory, so the - # notice has to name them. + # notice has to name them. Each image is priced by its own size: a flat 500 + # admitted twice as many screenshots as the model can take (a 1024x1024 image + # is ~1026 tokens on qwen3.6 — see navi/core/image_tokens.py). images_for_context = images context_content = user_message if images: @@ -506,9 +516,15 @@ session.context, profile.model ) available_tokens = int(self._window_for(profile) * 0.8) - current_tokens - max_images = max(0, available_tokens // 500) - if max_images < len(images): - images_for_context = images[:max_images] + fitted: list[str] = [] + for img in images: + cost = estimate_image_tokens(img) + if cost > available_tokens: + break + fitted.append(img) + available_tokens -= cost + if len(fitted) < len(images): + images_for_context = fitted context_content += "\n\n" + _inline_images_notice( len(images), len(images_for_context or []), saved_paths ) @@ -676,11 +692,18 @@ # undercounts code-heavy tool output and fires compression too late). # len(session.context) here is exactly what was sent — the new # assistant/tool messages are appended only below. + # `sent_tokens` lets the compressor notice a prompt the backend cut + # down instead of caching the cut size as the truth (Ollama keeps + # roughly half the window and reports only that). self._compressor.record_real_baseline( len(session.context), state.context_tokens, chars=self._compressor.context_chars(built_ctx), model=turn_ctx.resolved_model or profile.model, + sent_tokens=self._compressor.real_baseline_estimate( + session.context, built_ctx, turn_ctx.resolved_model or profile.model + ), + image_tokens=context_image_tokens(built_ctx), ) # Stopped mid-stream — save partial response and exit @@ -716,7 +739,20 @@ ) session.messages.append(assistant_msg) session.context.append(assistant_msg) - session.context_token_count = state.context_tokens or 0 + if self._compressor.context_truncated: + # A truncated prompt is reported as what the backend + # evaluated — roughly half the window — while the context + # that produced it was already over the window. Showing + # that number would drop the bar to ~50% at the exact + # moment the session is full; keep the honest estimate. + session.context_token_count = max( + state.context_tokens or 0, + self._compressor.real_baseline_estimate( + session.context, session.context, profile.model + ), + ) + else: + session.context_token_count = state.context_tokens or 0 # Final-turn intercept: the model stopped with bare text, but the # anti-stall/adaptive nudges never fired (they live inside the @@ -947,6 +983,11 @@ temperature=settings.context_summary_temperature, session_store=self._sessions, profile=profile, + # The backend cut the last prompt down (record_real_baseline saw it). + # Its token count is below the trigger, but the context it came from + # is over the window — compress now instead of waiting for a + # threshold that the truncated count will never reach. + context_truncated=self._compressor.pop_context_truncated(), ) events: list[AgentEvent] = [] for worker in self._workers: @@ -1016,7 +1057,9 @@ tasks = await load_tasks_for(session.id, session.user_id) return TodoUpdated(session_id=session.id, tasks=tasks) - async def _compression_events_preturn(self, session, llm, profile, session_id): + async def _compression_events_preturn( + self, session, llm, profile, session_id, pending_tokens: int = 0 + ): # real_baseline_estimate, not a character heuristic: the baseline is # valid here (the context only grew since the last LLM call), and a # character count undercounts code-heavy tool output — firing @@ -1026,11 +1069,15 @@ session.context, session.context, profile.model ) window = self._window_for(profile) + # `pending_tokens` prices the user message that has not been appended + # yet — this gate runs before it is. Without it, a message carrying + # images could overshoot the window in one step: the context was under + # the threshold a moment ago and the images were the only thing over. if ( settings.context_compression_enabled and len(session.context) > 2 and should_compress( - estimated_tokens, + estimated_tokens + pending_tokens, window, settings.context_compression_threshold, ) @@ -1046,11 +1093,14 @@ keep_recent=settings.context_keep_recent, threshold=settings.context_compression_threshold, max_context_tokens=window, - current_tokens=estimated_tokens, + # Judge the fallback on the same total that fired the gate. + current_tokens=estimated_tokens + pending_tokens, model=profile.model, ): return yield CompressionStarted( + # The context being compressed does not include the pending + # message yet, so report the stored context's own size. context_tokens=estimated_tokens, max_context_tokens=window, ) diff --git a/navi/core/compressor.py b/navi/core/compressor.py index f5f08db..36822f8 100644 --- a/navi/core/compressor.py +++ b/navi/core/compressor.py @@ -27,6 +27,7 @@ from .events import ContextCompressed from .token_calibration import calibration_for from .context_builder import truncate_oversized +from .image_tokens import context_image_tokens, estimate_image_tokens if TYPE_CHECKING: from navi.profiles.base import AgentProfile @@ -116,11 +117,63 @@ return [m for m in reference if id(m) in wanted] +def _split_turn_tail( + turn: list[Message], + keep_recent_messages: int, + *, + budget_tokens: int | None = None, + estimate: Callable[[list[Message]], int] | None = None, +) -> tuple[list[Message], list[Message], bool] | None: + """Split an in-flight turn into (dropped_middle, kept_tail, trimmed_by_budget). + + The kept shape mirrors the midturn partition: the user request head plus + the newest ``keep_recent_messages`` messages verbatim. + + ``budget_tokens`` bounds the kept region (head + tail) in the caller's + units. The count window alone is useless once messages are huge: twelve + messages at a sixth of the window each are two windows. With a budget the + oldest tail messages move into the summary until the kept region fits + (the newest message and the user head are always kept — dropping the + in-flight request or its latest tool result is worse than overrunning). + + Returns None when nothing has to be dropped, so callers keep their + "nothing to compress" semantics. + """ + if not turn: + return None + head_len = 1 if turn[0].role == "user" else 0 + count_start = max(head_len, len(turn) - keep_recent_messages) + tail_start = count_start + trimmed = False + if budget_tokens is not None and estimate is not None: + head = turn[:head_len] + while ( + tail_start < len(turn) - 1 + and estimate(head + turn[tail_start:]) > budget_tokens + ): + tail_start += 1 + trimmed = True + dropped = turn[head_len:tail_start] + if dropped: + # Never start the verbatim tail on a tool result whose call moved into + # the summary: keep the assistant tool_call beside its results (an + # orphaned tool message is invalid for chat-completion APIs). + while tail_start > head_len and turn[tail_start].role == "tool": + tail_start -= 1 + dropped = turn[head_len:tail_start] + if not dropped: + return None + return dropped, turn[:head_len] + turn[tail_start:], trimmed + + def partition_messages( messages: list[Message], keep_recent: int, keep_recent_messages: int | None = None, adaptive: bool = True, + *, + budget_tokens: int | None = None, + estimate: Callable[[list[Message]], int] | None = None, ) -> tuple[list[Message], list[Message]]: """ Returns (to_summarize, to_keep). @@ -130,6 +183,8 @@ Tool call groups (assistant + tool results) always stay together. When adaptive=True, important turns (user corrections, errors, critical tools) are kept longer and social/filler turns are compressed sooner. + ``budget_tokens``/``estimate`` bound the verbatim in-flight turn by size; + see ``_split_turn_tail``. Summary messages (is_summary=True) are not part of any turn: they carry already-compressed history, not a new user request. They always go into @@ -156,7 +211,12 @@ if len(turns) <= keep_recent: if keep_recent_messages is not None: - intra_turn = partition_current_turn_messages(turns, keep_recent_messages) + intra_turn = partition_current_turn_messages( + turns, + keep_recent_messages, + budget_tokens=budget_tokens, + estimate=estimate, + ) if intra_turn is not None: old, to_keep = intra_turn # A compression pass replaces every old summary, never keeps @@ -204,16 +264,16 @@ # partition_current_turn_messages. Without this, the in-flight turn stays verbatim # whenever turns > keep_recent, which is the 150->141 shallow-compression symptom. if keep_recent_messages is not None and recent_turns: - in_flight = recent_turns[-1] - if len(in_flight) > keep_recent_messages + 1: - head = [in_flight[0]] if in_flight and in_flight[0].role == "user" else [] - tail_start = max(len(head), len(in_flight) - keep_recent_messages) - to_summarize = to_summarize + in_flight[len(head):tail_start] - to_keep = ( - [m for turn in recent_turns[:-1] for m in turn] - + head - + in_flight[tail_start:] - ) + split = _split_turn_tail( + recent_turns[-1], + keep_recent_messages, + budget_tokens=budget_tokens, + estimate=estimate, + ) + if split is not None: + dropped, kept, _ = split + to_summarize = to_summarize + dropped + to_keep = [m for turn in recent_turns[:-1] for m in turn] + kept else: to_keep = [m for turn in recent_turns for m in turn] else: @@ -228,6 +288,9 @@ def partition_current_turn_messages( turns: list[list[Message]], keep_recent_messages: int, + *, + budget_tokens: int | None = None, + estimate: Callable[[list[Message]], int] | None = None, ) -> tuple[list[Message], list[Message]] | None: """ Mid-turn fallback for long autonomous tool loops. @@ -235,36 +298,55 @@ A long chain of assistant/tool iterations after one user message is one conversational turn, so turn-based compression may have nothing to compress. Keep the current user request and the newest messages verbatim, then summarize - older messages from the same in-flight turn. + older messages from the same in-flight turn. ``budget_tokens`` also trims a + turn made of a few huge messages, which the message-count window alone + cannot split (see ``_split_turn_tail``). """ if not turns: return None - current_turn = turns[-1] - if len(current_turn) <= keep_recent_messages + 1: + split = _split_turn_tail( + turns[-1], + keep_recent_messages, + budget_tokens=budget_tokens, + estimate=estimate, + ) + if split is None: return None + dropped, kept, trimmed = split - head = [current_turn[0]] if current_turn and current_turn[0].role == "user" else [] - tail_start = max(len(head), len(current_turn) - keep_recent_messages) - to_summarize = [m for turn in turns[:-1] for m in turn] + current_turn[len(head):tail_start] - to_keep = head + current_turn[tail_start:] + to_summarize = [m for turn in turns[:-1] for m in turn] + dropped + to_keep = kept - if len(to_summarize) < 2: + # A budget-driven trim may leave a single message to summarize; that message + # is exactly the oversized one the count window could not address. + if len(to_summarize) < (1 if trimmed else 2): return None return to_summarize, to_keep -def _format_for_summary(messages: list[Message]) -> tuple[str, list[str]]: +def _format_for_summary( + messages: list[Message], image_budget: int | None = None +) -> tuple[str, list[str]]: """ Render messages as plain text for the summarization prompt. Returns (text, images) where images is a flat list of base64 strings collected from all user messages. Vision-capable models will receive the images alongside the text; non-vision models silently ignore them. + + ``image_budget`` caps the total cost of the collected images in tokens. + Every image in the history used to be forwarded in one user message with + only the text capped, so a session with a dozen screenshots produced a + summarization prompt that could not fit its own window — the call was + silently truncated by the backend and the summary was written as if the + history had been read. Images over the budget stay out of the call; their + per-message "[+ N image(s)]" note still tells the summarizer they existed. """ lines: list[str] = [] images: list[str] = [] + remaining = image_budget if image_budget is None else max(0, image_budget) i = 0 while i < len(messages): m = messages[i] @@ -276,7 +358,14 @@ elif m.role == "user": if m.images: - images.extend(m.images) + for img in m.images: + if remaining is None: + images.append(img) + continue + cost = estimate_image_tokens(img) + if cost <= remaining: + images.append(img) + remaining -= cost img_note = f" [+ {len(m.images)} image(s)]" else: img_note = "" @@ -337,6 +426,17 @@ # Typical ollama_num_ctx is 65k–128k, so 32k of input leaves ample room for a 6k-token summary. _MAX_SUMMARY_INPUT_CHARS = 32_000 +# Share of the window the historical images may take in a summarization call +# (see _format_for_summary). The text side is capped separately, in chars. +_SUMMARY_IMAGE_BUDGET_FRAC = 0.25 + +# A call that reports fewer real prompt tokens than this share of our own +# estimate of the prompt was sent a truncated prompt (Ollama keeps roughly +# half the window and evaluates that). The estimate errs high on images, so +# the band is deliberately wide — a false positive costs one extra +# compression pass, a missed one costs the session. +_TRUNCATION_RATIO = 0.7 + # Hard-truncate fallbacks keep at most this fraction of the context window's # tokens of recent messages (the rest is dropped without summarizing). _HARD_TRUNCATE_TOKEN_FRAC = 0.5 @@ -434,21 +534,33 @@ substantial to compress. """ system_msgs = [m for m in context if m.role == "system"] + est = estimate or ContextCompressor.estimate_context_tokens + target_tokens = int( + (window or settings.ollama_num_ctx) * settings.context_compression_target + ) + # What the verbatim kept region may take: the target minus the system + # messages. Passed down so the intra-turn split also works when a few huge + # messages put the kept region over the target despite a small message count. + keep_budget = max(1, target_tokens - est(system_msgs)) to_summarize, to_keep = partition_messages( - context, keep_recent, keep_recent_messages=keep_recent_messages + context, + keep_recent, + keep_recent_messages=keep_recent_messages, + budget_tokens=keep_budget, + estimate=est, ) if keep_recent_messages is None: - est = estimate or ContextCompressor.estimate_context_tokens - target_tokens = int( - (window or settings.ollama_num_ctx) * settings.context_compression_target - ) kept_tokens = est(system_msgs + to_keep) _KEEP_RECENT_FLOOR = 2 while kept_tokens > target_tokens and keep_recent > _KEEP_RECENT_FLOOR: keep_recent -= 1 to_summarize, to_keep = partition_messages( - context, keep_recent, keep_recent_messages=keep_recent_messages + context, + keep_recent, + keep_recent_messages=keep_recent_messages, + budget_tokens=keep_budget, + estimate=est, ) kept_tokens = est(system_msgs + to_keep) @@ -457,6 +569,8 @@ context, keep_recent, keep_recent_messages=2, + budget_tokens=keep_budget, + estimate=est, ) if len(to_summarize) < 2: @@ -525,7 +639,15 @@ # If meta-summary fails, continue with raw summaries as-is pass - summary_text_input, images = _format_for_summary(to_summarize) + # Images forwarded to the summarizer share one user message with the + # formatted text. Give them a quarter of the window at most: the summary + # itself needs the rest, and a summarization call that overflows is + # silently truncated by Ollama (to half the window), which would produce a + # summary written from an arbitrary slice of the history. + image_budget = int((window or settings.ollama_num_ctx) * _SUMMARY_IMAGE_BUDGET_FRAC) + summary_text_input, images = _format_for_summary( + to_summarize, image_budget=image_budget + ) # Truncate oversized input so the summarizer LLM has room to generate output. # Keep head + tail, not just the head: to_summarize is chronological, so a @@ -584,8 +706,7 @@ while the real prompt sat above the window. """ chars = ContextCompressor.context_chars(context) - imgs = sum(500 * len(m.images) for m in context if m.images) - return chars // 3 + imgs + return chars // 3 + context_image_tokens(context) @staticmethod def estimate_tokens(context: list[Message], model=None) -> int: @@ -597,8 +718,7 @@ calibration falls back to ``DEFAULT_CHARS_PER_TOKEN``. """ chars = ContextCompressor.context_chars(context) - imgs = sum(500 * len(m.images) for m in context if m.images) - return calibration_for(model).tokens_for(chars) + imgs + return calibration_for(model).tokens_for(chars) + context_image_tokens(context) @staticmethod def estimate_tokens_delta(context: list[Message], model=None) -> int: @@ -610,8 +730,7 @@ window in another for exactly the tool-heavy turns they exist for. """ chars = ContextCompressor.context_chars(context) - imgs = sum(500 * len(m.images) for m in context if m.images) - return calibration_for(model).delta_for(chars) + imgs + return calibration_for(model).delta_for(chars) + context_image_tokens(context) def __init__(self) -> None: self._profile: "AgentProfile | None" = None @@ -622,11 +741,26 @@ # chars//3 estimate which undercounts code-heavy tool output and fires # compression too late. Cleared after compression (context shrank). self._real_baseline: tuple[int, int] | None = None + # Set when the last call reported far fewer tokens than we sent: the + # backend truncated the prompt. Consumed by the post-turn worker, which + # compresses unconditionally in that case (see record_real_baseline). + self._context_truncated = False def set_profile(self, profile: "AgentProfile | None") -> None: """Tell the compressor which profile is active so it can use profile-specific settings.""" self._profile = profile + @property + def context_truncated(self) -> bool: + """True when the last recorded call saw its prompt cut by the backend.""" + return self._context_truncated + + def pop_context_truncated(self) -> bool: + """Read and clear the truncation flag (the post-turn worker consumes it).""" + value = self._context_truncated + self._context_truncated = False + return value + def record_real_baseline( self, context_len: int, @@ -634,6 +768,8 @@ *, chars: int | None = None, model=None, + sent_tokens: int | None = None, + image_tokens: int = 0, ) -> None: """Cache the real prompt-token count returned by the last LLM call. @@ -646,12 +782,39 @@ (the built context, not just session.context). Together with ``real_tokens`` it feeds the per-model calibration line, which is what converts hypothetical contexts (the proposed kept region) into real - tokens. + tokens. Prompts that carried images are kept out of that fit: image + bytes are not in ``chars``, so their tokens would land in the fitted + intercept/slope and skew every later text estimate for that model. + + ``sent_tokens`` is our estimate of the prompt that was sent. When the + backend reports far fewer tokens than that, the prompt was truncated — + Ollama silently keeps about half the window and reports only what it + evaluated. Caching that number as the baseline used to pin the session + at ~50%: every later estimate started from the truncated count, the + 90% trigger never fired, and the model kept receiving a prompt whose + head (system prompt included) had been dropped. A truncated count is + therefore not recorded; the previous — honest — baseline stays, the + estimate keeps its size, and ``context_truncated`` tells the post-turn + worker to compress regardless of the threshold. """ - if real_tokens and real_tokens > 0: - self._real_baseline = (context_len, int(real_tokens)) - if chars: - calibration_for(model).observe(chars, real_tokens) + if not real_tokens or real_tokens <= 0: + return + if ( + sent_tokens + and sent_tokens > 0 + and real_tokens < sent_tokens * _TRUNCATION_RATIO + ): + self._context_truncated = True + log.warning( + "compressor.prompt_truncated", + sent_tokens=sent_tokens, + real_tokens=real_tokens, + model=model, + ) + return + self._real_baseline = (context_len, int(real_tokens)) + if chars and not image_tokens: + calibration_for(model).observe(chars, real_tokens) def real_baseline_estimate( self, @@ -665,15 +828,23 @@ no baseline is recorded or it is stale (the session shrank, e.g. right after compression — ``base_len`` exceeds the current length). Every branch is in real prompt tokens, the unit the window is measured in. + + Both the delta and the fallback are priced on the **truncated view** + (``truncate_oversized``), i.e. on what ``build()`` actually sends. The + raw stored context may hold a megabyte-long tool result that never + reaches the model; pricing it here made the gates fire for work the + compressor intentionally refuses (a single message cannot be split) + and — worse — made ``check_context_size`` kill the run with "context + too large" over a prompt the model never received. """ if self._real_baseline is not None: base_len, base_tokens = self._real_baseline if base_len <= len(session_context): delta = self.estimate_tokens_delta( - session_context[base_len:], model + truncate_oversized(session_context[base_len:]), model ) return base_tokens + delta - return self.estimate_tokens(built_ctx, model) + return self.estimate_tokens(truncate_oversized(built_ctx), model) async def compress_session( self, @@ -697,10 +868,13 @@ window_tokens = window or settings.ollama_num_ctx # Every size decision in this pass — the keep-region target, the # post-check, the truncate budgets — must be in real-token units, the - # same units the trigger fired in. Mixed units are what let a - # compression pass finish at "65% of the window" while the real prompt - # it produced was above the window. - est = lambda msgs: self.estimate_tokens(msgs, model) + # same units the trigger fired in, AND on the view the model is sent + # (``truncate_oversized``): pricing raw stored messages made a kept + # megabyte-long tool result look like the whole window and drop turns + # the model could have had. Mixed units are what let a compression + # pass finish at "65% of the window" while the real prompt it produced + # was above the window. + est = lambda msgs: self.estimate_tokens(truncate_oversized(msgs), model) # Attempt 1: normal compression try: @@ -794,7 +968,9 @@ """ effective_keep_recent = getattr(self._profile, "compression_keep_recent", None) or keep_recent window_tokens = max_context_tokens or settings.ollama_num_ctx - est = lambda msgs: self.estimate_tokens(msgs, model) + # Same unit as the caller's real-token estimate: the truncated view the + # model is actually sent, not the raw stored messages. + est = lambda msgs: self.estimate_tokens(truncate_oversized(msgs), model) if _plan_compression( context, effective_keep_recent, @@ -890,6 +1066,21 @@ kept_turns.insert(0, turn) used += turn_tokens + if kept_turns and used > budget_tokens: + # The newest turn alone is over budget (a first turn of a few huge + # messages): the loop above always keeps it whole, which used to + # make this fallback a no-op exactly when it was needed. Trim inside + # it — the user head and the newest messages stay verbatim. + newest = kept_turns[-1] + split = _split_turn_tail( + newest, + len(newest), + budget_tokens=budget_tokens, + estimate=est, + ) + if split is not None: + kept_turns[-1] = split[1] + to_keep = [m for turn in kept_turns for m in turn] if len(to_keep) == len(non_system): return None # already within budget @@ -1035,8 +1226,10 @@ truncate_oversized(new_context), model ) # The context just shrank, so the recorded real-token baseline is stale. - # Drop it — the next LLM call re-establishes a fresh one. + # Drop it — the next LLM call re-establishes a fresh one. The truncation + # flag goes with it: the prompt that will be sent is small again. self._real_baseline = None + self._context_truncated = False await session_store.save(session) log.info( diff --git a/navi/core/context_builder.py b/navi/core/context_builder.py index cd8d5f6..b89c603 100644 --- a/navi/core/context_builder.py +++ b/navi/core/context_builder.py @@ -38,9 +38,29 @@ # Roles whose content may be head/tail-truncated when a single message is too -# large for the context window. User messages carry intent and are never -# truncated; system/summary messages are already compact. -_TRUNCATABLE_ROLES = {"tool", "assistant"} +# large for the context window. System/summary messages are already compact. +# User messages carry intent, so their cap is deliberately more generous than +# tool output's (see `_truncation_budget`) — but leaving them uncapped at all +# turned a single oversized paste into a dead session: the size check refused +# every subsequent call while compression had nothing it could split. +_TRUNCATABLE_ROLES = {"tool", "assistant", "user"} + +# A user message may use up to this fraction of the window before its middle is +# elided for the LLM view (the stored history keeps the full text). +_USER_MESSAGE_WINDOW_FRAC = 1 / 3 + + +def _truncation_budget(role: str) -> int: + """Per-message view cap in tokens for a truncatable role.""" + budget = _config.settings.context_message_token_budget + if budget <= 0: + budget = max(1, _config.settings.ollama_num_ctx // 6) + if role == "user": + budget = max( + budget, + max(1, int(_config.settings.ollama_num_ctx * _USER_MESSAGE_WINDOW_FRAC)), + ) + return budget # Which stored system-role messages are infrastructure (and therefore dropped # from the LLM view) and which are addressed to the model. `task_note` carries @@ -50,11 +70,11 @@ _LLM_SYSTEM_SOURCES = {"task_note", "synapse_event"} -def _truncate_content(content: str, budget_tokens: int) -> str: +def _truncate_content(content: str, budget_tokens: int, *, label: str = "truncated") -> str: """Head/tail-truncate ``content`` to roughly ``budget_tokens`` tokens. Keeps the head and tail verbatim with a marker naming the elided middle, so - the model retains the beginning and end of a large tool result (usually the + the model retains the beginning and end of a large result (usually the most informative parts) without the whole thing blowing the context window. """ budget_chars = budget_tokens * 3 @@ -62,19 +82,20 @@ return content half = budget_chars // 2 dropped_tokens = (len(content) - 2 * half) // 3 - marker = f"\n[…truncated ~{dropped_tokens:,} tokens…]\n" + marker = f"\n[…{label} ~{dropped_tokens:,} tokens…]\n" return content[:half] + marker + content[-half:] def truncate_oversized(conv: list[Message]) -> list[Message]: - """Head/tail-truncate oversized tool/assistant messages for the LLM view. + """Head/tail-truncate oversized messages for the LLM view. A single huge tool result (e.g. reading a large file) can alone exceed the context window, and the turn/message-count-based compressor cannot shrink a single message. This per-build pass caps each truncatable - message at ``context_message_token_budget`` tokens (0 = num_ctx // 6), - keeping head + tail so the model still sees the start and end of the - result. Copies are returned — stored history (session.messages) is never + message at its role budget — ``context_message_token_budget`` tokens + (0 = num_ctx // 6) for tool/assistant output, at least num_ctx // 3 for + user text — keeping head + tail so the model still sees the start and end. + Copies are returned — stored history (session.messages) is never mutated, so the user still sees full content and reloads are unaffected. Module-level because the compressor reports the session's context size in @@ -82,16 +103,21 @@ the model is sent, and counting it as such shows a scary 800k over a 128k window. """ - budget = _config.settings.context_message_token_budget - if budget <= 0: - budget = max(1, _config.settings.ollama_num_ctx // 6) - cap_chars = budget * 3 out: list[Message] = [] for m in conv: - if m.role in _TRUNCATABLE_ROLES and len(m.content or "") > cap_chars: - out.append(m.model_copy(update={"content": _truncate_content(m.content or "", budget)})) - else: + if m.role not in _TRUNCATABLE_ROLES: out.append(m) + continue + budget = _truncation_budget(m.role) + if len(m.content or "") <= budget * 3: + out.append(m) + continue + label = "user message truncated" if m.role == "user" else "truncated" + out.append( + m.model_copy( + update={"content": _truncate_content(m.content or "", budget, label=label)} + ) + ) return out diff --git a/navi/core/image_tokens.py b/navi/core/image_tokens.py new file mode 100644 index 0000000..c3e7d09 --- /dev/null +++ b/navi/core/image_tokens.py @@ -0,0 +1,206 @@ +"""What an inline image costs the model, measured instead of guessed. + +Every size decision in the compressor prices images at a flat 500 tokens. +On the local vision model that constant is off by 2x for an ordinary +screenshot and by 8x for a large photo, and the error is invisible: image +bytes are not in ``context_chars``, so nothing else in the estimate sees +them. A session that carries a handful of images therefore sits below the +90% trigger while the prompt it sends is already past the window — and once +Ollama truncates that prompt, the reported ``prompt_eval_count`` is what gets +cached as the baseline, so the count never recovers. + +The measurement (Ollama 0.40.15, ``qwen3.6:35b``, ``prompt_eval_count`` of the +same prompt with and without an image, two calls each):: + + 256x256 -> 66 tokens 1024x768 -> 770 tokens + 512x512 -> 258 tokens 1536x1536 -> 2306 tokens + 1024x1024 -> 1026 tokens 2048x2048 -> 4098 tokens + 800x600 -> 477 tokens 3072x3072 -> 4098 tokens (downscaled) + +That is one token per 32x32 pixel block plus the two image markers, and a +downscale to at most 2048x2048 pixels before tokenizing. The estimate below +follows the same shape with *ceil* rounding, so it errs high rather than low: +over-counting costs an early compression, under-counting costs the session. + +The per-32px tile size is a property of the model's vision encoder, not a +universal law (llava-style encoders charge a flat ~256 for any image, and a +cloud model may charge differently). A model that prices images unlike +qwen3.6 will therefore be estimated a little high; the real-prompt baseline +corrects the total after the first successful call, and this estimate only +supplies the delta — so the residual is small and always in the safe +direction. +""" + +from __future__ import annotations + +import base64 +import math +import struct +from functools import lru_cache + +from navi.config import settings + +# One vision token per this many pixels per side (measured on qwen3.6). +_IMAGE_PATCH_PX = 32 +# Ollama downscales anything larger to this budget before tokenizing. +_MAX_IMAGE_PIXELS = 2048 * 2048 +# A floor keeps a tiny icon from pricing at ~0; a ceiling keeps a corrupt +# header from pricing an image at a whole window. +_MIN_IMAGE_TOKENS = 256 +_MAX_IMAGE_TOKENS = 4096 + +# Enough of the base64 payload to hold a PNG IHDR, a WebP chunk header or a +# JPEG SOF marker for the images this server produces (canvas, PIL — neither +# keeps EXIF). Only this prefix is kept as the cache key, never the payload. +_HEAD_CHARS = 4096 + +_NO_DIMS: tuple[int, int] | None = None + + +def context_image_tokens(messages) -> int: + """Total image cost of a message list, in real-prompt token units.""" + return sum( + estimate_image_tokens(img) + for m in messages + for img in (m.images or []) + ) + + +def estimate_image_tokens(data: str) -> int: + """Token cost of one base64 image (data-URI prefix accepted).""" + if not data: + return 0 + return _cost(len(data), data[:_HEAD_CHARS], int(settings.context_image_tokens)) + + +@lru_cache(maxsize=256) +def _cost(encoded_len: int, head: str, fallback: int) -> int: + """Cached by (length, header prefix) so one estimate does not re-parse. + + Two different images of the same encoded length and the same first 4 KB + are, for every format this server writes, the same image — and even a + collision only moves the estimate by a few hundred tokens. + """ + dims = _dimensions(head) + if dims is None: + return max(1, min(int(fallback), _MAX_IMAGE_TOKENS)) + return tokens_for_size(*dims) + + +def tokens_for_size(width: int, height: int) -> int: + """Estimated tokens for an image of ``width`` x ``height`` pixels.""" + if width <= 0 or height <= 0: + return _MIN_IMAGE_TOKENS + pixels = width * height + if pixels > _MAX_IMAGE_PIXELS: + scale = math.sqrt(_MAX_IMAGE_PIXELS / pixels) + width = max(1, int(width * scale)) + height = max(1, int(height * scale)) + tiles = _ceil_div(width, _IMAGE_PATCH_PX) * _ceil_div(height, _IMAGE_PATCH_PX) + return max(_MIN_IMAGE_TOKENS, min(_MAX_IMAGE_TOKENS, tiles + 2)) + + +def _ceil_div(value: int, divisor: int) -> int: + return -(-value // divisor) + + +def _dimensions(data: str) -> tuple[int, int] | None: + raw = _prefix_bytes(data) + if not raw: + return _NO_DIMS + for parser in (_png_size, _jpeg_size, _webp_size, _gif_size, _bmp_size): + dims = parser(raw) + if dims is not None: + return dims + return _NO_DIMS + + +def _prefix_bytes(data: str, max_bytes: int = 48 * 1024) -> bytes: + """Decode the head of a base64 payload — enough for every header we read.""" + s = data.strip() + if s.startswith("data:"): + comma = s.find(",") + if comma == -1: + return b"" + s = s[comma + 1:] + take = min(len(s), _HEAD_CHARS) + chunk = "".join(s[:take].split()) + chunk = chunk[: len(chunk) - len(chunk) % 4] + if not chunk: + return b"" + try: + return base64.b64decode(chunk)[:max_bytes] + except ValueError: # binascii.Error (bad padding / alphabet) is a ValueError + return b"" + + +def _png_size(b: bytes) -> tuple[int, int] | None: + if len(b) < 24 or b[:8] != b"\x89PNG\r\n\x1a\n" or b[12:16] != b"IHDR": + return None + return struct.unpack(">II", b[16:24]) + + +def _jpeg_size(b: bytes) -> tuple[int, int] | None: + if len(b) < 4 or b[:2] != b"\xff\xd8": + return None + i, n = 2, len(b) + while i + 4 <= n: + if b[i] != 0xFF: + i += 1 + continue + marker = b[i + 1] + if marker == 0xFF: # fill byte before the real marker + i += 1 + continue + if marker in (0x01, 0xD8) or 0xD0 <= marker <= 0xD7: + i += 2 # markers with no length field + continue + if marker in (0xD9, 0xDA): # end of image / scan: no SOF before this + return None + seg_len = int.from_bytes(b[i + 2:i + 4], "big") + if seg_len < 2: + return None + # SOF0..SOF15, minus DHT (C4), JPG (C8) and DAC (CC). + if 0xC0 <= marker <= 0xCF and marker not in (0xC4, 0xC8, 0xCC): + if i + 9 > n: + return None + height = int.from_bytes(b[i + 5:i + 7], "big") + width = int.from_bytes(b[i + 7:i + 9], "big") + return width, height + i += 2 + seg_len + return None + + +def _webp_size(b: bytes) -> tuple[int, int] | None: + if len(b) < 30 or b[:4] != b"RIFF" or b[8:12] != b"WEBP": + return None + fmt = b[12:16] + if fmt == b"VP8X": + width = 1 + int.from_bytes(b[24:27], "little") + height = 1 + int.from_bytes(b[27:30], "little") + return width, height + if fmt == b"VP8 ": + if b[23:26] != b"\x9d\x01\x2a": + return None + width = int.from_bytes(b[26:28], "little") & 0x3FFF + height = int.from_bytes(b[28:30], "little") & 0x3FFF + return width, height + if fmt == b"VP8L": + if b[20] != 0x2F: + return None + bits = int.from_bytes(b[21:25], "little") + return (bits & 0x3FFF) + 1, ((bits >> 14) & 0x3FFF) + 1 + return None + + +def _gif_size(b: bytes) -> tuple[int, int] | None: + if len(b) < 10 or b[:6] not in (b"GIF87a", b"GIF89a"): + return None + return struct.unpack(" tuple[int, int] | None: + if len(b) < 26 or b[:2] != b"BM": + return None + width, height = struct.unpack(" WorkerResult: if not settings.context_compression_enabled: return WorkerResult() - if ctx.context_tokens is None: + if ctx.context_tokens is None and not ctx.context_truncated: return WorkerResult() - if not should_compress(ctx.context_tokens, ctx.max_context_tokens, - settings.context_compression_threshold): + # A truncated prompt is the one case the threshold cannot see: the + # backend reports the size it evaluated (about half the window) while + # the context it came from was over the window. Compress now. + if not ctx.context_truncated and not should_compress( + ctx.context_tokens, ctx.max_context_tokens, + settings.context_compression_threshold, + ): return WorkerResult() compressor = ContextCompressor() diff --git a/tests/unit/core/test_agent.py b/tests/unit/core/test_agent.py index 47f698f..e06a757 100644 --- a/tests/unit/core/test_agent.py +++ b/tests/unit/core/test_agent.py @@ -763,6 +763,47 @@ assert not any(isinstance(ev, CompressionStarted) for ev in events) + @pytest.mark.asyncio + async def test_preturn_gate_prices_the_message_that_has_not_arrived_yet( + self, agent, session, monkeypatch + ): + """The pre-turn gate runs before the incoming user message is appended. + A message carrying images could overshoot the window in a single step — + the context was under the threshold the moment before and the images + were the only thing over — so its cost is added to the estimate.""" + import navi.core.agent as agent_mod + + for i in range(4): + session.context.append(Message(role="user", content=f"u{i}")) + session.context.append(Message(role="assistant", content="a" * 100)) + await agent._sessions.save(session) + + seen: list[int] = [] + + def _capture(tokens, *args, **kwargs): + seen.append(tokens) + return False + + monkeypatch.setattr(agent_mod, "should_compress", _capture) + + profile = agent._profiles.get("test") + baseline = agent._compressor.real_baseline_estimate( + session.context, session.context, profile.model + ) + events = [ + ev + async for ev in agent._compression_events_preturn( + session, + llm=agent._get_backend("ollama"), + profile=profile, + session_id=session.id, + pending_tokens=1234, + ) + ] + assert events == [] + assert seen == [baseline + 1234] + + # ─── run_ephemeral() tests ─────────────────────────────────────────────────── @@ -1077,15 +1118,19 @@ # ─── pasted images reach the session directory ─────────────────────────────── -def _png_b64(colour: tuple = (255, 0, 0, 128)) -> str: - """A pasted RGBA PNG, base64 — alpha present, so a re-encode would show up.""" +def _png_b64(colour: tuple = (255, 0, 0, 128), size: tuple[int, int] = (2, 2)) -> str: + """A pasted RGBA PNG, base64 — alpha present, so a re-encode would show up. + + ``size`` matters for budgeting: an image is priced by its pixel area + (~one token per 32x32 px), not a flat constant. + """ import base64 import io from PIL import Image buf = io.BytesIO() - Image.new("RGBA", (2, 2), colour).save(buf, format="PNG") + Image.new("RGBA", size, colour).save(buf, format="PNG") return base64.b64encode(buf.getvalue()).decode() @@ -1159,7 +1204,10 @@ backend = FakeLLMBackend(responses=["ok"]) agent._backends.register("ollama", backend) - async for _ in agent.run_stream(session.id, "look", images=[_png_b64()]): + # 1024x1024 costs ~1026 tokens — more than the 0.8 * 500 budget. + async for _ in agent.run_stream( + session.id, "look", images=[_png_b64(size=(1024, 1024))] + ): pass saved = await agent._sessions.get(session.id) diff --git a/tests/unit/core/test_compressor.py b/tests/unit/core/test_compressor.py index 355052a..fbc91fa 100644 --- a/tests/unit/core/test_compressor.py +++ b/tests/unit/core/test_compressor.py @@ -15,6 +15,18 @@ from tests.conftest_factory import FakeLLMBackend, make_profile +def _image_b64(width: int, height: int, fmt: str = "PNG") -> str: + """A real encoded image, so its size is read the way the estimator reads it.""" + import base64 + import io + + from PIL import Image + + buf = io.BytesIO() + Image.new("RGB", (width, height), (0, 0, 0)).save(buf, format=fmt) + return base64.b64encode(buf.getvalue()).decode() + + class TestShouldCompress: def test_below_threshold(self): assert should_compress(100, 1000, 0.7) is False @@ -273,6 +285,25 @@ text, _ = _format_for_summary(msgs) assert "old summary" in text + def test_image_budget_caps_the_forwarded_images(self): + """Every image in the history used to be forwarded to the summarizer + with only the text capped; a dozen screenshots made a prompt too big + for its own window, silently truncated, and the summary was written as + if the history had been read.""" + big = _image_b64(1024, 1024) # 1026 tokens + small = _image_b64(512, 512) # 258 tokens + msgs = [Message(role="user", content="look", images=[big, small])] + text, images = _format_for_summary(msgs, image_budget=300) + assert images == [small] + # The note still names both, so the summarizer knows what it missed. + assert "[+ 2 image(s)]" in text + + def test_image_budget_of_zero_forwards_nothing(self): + msgs = [Message(role="user", content="look", images=[_image_b64(512, 512)])] + text, images = _format_for_summary(msgs, image_budget=0) + assert images == [] + assert "[+ 1 image(s)]" in text + class TestCompressContext: async def test_nothing_to_compress(self): @@ -541,17 +572,40 @@ assert compressor.estimate_context_tokens(context) == 3 # 11 chars // 3 = 3 def test_estimate_context_tokens_with_images(self): + """A 1024x1024 image costs 1026 real tokens on qwen3.6, not the flat + 500 this server used to charge — see navi/core/image_tokens.py.""" compressor = ContextCompressor() - context = [Message(role="user", content="hi", images=["base64img"])] - assert compressor.estimate_context_tokens(context) == 500 # 2 chars // 3 = 0 + 500 + context = [Message(role="user", content="hi", images=[_image_b64(1024, 1024)])] + assert compressor.estimate_context_tokens(context) == 1026 # 2 chars // 3 = 0 def test_estimate_context_tokens_counts_each_image(self): - """Images count 500 tokens EACH, not 500 per message that happens to + """Each image is priced on its own, not once per message that happens to carry them — multi-image uploads (webclient sends several) used to be undercounted.""" compressor = ContextCompressor() - context = [Message(role="user", content="hi", images=["img1", "img2", "img3"])] - assert compressor.estimate_context_tokens(context) == 1500 + images = [_image_b64(512, 512) for _ in range(3)] + context = [Message(role="user", content="hi", images=images)] + assert compressor.estimate_context_tokens(context) == 3 * 258 + + def test_an_unreadable_image_falls_back_to_the_setting(self): + """A payload whose header we cannot parse must still cost something — + the default is a 1024x1024 image's price.""" + compressor = ContextCompressor() + context = [Message(role="user", content="hi", images=["not-an-image"])] + assert compressor.estimate_context_tokens(context) == 1024 + + def test_every_estimate_prices_images_the_same_way(self): + """/preturn, /midturn and check_context_size all read one of these three, + so they must agree on the image cost.""" + from navi.core.image_tokens import estimate_image_tokens + + img = _image_b64(1024, 1024) + context = [Message(role="user", content="x" * 180, images=[img])] + cost = estimate_image_tokens(img) + assert cost == 1026 + assert ContextCompressor.estimate_context_tokens(context) >= cost + assert ContextCompressor.estimate_tokens(context, "test") >= cost + assert ContextCompressor.estimate_tokens_delta(context, "test") >= cost @pytest.mark.asyncio async def test_compress_session_success(self): @@ -954,12 +1008,16 @@ """Even the keep_recent floor (2 turns) exceeds the target — the verbatim kept region cannot fit. compress_context returns the floor result (over target), and compress_session's safety net then drops oldest kept turns - by token budget so a compression firing always leaves real headroom.""" + by token budget so a compression firing always leaves real headroom. + + Sizes are chosen against the *view* estimate, the unit the window is + measured in: each 12k-char message is view-capped below num_ctx // 6. + """ prev_config, prev_cm = self._override_settings(ollama_num_ctx=10_000) try: - # target_tokens = 6_500. 6 turns, each assistant ~12_000 chars - # (~4_000 tokens) -> 2 turns ~8_000 > 6_500, so the floor (2) is - # still over target. compress_session must truncate further. + # target_tokens = 6_500. Each turn: user + 2 assistant messages, + # each assistant view-capped at ~1_670 tokens -> a turn is ~3_300, + # so the floor (2 turns, ~6_600) is still over target. big = "x" * 12_000 backend = FakeLLMBackend(responses=["Summary text"]) compressor = ContextCompressor() @@ -967,6 +1025,7 @@ for i in range(6): context.append(Message(role="user", content=f"u{i}")) context.append(Message(role="assistant", content=big)) + context.append(Message(role="assistant", content=big)) result = await compressor.compress_session( context=context, @@ -977,13 +1036,19 @@ ) assert result is not None new_context, _ = result - # Safety net brought the context under the target. - assert compressor.estimate_context_tokens(new_context) <= 6_500 + # Safety net brought the view under the target (the view — what the + # model is sent — is the unit every size decision uses now). + from navi.core.context_builder import truncate_oversized + + assert ( + compressor.estimate_tokens(truncate_oversized(new_context), "test") + <= 6_500 + ) # Only the newest turn fit the budget; older kept turns + the summary # were dropped (they were verbatim, not summarized — the documented # safety-net loss for the pathological "kept region alone exceeds # the target" case). - assert len(new_context) == 3 # system + newest user + its assistant + assert len(new_context) == 4 # system + newest user + its 2 assistants assert new_context[1].content == "u5" assert not any(m.is_summary for m in new_context) finally: @@ -1129,6 +1194,219 @@ assert c._real_baseline is None c.record_real_baseline(4, 0) assert c._real_baseline is None + + def test_a_truncated_prompt_is_not_cached_as_the_baseline(self): + """Ollama silently evaluates about half the window and reports only + that count. Caching it pinned the session at ~50%: every later estimate + started from the truncated number, the 90% trigger never fired, and the + model kept getting a prompt whose head had been dropped.""" + c = ContextCompressor() + c.record_real_baseline(4, 60_000) + c.record_real_baseline(6, 65_538, sent_tokens=131_072, model="qwen") + # The honest earlier baseline survives; the truncated count does not. + assert c._real_baseline == (4, 60_000) + assert c.context_truncated is True + # The post-turn worker consumes the flag and clears it. + assert c.pop_context_truncated() is True + assert c.context_truncated is False + + def test_a_normal_count_is_recorded_and_raises_no_flag(self): + c = ContextCompressor() + c.record_real_baseline(4, 120_000, sent_tokens=131_072) + assert c._real_baseline == (4, 120_000) + assert c.context_truncated is False + + def test_an_image_prompt_is_kept_out_of_the_calibration_fit(self): + """Image tokens are real prompt tokens but not characters, so a prompt + carrying images would land its whole image cost in the fitted + intercept/slope and skew every later text estimate for that model.""" + from navi.core.token_calibration import calibration_for + + c = ContextCompressor() + c.record_real_baseline( + 2, 60_000, chars=100_000, model="vision-model", image_tokens=1026 + ) + assert calibration_for("vision-model").observations == 0 + # ...but the baseline itself is still honest and gets recorded. + assert c._real_baseline == (2, 60_000) + + def test_delta_is_priced_as_the_view_the_model_is_sent(self): + """The first operation can fill the window with one huge tool result. + build() truncates it for the model, so the delta must be priced as that + truncated view — pricing the raw 300k chars made check_context_size kill + the run with "context too large" over a prompt never sent.""" + from navi.core.context_builder import truncate_oversized + + c = ContextCompressor() + c.record_real_baseline(1, 1_200) + big = "x" * 300_000 + session_context = [ + Message(role="user", content="read the file"), + Message( + role="assistant", + content="", + tool_calls=[ToolCallRequest(id="1", name="fs", arguments={})], + ), + Message(role="tool", name="fs", content=big, tool_call_id="1"), + ] + built = [Message(role="system", content="sys")] + truncate_oversized( + session_context + ) + + estimate = c.real_baseline_estimate(session_context, built) + raw_delta = c.estimate_tokens_delta(session_context[1:], None) + # The delta alone is a whole window of raw text; the view is a fraction. + assert estimate * 5 < 1_200 + raw_delta + # And the size check agrees with what will be sent. + c.check_context_size(built, session_context=session_context, window=65_536) + + def test_oversized_user_message_in_delta_no_longer_kills_the_session(self): + """A user paste bigger than the window is capped for the LLM view + (generously, num_ctx // 3), so it can no longer dead-end a session.""" + from navi.core.context_builder import truncate_oversized + + c = ContextCompressor() + c.record_real_baseline(1, 1_200) + session_context = [Message(role="user", content="x" * 300_000)] + built = [Message(role="system", content="sys")] + truncate_oversized( + session_context + ) + + estimate = c.real_baseline_estimate(session_context, built) + assert estimate < 65_536 + c.check_context_size(built, session_context=session_context, window=65_536) + + def test_fallback_without_baseline_prices_the_view(self): + from navi.core.context_builder import truncate_oversized + + c = ContextCompressor() + big = "x" * 300_000 + session_context = [Message(role="tool", name="fs", content=big)] + built = truncate_oversized(session_context) + assert c.real_baseline_estimate(session_context, built) * 5 < ( + c.estimate_tokens(session_context) + ) + + +class TestBudgetAwareInFlightSplit: + """The count window (keep_recent_messages) is useless once messages are + huge: a first turn of a few large results could not be shrunk at all, so a + gate that fired still ended in "compress_session: None".""" + + @staticmethod + def _override_settings(monkeypatch): + import navi.config as _config + import navi.core.compressor as _cm + from navi.config import Settings + + new = Settings( + _env_file=None, + ollama_num_ctx=4_000, + output_reserve_tokens=512, + navi_persona_file="", + context_compression_target=0.65, + ) + monkeypatch.setattr(_config, "settings", new) + monkeypatch.setattr(_cm, "settings", new) + return new + + @staticmethod + def _first_turn(): + turn = [Message(role="user", content="analyse these logs")] + for i in range(4): + turn.append( + Message( + role="assistant", + content="", + tool_calls=[ + ToolCallRequest(id=str(i), name="fs", arguments={}) + ], + ) + ) + turn.append( + Message(role="tool", name="fs", content="x" * 50_000, tool_call_id=str(i)) + ) + return turn + + def test_partition_splits_a_huge_turn_by_budget(self, monkeypatch): + from navi.core.compressor import partition_current_turn_messages + from navi.core.context_builder import truncate_oversized + + settings = self._override_settings(monkeypatch) + est = lambda msgs: ContextCompressor.estimate_tokens( + truncate_oversized(msgs), "m" + ) + turn = self._first_turn() + budget = int(settings.ollama_num_ctx * settings.context_compression_target) // 2 + + split = partition_current_turn_messages( + [turn], 12, budget_tokens=budget, estimate=est + ) + + assert split is not None + to_summarize, to_keep = split + # The user request head stays verbatim. + assert to_keep[0].role == "user" + # The newest tool result stays verbatim, together with its call. + assert to_keep[-1].role == "tool" + assert to_keep[-1].tool_call_id == "3" + assert to_keep[-2].role == "assistant" + # The kept region fits the budget; the middle went to the summary. + assert est(to_keep) <= budget * 1.5 + assert len(to_summarize) == len(turn) - len(to_keep) + + def test_would_compress_says_yes_for_the_huge_first_turn(self, monkeypatch): + from navi.core.context_builder import truncate_oversized + + settings = self._override_settings(monkeypatch) + turn = self._first_turn() + view = lambda msgs: ContextCompressor.estimate_tokens( + truncate_oversized(msgs), "m" + ) + c = ContextCompressor() + c.set_profile(None) + before = view(turn) + assert before > ( + settings.ollama_num_ctx * settings.context_compression_threshold + ) + + assert c.would_compress( + turn, + keep_recent=settings.context_keep_recent, + keep_recent_messages=12, + threshold=settings.context_compression_threshold, + max_context_tokens=settings.ollama_num_ctx, + current_tokens=before, + model="m", + ) + + @pytest.mark.asyncio + async def test_compression_actually_shrinks_the_huge_first_turn(self, monkeypatch): + from navi.core.context_builder import truncate_oversized + + settings = self._override_settings(monkeypatch) + turn = self._first_turn() + view = lambda msgs: ContextCompressor.estimate_tokens( + truncate_oversized(msgs), "m" + ) + c = ContextCompressor() + c.set_profile(None) + before = view(turn) + + result = await c.compress_session( + context=turn, + llm=FakeLLMBackend(responses=["summary"]), + model="m", + temperature=0.3, + keep_recent=settings.context_keep_recent, + keep_recent_messages=12, + window=settings.ollama_num_ctx, + ) + + assert result is not None + new_context, _ = result + assert view(new_context) < before + assert new_context[0].is_summary c.record_real_baseline(4, 5000) assert c._real_baseline == (4, 5000) diff --git a/tests/unit/core/test_context_builder.py b/tests/unit/core/test_context_builder.py index 358be4a..ff85867 100644 --- a/tests/unit/core/test_context_builder.py +++ b/tests/unit/core/test_context_builder.py @@ -490,10 +490,21 @@ assert len(original.content) == len(big) assert original.content == big - def test_user_messages_never_truncated(self): + def test_user_message_capped_more_generously_than_tool_output(self, monkeypatch): + """A paste bigger than the window can never reach the model whole. The + user cap is num_ctx // 3 (vs num_ctx // 6 for tool output): intent gets + more room, but an uncapped user message used to kill the session — the + size check refused every call and compression had nothing it could split. + """ + from navi.config import Settings + + monkeypatch.setattr( + "navi.config.settings", + Settings(_env_file=None, ollama_num_ctx=3_000, navi_persona_file=""), + ) builder = ContextBuilder(profile_registry=make_profile_registry()) profile = make_profile("test") - huge = "x" * 200_000 + huge = "x" * 20_000 ctx = [ Message(role="user", content=huge), Message(role="assistant", content=huge), @@ -501,8 +512,15 @@ result = builder.build(ctx, profile, mem=None) user_msgs = [m for m in result if m.role == "user"] asst = [m for m in result if m.role == "assistant"] - assert len(user_msgs[0].content) == len(huge) # user intent preserved verbatim - assert "truncated" in asst[0].content # assistant IS truncatable + # user: num_ctx // 3 tokens -> *3 chars, head + tail + marker + assert "user message truncated" in user_msgs[0].content + assert len(user_msgs[0].content) < 3_100 + # assistant: num_ctx // 6 tokens -> *3 chars + assert "truncated" in asst[0].content + assert len(asst[0].content) < 1_600 + assert len(user_msgs[0].content) > len(asst[0].content) + # Stored history keeps the full text; only the LLM view is capped. + assert ctx[0].content == huge def test_small_messages_left_intact(self): builder = ContextBuilder(profile_registry=make_profile_registry()) diff --git a/tests/unit/core/test_image_tokens.py b/tests/unit/core/test_image_tokens.py new file mode 100644 index 0000000..79c57f1 --- /dev/null +++ b/tests/unit/core/test_image_tokens.py @@ -0,0 +1,168 @@ +"""Pricing of inline images, measured against qwen3.6. + +The numbers in the parametrized table are the ones measured on the local +vision model (``prompt_eval_count`` of the same prompt with and without one +image); the estimator must reproduce them, because every window decision — +the compression trigger, the per-turn image budget, ``check_context_size`` — +is taken in these units. +""" + +import base64 +import struct + +import pytest + +from navi.config import Settings +from navi.core import image_tokens as image_tokens_module +from navi.core.image_tokens import ( + context_image_tokens, + estimate_image_tokens, + tokens_for_size, +) +from navi.llm.base import Message + + +def _b64(raw: bytes) -> str: + return base64.b64encode(raw).decode() + + +def _png_header(width: int, height: int) -> bytes: + return ( + b"\x89PNG\r\n\x1a\n" + + b"\x00\x00\x00\x0dIHDR" + + struct.pack(">II", width, height) + + b"\x08\x02\x00\x00\x00" + ) + + +def _jpeg_header(width: int, height: int) -> bytes: + return ( + b"\xff\xd8" # SOI + + b"\xff\xe0\x00\x04\x00\x00" # APP0, 2 payload bytes + + b"\xff\xc0\x00\x11\x08" # SOF0, 17-byte segment, 8-bit + + struct.pack(">HH", height, width) + + b"\x03\x01\x22\x00\x02\x11\x01\x03\x11\x01" + ) + + +class TestTokensForSize: + @pytest.mark.parametrize( + ("width", "height", "expected"), + [ + (512, 512, 258), + (800, 600, 477), + (1024, 768, 770), + (1024, 1024, 1026), + (1536, 1536, 2306), + (3072, 1024, 3074), + (4096, 512, 2050), + ], + ) + def test_matches_the_measured_costs(self, width, height, expected): + assert tokens_for_size(width, height) == expected + + def test_a_downscaled_image_costs_the_same_as_a_full_size_one(self): + # 3072x3072 is scaled to 2048x2048 before tokenizing (4098, capped). + assert tokens_for_size(3072, 3072) == tokens_for_size(2048, 2048) == 4096 + assert tokens_for_size(4096, 1024) == 4096 + + def test_a_tiny_icon_is_floored_not_priced_at_almost_nothing(self): + # The formula alone says 66; the floor keeps small images visible in + # the estimate instead of letting a wall of icons cost nothing. + assert tokens_for_size(256, 256) == 256 + assert tokens_for_size(1, 1) == 256 + + def test_a_degenerate_size_falls_back_to_the_floor(self): + assert tokens_for_size(0, 100) == 256 + assert tokens_for_size(-5, -5) == 256 + + +class TestHappensOnRealEncodedImages: + @pytest.mark.parametrize( + ("fmt", "size", "expected"), + [ + ("PNG", (1024, 768), 770), + ("JPEG", (512, 512), 258), + ], + ) + def test_a_genuinely_encoded_image_is_priced_by_its_header(self, fmt, size, expected): + import io + + from PIL import Image + + buf = io.BytesIO() + Image.new("RGB", size, (10, 20, 30)).save(buf, format=fmt) + assert estimate_image_tokens(_b64(buf.getvalue())) == expected + + def test_a_data_uri_prefix_is_accepted(self): + raw = _png_header(1024, 1024) + assert estimate_image_tokens("data:image/png;base64," + _b64(raw)) == 1026 + + +class TestHeaderParsers: + def test_png(self): + assert estimate_image_tokens(_b64(_png_header(800, 600))) == 477 + + def test_jpeg(self): + assert estimate_image_tokens(_b64(_jpeg_header(1024, 768))) == 770 + + def test_gif(self): + raw = b"GIF89a" + struct.pack("