diff --git a/docs/agent.md b/docs/agent.md index 153a762..7626fe2 100644 --- a/docs/agent.md +++ b/docs/agent.md @@ -5,30 +5,33 @@ ## Entry points ### `run_stream(session_id, user_message)` → `AsyncGenerator[AgentEvent]` -Streaming. Yields `AgentEvent` objects in real time. Used by the WebSocket handler. Runs the planning phase if `profile.planning_enabled = True`. +Streaming. Yields `AgentEvent` objects in real time. Used by the WebSocket handler. Runs the planning phase when `(_is_first_message or profile.planning_enabled) and not _is_casual` (always on the first user message; never for casual greetings). `profile.planning_mandatory` forces `force_plan` on every turn. ### `run(session_id, user_message)` → `str` -Non-streaming. Full tool-calling loop, returns final text. No planning phase. +Non-streaming. Delegates to `run_stream()` and returns the final text. Planning and the full tool loop run; events are consumed internally, not yielded. -### `run_ephemeral(user_message, profile_id)` → `tuple[str, bool]` +### `run_ephemeral(user_message, profile_id, *, max_iterations=40, exclude_tools=(), briefing=None, custom_system_prompt=None, inherit_system_prompt=False, context_transfer=None, parent_session_id=None, timeout_seconds=300.0)` → `tuple[str, bool]` Non-persistent subagent. Temporary in-memory context. Called by `SpawnAgentTool`. Returns `(result_text, completed_normally)`. `completed_normally` is `False` if the subagent hit the iteration limit or timed out. -`spawn_agent.profile_id` is optional. If omitted, `SpawnAgentTool` resolves the parent session's current profile. If provided, the subagent uses the selected profile's model, `subagent_system_prompt`, planning flags, and tool set. Its tools come from that profile's `subagent_tools`, falling back to `enabled_tools` when `subagent_tools` is empty. +`spawn_agent.profile_id` is optional. If omitted, `SpawnAgentTool` resolves the parent session's current profile. If provided, the subagent uses the selected profile's model, `subagent_system_prompt`, planning flags, and tool set. Its tools come from that profile's `tools.subagent`, falling back to `tools.agent` when `tools.subagent` is empty. `exclude_tools` removes specific tools from that set; `briefing`/`custom_system_prompt`/`inherit_system_prompt` shape the system prompt; `context_transfer` injects a synthetic user/assistant exchange before the task message. When spawned from a persistent parent session, session-aware tools run under the parent session id so file tools resolve the user's session directory rather than a `subagent_*` directory. `run_ephemeral` reads the parent session from the DB when `parent_session_id` is provided, so session-aware tools (filesystem, todo, scratchpad) operate on the parent's data. +### `compact_stream(session_id)` → `AsyncGenerator[AgentEvent]` +Forced context compression triggered by the `{"type":"compact"}` WebSocket control message (TUI `/compact`, `Ctrl+X C`). Bypasses the token threshold and runs the compressor immediately, emitting `CompressionStarted` + `ContextCompressed`. Raises `NothingToCompactError` (surfaced as an error frame) when there is nothing to compress. + ### ContextVar restoration -`run_ephemeral` saves the parent's `current_session_id`, `current_model`, `current_user_id`, `current_user_role`, and `current_user_info` before starting and restores them in a `finally` block. This prevents background tasks or the next parent iteration from inheriting stale subagent IDs. +`run_ephemeral` saves the parent's `current_session_id`, `current_model`, `current_working_directory`, `current_user_id`, `current_user_role`, and `current_user_info` before starting and restores them in a `finally` block. This prevents background tasks or the next parent iteration from inheriting stale subagent IDs. `run_stream` likewise sets and resets `current_working_directory` from the message `cwd`. --- ## Planning phase (`_run_planning`) -Runs before the tool loop when `profile.planning_enabled = True`. +Runs before the tool loop when `(_is_first_message or profile.planning_enabled) and not _is_casual`. `profile.planning_mandatory` forces `force_plan` on every turn (suppresses the `DIRECT` early-return only, not the observe skip — see `profiles.md`). ### Phase 1 — Analysis LLM receives the user request with a classification prompt. Outputs: @@ -88,17 +91,22 @@ ``` Each iteration: - 1. Check stop_event → yield StreamStopped if set + 1. Mid-turn compression (iteration > 0): estimate tokens via real_baseline_estimate, + and if over threshold + would_compress() → emit CompressionStarted, compress, save 2. Build context: _build_context() injects iteration budget and goal anchor (if due) 3. Check anti-stall: if stalled, append warning message to context 4. Inject queued adaptive re-plan message (if a step failed last iteration) - 5. llm.stream_complete(context, tool_schemas) + 5. check_context_size(built_ctx) → raise ContextTooLargeError if it won't fit + (surfaced as a synthesized assistant response + StreamEnd) + 6. llm.stream_complete(context, tool_schemas) → ThinkingDelta/ThinkingEnd events during reasoning → TextDelta events during text generation - 6a. No tool calls → save session, yield StreamEnd, run workers, return - 6b. Tool calls → execute each, yield ToolEvent, append results to context - 7. Update anti-stall counters, detect newly-failed todo steps - 8. Check if profile switched → reload profile + tools + → ModelInfo event when the resolved model changes mid-turn + 7. Record real prompt_tokens baseline (record_real_baseline) for the next estimate + 8a. No tool calls → save session, yield StreamEnd, run workers, return + 8b. Tool calls → execute each, yield ToolEvent, append results to context + 9. Update anti-stall counters, detect newly-failed todo steps + 10. Check if profile switched → reload profile + tools ``` ### Sub-agent event forwarding @@ -111,20 +119,20 @@ `run_stream()` wraps the LLM generator with `_iter_stream_guarded()`, which provides two safety layers: 1. **Stop-event polling during prefill.** Ollama emits no chunks during prefill, so a plain `await` on the first token can block for minutes. The wrapper polls `stop_event` every second so the user's Stop button works even during silent prefill. -2. **Hard timeouts.** `first_chunk_timeout` (default 120 s) caps prefill wait time. `chunk_timeout` (default 60 s) caps gaps between subsequent tokens. On timeout the generator is closed, terminating the HTTP connection to Ollama so GPU load drops to idle. +2. **Hard timeouts.** `first_chunk_timeout` (default 90 s) caps prefill wait time. `chunk_timeout` (default 60 s) caps gaps between subsequent tokens. On timeout the generator is closed, terminating the HTTP connection to Ollama so GPU load drops to idle. | Env var | Default | Purpose | |---|---|---| -| `LLM_STREAM_FIRST_CHUNK_TIMEOUT` | `120` | Max seconds to wait for the first token | +| `LLM_STREAM_FIRST_CHUNK_TIMEOUT` | `90` | Max seconds to wait for the first token | | `LLM_STREAM_CHUNK_TIMEOUT` | `60` | Max seconds between tokens after the first | --- ## Workers -Run sequentially after `StreamEnd`. Currently: `CompressionWorker`. +Run sequentially after `StreamEnd`. Currently: `CompressionWorker`. Workers receive a `WorkerContext` carrying the active `profile`, so `CompressionWorker` applies per-profile compression overrides (`compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file`). -Pre-turn compression also runs at the start of `run_stream()` if `session.context_token_count` exceeds the threshold. See [`sessions.md`](sessions.md). +Pre-turn compression also runs at the start of `run_stream()`: it estimates tokens via `estimate_context_tokens(session.context)` (not the stored `context_token_count`) and, when over threshold, is guarded by `would_compress()` before emitting `CompressionStarted`. See [`sessions.md`](sessions.md). --- @@ -139,3 +147,6 @@ ### System prompt caching The built system prompt string is cached per profile ID in `ContextBuilder` to avoid rebuilding on every turn. The cache is invalidated when the profile is reloaded (e.g. after `switch_profile` or hot-reload). This saves ~1–2 ms per turn for profiles with large system prompts. + +### Per-message view truncation +`ContextBuilder.build()` head/tail-truncates any single `tool`/`assistant` message whose estimated size exceeds `CONTEXT_MESSAGE_TOKEN_BUDGET` (default `0` → auto, `OLLAMA_NUM_CTX // 6`) in the *built* context only. Stored history is never mutated — a copy with a `[…truncated ~N tokens…]` marker is sent to the LLM. This prevents one huge tool result from alone blowing the window; full context compression still handles the cumulative case. diff --git a/docs/config.md b/docs/config.md index cca852a..995ac41 100644 --- a/docs/config.md +++ b/docs/config.md @@ -69,10 +69,10 @@ | Variable | Type | Default | Description | |---|---|---|---| | `LLM_COMPLETE_TIMEOUT` | int | `120` | Seconds before a non-streaming `complete()` call times out | -| `LLM_STREAM_FIRST_CHUNK_TIMEOUT` | int | `180` | Seconds to wait for the first token of a streaming call (prefill phase) | +| `LLM_STREAM_FIRST_CHUNK_TIMEOUT` | int | `90` | Seconds to wait for the first token of a streaming call (prefill phase) | | `LLM_STREAM_CHUNK_TIMEOUT` | int | `60` | Max seconds between consecutive tokens in a streaming call | -Large contexts can take 60–90 s to prefill; `LLM_STREAM_FIRST_CHUNK_TIMEOUT=180` is a safe upper bound for local Ollama. Reduce if using a fast cloud endpoint. +Large contexts can take 60–90 s to prefill; `LLM_STREAM_FIRST_CHUNK_TIMEOUT=90` covers the common case for local Ollama. Increase (e.g. 180) for very large contexts or slow GPUs; reduce for fast cloud endpoints. ## Security / Sandboxing @@ -135,6 +135,7 @@ | `CONTEXT_SUMMARY_TEMPERATURE` | float | `0.3` | Temperature for the summarization LLM call | | `CONTEXT_SUMMARY_MAX_TOKENS` | int | `4000` | Max output tokens for the summary LLM call | | `OUTPUT_RESERVE_TOKENS` | int | `2048` | Headroom reserved for model response in context size checks | +| `CONTEXT_MESSAGE_TOKEN_BUDGET` | int | `0` | Per-message token budget for the LLM context view. A single `tool`/`assistant` message whose estimated size exceeds this is head/tail-truncated in the *built* context only (stored history is never mutated) so one huge tool result cannot alone blow the window. `0` = auto (`OLLAMA_NUM_CTX // 6`). | ## Gmail diff --git a/docs/mechanics.md b/docs/mechanics.md index 09e8c6a..e3a8e46 100644 --- a/docs/mechanics.md +++ b/docs/mechanics.md @@ -23,16 +23,18 @@ | Mechanic | Description | Config / Flags | Files | Docs | |---|---|---|---|---| | **Streaming entry point** | `run_stream()` — yields `AgentEvent` objects in real time. Loads session, runs planning (if enabled), tool loop, workers. | `profile.max_iterations`, `profile.llm_backend`, `profile.model`, `profile.temperature` | `agent.py` | ✅ | -| **Non-streaming entry point** | `run()` — same loop but returns plain string. No planning phase, no events. | Same as above | `agent.py` | ✅ | +| **Non-streaming entry point** | `run()` — delegates to `run_stream()` and returns the final text. Planning and the full tool loop run; events are consumed internally, not yielded. | Same as above | `agent.py` | ✅ | | **Streaming guard wrapper** | Wraps `llm.stream_complete()` with two safety layers: (1) polls `stop_event` every second during prefill so the Stop button works even when the model emits no chunks, and (2) hard `first_chunk_timeout`/`chunk_timeout` deadlines that close the HTTP connection to Ollama so GPU load drops. | `LLM_STREAM_FIRST_CHUNK_TIMEOUT`, `LLM_STREAM_CHUNK_TIMEOUT` | `agent.py` | ✅ | | **Subagent thinking stall detector** | Monitors subagent streaming; if only `thinking` output is emitted for 60 s or 12 000 chars without text/tool calls, aborts the subagent to prevent endless internal-token loops on local models. | Hard-coded `_SUBAGENT_THINKING_STALL_SECONDS=60.0`, `_SUBAGENT_THINKING_STALL_CHARS=12000` | `agent.py` | ❌ | | **Cooperative stop** | Checks `current_stop_event` (asyncio.Event) before each LLM call, during streaming, and after tool execution. Uses clean generator close — never `task.cancel()`. | None | `agent.py` | ✅ | -| **First-message forced planning** | Planning phase always runs on the first user message in a session regardless of `profile.planning_enabled`. | `profile.planning_enabled` (only affects subsequent turns) | `agent.py` | ✅ | +| **Planning entry condition** | Planning runs when `(_is_first_message or profile.planning_enabled) and not _is_casual` — i.e. always on the first user message regardless of the flag, on later turns only when enabled, and never for casual greetings. `profile.planning_mandatory` forces `force_plan` on every turn. | `profile.planning_enabled`, `profile.planning_mandatory` | `agent.py` | ✅ | | **Profile reload mid-session** | After each tool execution batch, checks DB for profile ID change (e.g. from `switch_profile`). If changed, reloads profile, tools, schemas, and backend for next iteration. | None | `agent.py` | ✅ | -| **Pre-turn context compression** | Before assistant reply, checks `session.context_token_count` against threshold and compresses if exceeded. | `CONTEXT_COMPRESSION_ENABLED`, `OLLAMA_NUM_CTX`, `CONTEXT_COMPRESSION_THRESHOLD` | `agent.py` | ✅ | -| **Mid-turn context compression** | On iterations > 0, estimates tokens and triggers compression with `keep_recent_messages=max(12, CONTEXT_KEEP_RECENT*2)`. For long autonomous loops where the entire conversation is one turn. | Same as above + `CONTEXT_KEEP_RECENT` | `agent.py` | ❌ | -| **Context size check with output reserve** | Raises `ContextTooLargeError` if estimated input tokens exceed `OLLAMA_NUM_CTX - OUTPUT_RESERVE_TOKENS`. Images counted at 500 tokens each. | `OLLAMA_NUM_CTX`, `OUTPUT_RESERVE_TOKENS` | `agent.py` | ⚠️ | -| **Local token estimation** | Conservative estimate: `chars // 4 + imgs * 500`. Used for preflight context size checks. | None | `agent.py` | ❌ | +| **Pre-turn context compression** | Before the assistant reply, estimates tokens (`estimate_context_tokens(session.context)`) and compresses when over threshold. Guarded by `would_compress()` so `CompressionStarted` is only emitted when the partition can actually shrink the stored context. | `CONTEXT_COMPRESSION_ENABLED`, `OLLAMA_NUM_CTX`, `CONTEXT_COMPRESSION_THRESHOLD` | `agent.py` | ✅ | +| **Mid-turn context compression** | On iterations > 0, estimates tokens via `real_baseline_estimate(session.context, preflight_ctx)` (real prompt_tokens bulk + heuristic delta) and compresses with `keep_recent_messages=max(12, CONTEXT_KEEP_RECENT*2)`. Guarded by `would_compress()`. For long autonomous loops where the entire conversation is one turn. | Same as above + `CONTEXT_KEEP_RECENT` | `agent.py` | ✅ | +| **Forced `/compact`** | `compact_stream()` — bypasses the token threshold and compresses immediately on client demand (`{"type":"compact"}` WS control message). Emits `CompressionStarted` + `ContextCompressed`; raises `NothingToCompactError` when there is nothing to compress. Bound to `Ctrl+X C` in the TUI. | None | `agent.py`, `websocket.py` | ✅ | +| **Context size check with output reserve** | Raises `ContextTooLargeError` if estimated input tokens exceed `OLLAMA_NUM_CTX - OUTPUT_RESERVE_TOKENS`. When the caller passes `session_context`, the total uses `real_baseline_estimate()` (real bulk + heuristic delta) instead of the chars//3 estimate. Images counted at 500 tokens each. | `OLLAMA_NUM_CTX`, `OUTPUT_RESERVE_TOKENS` | `compressor.py` | ✅ | +| **Real-token baseline estimator** | Records the real `prompt_tokens` returned by each LLM call (`record_real_baseline(len(session.context), …)`) and reuses them as the bulk of the next estimate, heuristicking only the messages appended since. Used by the mid-turn gate and `check_context_size`. Cleared after compression (context shrank). | None | `compressor.py`, `agent.py` | ✅ | +| **Local token estimation** | Conservative estimate `chars // 3 + imgs * 500` (`ContextCompressor.estimate_context_tokens`, a staticmethod). Used as the fallback when no real baseline exists and for the per-message delta. | None | `compressor.py` | ✅ | | **Anti-stall detection** | Tracks two signals: (1) consecutive iterations with no todo status change, (2) identical tool call signatures. When either hits threshold, injects a hard warning system message. | `profile.anti_stall_enabled`, `profile.anti_stall_threshold` | `agent.py` | ✅ | | **Adaptive replan on failure** | Detects newly-failed todo steps after each tool batch and queues a re-planning system message for the next iteration. | `profile.adaptive_replan_enabled` | `agent.py` | ✅ | | **Goal anchoring** | Injects `[Goal anchor]` system message with original request + todo state every N iterations. | `profile.goal_anchoring_enabled`, `profile.goal_anchoring_interval` | `agent.py` | ✅ | @@ -96,16 +98,24 @@ | **Iteration budget message** | Appends `[Iteration N/M — K after this one]` with escalating urgency when ≤2 or ≤5 remaining. | `profile.iteration_budget_enabled` | `context_builder.py` | ✅ | | **Session context injection** | Appends session ID and exact `session_files_dir/{session_id}/` path. | `SESSION_FILES_DIR` | `context_builder.py` | ❌ | -## Context Compression (`navi/core/compressor.py`) +## Context Compression (`navi/core/compressor.py`, `navi/workers/compressor.py`) | Mechanic | Description | Config / Flags | Files | Docs | |---|---|---|---|---| | **Threshold-based trigger** | Returns `True` when `context_tokens >= max_context_tokens * threshold`. | `CONTEXT_COMPRESSION_THRESHOLD` | `compressor.py` | ✅ | -| **Turn-based partitioning** | Groups messages into turns. Keeps last `keep_recent` turns verbatim; older go to summarization. Tool call groups never split. | `CONTEXT_KEEP_RECENT` | `compressor.py` | ❌ | -| **Mid-turn fallback partitioning** | For long autonomous loops where entire conversation is one turn: keeps current request + newest N messages verbatim, summarizes older messages from same turn. | `keep_recent_messages` param | `compressor.py` | ❌ | -| **Summary input formatter** | Renders messages as plain text for summarizer: preserves summaries, notes image counts, renders tool calls compactly, collects base64 images for vision models. | None | `compressor.py` | ❌ | -| **Summary input truncate** | Hard cap of 24 000 chars on formatted input sent to summarizer LLM. | Hard-coded `_MAX_SUMMARY_INPUT_CHARS=24000` | `compressor.py` | ❌ | -| **LLM-based summarization** | Calls LLM with structured summarization prompt and replaces old messages with `is_summary=True` user message. | `CONTEXT_SUMMARY_TEMPERATURE`, `CONTEXT_SUMMARY_MAX_TOKENS` | `compressor.py` | ✅ | +| **Turn-based partitioning** | Groups messages into turns. Keeps last `keep_recent` turns verbatim; older go to summarization. Tool call groups never split. Adaptive: `_turn_importance` scores turns and swaps an important old turn into the kept set in place of a filler-recent one. | `CONTEXT_KEEP_RECENT`, `profile.compression_keep_recent` | `compressor.py` | ✅ | +| **Intra-turn fallback partitioning** | `partition_current_turn_messages`: for long autonomous loops where the entire conversation is one turn, keeps the current request + newest N messages verbatim and summarizes older messages from the same turn. | `keep_recent_messages` param | `compressor.py` | ✅ | +| **Honest `CompressionStarted` guard** | `would_compress()` runs only the partition decision + token-budget check (no LLM) so the agent emits `CompressionStarted` only when the partition can actually shrink the stored context — never a misleading "compression" status with no work done. | None | `compressor.py` | ✅ | +| **Token-budget hard-truncate fallback** | When `compress_session` gets a partition no-op but the token gate fired, drops oldest turns (keeping system + newest whole turns) until under `_HARD_TRUNCATE_TOKEN_FRAC=0.5` of the window. Last-resort, no LLM. | `OLLAMA_NUM_CTX` | `compressor.py` | ✅ | +| **Per-message view truncation** | `ContextBuilder._truncate_oversized` head/tail-truncates a single `tool`/`assistant` message whose estimated size exceeds the per-message budget in the *built* context only — stored history is never mutated. 0 = auto (`OLLAMA_NUM_CTX // 6`). | `CONTEXT_MESSAGE_TOKEN_BUDGET` | `context_builder.py` | ✅ | +| **Profile-aware compression overrides** | `compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file` override the global defaults inside `compress_context` / `compress_session` / the summary system prompt. `Agent.set_profile()` propagates the active profile to the compressor. | `profile.compression_keep_recent`, `profile.compression_max_tokens`, `profile.compression_prompt_file` | `compressor.py`, `profiles/base.py` | ✅ | +| **Real-token baseline estimator** | `record_real_baseline()` caches the last call's real `prompt_tokens` keyed by `len(session.context)`; `real_baseline_estimate()` returns `real_tokens + heuristic(messages since)` so code-heavy contexts are estimated accurately instead of undercounted by chars//3. Cleared after compression. | None | `compressor.py`, `agent.py` | ✅ | +| **Summary input formatter** | Renders messages as plain text for summarizer: preserves summaries, notes image counts, renders tool calls compactly, collects base64 images for vision models. Critical tool results (`is_compression_critical` or critical tool names) survive verbatim up to 4000 chars; others capped at 300. | None | `compressor.py` | ✅ | +| **Summary input truncate** | Hard cap of 24 000 chars on formatted input sent to summarizer LLM. | Hard-coded `_MAX_SUMMARY_INPUT_CHARS=24000` | `compressor.py` | ✅ | +| **Meta-summary consolidation** | When `to_summarize` contains multiple existing summaries totaling > `_META_SUMMARY_THRESHOLD=8000` chars, consolidates them into one via `_meta_summarize` first so old summaries don't crowd the summarizer input. | None | `compressor.py` | ✅ | +| **LLM-based summarization** | Calls LLM with structured summarization prompt (`think=False` for speed) and replaces old messages with an `is_summary=True` user message. Failure is non-fatal. | `CONTEXT_SUMMARY_TEMPERATURE`, `CONTEXT_SUMMARY_MAX_TOKENS`, `profile.compression_max_tokens` | `compressor.py` | ✅ | +| **Compression result events** | `CompressionStarted` (status) and `ContextCompressed` (carries `summary`, `messages_before`, `messages_after`, `context_tokens`, `max_context_tokens`). The TUI renders the `summary` as a `Context compressed: N → M messages` card. | None | `events.py`, `tui/renderers/summary.py` | ✅ | +| **Archive-on-compress** | `compress_and_save_session` archives old `session.messages` rows when `session_messages_window` is exceeded, and appends an `is_compression=True` system marker to `session.messages`. | `SESSION_MESSAGES_WINDOW` | `compressor.py` | ✅ | ## Tool Execution (`navi/core/tool_executor.py`) @@ -234,8 +244,8 @@ | Mechanic | Description | Config / Flags | Files | Docs | |---|---|---|---|---| -| **Worker base class** | Abstract base for post-response background tasks. Receives `WorkerContext`, may mutate session, return events. | None | `base.py` | ⚠️ | -| **`CompressionWorker`** | Post-turn compression. Replaces old context with summary, resets token count, appends marker. | `CONTEXT_COMPRESSION_ENABLED`, `CONTEXT_COMPRESSION_THRESHOLD`, `CONTEXT_KEEP_RECENT`, `CONTEXT_SUMMARY_TEMPERATURE`, `CONTEXT_SUMMARY_MAX_TOKENS` | `compressor.py` | ✅ | +| **Worker base class** | Abstract base for post-response background tasks. Receives `WorkerContext` (now carrying the active `profile`), may mutate session, return events. | None | `base.py` | ✅ | +| **`CompressionWorker`** | Post-turn compression. Calls `compress_context` with `keep_recent_messages=max(12, CONTEXT_KEEP_RECENT*2)` and `profile=ctx.profile`, mirroring the mid-turn path so a single long autonomous turn compresses too. Replaces old context with summary, marks dropped messages `is_context=False`, resets token count, appends `is_compression=True` marker. | `CONTEXT_COMPRESSION_ENABLED`, `CONTEXT_COMPRESSION_THRESHOLD`, `CONTEXT_KEEP_RECENT`, `CONTEXT_SUMMARY_TEMPERATURE`, `CONTEXT_SUMMARY_MAX_TOKENS`, `profile.compression_*` | `compressor.py` | ✅ | | **Worker auto-discovery** | Scans `navi/workers/*.py` and auto-instantiates non-abstract `Worker` subclasses. | None | `__init__.py` | ❌ | ## KV Store (`navi/store/`) @@ -248,10 +258,11 @@ | Mechanic | Description | Config / Flags | Files | Docs | |---|---|---|---|---| -| **Streaming protocol** | Full-duplex: client sends `message`, server emits `stream_start`, `thinking_delta`, `stream_delta`, `tool_call`, `stream_end`, etc. | `_HEARTBEAT_INTERVAL=20.0`, `_MAX_REPLAY_EVENTS=500`, `_MAX_IMAGES=10`, `_MAX_IMAGE_BYTES=5MB` | `websocket.py` | ✅ | +| **Streaming protocol** | Full-duplex: client sends `{"type":"message"}` or `{"type":"compact"}`, server emits `stream_start`, `thinking_delta`, `stream_delta`, `tool_call`, `compression_started`, `context_compressed`, `model_info`, `todo_updated`, `stream_end`, etc. | `_HEARTBEAT_INTERVAL=20.0`, `_MAX_REPLAY_EVENTS=500`, `_MAX_IMAGES=8`, `_MAX_IMAGE_BYTES_TOTAL=50MB` | `websocket.py` | ✅ | +| **Forced compact control message** | `{"type":"compact"}` runs the context compressor immediately, bypassing the token threshold; streams back `CompressionStarted` + `ContextCompressed`. Does not send `stream_start`; rejected with an error if a run is already active. | None | `websocket.py` | ✅ | | **Stop session** | `POST /sessions/{id}/stop` sets stop_event cooperatively. | None | `websocket.py` | ✅ | -| **Reconnect/replay** | On connect, if run active, replays buffered events before live stream. | `_MAX_REPLAY_EVENTS=500` | `websocket.py` | ✅ | -| **Image upload validation** | Max 10 images, 5MB each. Strips `data:...;base64,` prefix. | `_MAX_IMAGES=10`, `_MAX_IMAGE_BYTES=5242880` | `websocket.py` | ✅ | +| **Reconnect/replay** | On connect, if run active, replays buffered events before live stream, then sends `session_sync` with `session_id`/`profile_id`. | `_MAX_REPLAY_EVENTS=500` | `websocket.py` | ✅ | +| **Image upload validation** | Max 8 images, 50 MB total payload. Strips `data:...;base64,` prefix. | `_MAX_IMAGES=8`, `_MAX_IMAGE_BYTES_TOTAL=52428800` | `websocket.py` | ✅ | | **Image context annotation** | Appends note telling model that N images are already in multimodal context. | None | `websocket.py` | ❌ | | **File context annotation** | Appends `[Uploaded files on disk: ...]` to user content. | None | `websocket.py` | ⚠️ | | **Concurrent run guard** | Rejects new messages if `_runs` or `_busy_sessions` already contains session ID. | None | `websocket.py` | ✅ | @@ -352,58 +363,26 @@ ## Undocumented Mechanics Summary -These mechanisms have **no documentation** in `docs/`: +These mechanisms are marked `❌` in the catalog above (no dedicated doc section): -1. **Streaming guard wrapper** (`agent.py`) — prefill polling + hard timeouts -2. **Subagent thinking stall detector** (`agent.py`) — aborts subagent after 60s/12k chars of pure thinking +1. **Subagent thinking stall detector** (`agent.py`) — aborts subagent after 60s/12k chars of pure thinking +2. **Todo status snapshot / failed-steps tracking / progress injection** (`agent.py`) — anti-stall + adaptive replan signals 3. **Memory facts deduplication** (`agent.py`) — `_injected_fact_ids` per turn -4. **Mid-turn context compression** (`agent.py`) — compression during autonomous loops with doubled keep-recent -5. **Context injection collection (parallel)** (`agent.py`) — concurrent context providers + memory -6. **Security policy message** (`context_builder.py`) — dynamic sandbox/allowlist message -7. **User context message** (`context_builder.py`) — injects user profile data -8. **MCP context message** (`context_builder.py`) — combines handshake + overlay instructions -9. **Memory facts message (length limits)** (`context_builder.py`) — skip short messages, limit results -10. **System prompt caching** (`context_builder.py`) — per-profile cache -11. **Goal anchor builder** (`context_builder.py`) — constructs goal anchor message -12. **Session context injection** (`context_builder.py`) — injects session files path -13. **Turn-based partitioning** (`compressor.py`) — turn grouping for compression -14. **Mid-turn fallback partitioning** (`compressor.py`) — same-turn compression for long loops -15. **Summary input formatter** (`compressor.py`) — message rendering for summarizer -16. **Summary input truncate** (`compressor.py`) — 24k char cap -17. **Tool name resolution (MCP aliases)** (`tool_executor.py`) — 3 heuristic fallbacks -18. **Tool middleware hooks** (`tool_executor.py`) — before/after execute -19. **Image message generation** (`tool_executor.py`) — synthesizes vision message from tool result -20. **Streaming tool execution** (`tool_executor.py`) — yields ToolEvent alongside messages -21. **AIHelper single-turn wrapper** (`ai_helper.py`) -22. **AIHelper `ask()` timeout** (`ai_helper.py`) — 120s hard limit -23. **AIHelper `ask_json()`** (`ai_helper.py`) -24. **AIHelper token usage emission** (`ai_helper.py`) -25. **AIHelper JSON extractor** (`ai_helper.py`) -26. **Orchestrator stub** (`orchestrator.py`) -27. **EventBus pub/sub** (`event_bus.py`) -28. **Image upload validation** (`websocket.py`) — 10 images, 5MB each -29. **Image context annotation** (`websocket.py`) — note about inline images -30. **File context annotation** (`websocket.py`) — uploaded files list -31. **Concurrent run guard** (`websocket.py`) -32. **User ContextVar propagation** (`websocket.py`) -33. **MCP manager & tool registration** (`deps.py`) — lazy init + clear/re-register -34. **Worker auto-discovery** (`workers/__init__.py`) -35. **`ToolMiddleware` ABC** (`tools/_internal/middleware.py`) -36. **`LoggingMiddleware`** (`tools/_internal/logging_middleware.py`) -37. **Profile overrides persistence** (`profiles/_overrides.py`) — DB table for admin toggles -38. **Plan step parser** (`planning.py`) -39. **Planning debug data logging** (`planning.py`) -40. **Knowledge store rules** (`planning.py`) -41. **Context transfer priming** (`agent.py` subagent) -42. **ContextVar restoration** (`agent.py` subagent) -43. **FastAPI CORS** (`main.py`) -44. **Static file mounting** (`main.py`) -45. **Startup lifecycle** (`main.py`) — table creation retries, MCP connect, override apply -46. **Shutdown lifecycle** (`main.py`) -47. **Local token estimation** (`agent.py`) -48. **Recall message wrapping** (`agent.py`) -49. **Cross-profile awareness** (`context_builder.py`) -50. **MCP context message** (`context_builder.py`) +4. **Context injection collection (parallel)** (`agent.py`) — concurrent context providers + memory +5. **Recall message wrapping** (`agent.py`) — `[Scheduled recall …]` prefix +6. **Context transfer priming** (`agent.py` subagent) — synthetic exchange before task +7. **Memory facts message / goal anchor / security policy / user context / MCP context / session context injection** (`context_builder.py`) — per-turn system messages +8. **Plan step parser / planning debug data / knowledge store rules** (`planning.py`) +9. **Tool name resolution / sequential & streaming execution / middleware hooks / image message generation** (`tool_executor.py`) +10. **AIHelper** (`ai_helper.py`) — single-turn wrapper, `ask()` 120s timeout, `ask_json()`, token usage emission, JSON extractor +11. **Orchestrator stub** (`orchestrator.py`) — placeholder, raises `NotImplementedError` +12. **EventBus** (`event_bus.py`) — async pub/sub + global singleton +13. **`ToolMiddleware` / `LoggingMiddleware`** (`tools/_internal/`) +14. **Worker auto-discovery** (`workers/__init__.py`) — scans `navi/workers/*.py` +15. **Image context annotation / user ContextVar propagation** (`websocket.py`) +16. **MCP manager & tool registration** (`deps.py`) — lazy init + clear/re-register +17. **FastAPI app setup / CORS / static file mounting / startup & shutdown lifecycle** (`main.py`) +18. **Runtime profile overrides** (`profiles/_overrides.py`) — DB table for admin toggles --- @@ -422,8 +401,12 @@ | **Track task progress across turns** | `TodoTool` + planning auto-populate | | **Detect model looping** | Anti-stall detector + todo snapshot comparison | | **Auto-replan on failures** | Adaptive replan + todo failed-steps tracking | -| **Compress long conversations** | `CompressionWorker` + `compress_context()` | -| **Summarize old turns** | `compress_context()` with LLM-based summarization | +| **Compress long conversations** | `CompressionWorker` + `compress_context()` + forced `/compact` (`compact_stream`) | +| **Summarize old turns** | `compress_context()` with LLM-based summarization + meta-summary consolidation | +| **Cap one huge tool result** | Per-message view truncation (`ContextBuilder._truncate_oversized`, `CONTEXT_MESSAGE_TOKEN_BUDGET`) | +| **Shrink context when summarizer fails** | Token-budget hard-truncate fallback (`_hard_truncate`) | +| **Avoid lying "compression" status** | `would_compress()` guard before `CompressionStarted` | +| **Estimate context tokens accurately** | Real-token baseline (`record_real_baseline` / `real_baseline_estimate`) | | **Schedule future work** | `RecallScheduler` + `schedule_recall` / `manage_recall` tools | | **Run headless agent** | `Agent.run_stream()` with `is_recall=True` | | **Stream events to client** | `_AgentRun` queue + WebSocket protocol | @@ -464,4 +447,4 @@ --- -*Generated 2026-05-16. To update after adding a new mechanic: add a row to the appropriate section and update the cross-reference index if relevant.* +*Last updated 2026-07-12. To update after adding a new mechanic: add a row to the appropriate section and update the cross-reference index if relevant.* diff --git a/docs/sessions.md b/docs/sessions.md index 2a0068c..fe030ea 100644 --- a/docs/sessions.md +++ b/docs/sessions.md @@ -23,14 +23,18 @@ Messages in `session.messages` carry optional flags beyond role/content: -| Flag | Purpose | -|---|---| -| `is_plan: bool` | Message is a planning phase output (shown as plan card in UI, not text) | -| `is_compression: bool` | Marker message injected when context compression ran | -| `is_summary: bool` | A summary message replacing compressed history in `session.context` | -| `is_recall: bool` | Message was generated by a scheduled recall (styled differently in UI) | -| `thinking: str \| None` | LLM reasoning captured during a tool-calling turn | -| `metadata: dict` | Tool result metadata (e.g. `is_image`, `base64`) | +| Flag | Default | Purpose | +|---|---|---| +| `is_context: bool` | `True` | Whether the message is part of `session.context`. Dropped messages are marked `False` rather than deleted — `session.context` is rebuilt from `messages` on load (`[m for m in all if m.is_context]`). | +| `is_display: bool` | `True` | Whether the message is shown in the UI. Summary messages added by compression are `False`. | +| `is_plan: bool` | `False` | Message is a planning phase output (shown as plan card in UI, not text) | +| `is_compression: bool` | `False` | Marker message injected when context compression ran (carries the summary text, `is_context=False`) | +| `is_summary: bool` | `False` | A summary message replacing compressed history in `session.context` (`role=user`) | +| `is_compression_critical: bool` | `False` | Tool result kept verbatim (up to 4000 chars) by the summary formatter instead of capped | +| `is_recall: bool` | `False` | Message was generated by a scheduled recall (styled differently in UI) | +| `thinking: str \| None` | `None` | LLM reasoning captured during a tool-calling turn | +| `metadata: dict` | `{}` | Tool result metadata (e.g. `is_image`, `base64`, `step_text`) | +| `sequence_number`, `elapsed_seconds`, `tool_call_count`, `token_count`, `tool_calls`, `tool_call_id`, `name`, `files`, `images` | — | Per-message bookkeeping (ordering, metrics, tool-call payloads, attachments) | ## Dual-buffer design @@ -60,9 +64,11 @@ - `count_all(user_id=None, is_admin=False, search=None)` → total matching sessions - `search_list(limit, offset, user_id=None, is_admin=False, search=None, sort_by="last_active", sort_order="desc")` → paginated, filtered, sorted sessions - `delete(session_id)` → `bool` -- `list_page(user_id=None, is_admin=False, limit=50, offset=0)` → paginated list with `has_more` flag +- `list_page(user_id=None, is_admin=False, limit=50, offset=0, profile_id=None)` → paginated list with `has_more` flag; `profile_id` filters to one profile - `set_pinned(session_id, pinned)` → `bool` - `set_name(session_id, name)` → `bool` +- `archive_old_messages(session_id, threshold)` → moves messages with `sequence_number < threshold` out of the hot table (called by `compress_and_save_session` when `session_messages_window` is exceeded) +- `get_archived_messages(session_id, ...)` → read archived messages back Requires `DATABASE_URL` env variable (e.g. `postgresql://user:pass@localhost/navi`). @@ -78,39 +84,62 @@ ### When it triggers -Two trigger points: +Three trigger points: -1. **Pre-turn** (in `run_stream()`): before calling the LLM, checks `session.context_token_count` against the threshold. Compresses if `tokens >= num_ctx * threshold`. -2. **Post-turn** (via `CompressionWorker`): after `StreamEnd`, the worker re-checks and compresses if needed. +1. **Pre-turn** (in `run_stream()` → `_compression_events_preturn`): before the first LLM call of a turn, estimates tokens via `estimate_context_tokens(session.context)` (not the stored `context_token_count`) and compresses when `tokens >= num_ctx * threshold`. Guarded by `would_compress()` so `CompressionStarted` is only emitted when the partition can actually shrink the stored context. +2. **Mid-turn** (in `run_stream()` → `_compression_events_midturn`, every iteration > 0): estimates tokens via `real_baseline_estimate(session.context, preflight_ctx)` — real `prompt_tokens` from the previous call (bulk) + a heuristic delta for messages appended since — and compresses with `keep_recent_messages=max(12, context_keep_recent*2)`. This is what keeps long autonomous loops (one user message + many tool iterations = one turn) from exhausting the window. +3. **Post-turn** (via `CompressionWorker`): after `StreamEnd`, the worker re-checks (using the real `context_tokens` from the last call) and compresses if needed, mirroring the mid-turn `keep_recent_messages`. + +A fourth, on-demand path is **forced `/compact`** (`compact_stream()`): the client sends `{"type":"compact"}`, bypassing the threshold entirely; emits `CompressionStarted` + `ContextCompressed`; raises `NothingToCompactError` when there is nothing to compress. Config values (`settings`): - `context_compression_enabled: bool = True` - `context_compression_threshold: float = 0.70` — trigger at 70% of `ollama_num_ctx` -- `context_keep_recent: int = 10` — keep last N conversational turns verbatim +- `context_keep_recent: int = 8` — keep last N conversational turns verbatim - `context_summary_temperature: float = 0.3` +- `context_summary_max_tokens: int = 4000` — max output tokens for the summary LLM call +- `output_reserve_tokens: int = 2048` — headroom reserved for the response in `check_context_size` +- `context_message_token_budget: int = 0` — per-message view truncation budget (`0` = auto, `ollama_num_ctx // 6`) + +Per-profile overrides (`AgentProfile`): `compression_keep_recent`, `compression_max_tokens`, `compression_prompt_file` — applied inside `compress_context` / `compress_session` / the summary system prompt. navi_code uses `compression_keep_recent=12`. + +### Context size guard + +Before every LLM call, `check_context_size(built_ctx, session_context=session.context)` raises `ContextTooLargeError` when the estimated input exceeds `ollama_num_ctx - output_reserve_tokens`. The total uses `real_baseline_estimate()` when a baseline is available (real bulk + heuristic delta), falling back to the `chars // 3 + imgs*500` estimate. The error is surfaced to the user as a synthesized assistant response + `StreamEnd` rather than a raw system error. ### Compression algorithm -`compress_context(context, llm, model, temperature, keep_recent)`: +`compress_context(context, llm, model, temperature, keep_recent, *, max_tokens=None, keep_recent_messages=None, profile=None)`: -1. Partition messages into `to_summarize` (old turns) and `to_keep` (recent `keep_recent` turns). +1. Resolve effective `keep_recent`/`max_tokens` from `profile.compression_*` overrides. Partition messages into `to_summarize` (old turns) and `to_keep` (recent `keep_recent` turns). - A "turn" = one user message + all following assistant/tool messages up to the next user message. - Tool call groups (assistant + results) are never split across the partition. - - Existing summary messages are folded into the next pass. -2. Format `to_summarize` as plain text (tool calls shown as compact previews, max 120 chars for args, max 300 chars for results). -3. Truncate formatted input to `_MAX_SUMMARY_INPUT_CHARS = 12_000` chars. -4. Call `llm.complete()` with `think=False` to produce a bullet-point summary. -5. Replace `to_summarize` with a single summary message (`role=user`, `is_summary=True`). -6. Return `system_msgs + [summary_msg] + to_keep`. + - **Adaptive partitioning:** `_turn_importance` scores each turn; an important old turn can be swapped into the kept set in place of a filler-recent one. + - **Intra-turn fallback** (`partition_current_turn_messages`, when `keep_recent_messages` is set): for a single long turn, keeps the current request + newest N messages verbatim and summarizes older messages from the same turn. +2. **Meta-summary:** if `to_summarize` contains multiple existing summaries totaling > `_META_SUMMARY_THRESHOLD = 8000` chars, consolidate them into one via `_meta_summarize` first so old summaries don't crowd the summarizer input. +3. Format `to_summarize` as plain text. Tool calls are shown as compact previews (max 120 chars for args). **Critical** tool results (`is_compression_critical=True` or critical tool names) survive verbatim up to 4000 chars; others are capped at 300 chars. Base64 images are collected for vision models. +4. Truncate formatted input to `_MAX_SUMMARY_INPUT_CHARS = 24_000` chars. +5. Call `llm.complete()` with `think=False` to produce a bullet-point summary. +6. Replace `to_summarize` with a single summary message (`role=user`, `is_summary=True`). +7. Return `system_msgs + [summary_msg] + to_keep`. + +`compress_session` wraps `compress_context` with retry + a **token-budget hard-truncate fallback** (`_hard_truncate`): if the LLM summarization fails twice, it drops oldest turns (keeping system + newest whole turns) until under `_HARD_TRUNCATE_TOKEN_FRAC = 0.5` of the window — a last resort with no LLM call. + +`compress_and_save_session` then mutates the session: replaces `session.context`, marks dropped messages `is_context=False`, appends the summary (`is_display=False`) and an `is_compression=True` system marker to `session.messages`, resets `context_token_count`, archives old messages when `session_messages_window` is exceeded, and persists. It also clears the real-token baseline (the context just shrank). If compression fails, the exception propagates to `CompressionWorker`, which logs a warning and continues — compression failure is non-fatal. ### What is never compressed - `session.messages` — full history is always intact. -- The last `context_keep_recent` conversational turns. +- The last `context_keep_recent` conversational turns (or the intra-turn recent window). - System messages (never stored in context anyway). +### Events + +- `CompressionStarted` — status signal (`context_tokens`, `max_context_tokens`). +- `ContextCompressed` — result (`summary`, `messages_before`, `messages_after`, `context_tokens`, `max_context_tokens`). The TUI renders `summary` as a `Context compressed: N → M messages` card. + --- ## Session file uploads