context: price images by area and detect backend-truncated prompts
...
Every window decision priced an inline image at a flat 500 tokens, while a
1024x1024 screenshot costs ~1026 on qwen3.6 and a 2048x2048 one ~4098. The
undercount kept sessions below the 90% trigger; Ollama then silently truncated
the prompt to about half the window (dropping the system prompt), and the
truncated prompt_eval_count was cached as the session baseline, pinning the
estimate at ~50% for good - the session never compressed while the model was
already told the context was full.
- image_tokens.py prices each image from its header (one token per 32x32 px,
downscaled to <=2048^2, clamped to 256..4096); unreadable headers fall back
to CONTEXT_IMAGE_TOKENS. Used by both estimators, the delta and
check_context_size; the per-turn image budget fits images by their own cost.
- A prompt reported far below what was sent is a truncation: it is not cached
as the baseline, and the post-turn worker compresses regardless of the
threshold. The session keeps the honest token count.
- The pre-turn gate prices the incoming message and its images, which are not
in session.context yet.
- The summarizer receives images under a window-scaled budget, and prompts
carrying images stay out of the calibration fit (their bytes are not in
chars and would skew it).
- Oversized user messages are capped for the LLM view (at least num_ctx/3),
so a paste bigger than the window no longer dead-ends a session.
- The in-flight partition also trims by token budget: a gate that fires can
now actually shrink a turn made of a few huge messages.
Eugene Sukhodolskiy
committed
1 day ago