| 2026-10-08 |

startup, llm, profiles, todo: five more the prod logs gave up
...
The same 5-day sweep of prod's journal as 5e51f87, continued. Every one of
these failed silently: no error in the UI, just a line in the log or a hole
in the database.
complete()/embed() retry a connection error before falling back.
stream_complete has always given its first chunk two attempts, but the
non-streaming path made exactly one per (server, model), and with a single
configured server (ollama.com) one ReadTimeout raised "All backends
exhausted: ReadTimeout" outright. That is the whole story of 2026-10-08,
when 27 calls died that way in a day — 23 of them memory summarisation, the
rest planning. Attempts now come from settings.llm_complete_retries (2) with
llm_retry_backoff_sec (2.0) between them; a missing model is still not
retried (a 404 is an answer, not a hiccup), and a lone server is still never
blacklisted — blacklisting it would block the next request for _TTL.
Startup waits for Postgres. create_container() opens the pool eagerly and
the retry loop for the DDL tables sits below it, so a host that comes up
before docker takes the whole lifespan with it. On 2026-10-07 15:19:40 the
agent ran `sudo reboot` on itself at the user's request ("Перезагрузи
себя"); the host booted 15:20:03, navi started at 15:20:18, hit
ConnectionRefusedError on the pool and logged "Application startup failed.
Exiting." Connection errors now retry for ~30s; a genuinely broken config
(a missing DATABASE_URL) still fails fast.
The profile loader reports its tally. A profile dropped by an unreadable
config.json is skipped by design so it cannot take the server down, but on
that same 15:20 boot four of them — modeler_3d, navi_code, secretary,
server_admin — were dropped with `Extra data: line 125 column 1`, and the
only trace was one error line each: the UI simply showed four profiles
fewer.
todo recovers a dropped `op`. Of the 3385 todo calls stored on prod, 73
arrived with the discriminator flattened away ({"index": 1, "status":
"in_progress"}, {"action": "view"}, {"": "add", "tasks": [...]}) and got a
bare "Unknown op: None" — a wasted round trip for an intent the remaining
arguments state plainly. The error message now names the five ops and what
each needs.
Message.created_at is stamped at creation. It defaulted to None and only
some construction sites filled it, so 85% of stored rows had no timestamp at
all (every tool message, 14 168 of 14 168) and no query could slice history
by time or measure a pause. Loading a row passes the column through
explicitly, so NULL rows stay NULL instead of being restamped with "now".
Every new test was checked to fail with its fix reverted. Full suite green
(1466 passed, 1 skipped). No frontend or dependency change, so the deploy is
a pull and a restart.
Eugene Sukhodolskiy
committed
6 hours ago
|
| 2026-07-10 |

tui: show the currently-served model in the status panel
...
The status panel's Model line was fed the global ollama_default_model, not the
session/profile model, and the server never told the client which model
actually served a call. Now:
- Backends stamp the resolved model onto LLMChunk (first chunk) / LLMResponse.
The fallback backend reports the model that survived its server+model
priority list (may differ from the profile's first choice).
- New ModelInfo event ({"type":"model_info","model":...}) emitted once per
turn from agent._consume_stream, re-emitted only when the model changes
across iterations. Additive WS event — old clients ignore it.
- TUI: attach_session/switch fetch the profile's configured model (first of
profile.model) via api.get_profile_model so the panel shows a value before
the first request; model_info then refines it to the actually-served model.
Not forwarded to the chat panel. raw CLI prints "[model] ...".
Co-Authored-By: Claude <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 10 Jul
|
| 2026-05-24 |
Raise first-chunk timeout to 90s and retry same server+model before fallback
...
- config.py: llm_stream_first_chunk_timeout 180s → 90s
- fallback.py stream_complete: wrap gen.__anext__() in asyncio.wait_for()
with llm_stream_first_chunk_timeout; on TimeoutError or LLMConnectionError
sleep 2s and retry once on the same server+model before blacklisting/fallback
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 24 May
|
| 2026-05-21 |
FallbackOllamaBackend: do not blacklist single server, empty file fallback
...
- When only one Ollama server is configured, LLMConnectionError no longer
adds it to the dead-server blacklist. This fixes the bug where a
transient failure permanently blocked all requests until server restart.
- LLMModelNotFoundError on a single server is also not blacklisted.
- _discover_backends now falls back to settings.ollama_host when the
ollama_backends_file is empty, missing, or returns no valid servers.
- Added unit tests covering single-server no-blacklist, multi-server
blacklist, model-not-found no-blacklist, and empty-file fallback.
400 passed, 1 skipped
Eugene Sukhodolskiy
committed
on 21 May
|
| 2026-05-12 |
Remove dead LLMBackend.stream() method
...
The method was defined on all backends (Ollama, FallbackOllama, OpenAI)
and in the base LLMBackend interface, but was never called by agent.py
or messages.py. stream_complete() covers all streaming use cases.
- navi/llm/base.py: remove abstract stream() method
- navi/llm/ollama.py: remove OllamaBackend.stream()
- navi/llm/fallback.py: remove FallbackOllamaBackend.stream()
- navi/llm/openai_backend.py: remove OpenAIBackend.stream()
- docs/tech_debt_review: mark item 54 as fixed
236 tests passing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 12 May
|
| 2026-05-11 |
Fix ollama_backends / FallbackOllamaBackend issues
...
- registry.py: always use FallbackOllamaBackend (unified backend).
Enables model priority lists in all deployments, not just multi-server.
- agent.py: add missing think=profile.think_enabled to run() (REST endpoint).
- compressor.py: fix model param type (str → list[str] | str | None).
- fallback.py: harden load_servers_from_file against missing/bad JSON files
and entries without host. Add clear_blacklists() for manual reset.
- admin.py: add POST /admin/ollama/clear-blacklists endpoint.
- tech_debt_review: document dead stream() methods.
- tests: add tests for single-server fallback, bad file handling,
missing host skipping, and blacklist clearing.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 11 May
|
| 2026-04-29 |
Align Ollama HTTP timeout with LLM timeouts
Eugene Sukhodolskiy
committed
on 29 Apr
|

Stability fixes batch — tech debt review 2026-04-29
...
Critical:
- Concurrent WS run race guard (#1)
- Tool task cancellation on generator teardown (#2)
- StopAsyncIteration kills fallback chain (#3)
- Session loading race with _lastLoadId guard (#4)
- ContentCard .match() crash on non-string result (#5)
- Image data type guard in buildMessageList (#6)
High:
- Cap WS replay buffer at 500 events (#7)
- Deduplicate memory extraction task with asyncio.Lock (#9)
- TTL-based fallback blacklisting (5 min) (#10)
- Subagent tool exception isolation (#11)
- Inline image size/count validation on WS (#12)
- Clean up orphaned file on DB insert failure (#13)
- Deep watch streamingMsg for auto-scroll (#14)
- WS_SCHEME wss:// support for HTTPS (#15)
- Sending guard against duplicate message sends (#16)
- Global unhandledrejection listener in API layer (#17)
Medium:
- Cap planning_logs at 20 entries (#22)
- Store cleanup_loop task reference (#23)
- BaseException → Exception in _run_with_sentinel (#24)
- Propagate SystemExit in agent loop (#25)
- Configurable output_reserve_tokens (#26)
- Always reloadSession on session_sync (#30)
- FIFO queue for confirm dialogs (#31)
- Reset body.overflow on ImageLightbox unmount (#32)
- try/finally in fallback copy (#33)
- _isConnecting guard in WS send() (#34)
Low:
- Lazy-init deps.py singletons (#36)
- Replace __import__ with direct imports (#38)
- Preserve token count 0 in ollama.py (#39)
- Clear orphaned streamingMsg on reconnect reload (#43)
- Escape single quote in UserMessage (#44)
- Polyfill-free findLast replacement (#48)
- Match <table> tags with attributes in markdown (#49)
- Attach copy buttons only when msg.done (#50)
- Fix hasMeta falsy-0 bug (#53)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 29 Apr
|
| 2026-04-28 |
Wire pgvector semantic search into memory system
...
- Add vector(768) column + HNSW index to memory_facts
- Add LLMBackend.embed() with Ollama + fallback implementation
- MemoryStore: cosine-distance search with ILIKE fallback
- New memory tool params: source, confidence, expires_days, source_context
- Update extractor, sqlite_store, deps wiring
- Add pgvector to requirements
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 28 Apr
|
| 2026-04-26 |
changed llm & new ollama param
Eugene Sukhodolskiy
committed
on 26 Apr
|
| 2026-04-25 |
Tune profile sampling configs
Eugene Sukhodolskiy
committed
on 25 Apr
|
| 2026-04-24 |

Add Ollama multi-server fallback with in-memory blacklisting
...
- New FallbackOllamaBackend (navi/llm/fallback.py): tries servers and
models in priority order; on LLMConnectionError blacklists the server
for the process lifetime, on LLMModelNotFoundError blacklists the
(server, model) pair — eliminates latency from repeated failed probes
- OllamaBackend now raises typed LLMConnectionError / LLMModelNotFoundError
instead of bare LLMBackendError; accepts list[str] | str | None for model
- AgentProfile.model changed from str to list[str] (str auto-normalised);
all profiles updated to ["gemma4:31b-cloud", "gemma4:26b-a4b-it-q4_K_M"]
- New config field OLLAMA_BACKENDS_FILE: path to [{host, api_key?}] JSON;
when set, registry creates FallbackOllamaBackend instead of OllamaBackend
- ollama_backends.json template added (gitignored — contains API key)
- current_model ContextVar type widened to list[str] | str | None
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Eugene Sukhodolskiy
committed
on 24 Apr
|