
..
Modelfile.qwen-128k	llm: routes that name a host, a backup and a window, and a profile that reads them	23 hours ago
README.md	llm: routes that name a host, a backup and a window, and a profile that reads them	23 hours ago

Ollama on the fallback host

192.168.1.129 serves the local backup for the default route: when the cloud (glm-5.3-flash:cloud) is unreachable or out of quota, qwen3.6:35b-128k keeps Navi usable with the same 128k window.

The model

Modelfile.qwen-128k derives a tag from the base model — same blobs, a new manifest, no extra disk. Apply it once per host (and again after editing):

ollama create qwen3.6:35b-128k -f deploy/ollama/Modelfile.qwen-128k

What it changes and why is documented inline. In short: 131072 context, MTP drafting off, presence_penalty 0 instead of 1.5.

Server settings

The daemon needs a drop-in (/etc/systemd/system/ollama.service.d/override.conf):

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=f16"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_KEEP_ALIVE=30m"

KV_CACHE_TYPE=f16 — published benchmarks put q8_0 level with f16 on this model, but on this host quantized KV is a regression: measured at a 128k slot, q8_0 gave 37 tok/s against 70 on f16, and llama.cpp split the graph 24 ways (vs 2) with a 443 MiB compute buffer instead of 92 MiB — the quantized KV decode path is evidently not taking the fast kernel for this model's 256-dim heads. The f16 cache costs ~2.5 GiB at 128k and still leaves ~1.5 GiB free on a 24 GiB card, which is enough for the embedding model.
NUM_PARALLEL=1 — num_ctx is the window of one slot and -c is num_ctx × np. Two slots would need 262144 tokens of cache for a 128k request. A second concurrent request waits instead.
Quantization, when it is used, touches only the 10 attention layers; the SSM state is always f32 (~63 MiB per slot).

Measured on this host

RTX 24 GiB, Q4_K_M, 41/42 layers on GPU, 128k slot: ~70 tok/s decode, ~1350 tok/s prefill (86k-token prompt), ~2.5 GiB of f16 KV, ~1.5 GiB of VRAM left. Decode speed is not monotonic in num_ctx — 32k runs at ~150 tok/s, 34k at ~92, 36k at ~150 again. Reproducible, cause not identified; do not assume a bigger window is proportionally slower.