# Ollama on the fallback host

`192.168.1.129` serves the local backup for the `default` route: when the cloud
(`glm-5.3-flash:cloud`) is unreachable or out of quota, `qwen3.6:35b-128k` keeps
Navi usable with the **same 128k window**.

## The model

`Modelfile.qwen-128k` derives a tag from the base model — same blobs, a new
manifest, no extra disk. Apply it once per host (and again after editing):

```bash
ollama create qwen3.6:35b-128k -f deploy/ollama/Modelfile.qwen-128k
```

What it changes and why is documented inline. In short: 131072 context, MTP
drafting off, `presence_penalty` 0 instead of 1.5.

## Server settings

The daemon needs a drop-in (`/etc/systemd/system/ollama.service.d/override.conf`):

```ini
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=f16"
Environment="OLLAMA_NUM_PARALLEL=1"
Environment="OLLAMA_KEEP_ALIVE=30m"
```

- **`KV_CACHE_TYPE=f16`** — published benchmarks put q8_0 level with f16 on this
  model, but **on this host quantized KV is a regression**: measured at a 128k
  slot, q8_0 gave 37 tok/s against 70 on f16, and llama.cpp split the graph 24
  ways (vs 2) with a 443 MiB compute buffer instead of 92 MiB — the quantized
  KV decode path is evidently not taking the fast kernel for this model's
  256-dim heads. The f16 cache costs ~2.5 GiB at 128k and still leaves ~1.5 GiB
  free on a 24 GiB card, which is enough for the embedding model.
- **`NUM_PARALLEL=1`** — `num_ctx` is the window of *one* slot and `-c` is
  `num_ctx × np`. Two slots would need 262144 tokens of cache for a 128k
  request. A second concurrent request waits instead.
- Quantization, when it is used, touches only the 10 attention layers; the SSM
  state is always f32 (~63 MiB per slot).

## Measured on this host

RTX 24 GiB, Q4_K_M, 41/42 layers on GPU, 128k slot: **~70 tok/s decode,
~1350 tok/s prefill** (86k-token prompt), ~2.5 GiB of f16 KV, ~1.5 GiB of VRAM
left. Decode speed is not monotonic in `num_ctx` — 32k runs at ~150 tok/s, 34k
at ~92, 36k at ~150 again. Reproducible, cause not identified; do not assume a
bigger window is proportionally slower.