Newer
Older
gnexus-book / 90-maintenance / pending-changes / 20261008213150-aa56e649.json
{
  "id": "20261008213150-aa56e649",
  "kind": "doc",
  "target": "10-systems/hosts/jellyfin.md",
  "summary": "jellyfin: root cause OOM-цикла найден (Trickplay task), применены cgroup-лимиты 8G/10G, задача остановлена и переведена на ручной запуск",
  "reason": "Реакция на ram_high алерт 2026-10-08 21:18 UTC: найдена первопричина ежедневного OOM-цикла (задача Trickplay), применены меры (остановка задачи, снятие триггера, cgroup-лимиты MemoryHigh=8G/MemoryMax=10G), очистка /tmp/jellyfin; добавлен API-ключ в gnexus-creds",
  "payload": {
    "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: \"2026-10-08\"\nreview_interval: \"90d\"\nconfidence: high\nsource_of_truth: ssh-host\n---\n\n# jellyfin\n\nHome LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).\n\n## Identity\n\n- Hostname: `jellyfin`\n- IP: `192.168.1.153`\n- User: `ubuntu` (sudo with same password)\n- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07)\n- REST API key: gnexus-creds secret `Jellyfin (192.168.1.153) — API key «Navi»` (id `da9bc6a5-648e-44d7-bc4f-6084f83c6717`; key name «Navi» existed in the Jellyfin `ApiKeys` table since 2026-04-11, saved to creds 2026-10-08)\n\n## Platform\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel: 5.15.0-112-generic\n- RAM: 16 GB\n- Swap: 1.4 GB (swappiness 60)\n- Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs)\n- Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130)\n- Uptime: long-running (40+ days as of 2026-10-08)\n\n## Services\n\n- `jellyfin.service` — Jellyfin Media Server 10.10.3, HTTP port `8096` (health endpoint `/health` returns 200 \"Healthy\")\n- `ssh.service` — port 22\n- `hard-monitor.service` — GHard Monitor agent\n\n## Monitoring\n\n- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s.\n- Not registered as a health-check service on the panel (only server metrics).\n\n## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)\n\nDiagnosed by navi rei via SSH after repeated `ram_high`/`swap_high` events (2026-10-07, 2026-10-08):\n\n- **Root cause:** the scheduled task **«Сгенерировать изображения для Trickplay»** (Trickplay image generation, task id `64f5f44cd30dc273cb9890205473bbcc`). It has a DailyTrigger at **03:00 UTC (06:00 Kyiv)** and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: `-vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin/<id>/%08d.jpg`) reaches certain files (observed in the large One Piece library), it balloons to **~14.5–14.8 GB RSS**; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.\n- **Confirmed OOM data points:** 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by `.NET TP Worker` via page-cache fault path).\n- **Reproduction attempt 2026-10-08 (navi rei):** running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did **not** reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.\n- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.\n\n## Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to `ram_high` alert)\n\n1. **Stopped the running Trickplay task** via REST API (`DELETE /ScheduledTasks/Running/64f5f44c...` → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.\n2. **Removed the task's daily trigger** (`PUT /ScheduledTasks/64f5f44c.../Triggers` with `[]`) — trickplay generation is now **manual-only** until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.\n3. **systemd cgroup memory limits** on `jellyfin.service`: `sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G` — applied live (no restart) and persistent (drop-ins `/etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf`). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: `sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity`.\n4. **Cleaned stale trickplay temp dirs**: `/tmp/jellyfin/trickplay_*` (1.3 GB of partial images left by OOM-killed processes).\n5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).\n\nResult: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.\n\n## Swap clearing procedure (validated 2026-10-08)\n\nAfter each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:\n\n1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).\n2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0.\n3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`.\n\nResult 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure.\n\n## Recommended fixes — status\n\n1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation).\n2. Lower `vm.swappiness` to 10 — still proposed, not applied.\n3. Trickplay task: **stopped + daily trigger removed 2026-10-08** (manual-only now). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.\n4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.\n\n## Notes\n\n- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).\n- No docker on this host; Jellyfin runs as a native systemd service.\n- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation)."
  },
  "status": "applied",
  "created_at": "2026-10-08T21:31:50Z",
  "updated_at": "2026-10-08T21:31:52Z",
  "applied_at": "2026-10-08T21:31:52Z",
  "validation": {
    "status": "ok",
    "issue_count": 0,
    "issues": []
  }
}