diff --git a/10-systems/hosts/jellyfin.md b/10-systems/hosts/jellyfin.md index 0edc49c..cfd63c0 100644 --- a/10-systems/hosts/jellyfin.md +++ b/10-systems/hosts/jellyfin.md @@ -1,7 +1,7 @@ --- owner: gmikcon status: active -last_reviewed: 2026-10-07 +last_reviewed: "2026-10-08" review_interval: "90d" confidence: high source_of_truth: ssh-host @@ -38,23 +38,35 @@ - GHard panel: server `jellyfin`, **id 39**, status online, interval 30s. - Not registered as a health-check service on the panel (only server metrics). -## Known issue: daily OOM-kill cycle (documented 2026-10-07) +## Known issue: daily OOM-kill cycle (documented 2026-10-07, confirmed 2026-10-08) -Diagnosed by navi rei via SSH (read-only) after repeated `ram_high`/`swap_high`/`swap_recovered` events: +Diagnosed by navi rei via SSH after repeated `ram_high`/`swap_high`/`swap_recovered` events: -- **Pattern:** every evening (~20:48–21:12 UTC, drifting later each day) an `ffmpeg` process spawned by Jellyfin (uid 109) grows to **~14.5–14.8 GB RSS** on the 16 GB box; the kernel OOM-killer kills it, `jellyfin.service` fails with `oom-kill` and systemd restarts it immediately. Swap fills to 100% during the ramp-up and then drains over ~24 h — producing daily `swap_high` → `swap_recovered` event pairs on the panel. +- **Pattern:** every evening (~20:48–21:18 UTC, drifting later each day) an `ffmpeg` process spawned by Jellyfin (uid 109) grows to **~14.5–14.8 GB RSS** on the 16 GB box; the kernel OOM-killer kills it, and systemd restarts the service if needed. Swap fills to 100% during the ramp-up and then drains over ~24 h — producing daily `swap_high` → `swap_recovered` event pairs on the panel. +- **Confirmed data points:** 2026-10-07 21:10:41 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:18:09 UTC (ffmpeg pid 886034, anon-rss 14.72 GB). Drift continues (~+8 min/day), consistent with a slow leak in the transcoding path. - **Not cron/timers:** `/etc/cron.d`, user crontabs and systemd timers are standard; the trigger is Jellyfin activity itself (library scan with ~140 ffprobe calls + evening remux playback sessions). -- **Slow memory leak suspected:** the kill time drifts later each day (20:48 → 21:10 over several days), consistent with a leak in the transcoding path. -- **Current mitigation:** none applied — systemd auto-restart masks the outage; playback interruptions are brief. +- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself usually survives (main PID unchanged, 40+ days uptime), logs an `ffmpeg image extraction failed` error and continues. Playback interruptions are brief. +- **Current mitigation:** none structural — systemd auto-restart masks the outage. See "Swap clearing procedure" below for the routine post-incident cleanup. + +## Swap clearing procedure (validated 2026-10-08) + +After each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei: + +1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily). +2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0. +3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`. + +Result 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. ## Recommended fixes (proposed, NOT applied — owner approval required) -1. systemd drop-in for `jellyfin.service`: `MemoryMax=4G` (or similar) so a runaway ffmpeg is capped before global OOM. +1. systemd drop-in for `jellyfin.service`: `MemoryMax=4G` (or similar) so a runaway ffmpeg is capped before global OOM. Note: the existing drop-in `/etc/systemd/system/jellyfin.service.d/jellyfin.service.conf` is the vendor default (all options commented out) — a new drop-in file (e.g. `memlimit.conf`) is the cleanest place. 2. Lower `vm.swappiness` to 10 (swap currently fills to 100% before OOM fires). -3. In Jellyfin settings: enable hardware-accelerated transcoding and/or cap transcode threads; review the library-scan schedule. +3. In Jellyfin settings: enable hardware-accelerated transcoding and/or cap transcode threads; review the library-scan schedule. The runaway process is the **trickplay/thumbnail extraction** ffmpeg (`-vf fps=0.1 ... -c:v mjpeg` over `/tmp/jellyfin//%08d.jpg`) — disabling or limiting trickplay generation for the large One Piece library is a targeted candidate. 4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan). ## Notes - Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident). - No docker on this host; Jellyfin runs as a native systemd service. +- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `swap_high` warning for 2026-10-08 can be treated as resolved. diff --git a/90-maintenance/pending-changes/20261008212410-8de03f89.json b/90-maintenance/pending-changes/20261008212410-8de03f89.json new file mode 100644 index 0000000..451c5fd --- /dev/null +++ b/90-maintenance/pending-changes/20261008212410-8de03f89.json @@ -0,0 +1,19 @@ +{ + "id": "20261008212410-8de03f89", + "kind": "doc", + "target": "10-systems/hosts/jellyfin.md", + "summary": "jellyfin: подтверждён ежедневный OOM-цикл (второй инцидент 2026-10-08), добавлена проверенная процедура очистки swap, уточнён процесс-виновник (trickplay ffmpeg)", + "reason": "Ежедневный OOM-цикл подтверждён вторым инцидентом (2026-10-08 21:18 UTC, ffmpeg 14.72 GB); swap_high 2026-10-08 обработан: swap очищен безопасной процедурой swapoff/swapon, сервис здоров. Добавлена проверенная процедура очистки swap и уточнён виновник (trickplay-ffmpeg).", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: \"2026-10-08\"\nreview_interval: \"90d\"\nconfidence: high\nsource_of_truth: ssh-host\n---\n\n# jellyfin\n\nHome LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).\n\n## Identity\n\n- Hostname: `jellyfin`\n- IP: `192.168.1.153`\n- User: `ubuntu` (sudo with same password)\n- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07)\n\n## Platform\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel: 5.15.0-112-generic\n- RAM: 16 GB\n- Swap: 1.4 GB (swappiness 60)\n- Disks: `/` 29 GB (~48% used), `/home` 161 GB (~0.5% used)\n- Uptime: long-running (40+ days as of 2026-10-07)\n\n## Services\n\n- `jellyfin.service` — Jellyfin Media Server, HTTP port `8096` (health endpoint `/health` returns 200 \"Healthy\")\n- `ssh.service` — port 22\n- `hard-monitor.service` — GHard Monitor agent\n\n## Monitoring\n\n- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s.\n- Not registered as a health-check service on the panel (only server metrics).\n\n## Known issue: daily OOM-kill cycle (documented 2026-10-07, confirmed 2026-10-08)\n\nDiagnosed by navi rei via SSH after repeated `ram_high`/`swap_high`/`swap_recovered` events:\n\n- **Pattern:** every evening (~20:48–21:18 UTC, drifting later each day) an `ffmpeg` process spawned by Jellyfin (uid 109) grows to **~14.5–14.8 GB RSS** on the 16 GB box; the kernel OOM-killer kills it, and systemd restarts the service if needed. Swap fills to 100% during the ramp-up and then drains over ~24 h — producing daily `swap_high` → `swap_recovered` event pairs on the panel.\n- **Confirmed data points:** 2026-10-07 21:10:41 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:18:09 UTC (ffmpeg pid 886034, anon-rss 14.72 GB). Drift continues (~+8 min/day), consistent with a slow leak in the transcoding path.\n- **Not cron/timers:** `/etc/cron.d`, user crontabs and systemd timers are standard; the trigger is Jellyfin activity itself (library scan with ~140 ffprobe calls + evening remux playback sessions).\n- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself usually survives (main PID unchanged, 40+ days uptime), logs an `ffmpeg image extraction failed` error and continues. Playback interruptions are brief.\n- **Current mitigation:** none structural — systemd auto-restart masks the outage. See \"Swap clearing procedure\" below for the routine post-incident cleanup.\n\n## Swap clearing procedure (validated 2026-10-08)\n\nAfter each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:\n\n1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).\n2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0.\n3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`.\n\nResult 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure.\n\n## Recommended fixes (proposed, NOT applied — owner approval required)\n\n1. systemd drop-in for `jellyfin.service`: `MemoryMax=4G` (or similar) so a runaway ffmpeg is capped before global OOM. Note: the existing drop-in `/etc/systemd/system/jellyfin.service.d/jellyfin.service.conf` is the vendor default (all options commented out) — a new drop-in file (e.g. `memlimit.conf`) is the cleanest place.\n2. Lower `vm.swappiness` to 10 (swap currently fills to 100% before OOM fires).\n3. In Jellyfin settings: enable hardware-accelerated transcoding and/or cap transcode threads; review the library-scan schedule. The runaway process is the **trickplay/thumbnail extraction** ffmpeg (`-vf fps=0.1 ... -c:v mjpeg` over `/tmp/jellyfin//%08d.jpg`) — disabling or limiting trickplay generation for the large One Piece library is a targeted candidate.\n4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan).\n\n## Notes\n\n- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).\n- No docker on this host; Jellyfin runs as a native systemd service.\n- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `swap_high` warning for 2026-10-08 can be treated as resolved." + }, + "status": "applied", + "created_at": "2026-10-08T21:24:10Z", + "updated_at": "2026-10-08T21:24:12Z", + "applied_at": "2026-10-08T21:24:12Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}