diff --git a/10-systems/hosts/jellyfin.md b/10-systems/hosts/jellyfin.md index cfd63c0..9af2b2c 100644 --- a/10-systems/hosts/jellyfin.md +++ b/10-systems/hosts/jellyfin.md @@ -17,6 +17,7 @@ - IP: `192.168.1.153` - User: `ubuntu` (sudo with same password) - Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07) +- REST API key: gnexus-creds secret `Jellyfin (192.168.1.153) — API key «Navi»` (id `da9bc6a5-648e-44d7-bc4f-6084f83c6717`; key name «Navi» existed in the Jellyfin `ApiKeys` table since 2026-04-11, saved to creds 2026-10-08) ## Platform @@ -24,12 +25,13 @@ - Kernel: 5.15.0-112-generic - RAM: 16 GB - Swap: 1.4 GB (swappiness 60) -- Disks: `/` 29 GB (~48% used), `/home` 161 GB (~0.5% used) -- Uptime: long-running (40+ days as of 2026-10-07) +- Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs) +- Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130) +- Uptime: long-running (40+ days as of 2026-10-08) ## Services -- `jellyfin.service` — Jellyfin Media Server, HTTP port `8096` (health endpoint `/health` returns 200 "Healthy") +- `jellyfin.service` — Jellyfin Media Server 10.10.3, HTTP port `8096` (health endpoint `/health` returns 200 "Healthy") - `ssh.service` — port 22 - `hard-monitor.service` — GHard Monitor agent @@ -38,15 +40,24 @@ - GHard panel: server `jellyfin`, **id 39**, status online, interval 30s. - Not registered as a health-check service on the panel (only server metrics). -## Known issue: daily OOM-kill cycle (documented 2026-10-07, confirmed 2026-10-08) +## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08) -Diagnosed by navi rei via SSH after repeated `ram_high`/`swap_high`/`swap_recovered` events: +Diagnosed by navi rei via SSH after repeated `ram_high`/`swap_high` events (2026-10-07, 2026-10-08): -- **Pattern:** every evening (~20:48–21:18 UTC, drifting later each day) an `ffmpeg` process spawned by Jellyfin (uid 109) grows to **~14.5–14.8 GB RSS** on the 16 GB box; the kernel OOM-killer kills it, and systemd restarts the service if needed. Swap fills to 100% during the ramp-up and then drains over ~24 h — producing daily `swap_high` → `swap_recovered` event pairs on the panel. -- **Confirmed data points:** 2026-10-07 21:10:41 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:18:09 UTC (ffmpeg pid 886034, anon-rss 14.72 GB). Drift continues (~+8 min/day), consistent with a slow leak in the transcoding path. -- **Not cron/timers:** `/etc/cron.d`, user crontabs and systemd timers are standard; the trigger is Jellyfin activity itself (library scan with ~140 ffprobe calls + evening remux playback sessions). -- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself usually survives (main PID unchanged, 40+ days uptime), logs an `ffmpeg image extraction failed` error and continues. Playback interruptions are brief. -- **Current mitigation:** none structural — systemd auto-restart masks the outage. See "Swap clearing procedure" below for the routine post-incident cleanup. +- **Root cause:** the scheduled task **«Сгенерировать изображения для Trickplay»** (Trickplay image generation, task id `64f5f44cd30dc273cb9890205473bbcc`). It has a DailyTrigger at **03:00 UTC (06:00 Kyiv)** and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: `-vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin//%08d.jpg`) reaches certain files (observed in the large One Piece library), it balloons to **~14.5–14.8 GB RSS**; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger. +- **Confirmed OOM data points:** 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by `.NET TP Worker` via page-cache fault path). +- **Reproduction attempt 2026-10-08 (navi rei):** running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did **not** reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue. +- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise. + +## Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to `ram_high` alert) + +1. **Stopped the running Trickplay task** via REST API (`DELETE /ScheduledTasks/Running/64f5f44c...` → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle. +2. **Removed the task's daily trigger** (`PUT /ScheduledTasks/64f5f44c.../Triggers` with `[]`) — trickplay generation is now **manual-only** until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI. +3. **systemd cgroup memory limits** on `jellyfin.service`: `sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G` — applied live (no restart) and persistent (drop-ins `/etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf`). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: `sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity`. +4. **Cleaned stale trickplay temp dirs**: `/tmp/jellyfin/trickplay_*` (1.3 GB of partial images left by OOM-killed processes). +5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below). + +Result: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy. ## Swap clearing procedure (validated 2026-10-08) @@ -58,15 +69,15 @@ Result 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. -## Recommended fixes (proposed, NOT applied — owner approval required) +## Recommended fixes — status -1. systemd drop-in for `jellyfin.service`: `MemoryMax=4G` (or similar) so a runaway ffmpeg is capped before global OOM. Note: the existing drop-in `/etc/systemd/system/jellyfin.service.d/jellyfin.service.conf` is the vendor default (all options commented out) — a new drop-in file (e.g. `memlimit.conf`) is the cleanest place. -2. Lower `vm.swappiness` to 10 (swap currently fills to 100% before OOM fires). -3. In Jellyfin settings: enable hardware-accelerated transcoding and/or cap transcode threads; review the library-scan schedule. The runaway process is the **trickplay/thumbnail extraction** ffmpeg (`-vf fps=0.1 ... -c:v mjpeg` over `/tmp/jellyfin//%08d.jpg`) — disabling or limiting trickplay generation for the large One Piece library is a targeted candidate. -4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan). +1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation). +2. Lower `vm.swappiness` to 10 — still proposed, not applied. +3. Trickplay task: **stopped + daily trigger removed 2026-10-08** (manual-only now). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps. +4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open. ## Notes - Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident). - No docker on this host; Jellyfin runs as a native systemd service. -- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `swap_high` warning for 2026-10-08 can be treated as resolved. +- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation). diff --git a/90-maintenance/pending-changes/20261008213150-aa56e649.json b/90-maintenance/pending-changes/20261008213150-aa56e649.json new file mode 100644 index 0000000..f21ebe9 --- /dev/null +++ b/90-maintenance/pending-changes/20261008213150-aa56e649.json @@ -0,0 +1,19 @@ +{ + "id": "20261008213150-aa56e649", + "kind": "doc", + "target": "10-systems/hosts/jellyfin.md", + "summary": "jellyfin: root cause OOM-цикла найден (Trickplay task), применены cgroup-лимиты 8G/10G, задача остановлена и переведена на ручной запуск", + "reason": "Реакция на ram_high алерт 2026-10-08 21:18 UTC: найдена первопричина ежедневного OOM-цикла (задача Trickplay), применены меры (остановка задачи, снятие триггера, cgroup-лимиты MemoryHigh=8G/MemoryMax=10G), очистка /tmp/jellyfin; добавлен API-ключ в gnexus-creds", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: \"2026-10-08\"\nreview_interval: \"90d\"\nconfidence: high\nsource_of_truth: ssh-host\n---\n\n# jellyfin\n\nHome LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).\n\n## Identity\n\n- Hostname: `jellyfin`\n- IP: `192.168.1.153`\n- User: `ubuntu` (sudo with same password)\n- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07)\n- REST API key: gnexus-creds secret `Jellyfin (192.168.1.153) — API key «Navi»` (id `da9bc6a5-648e-44d7-bc4f-6084f83c6717`; key name «Navi» existed in the Jellyfin `ApiKeys` table since 2026-04-11, saved to creds 2026-10-08)\n\n## Platform\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel: 5.15.0-112-generic\n- RAM: 16 GB\n- Swap: 1.4 GB (swappiness 60)\n- Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs)\n- Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130)\n- Uptime: long-running (40+ days as of 2026-10-08)\n\n## Services\n\n- `jellyfin.service` — Jellyfin Media Server 10.10.3, HTTP port `8096` (health endpoint `/health` returns 200 \"Healthy\")\n- `ssh.service` — port 22\n- `hard-monitor.service` — GHard Monitor agent\n\n## Monitoring\n\n- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s.\n- Not registered as a health-check service on the panel (only server metrics).\n\n## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)\n\nDiagnosed by navi rei via SSH after repeated `ram_high`/`swap_high` events (2026-10-07, 2026-10-08):\n\n- **Root cause:** the scheduled task **«Сгенерировать изображения для Trickplay»** (Trickplay image generation, task id `64f5f44cd30dc273cb9890205473bbcc`). It has a DailyTrigger at **03:00 UTC (06:00 Kyiv)** and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: `-vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin//%08d.jpg`) reaches certain files (observed in the large One Piece library), it balloons to **~14.5–14.8 GB RSS**; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.\n- **Confirmed OOM data points:** 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by `.NET TP Worker` via page-cache fault path).\n- **Reproduction attempt 2026-10-08 (navi rei):** running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did **not** reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.\n- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.\n\n## Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to `ram_high` alert)\n\n1. **Stopped the running Trickplay task** via REST API (`DELETE /ScheduledTasks/Running/64f5f44c...` → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.\n2. **Removed the task's daily trigger** (`PUT /ScheduledTasks/64f5f44c.../Triggers` with `[]`) — trickplay generation is now **manual-only** until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.\n3. **systemd cgroup memory limits** on `jellyfin.service`: `sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G` — applied live (no restart) and persistent (drop-ins `/etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf`). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: `sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity`.\n4. **Cleaned stale trickplay temp dirs**: `/tmp/jellyfin/trickplay_*` (1.3 GB of partial images left by OOM-killed processes).\n5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).\n\nResult: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.\n\n## Swap clearing procedure (validated 2026-10-08)\n\nAfter each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:\n\n1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).\n2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0.\n3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`.\n\nResult 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure.\n\n## Recommended fixes — status\n\n1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation).\n2. Lower `vm.swappiness` to 10 — still proposed, not applied.\n3. Trickplay task: **stopped + daily trigger removed 2026-10-08** (manual-only now). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.\n4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.\n\n## Notes\n\n- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).\n- No docker on this host; Jellyfin runs as a native systemd service.\n- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation)." + }, + "status": "applied", + "created_at": "2026-10-08T21:31:50Z", + "updated_at": "2026-10-08T21:31:52Z", + "applied_at": "2026-10-08T21:31:52Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}