diff --git a/10-systems/hosts/jellyfin.md b/10-systems/hosts/jellyfin.md index ff456c7..7cac778 100644 --- a/10-systems/hosts/jellyfin.md +++ b/10-systems/hosts/jellyfin.md @@ -38,7 +38,7 @@ ## Monitoring - GHard panel: server `jellyfin`, **id 39**, status online, interval 30s. -- Not registered as a health-check service on the panel (only server metrics). +- Health-check service on the panel: **service id 7**, url `http://192.168.1.153:8096/health` (added 2026-10-10 after the web-UI freeze incident; public `https://jellyfin.gnexus.space`). Catches service-level hangs while server metrics still look online. ## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08) @@ -73,6 +73,7 @@ - **API note:** updating task triggers is `POST /ScheduledTasks/{id}/Triggers` (PUT returns 405); body `[]` clears all triggers. - **Trigger persistence warning:** the Trickplay DailyTrigger removed on 2026-10-08 was **back on 2026-10-10** with no service restart in between (main PID up since Oct 07). Cause unclear (UI re-enable or config rewrite by another task). After removing any trigger, re-verify the next day. - **Observation:** the 21:14 UTC daily maintenance window stacks many tasks at once (library scan, subtitles, lyrics, audio normalization, box sets all fired 21:14–21:16 on Oct 09) — consider staggering them. +- **Follow-up:** Jellyfin added to the panel health checker as service id 7 (`http://192.168.1.153:8096/health`) so a frozen web UI is detected while server metrics still look online. Result 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running. @@ -99,4 +100,4 @@ - Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident). - No docker on this host; Jellyfin runs as a native systemd service. - 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation). -- 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. +- 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. Same day: Jellyfin added to the panel health checker (service id 7). diff --git a/90-maintenance/pending-changes/20261010120824-e3f60f28.json b/90-maintenance/pending-changes/20261010120824-e3f60f28.json new file mode 100644 index 0000000..a8f2e69 --- /dev/null +++ b/90-maintenance/pending-changes/20261010120824-e3f60f28.json @@ -0,0 +1,19 @@ +{ + "id": "20261010120824-e3f60f28", + "kind": "doc", + "target": "10-systems/hosts/jellyfin.md", + "summary": "Monitoring section: jellyfin now health-checked on the panel as service id 7; follow-up note in incident section", + "reason": "Jellyfin добавлен в health-чекер панели как сервис id 7 (http://192.168.1.153:8096/health) 2026-10-10 — после инцидента с зависшим веб-UI, который сервер-метрики не ловили.", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: \"2026-10-10\"\nreview_interval: \"90d\"\nconfidence: high\nsource_of_truth: ssh-host\n---\n\n# jellyfin\n\nHome LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).\n\n## Identity\n\n- Hostname: `jellyfin`\n- IP: `192.168.1.153`\n- User: `ubuntu` (sudo with same password)\n- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07)\n- REST API key: gnexus-creds secret `Jellyfin (192.168.1.153) — API key «Navi»` (id `da9bc6a5-648e-44d7-bc4f-6084f83c6717`; key name «Navi» existed in the Jellyfin `ApiKeys` table since 2026-04-11, saved to creds 2026-10-08)\n\n## Platform\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel: 5.15.0-112-generic\n- RAM: 16 GB\n- Swap: 1.4 GB (swappiness 60)\n- Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs)\n- Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130)\n- Uptime: long-running (40+ days as of 2026-10-08)\n\n## Services\n\n- `jellyfin.service` — Jellyfin Media Server 10.10.3, HTTP port `8096` (health endpoint `/health` returns 200 \"Healthy\")\n- `ssh.service` — port 22\n- `hard-monitor.service` — GHard Monitor agent\n\n## Monitoring\n\n- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s.\n- Health-check service on the panel: **service id 7**, url `http://192.168.1.153:8096/health` (added 2026-10-10 after the web-UI freeze incident; public `https://jellyfin.gnexus.space`). Catches service-level hangs while server metrics still look online.\n\n## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)\n\nDiagnosed by navi rei via SSH after repeated `ram_high`/`swap_high` events (2026-10-07, 2026-10-08):\n\n- **Root cause:** the scheduled task **«Сгенерировать изображения для Trickplay»** (Trickplay image generation, task id `64f5f44cd30dc273cb9890205473bbcc`). It has a DailyTrigger at **03:00 UTC (06:00 Kyiv)** and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: `-vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin//%08d.jpg`) reaches certain files (observed in the large One Piece library), it balloons to **~14.5–14.8 GB RSS**; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.\n- **Confirmed OOM data points:** 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by `.NET TP Worker` via page-cache fault path).\n- **Reproduction attempt 2026-10-08 (navi rei):** running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did **not** reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.\n- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.\n\n## Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to `ram_high` alert)\n\n1. **Stopped the running Trickplay task** via REST API (`DELETE /ScheduledTasks/Running/64f5f44c...` → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.\n2. **Removed the task's daily trigger** (`POST /ScheduledTasks/64f5f44c.../Triggers` with `[]`) — trickplay generation is now **manual-only** until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.\n3. **systemd cgroup memory limits** on `jellyfin.service`: `sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G` — applied live (no restart) and persistent (drop-ins `/etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf`). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: `sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity`.\n4. **Cleaned stale trickplay temp dirs**: `/tmp/jellyfin/trickplay_*` (1.3 GB of partial images left by OOM-killed processes).\n5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).\n\nResult: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.\n\n## Incident 2026-10-10: AudioNormalization (ebur128 on DSD256) froze the web UI — RESOLVED\n\nDiagnosed by navi rei after the owner asked «что с jellyfin?»; public `https://jellyfin.gnexus.space/health` timed out, VM looked \"online\" in GHard.\n\n- **Symptom:** Jellyfin web UI unresponsive — `curl /health` on **localhost** returned 000 even with 25 s timeout, while `systemctl` reported the service `active`. CPU 90–99%, load up to 40 (20 vCPU), swap 100%.\n- **Cause:** scheduled task **«Нормализация звука»** (AudioNormalization, id `ec2f221fd8e7706b3d3afd2c4591b4d7`, IntervalTrigger 24 h) started 2026-10-09 21:14 UTC. Its ffmpeg ran loudness analysis (`-f concat ... -af ebur128=framelog=verbose -f null -`) over a concat list of **DSD256 `.dsf` music files** (Audioslave 2010 remaster) — DSD decodes pathologically slowly. Runtime **14 h 44 min**, 271% CPU, **7.3 GB RSS** (45% of RAM). In parallel, a library-scan ffprobe spent 4.5 h+ probing `Alien.Earth.S01E03` over NFS. The Kestrel event loop was starved → web UI dead.\n- **Fix applied 2026-10-10 ~11:59 UTC (owner approved):**\n 1. `kill -9` on the runaway ffmpeg (pid 902247) and stuck ffprobe (pid 905977) → `/health` returned **200 within seconds** (localhost and via the public proxy, 2 ms).\n 2. Swap cleared per the procedure below (551 MiB → 0; RAM had 13 GB available).\n 3. **Removed the AudioNormalization trigger** (`POST /ScheduledTasks/ec2f221f.../Triggers` with `[]` → 204) — task is manual-only now. The task itself completed at 11:59:34 UTC right after the kill.\n 4. **Re-removed the Trickplay daily trigger** — see warning below. Its current run (69%) was left going: it is nice-ed, single-threaded, and bounded by the cgroup limits.\n- **API note:** updating task triggers is `POST /ScheduledTasks/{id}/Triggers` (PUT returns 405); body `[]` clears all triggers.\n- **Trigger persistence warning:** the Trickplay DailyTrigger removed on 2026-10-08 was **back on 2026-10-10** with no service restart in between (main PID up since Oct 07). Cause unclear (UI re-enable or config rewrite by another task). After removing any trigger, re-verify the next day.\n- **Observation:** the 21:14 UTC daily maintenance window stacks many tasks at once (library scan, subtitles, lyrics, audio normalization, box sets all fired 21:14–21:16 on Oct 09) — consider staggering them.\n- **Follow-up:** Jellyfin added to the panel health checker as service id 7 (`http://192.168.1.153:8096/health`) so a frozen web UI is detected while server metrics still look online.\n\nResult 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running.\n\n## Swap clearing procedure (validated 2026-10-08)\n\nAfter each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:\n\n1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).\n2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0.\n3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`.\n\nResult 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. Reused 2026-10-10 (551 MiB → 0).\n\n## Recommended fixes — status\n\n1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation).\n2. Lower `vm.swappiness` to 10 — still proposed, not applied.\n3. Trickplay task: stopped + daily trigger removed 2026-10-08; **trigger reappeared and was re-removed 2026-10-10** (manual-only now; verify persistence). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.\n4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.\n5. **AudioNormalization: trigger removed 2026-10-10 (manual-only)** — ebur128 analysis of DSD256 music is pathologically slow (14+ h, 7+ GB RSS). Re-enable only after excluding DSD/SACD files from loudness analysis (e.g. separate library without normalization) or on much faster hardware.\n\n## Notes\n\n- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).\n- No docker on this host; Jellyfin runs as a native systemd service.\n- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation).\n- 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. Same day: Jellyfin added to the panel health checker (service id 7)." + }, + "status": "applied", + "created_at": "2026-10-10T12:08:24Z", + "updated_at": "2026-10-10T12:08:25Z", + "applied_at": "2026-10-10T12:08:25Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}