diff --git a/10-systems/hosts/jellyfin.md b/10-systems/hosts/jellyfin.md index 7cac778..1360e6e 100644 --- a/10-systems/hosts/jellyfin.md +++ b/10-systems/hosts/jellyfin.md @@ -1,7 +1,7 @@ --- owner: gmikcon status: active -last_reviewed: "2026-10-10" +last_reviewed: "2026-11-07" review_interval: "90d" confidence: high source_of_truth: ssh-host @@ -27,7 +27,7 @@ - Swap: 1.4 GB (swappiness 60) - Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs) - Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130) -- Uptime: long-running (40+ days as of 2026-10-08) +- Uptime: long-running (40+ days as of 2026-10-08); rebooted 2026-11-07 ~05:20 UTC (home-wide network outage 05:21–05:24) — cgroup MemoryHigh/MemoryMax drop-ins survived the reboot. ## Services @@ -77,6 +77,17 @@ Result 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running. +## Episode 2026-11-07: library validation ReadTimeout + CPU storm — OBSERVED, self-recovered + +Diagnosed by navi rei (reaction session to panel event #1381 `cpu_high` 91% at 18:05:51 UTC; full triage in `workspace/incident_jellyfin_2026-11-07_readtimeout.md`). + +- **Trigger:** scheduled task **«Сканирование медиатеки»** («Validating media library», id `7738148f`) started **17:21:27 UTC** (schedule drifts ~2 min/day). Second availability episode of this class (first: 2026-10-26, 30.6 s ReadTimeout), and this time also a CPU phase. +- **Availability episode:** `service_down` (ReadTimeout, **30.5 s**) 17:22:31→17:23:01 UTC during the ffprobe phase; self-recovered, NRestarts=0. +- **CPU phase:** 17:22 → ~18:08+ UTC — **8–12 parallel ffmpeg children** (thumbnail `-vframes 1` and chapter-image extraction) over a **new serials batch on NFS NetTera8** (Twin Peaks S03, The Boys S01/S03, GoT S01, House of Cards S04, Peaky Blinders S02, HTGAWM S01, One Piece S01E300…), each 100–270% CPU; load1 peak **77** (on 20 cores), panel CPU peak **92%**; NFS read steady 30–50 MB/s for ~45 min. `/health` stayed 200 (4.8–6 ms) through the whole CPU phase. +- **State at 18:07–18:08 UTC:** task Running **95.8%** — left to finish on its own, no intervention. Trickplay `64f5f44c` still **Idle with empty triggers** (did not reappear after the 08/10.10 removals); AudioNormalization still manual-only. RAM available 14.3 GB, swap 0, `oom_kill=0`. +- **Verdict:** load, not fault — the 20-core Xeon absorbing a 12-ffmpeg preview storm; MemoryHigh throttling working as designed. Unlike 2026-10-10, Kestrel did not degrade under ffmpeg storm (the 10-10 freeze was the concat-loudness DSD path). +- **Related finding (same day, from the parallel session):** NetTera8 (192.168.1.131) is **100% full** (35 MB free of 7.3 TB; panel shows 95% due to ext4 reserved blocks — trust `df` from the host); metadata writes to it fail with Permission denied. + ## Swap clearing procedure (validated 2026-10-08) After each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei: @@ -91,9 +102,11 @@ 1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation). 2. Lower `vm.swappiness` to 10 — still proposed, not applied. -3. Trickplay task: stopped + daily trigger removed 2026-10-08; **trigger reappeared and was re-removed 2026-10-10** (manual-only now; verify persistence). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps. +3. Trickplay task: stopped + daily trigger removed 2026-10-08; **trigger reappeared and was re-removed 2026-10-10** (manual-only now; **verified still cleared 2026-11-07**). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps. 4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open. 5. **AudioNormalization: trigger removed 2026-10-10 (manual-only)** — ebur128 analysis of DSD256 music is pathologically slow (14+ h, 7+ GB RSS). Re-enable only after excluding DSD/SACD files from loudness analysis (e.g. separate library without normalization) or on much faster hardware. +6. **Library scan/validate triggers → night hours** — proposed 2026-11-07 after the cpu_high 91% + service_down 30.5 s episodes (both from «Сканирование медиатеки»); a new content batch makes the scan heavy for ~45+ min in daytime. Owner decision, not applied. +7. **NetTera8 (192.168.1.131) is 100% full** — `df` from the jellyfin host (35 MB free of 7.3 TB; ext4-reserved blocks mask it on the panel); Jellyfin metadata writes to it fail. Needs cleanup/expansion — flagged 2026-11-07, owner decision. ## Notes @@ -101,3 +114,4 @@ - No docker on this host; Jellyfin runs as a native systemd service. - 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation). - 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. Same day: Jellyfin added to the panel health checker (service id 7). +- 2026-11-07: reboot ~05:20 UTC (home network outage), then library-scan episode 17:21–18:08+ UTC (ReadTimeout 30.5 s + cpu_high 91%, self-recovered; Trickplay trigger still cleared) — see Episode section; SSH triage by navi rei, no changes on the host. diff --git a/90-maintenance/pending-changes/20261107180932-5815ad6a.json b/90-maintenance/pending-changes/20261107180932-5815ad6a.json new file mode 100644 index 0000000..986811e --- /dev/null +++ b/90-maintenance/pending-changes/20261107180932-5815ad6a.json @@ -0,0 +1,19 @@ +{ + "id": "20261107180932-5815ad6a", + "kind": "doc", + "target": "10-systems/hosts/jellyfin.md", + "summary": "jellyfin: 2026-11-07 scan episode (ReadTimeout 30.5s + cpu_high 91%) triaged; Trickplay trigger still cleared; new recs: night-hour scan triggers, NetTera8 100% full", + "reason": "Episode 2026-11-07 triaged by rei via SSH (cpu_high 91%, load 77, 12 ffmpeg during library scan 17:21 UTC; task left to finish; Trickplay trigger verified still cleared; NetTera8 100% full)", + "payload": { + "content": "---\nowner: gmikcon\nstatus: active\nlast_reviewed: \"2026-11-07\"\nreview_interval: \"90d\"\nconfidence: high\nsource_of_truth: ssh-host\n---\n\n# jellyfin\n\nHome LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).\n\n## Identity\n\n- Hostname: `jellyfin`\n- IP: `192.168.1.153`\n- User: `ubuntu` (sudo with same password)\n- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07)\n- REST API key: gnexus-creds secret `Jellyfin (192.168.1.153) — API key «Navi»` (id `da9bc6a5-648e-44d7-bc4f-6084f83c6717`; key name «Navi» existed in the Jellyfin `ApiKeys` table since 2026-04-11, saved to creds 2026-10-08)\n\n## Platform\n\n- OS: Ubuntu 22.04.4 LTS\n- Kernel: 5.15.0-112-generic\n- RAM: 16 GB\n- Swap: 1.4 GB (swappiness 60)\n- Disks: `/` 29 GB (~44% used), `/home` 161 GB (~0.5% used); `/tmp` is on the root fs (not tmpfs)\n- Storage: media on NFS mounts (`jf_storage/NetTera8` from 192.168.1.131, `secraid5_storage`/`jf_cache` from 192.168.1.130)\n- Uptime: long-running (40+ days as of 2026-10-08); rebooted 2026-11-07 ~05:20 UTC (home-wide network outage 05:21–05:24) — cgroup MemoryHigh/MemoryMax drop-ins survived the reboot.\n\n## Services\n\n- `jellyfin.service` — Jellyfin Media Server 10.10.3, HTTP port `8096` (health endpoint `/health` returns 200 \"Healthy\")\n- `ssh.service` — port 22\n- `hard-monitor.service` — GHard Monitor agent\n\n## Monitoring\n\n- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s.\n- Health-check service on the panel: **service id 7**, url `http://192.168.1.153:8096/health` (added 2026-10-10 after the web-UI freeze incident; public `https://jellyfin.gnexus.space`). Catches service-level hangs while server metrics still look online.\n\n## Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)\n\nDiagnosed by navi rei via SSH after repeated `ram_high`/`swap_high` events (2026-10-07, 2026-10-08):\n\n- **Root cause:** the scheduled task **«Сгенерировать изображения для Trickplay»** (Trickplay image generation, task id `64f5f44cd30dc273cb9890205473bbcc`). It has a DailyTrigger at **03:00 UTC (06:00 Kyiv)** and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: `-vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin//%08d.jpg`) reaches certain files (observed in the large One Piece library), it balloons to **~14.5–14.8 GB RSS**; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.\n- **Confirmed OOM data points:** 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by `.NET TP Worker` via page-cache fault path).\n- **Reproduction attempt 2026-10-08 (navi rei):** running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did **not** reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.\n- **Impact:** the kernel kills only the runaway `ffmpeg` child; `jellyfin.service` itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.\n\n## Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to `ram_high` alert)\n\n1. **Stopped the running Trickplay task** via REST API (`DELETE /ScheduledTasks/Running/64f5f44c...` → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.\n2. **Removed the task's daily trigger** (`POST /ScheduledTasks/64f5f44c.../Triggers` with `[]`) — trickplay generation is now **manual-only** until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.\n3. **systemd cgroup memory limits** on `jellyfin.service`: `sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G` — applied live (no restart) and persistent (drop-ins `/etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf`). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: `sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity`.\n4. **Cleaned stale trickplay temp dirs**: `/tmp/jellyfin/trickplay_*` (1.3 GB of partial images left by OOM-killed processes).\n5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).\n\nResult: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.\n\n## Incident 2026-10-10: AudioNormalization (ebur128 on DSD256) froze the web UI — RESOLVED\n\nDiagnosed by navi rei after the owner asked «что с jellyfin?»; public `https://jellyfin.gnexus.space/health` timed out, VM looked \"online\" in GHard.\n\n- **Symptom:** Jellyfin web UI unresponsive — `curl /health` on **localhost** returned 000 even with 25 s timeout, while `systemctl` reported the service `active`. CPU 90–99%, load up to 40 (20 vCPU), swap 100%.\n- **Cause:** scheduled task **«Нормализация звука»** (AudioNormalization, id `ec2f221fd8e7706b3d3afd2c4591b4d7`, IntervalTrigger 24 h) started 2026-10-09 21:14 UTC. Its ffmpeg ran loudness analysis (`-f concat ... -af ebur128=framelog=verbose -f null -`) over a concat list of **DSD256 `.dsf` music files** (Audioslave 2010 remaster) — DSD decodes pathologically slowly. Runtime **14 h 44 min**, 271% CPU, **7.3 GB RSS** (45% of RAM). In parallel, a library-scan ffprobe spent 4.5 h+ probing `Alien.Earth.S01E03` over NFS. The Kestrel event loop was starved → web UI dead.\n- **Fix applied 2026-10-10 ~11:59 UTC (owner approved):**\n 1. `kill -9` on the runaway ffmpeg (pid 902247) and stuck ffprobe (pid 905977) → `/health` returned **200 within seconds** (localhost and via the public proxy, 2 ms).\n 2. Swap cleared per the procedure below (551 MiB → 0; RAM had 13 GB available).\n 3. **Removed the AudioNormalization trigger** (`POST /ScheduledTasks/ec2f221f.../Triggers` with `[]` → 204) — task is manual-only now. The task itself completed at 11:59:34 UTC right after the kill.\n 4. **Re-removed the Trickplay daily trigger** — see warning below. Its current run (69%) was left going: it is nice-ed, single-threaded, and bounded by the cgroup limits.\n- **API note:** updating task triggers is `POST /ScheduledTasks/{id}/Triggers` (PUT returns 405); body `[]` clears all triggers.\n- **Trigger persistence warning:** the Trickplay DailyTrigger removed on 2026-10-08 was **back on 2026-10-10** with no service restart in between (main PID up since Oct 07). Cause unclear (UI re-enable or config rewrite by another task). After removing any trigger, re-verify the next day.\n- **Observation:** the 21:14 UTC daily maintenance window stacks many tasks at once (library scan, subtitles, lyrics, audio normalization, box sets all fired 21:14–21:16 on Oct 09) — consider staggering them.\n- **Follow-up:** Jellyfin added to the panel health checker as service id 7 (`http://192.168.1.153:8096/health`) so a frozen web UI is detected while server metrics still look online.\n\nResult 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running.\n\n## Episode 2026-11-07: library validation ReadTimeout + CPU storm — OBSERVED, self-recovered\n\nDiagnosed by navi rei (reaction session to panel event #1381 `cpu_high` 91% at 18:05:51 UTC; full triage in `workspace/incident_jellyfin_2026-11-07_readtimeout.md`).\n\n- **Trigger:** scheduled task **«Сканирование медиатеки»** («Validating media library», id `7738148f`) started **17:21:27 UTC** (schedule drifts ~2 min/day). Second availability episode of this class (first: 2026-10-26, 30.6 s ReadTimeout), and this time also a CPU phase.\n- **Availability episode:** `service_down` (ReadTimeout, **30.5 s**) 17:22:31→17:23:01 UTC during the ffprobe phase; self-recovered, NRestarts=0.\n- **CPU phase:** 17:22 → ~18:08+ UTC — **8–12 parallel ffmpeg children** (thumbnail `-vframes 1` and chapter-image extraction) over a **new serials batch on NFS NetTera8** (Twin Peaks S03, The Boys S01/S03, GoT S01, House of Cards S04, Peaky Blinders S02, HTGAWM S01, One Piece S01E300…), each 100–270% CPU; load1 peak **77** (on 20 cores), panel CPU peak **92%**; NFS read steady 30–50 MB/s for ~45 min. `/health` stayed 200 (4.8–6 ms) through the whole CPU phase.\n- **State at 18:07–18:08 UTC:** task Running **95.8%** — left to finish on its own, no intervention. Trickplay `64f5f44c` still **Idle with empty triggers** (did not reappear after the 08/10.10 removals); AudioNormalization still manual-only. RAM available 14.3 GB, swap 0, `oom_kill=0`.\n- **Verdict:** load, not fault — the 20-core Xeon absorbing a 12-ffmpeg preview storm; MemoryHigh throttling working as designed. Unlike 2026-10-10, Kestrel did not degrade under ffmpeg storm (the 10-10 freeze was the concat-loudness DSD path).\n- **Related finding (same day, from the parallel session):** NetTera8 (192.168.1.131) is **100% full** (35 MB free of 7.3 TB; panel shows 95% due to ext4 reserved blocks — trust `df` from the host); metadata writes to it fail with Permission denied.\n\n## Swap clearing procedure (validated 2026-10-08)\n\nAfter each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily `swap_high` warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:\n\n1. Verify enough free RAM first: `free -m` (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).\n2. `sudo swapoff /swapfile && sudo swapon /swapfile` — forces all swap pages back into RAM, swap usage drops to 0.\n3. Verify: `free -m`, `swapon --show`, `curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health` (expect 200), `systemctl is-active jellyfin.service`.\n\nResult 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. Reused 2026-10-10 (551 MiB → 0).\n\n## Recommended fixes — status\n\n1. ~~systemd MemoryMax drop-in~~ → **APPLIED 2026-10-08** as `MemoryHigh=8G` / `MemoryMax=10G` (see Mitigation).\n2. Lower `vm.swappiness` to 10 — still proposed, not applied.\n3. Trickplay task: stopped + daily trigger removed 2026-10-08; **trigger reappeared and was re-removed 2026-10-10** (manual-only now; **verified still cleared 2026-11-07**). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.\n4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.\n5. **AudioNormalization: trigger removed 2026-10-10 (manual-only)** — ebur128 analysis of DSD256 music is pathologically slow (14+ h, 7+ GB RSS). Re-enable only after excluding DSD/SACD files from loudness analysis (e.g. separate library without normalization) or on much faster hardware.\n6. **Library scan/validate triggers → night hours** — proposed 2026-11-07 after the cpu_high 91% + service_down 30.5 s episodes (both from «Сканирование медиатеки»); a new content batch makes the scan heavy for ~45+ min in daytime. Owner decision, not applied.\n7. **NetTera8 (192.168.1.131) is 100% full** — `df` from the jellyfin host (35 MB free of 7.3 TB; ext4-reserved blocks mask it on the panel); Jellyfin metadata writes to it fail. Needs cleanup/expansion — flagged 2026-11-07, owner decision.\n\n## Notes\n\n- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).\n- No docker on this host; Jellyfin runs as a native systemd service.\n- 2026-10-08: swap cleared manually after the daily OOM (see procedure above); `ram_high` 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation).\n- 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. Same day: Jellyfin added to the panel health checker (service id 7).\n- 2026-11-07: reboot ~05:20 UTC (home network outage), then library-scan episode 17:21–18:08+ UTC (ReadTimeout 30.5 s + cpu_high 91%, self-recovered; Trickplay trigger still cleared) — see Episode section; SSH triage by navi rei, no changes on the host.\n" + }, + "status": "applied", + "created_at": "2026-11-07T18:09:32Z", + "updated_at": "2026-11-07T18:09:35Z", + "applied_at": "2026-11-07T18:09:35Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}