Newer
Older
gnexus-book / 10-systems / hosts / jellyfin.md

owner: gmikcon status: active last_reviewed: "2026-11-07" review_interval: "90d" confidence: high

source_of_truth: ssh-host

jellyfin

Home LAN media server running Jellyfin — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).

Identity

  • Hostname: jellyfin
  • IP: 192.168.1.153
  • User: ubuntu (sudo with same password)
  • Credentials: gnexus-creds secret jellyfin (192.168.1.153) — SSH ubuntu (id 84a01b86-ec1d-41d1-927b-253f9b4c1770, SSH = sudo password; verified by SSH login 2026-10-07)
  • REST API key: gnexus-creds secret Jellyfin (192.168.1.153) — API key «Navi» (id da9bc6a5-648e-44d7-bc4f-6084f83c6717; key name «Navi» existed in the Jellyfin ApiKeys table since 2026-04-11, saved to creds 2026-10-08)

Platform

  • OS: Ubuntu 22.04.4 LTS
  • Kernel: 5.15.0-112-generic
  • RAM: 16 GB
  • Swap: 1.4 GB (swappiness 60)
  • Disks: / 29 GB (~44% used), /home 161 GB (~0.5% used); /tmp is on the root fs (not tmpfs)
  • Storage: media on NFS mounts (jf_storage/NetTera8 from 192.168.1.131, secraid5_storage/jf_cache from 192.168.1.130)
  • Uptime: long-running (40+ days as of 2026-10-08); rebooted 2026-11-07 ~05:20 UTC (home-wide network outage 05:21–05:24) — cgroup MemoryHigh/MemoryMax drop-ins survived the reboot.

Services

  • jellyfin.service — Jellyfin Media Server 10.10.3, HTTP port 8096 (health endpoint /health returns 200 "Healthy")
  • ssh.service — port 22
  • hard-monitor.service — GHard Monitor agent

Monitoring

  • GHard panel: server jellyfin, id 39, status online, interval 30s.
  • Health-check service on the panel: service id 7, url http://192.168.1.153:8096/health (added 2026-10-10 after the web-UI freeze incident; public https://jellyfin.gnexus.space). Catches service-level hangs while server metrics still look online.

Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)

Diagnosed by navi rei via SSH after repeated ram_high/swap_high events (2026-10-07, 2026-10-08):

  • Root cause: the scheduled task «Сгенерировать изображения для Trickplay» (Trickplay image generation, task id 64f5f44cd30dc273cb9890205473bbcc). It has a DailyTrigger at 03:00 UTC (06:00 Kyiv) and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: -vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin/<id>/%08d.jpg) reaches certain files (observed in the large One Piece library), it balloons to ~14.5–14.8 GB RSS; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.
  • Confirmed OOM data points: 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by .NET TP Worker via page-cache fault path).
  • Reproduction attempt 2026-10-08 (navi rei): running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did not reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.
  • Impact: the kernel kills only the runaway ffmpeg child; jellyfin.service itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.

Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to ram_high alert)

  1. Stopped the running Trickplay task via REST API (DELETE /ScheduledTasks/Running/64f5f44c... → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.
  2. Removed the task's daily trigger (POST /ScheduledTasks/64f5f44c.../Triggers with []) — trickplay generation is now manual-only until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.
  3. systemd cgroup memory limits on jellyfin.service: sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G — applied live (no restart) and persistent (drop-ins /etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity.
  4. Cleaned stale trickplay temp dirs: /tmp/jellyfin/trickplay_* (1.3 GB of partial images left by OOM-killed processes).
  5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).

Result: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.

Incident 2026-10-10: AudioNormalization (ebur128 on DSD256) froze the web UI — RESOLVED

Diagnosed by navi rei after the owner asked «что с jellyfin?»; public https://jellyfin.gnexus.space/health timed out, VM looked "online" in GHard.

  • Symptom: Jellyfin web UI unresponsive — curl /health on localhost returned 000 even with 25 s timeout, while systemctl reported the service active. CPU 90–99%, load up to 40 (20 vCPU), swap 100%.
  • Cause: scheduled task «Нормализация звука» (AudioNormalization, id ec2f221fd8e7706b3d3afd2c4591b4d7, IntervalTrigger 24 h) started 2026-10-09 21:14 UTC. Its ffmpeg ran loudness analysis (-f concat ... -af ebur128=framelog=verbose -f null -) over a concat list of DSD256 .dsf music files (Audioslave 2010 remaster) — DSD decodes pathologically slowly. Runtime 14 h 44 min, 271% CPU, 7.3 GB RSS (45% of RAM). In parallel, a library-scan ffprobe spent 4.5 h+ probing Alien.Earth.S01E03 over NFS. The Kestrel event loop was starved → web UI dead.
  • Fix applied 2026-10-10 ~11:59 UTC (owner approved):
    1. kill -9 on the runaway ffmpeg (pid 902247) and stuck ffprobe (pid 905977) → /health returned 200 within seconds (localhost and via the public proxy, 2 ms).
    2. Swap cleared per the procedure below (551 MiB → 0; RAM had 13 GB available).
    3. Removed the AudioNormalization trigger (POST /ScheduledTasks/ec2f221f.../Triggers with [] → 204) — task is manual-only now. The task itself completed at 11:59:34 UTC right after the kill.
    4. Re-removed the Trickplay daily trigger — see warning below. Its current run (69%) was left going: it is nice-ed, single-threaded, and bounded by the cgroup limits.
  • API note: updating task triggers is POST /ScheduledTasks/{id}/Triggers (PUT returns 405); body [] clears all triggers.
  • Trigger persistence warning: the Trickplay DailyTrigger removed on 2026-10-08 was back on 2026-10-10 with no service restart in between (main PID up since Oct 07). Cause unclear (UI re-enable or config rewrite by another task). After removing any trigger, re-verify the next day.
  • Observation: the 21:14 UTC daily maintenance window stacks many tasks at once (library scan, subtitles, lyrics, audio normalization, box sets all fired 21:14–21:16 on Oct 09) — consider staggering them.
  • Follow-up: Jellyfin added to the panel health checker as service id 7 (http://192.168.1.153:8096/health) so a frozen web UI is detected while server metrics still look online.

Result 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running.

Episode 2026-11-07: library validation ReadTimeout + CPU storm — OBSERVED, self-recovered

Diagnosed by navi rei (reaction session to panel event #1381 cpu_high 91% at 18:05:51 UTC; full triage in workspace/incident_jellyfin_2026-11-07_readtimeout.md).

  • Trigger: scheduled task «Сканирование медиатеки» («Validating media library», id 7738148f) started 17:21:27 UTC (schedule drifts ~2 min/day). Second availability episode of this class (first: 2026-10-26, 30.6 s ReadTimeout), and this time also a CPU phase.
  • Availability episode: service_down (ReadTimeout, 30.5 s) 17:22:31→17:23:01 UTC during the ffprobe phase; self-recovered, NRestarts=0.
  • CPU phase: 17:22 → ~18:08+ UTC — 8–12 parallel ffmpeg children (thumbnail -vframes 1 and chapter-image extraction) over a new serials batch on NFS NetTera8 (Twin Peaks S03, The Boys S01/S03, GoT S01, House of Cards S04, Peaky Blinders S02, HTGAWM S01, One Piece S01E300…), each 100–270% CPU; load1 peak 77 (on 20 cores), panel CPU peak 92%; NFS read steady 30–50 MB/s for ~45 min. /health stayed 200 (4.8–6 ms) through the whole CPU phase.
  • State at 18:07–18:08 UTC: task Running 95.8% — left to finish on its own, no intervention. Trickplay 64f5f44c still Idle with empty triggers (did not reappear after the 08/10.10 removals); AudioNormalization still manual-only. RAM available 14.3 GB, swap 0, oom_kill=0.
  • Verdict: load, not fault — the 20-core Xeon absorbing a 12-ffmpeg preview storm; MemoryHigh throttling working as designed. Unlike 2026-10-10, Kestrel did not degrade under ffmpeg storm (the 10-10 freeze was the concat-loudness DSD path).
  • Related finding (same day, from the parallel session): NetTera8 (192.168.1.131) is 100% full (35 MB free of 7.3 TB; panel shows 95% due to ext4 reserved blocks — trust df from the host); metadata writes to it fail with Permission denied.

Swap clearing procedure (validated 2026-10-08)

After each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily swap_high warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:

  1. Verify enough free RAM first: free -m (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).
  2. sudo swapoff /swapfile && sudo swapon /swapfile — forces all swap pages back into RAM, swap usage drops to 0.
  3. Verify: free -m, swapon --show, curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health (expect 200), systemctl is-active jellyfin.service.

Result 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. Reused 2026-10-10 (551 MiB → 0).

  1. systemd MemoryMax drop-in → APPLIED 2026-10-08 as MemoryHigh=8G / MemoryMax=10G (see Mitigation).
  2. Lower vm.swappiness to 10 — still proposed, not applied.
  3. Trickplay task: stopped + daily trigger removed 2026-10-08; trigger reappeared and was re-removed 2026-10-10 (manual-only now; verified still cleared 2026-11-07). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.
  4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.
  5. AudioNormalization: trigger removed 2026-10-10 (manual-only) — ebur128 analysis of DSD256 music is pathologically slow (14+ h, 7+ GB RSS). Re-enable only after excluding DSD/SACD files from loudness analysis (e.g. separate library without normalization) or on much faster hardware.
  6. Library scan/validate triggers → night hours — proposed 2026-11-07 after the cpu_high 91% + service_down 30.5 s episodes (both from «Сканирование медиатеки»); a new content batch makes the scan heavy for ~45+ min in daytime. Owner decision, not applied.
  7. NetTera8 (192.168.1.131) is 100% full — df from the jellyfin host (35 MB free of 7.3 TB; ext4-reserved blocks mask it on the panel); Jellyfin metadata writes to it fail. Needs cleanup/expansion — flagged 2026-11-07, owner decision.

Notes

  • Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).
  • No docker on this host; Jellyfin runs as a native systemd service.
  • 2026-10-08: swap cleared manually after the daily OOM (see procedure above); ram_high 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation).
  • 2026-10-10: AudioNormalization froze the web UI for ~15 h (see Incident section); resolved by killing its ffmpeg + clearing swap + removing its trigger; Trickplay trigger re-removed. Same day: Jellyfin added to the panel health checker (service id 7).
  • 2026-11-07: reboot ~05:20 UTC (home network outage), then library-scan episode 17:21–18:08+ UTC (ReadTimeout 30.5 s + cpu_high 91%, self-recovered; Trickplay trigger still cleared) — see Episode section; SSH triage by navi rei, no changes on the host.