Newer
Older
gnexus-book / 10-systems / hosts / jellyfin.md

owner: gmikcon status: active last_reviewed: "2026-10-08" review_interval: "90d" confidence: high

source_of_truth: ssh-host

jellyfin

Home LAN media server running Jellyfin — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).

Identity

  • Hostname: jellyfin
  • IP: 192.168.1.153
  • User: ubuntu (sudo with same password)
  • Credentials: gnexus-creds secret jellyfin (192.168.1.153) — SSH ubuntu (id 84a01b86-ec1d-41d1-927b-253f9b4c1770, SSH = sudo password; verified by SSH login 2026-10-07)
  • REST API key: gnexus-creds secret Jellyfin (192.168.1.153) — API key «Navi» (id da9bc6a5-648e-44d7-bc4f-6084f83c6717; key name «Navi» existed in the Jellyfin ApiKeys table since 2026-04-11, saved to creds 2026-10-08)

Platform

  • OS: Ubuntu 22.04.4 LTS
  • Kernel: 5.15.0-112-generic
  • RAM: 16 GB
  • Swap: 1.4 GB (swappiness 60)
  • Disks: / 29 GB (~44% used), /home 161 GB (~0.5% used); /tmp is on the root fs (not tmpfs)
  • Storage: media on NFS mounts (jf_storage/NetTera8 from 192.168.1.131, secraid5_storage/jf_cache from 192.168.1.130)
  • Uptime: long-running (40+ days as of 2026-10-08)

Services

  • jellyfin.service — Jellyfin Media Server 10.10.3, HTTP port 8096 (health endpoint /health returns 200 "Healthy")
  • ssh.service — port 22
  • hard-monitor.service — GHard Monitor agent

Monitoring

  • GHard panel: server jellyfin, id 39, status online, interval 30s.
  • Not registered as a health-check service on the panel (only server metrics).

Known issue: daily OOM-kill cycle — ROOT CAUSE IDENTIFIED (2026-10-08)

Diagnosed by navi rei via SSH after repeated ram_high/swap_high events (2026-10-07, 2026-10-08):

  • Root cause: the scheduled task «Сгенерировать изображения для Trickplay» (Trickplay image generation, task id 64f5f44cd30dc273cb9890205473bbcc). It has a DailyTrigger at 03:00 UTC (06:00 Kyiv) and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: -vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin/<id>/%08d.jpg) reaches certain files (observed in the large One Piece library), it balloons to ~14.5–14.8 GB RSS; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger.
  • Confirmed OOM data points: 2026-10-07 21:12:08 UTC (ffmpeg pid 870980, anon-rss 14.52 GB); 2026-10-08 21:19:37 UTC (ffmpeg pid 886034, anon-rss 14.72 GB, invoked by .NET TP Worker via page-cache fault path).
  • Reproduction attempt 2026-10-08 (navi rei): running the exact same ffmpeg command on the two suspect files (One Piece S01E0749, S01E0338) in an isolated 2 GB cgroup did not reproduce the blowup (RSS flat ~50 MB, clean exit). The explosion depends on runtime state (NFS page-cache pressure / file state at that moment), not the file alone. The underlying ffmpeg/Jellyfin memory blowup remains an open upstream issue.
  • Impact: the kernel kills only the runaway ffmpeg child; jellyfin.service itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.

Mitigation applied 2026-10-08 21:18–21:30 UTC (navi rei, autonomous reaction to ram_high alert)

  1. Stopped the running Trickplay task via REST API (DELETE /ScheduledTasks/Running/64f5f44c... → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.
  2. Removed the task's daily trigger (PUT /ScheduledTasks/64f5f44c.../Triggers with []) — trickplay generation is now manual-only until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.
  3. systemd cgroup memory limits on jellyfin.service: sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G — applied live (no restart) and persistent (drop-ins /etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity.
  4. Cleaned stale trickplay temp dirs: /tmp/jellyfin/trickplay_* (1.3 GB of partial images left by OOM-killed processes).
  5. Swap was already 0 by 21:29 UTC (cleared at 21:22 UTC — see procedure below).

Result: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.

Swap clearing procedure (validated 2026-10-08)

After each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily swap_high warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:

  1. Verify enough free RAM first: free -m (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).
  2. sudo swapoff /swapfile && sudo swapon /swapfile — forces all swap pages back into RAM, swap usage drops to 0.
  3. Verify: free -m, swapon --show, curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health (expect 200), systemctl is-active jellyfin.service.

Result 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure.

  1. systemd MemoryMax drop-in → APPLIED 2026-10-08 as MemoryHigh=8G / MemoryMax=10G (see Mitigation).
  2. Lower vm.swappiness to 10 — still proposed, not applied.
  3. Trickplay task: stopped + daily trigger removed 2026-10-08 (manual-only now). Hardware-accelerated transcoding / thread caps — still open. The ffmpeg memory blowup itself is upstream; candidate for a Jellyfin bug report with the OOM dumps.
  4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan) — still open.

Notes

  • Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident).
  • No docker on this host; Jellyfin runs as a native systemd service.
  • 2026-10-08: swap cleared manually after the daily OOM (see procedure above); ram_high 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation).