owner: gmikcon status: active last_reviewed: "2026-11-07" review_interval: "90d" confidence: high
Home LAN media server running Jellyfin — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere).
jellyfin192.168.1.153ubuntu (sudo with same password)jellyfin (192.168.1.153) — SSH ubuntu (id 84a01b86-ec1d-41d1-927b-253f9b4c1770, SSH = sudo password; verified by SSH login 2026-10-07)Jellyfin (192.168.1.153) — API key «Navi» (id da9bc6a5-648e-44d7-bc4f-6084f83c6717; key name «Navi» existed in the Jellyfin ApiKeys table since 2026-04-11, saved to creds 2026-10-08)/ 29 GB (~44% used), /home 161 GB (~0.5% used); /tmp is on the root fs (not tmpfs)jf_storage/NetTera8 from 192.168.1.131, secraid5_storage/jf_cache from 192.168.1.130)jellyfin.service — Jellyfin Media Server 10.10.3, HTTP port 8096 (health endpoint /health returns 200 "Healthy")ssh.service — port 22hard-monitor.service — GHard Monitor agentjellyfin, id 39, status online, interval 30s.http://192.168.1.153:8096/health (added 2026-10-10 after the web-UI freeze incident; public https://jellyfin.gnexus.space). Catches service-level hangs while server metrics still look online.Diagnosed by navi rei via SSH after repeated ram_high/swap_high events (2026-10-07, 2026-10-08):
64f5f44cd30dc273cb9890205473bbcc). It has a DailyTrigger at 03:00 UTC (06:00 Kyiv) and grinds through the library for ~18 h; when its ffmpeg child (trickplay extraction: -vf fps=0.1 ... -c:v mjpeg -f image2 /tmp/jellyfin/<id>/%08d.jpg) reaches certain files (observed in the large One Piece library), it balloons to ~14.5–14.8 GB RSS; the kernel OOM-killer kills it (~21:10–21:20 UTC), the task loop moves on / retries — producing the daily cycle. Task log confirmed: «Cancelled after 1101 minutes» = started 03:00 UTC, matches the trigger..NET TP Worker via page-cache fault path).ffmpeg child; jellyfin.service itself survives (main PID unchanged, 40+ days uptime). Playback interruptions are brief; the real cost is swap filling to 100% and the daily alert noise.ram_high alert)DELETE /ScheduledTasks/Running/64f5f44c... → 204; it had been running 1101 min). Verified: no ffmpeg processes, task state Idle.POST /ScheduledTasks/64f5f44c.../Triggers with []) — trickplay generation is now manual-only until the owner decides. Rollback: re-add the DailyTrigger 03:00 via API/UI.jellyfin.service: sudo systemctl set-property jellyfin.service MemoryHigh=8G MemoryMax=10G — applied live (no restart) and persistent (drop-ins /etc/systemd/system.control/jellyfin.service.d/50-Memory{High,Max}.conf). Effect: a runaway ffmpeg is OOM-killed inside the service cgroup at 10 GB instead of triggering a global OOM at ~15 GB; MemoryHigh=8G adds reclaim pressure earlier. Chosen over the previously proposed 4G to keep headroom for legit 4K remux/transcode work. Rollback: sudo systemctl set-property jellyfin.service MemoryHigh=infinity MemoryMax=infinity./tmp/jellyfin/trickplay_* (1.3 GB of partial images left by OOM-killed processes).Result: RAM 13% (2.2/16.7 GB), swap 0, load 0.46, no ffmpeg processes, service healthy.
Diagnosed by navi rei after the owner asked «что с jellyfin?»; public https://jellyfin.gnexus.space/health timed out, VM looked "online" in GHard.
curl /health on localhost returned 000 even with 25 s timeout, while systemctl reported the service active. CPU 90–99%, load up to 40 (20 vCPU), swap 100%.ec2f221fd8e7706b3d3afd2c4591b4d7, IntervalTrigger 24 h) started 2026-10-09 21:14 UTC. Its ffmpeg ran loudness analysis (-f concat ... -af ebur128=framelog=verbose -f null -) over a concat list of DSD256 .dsf music files (Audioslave 2010 remaster) — DSD decodes pathologically slowly. Runtime 14 h 44 min, 271% CPU, 7.3 GB RSS (45% of RAM). In parallel, a library-scan ffprobe spent 4.5 h+ probing Alien.Earth.S01E03 over NFS. The Kestrel event loop was starved → web UI dead.kill -9 on the runaway ffmpeg (pid 902247) and stuck ffprobe (pid 905977) → /health returned 200 within seconds (localhost and via the public proxy, 2 ms).POST /ScheduledTasks/ec2f221f.../Triggers with [] → 204) — task is manual-only now. The task itself completed at 11:59:34 UTC right after the kill.POST /ScheduledTasks/{id}/Triggers (PUT returns 405); body [] clears all triggers.http://192.168.1.153:8096/health) so a frozen web UI is detected while server metrics still look online.Result 2026-10-10 12:00 UTC: health 200, RAM 1.8/15 Gi, swap 0, load falling (5.4), only the nice-ed trickplay ffmpeg running.
Diagnosed by navi rei (reaction session to panel event #1381 cpu_high 91% at 18:05:51 UTC; full triage in workspace/incident_jellyfin_2026-11-07_readtimeout.md).
7738148f) started 17:21:27 UTC (schedule drifts ~2 min/day). Second availability episode of this class (first: 2026-10-26, 30.6 s ReadTimeout), and this time also a CPU phase.service_down (ReadTimeout, 30.5 s) 17:22:31→17:23:01 UTC during the ffprobe phase; self-recovered, NRestarts=0.-vframes 1 and chapter-image extraction) over a new serials batch on NFS NetTera8 (Twin Peaks S03, The Boys S01/S03, GoT S01, House of Cards S04, Peaky Blinders S02, HTGAWM S01, One Piece S01E300…), each 100–270% CPU; load1 peak 77 (on 20 cores), panel CPU peak 92%; NFS read steady 30–50 MB/s for ~45 min. /health stayed 200 (4.8–6 ms) through the whole CPU phase.64f5f44c still Idle with empty triggers (did not reappear after the 08/10.10 removals); AudioNormalization still manual-only. RAM available 14.3 GB, swap 0, oom_kill=0.df from the host); metadata writes to it fail with Permission denied.After each OOM cycle the swapfile stays ~100% full with cold pages (mostly jellyfin) and drains only over ~24 h, producing a daily swap_high warning. Safe manual cleanup, applied 2026-10-08 21:22 UTC by navi rei:
free -m (need available > swap used; after OOM RAM is ~1.3/16 GB, so 564 Mi–1.3 Gi swap fits easily).sudo swapoff /swapfile && sudo swapon /swapfile — forces all swap pages back into RAM, swap usage drops to 0.free -m, swapon --show, curl -s -o /dev/null -w '%{http_code}' http://localhost:8096/health (expect 200), systemctl is-active jellyfin.service.Result 2026-10-08: swap 0 B used, RAM used 1.25→1.75 GB, service healthy, no degradation. Do NOT run swapoff while RAM is under pressure. Reused 2026-10-10 (551 MiB → 0).
MemoryHigh=8G / MemoryMax=10G (see Mitigation).vm.swappiness to 10 — still proposed, not applied.df from the jellyfin host (35 MB free of 7.3 TB; ext4-reserved blocks mask it on the panel); Jellyfin metadata writes to it fail. Needs cleanup/expansion — flagged 2026-11-07, owner decision.ram_high 21:18 UTC resolved the same evening by stopping the Trickplay task + cgroup limits (see Mitigation).