diff --git a/10-systems/hosts/jellyfin.md b/10-systems/hosts/jellyfin.md index d479734..0edc49c 100644 --- a/10-systems/hosts/jellyfin.md +++ b/10-systems/hosts/jellyfin.md @@ -1,41 +1,60 @@ --- owner: gmikcon status: active -last_reviewed: 2026-10-08 -review_interval: 90d +last_reviewed: 2026-10-07 +review_interval: "90d" confidence: high source_of_truth: ssh-host --- -# Jellyfin VM +# jellyfin -KVM VM on `hp-proliant-dl380-g6` (libvirt), LAN `192.168.1.153`, Jellyfin 10.10.3, systemd unit `jellyfin`, port 8096. Public URL: https://jellyfin.gnexus.space (nginx on internal-proxy-vps `.226` → `http://192.168.1.153:8096`). SSH: `ubuntu@192.168.1.153` (password same as Navi swarm peers). Verified live 2026-09-29. +Home LAN media server running **Jellyfin** — the family media library and streaming backend (watched via Jellyfin clients; torrent downloads arrive via Transmission elsewhere). -На hard-panel (GHard Monitor): сервер **id 39**, имя `jellyfin`, 16 ГБ RAM, агент интервал 30 с. +## Identity -## Health checks +- Hostname: `jellyfin` +- IP: `192.168.1.153` +- User: `ubuntu` (sudo with same password) +- Credentials: gnexus-creds secret `jellyfin (192.168.1.153) — SSH ubuntu` (id `84a01b86-ec1d-41d1-927b-253f9b4c1770`, SSH = sudo password; verified by SSH login 2026-10-07) -- `GET /health` → 200 "Healthy" -- `GET /System/Info/Public` → JSON with version -- Logs: `/var/log/jellyfin/jellyfinYYYYMMDD*.log` (rotated ~10 MB) +## Platform -## Storage (NFS mounts on the VM) +- OS: Ubuntu 22.04.4 LTS +- Kernel: 5.15.0-112-generic +- RAM: 16 GB +- Swap: 1.4 GB (swappiness 60) +- Disks: `/` 29 GB (~48% used), `/home` 161 GB (~0.5% used) +- Uptime: long-running (40+ days as of 2026-10-07) -| Mount on VM | NFS export | Note (2026-09-29) | -| --- | --- | --- | -| `/home/ubuntu/jf_storage/firstraid5_storage` | `192.168.1.130:/mnt/firstraid5` | 100% full, 0 B free; `music/` folder is empty | -| `/home/ubuntu/jf_storage/secraid5_storage` | `192.168.1.130:/mnt/secraid5/jellyfin_storage` | 126 GB free (94% used) | -| `/home/ubuntu/jf_cache` | `192.168.1.130:/mnt/secraid5/jellyfin_cache` | 99% free | -| `/home/ubuntu/jf_storage/NetTera8` | `192.168.1.131:/mnt/NetTera8` | 100% full (7.3 TB used) | +## Services -## Known incident (2026-09-29): NFS hang from gprint +- `jellyfin.service` — Jellyfin Media Server, HTTP port `8096` (health endpoint `/health` returns 200 "Healthy") +- `ssh.service` — port 22 +- `hard-monitor.service` — GHard Monitor agent -Symptom: Jellyfin "not working" — library scans/streams hang, processes stuck in D-state. Root cause: `nfs-server.service` on `gprint` (192.168.1.131, NetTera8 export) was `inactive (dead)` since 14:51 UTC (stopped during the 2026-09-29 gprint cleanup). The NetTera8 NFS mount on the Jellyfin VM hung on any access (`df`, `stat`, `ls` all timed out). +## Monitoring -Fix: start nfs-server on gprint. Gotcha: first start failed with `errno 12 (Cannot allocate memory)` — gprint has only 1.9 GiB RAM and 1.4 GiB was in buff/cache. `sync; echo 3 > /proc/sys/vm/drop_caches` freed 1.3 GiB and the service started. +- GHard panel: server `jellyfin`, **id 39**, status online, interval 30s. +- Not registered as a health-check service on the panel (only server metrics). -Rule: if Jellyfin hangs, check `df` on all NFS mounts of the VM first, then `rpcinfo -p 192.168.1.131` (must list nfs/mountd, not just portmapper). +## Known issue: daily OOM-kill cycle (documented 2026-10-07) -## Incident 2026-10-07 21:11 UTC: transient RAM spike, self-recovered +Diagnosed by navi rei via SSH (read-only) after repeated `ram_high`/`swap_high`/`swap_recovered` events: -hard-panel alert `ram_high` 92.23% (≥90%) on panel id 39 → Synapse → Navi rei. Через ~90 с панель сама записала `ram_recovered` (RAM 4.52%) и `swap_recovered` (swap 3.3%) в 21:12:38. Характер: разовый транзиент — RAM 92%→5% (~14.5 ГБ освободилось), swap упал с хронических ≥40% до 3%. Топ-процессы после: jellyfin ~2.3% mem, CPU 0%. Похоже на page-cache/транскод-всплеск с последующей выгрузкой; своп ≥40% наблюдался до этого (событие swap_high за пределами окна журнала). Действий не потребовалось, владелец уведомлён. Если ram_high на jellyfin начнёт повторяться регулярно — смотреть логи Jellyfin (`/var/log/jellyfin/`) и активные транскод-сессии. +- **Pattern:** every evening (~20:48–21:12 UTC, drifting later each day) an `ffmpeg` process spawned by Jellyfin (uid 109) grows to **~14.5–14.8 GB RSS** on the 16 GB box; the kernel OOM-killer kills it, `jellyfin.service` fails with `oom-kill` and systemd restarts it immediately. Swap fills to 100% during the ramp-up and then drains over ~24 h — producing daily `swap_high` → `swap_recovered` event pairs on the panel. +- **Not cron/timers:** `/etc/cron.d`, user crontabs and systemd timers are standard; the trigger is Jellyfin activity itself (library scan with ~140 ffprobe calls + evening remux playback sessions). +- **Slow memory leak suspected:** the kill time drifts later each day (20:48 → 21:10 over several days), consistent with a leak in the transcoding path. +- **Current mitigation:** none applied — systemd auto-restart masks the outage; playback interruptions are brief. + +## Recommended fixes (proposed, NOT applied — owner approval required) + +1. systemd drop-in for `jellyfin.service`: `MemoryMax=4G` (or similar) so a runaway ffmpeg is capped before global OOM. +2. Lower `vm.swappiness` to 10 (swap currently fills to 100% before OOM fires). +3. In Jellyfin settings: enable hardware-accelerated transcoding and/or cap transcode threads; review the library-scan schedule. +4. Consider upgrading RAM if transcoding demand is real (16 GB is tight for 4K remux + scan). + +## Notes + +- Host was missing from gnexus-book inventory until 2026-10-07 (added by navi rei after the OOM incident). +- No docker on this host; Jellyfin runs as a native systemd service.