| 2026-11-03 |
Add gstation-tts node doc: pipeline layout, 7 verified traps, v8 run book and CER baseline.
...
Session 2026-11-03 verified live: HF_HOME trap broke two training launches; sparse blob, NUL logs, NFS-after-reboot and /tmp-wipe traps all hit in one night. Persisting the run book to gnexus-book so future sessions don't rediscover them.
|
| 2026-10-31 |
gitbucket.md: 9th crawler episode (31.10 03:30 UTC) + correction of ep8 timeline (one continuous ~3.5h crawl, ~127k req)
...
9th crawler episode on gitbucket (GHard #1152, 2026-10-31 03:30 UTC, cpu 97.1%): 4x GET /root/smart-home-server/archive/devices/*.zip (3x sha 44250a7a, 1x 8a945d0d), 8x EofException/Broken pipe, java 333.7% CPU (~3 ZIP builds), load peak 7.05, no restart, self-recovered 03:31:51. Correction: ep8 did not self-extinguish 00:09 — enumeration continued ~440-480/min through 00-02 (29k req/h), one continuous crawl 23:58-03:35+, ~126.8k req total, client 192.168.1.100% 192.168.1.150, GPTBot/1.4 + Chrome/Edge 142-145 rotation. Triage: read-only ssh ubuntu@192.168.1.156 (bserv@192.168.1.130 creds) and nginx access.log on 192.168.1.183. Incident journal: workspace/incident_gitbucket_2026-10-31_ep9_crawl.md.
|
gstation: новый хост — семейный игровой ПК на Bazzite (192.168.1.135)
...
Новый хост в инвентаре (hosts/gstation) + новый док 10-systems/hosts/gstation.md. Факты из SSH-сессий 2026-10-31: Bazzite 44.20261026, i5-3470/19GB/RX 580 2048SP, HDD NTFS /var/mnt/games, статический IP .135, GDM autologin, отключённые lock/hot corners/hotedge/welcome-тур (+welcome-fix autostart), ТВ underscan через GNOME underscanning (tvmode/tvtest), 35 игр через play.sh + Heroic couch, геймпад RZTK спарен (Home+X).
|
gitbucket: 8th archive-burst episode (30.10 23:57 UTC), nextcloud 61-s blip correlation confirmed twice, pending-decision counter updated
...
8th archive-burst episode 2026-10-30 23:57 UTC (panel #1136/#1137, reaction events 63c2c371 + 34367587): added episode to the list, recorded the 2nd confirmed 61-s nextcloud blip coincidence (27.10 + 30.10), updated pending-decision counter to 5 consecutive days (26–30 Oct). Sources: SSH triage on 192.168.1.156, nginx access.log on 192.168.1.183, hard-panel service_incidents.
|
gitbucket host doc: 8th archive-burst episode 2026-10-30 23:57 UTC (short burst, self-recovered ~7 min), GPTBot/1.4 crawl attribution, pending owner decision now 5 consecutive days
...
Episode 8 of the archive-burst class confirmed 2026-10-30 23:57 UTC (panel event #1136, diagnosed by navi rei via SSH + nginx logs): short burst of 20 archive/*.zip requests amid GPTBot crawl, peak load1 11.25, self-recovered ~7 min. Updates episode count, traffic-source note and pending owner decision (5 consecutive days).
|
gitbucket: 8th archive-crawler CPU episode (30.10 23:57 UTC), series 5 consecutive days
...
8th episode of the archive-crawler CPU class: 2026-10-30 23:57 UTC burst (cpu_high 91% #1136, load_high 2.11×cores #1137, cpu_recovered #1140), confirmed via read-only SSH journalctl on .156 — same /archive/*.zip signature, self-recovered in ~3.5 min. Series now 5 consecutive days (26–30 Oct).
|
| 2026-10-30 |
Document hpsa controller reset failure mode (DL380 G6) with 2026-10-30 incident
...
Reaction session: load_high on files (#36) and 5 other VMs traced to hpsa controller logical-volume resets on bserv (DL380 G6). Incident 2026-10-30 13:06-13:15 UTC, self-recovered. Documented signature, timeline, triage playbook.
|
Добавлен хост fuckjira (OpenProject 17, 192.168.1.175) в inventory hosts
...
Реакция на событие hard-panel container_added (bd255c26, openproject-web-1, server_id 29). Проверено: контейнер Up 13 days (healthy), стек OpenProject 17 цел; removed/added 13:11:41-42 UTC — артефакт ре-регистрации инвентаря панели на фоне load-всплеска 13:10-13:12 (hive ReadTimeout), реального рестарта не было (uptime непрерывный). SSH-доступа к хосту в gnexus-creds нет.
|
| 2026-10-29 |
gitbucket: 7th archive-crawl episode (29.10, ~4.5 h sustained, self-recovered)
...
Reaction session rei on GHard event #1100 (cpu_high 90.5%, server 44 gitbucket, 2026-10-29 10:40:57 UTC). SSH triage confirmed the known archive-crawl class (7th episode since 2026-10-12) with a new signature: sustained ~4.5 h crawl 06:00–10:42 UTC, 5248 /archive/*.zip requests from 192.168.1.150 via public route, 29 Broken pipe/EofException errors, load decayed 5.07→0.66 by 10:48 UTC, self-recovered, no restart needed. Known behavior section updated with 7th episode, new signature, and owner recommendation (4 consecutive days of episodes).
|
| 2026-10-28 |
gitbucket: corrected archive-burst section — 6th episode of class, client attributed, proxy is .183 not .226
...
Correction after cross-checking memory of prior sessions: the 28.10 episode is the 6th of the /archive/*.zip class since 12.10; client already attributed (192.168.1.150 via ovpn VM SNAT, browser UA rotation; crawler UAs in 25.10 storm); actual nginx proxy is 192.168.1.183 (inventory .226 entry stale — .226 does not answer SSH/ICMP). Removed the incorrect "client IPs not identified" statement.
|
gitbucket: known behavior — load_high episodes from archive/*.zip download bursts (2026-10-26/27/28)
...
Diagnosed by navi rei during reaction to panel event #1097 (load_high 2.1x, server 44). journalctl evidence: bursts of GET /root/<repo>/archive/<path>/<sha>.zip through git.gnexus.space, clients abort (EofException/Broken pipe), GitBucket keeps building ZIPs -> java 325% CPU, load1 9.8. Self-recovered in ~90 s, HTTP 200 throughout, no restart needed. Triage recipe and pending owner decision (nginx rate-limit on .226) recorded.
|
| 2026-10-27 |
gitbucket.md: add 2026-10-27 17:50 UTC recurrence (5th /archive/*.zip cpu_high episode, self-recovered, crawler via ovpn VM .150)
...
Recurrence 2026-10-27 17:50 UTC (5th /archive/*.zip cpu_high episode, self-recovered, crawler via ovpn VM .150) — triage by navi rei during reaction session on panel event #1091.
|
| 2026-10-26 |
jellyfin host: OOM fix applied 2026-10-08, daily OOM cycle stopped; 2x/day library validation can stall /health 30s
...
SSH-триаж инцидента 2026-10-26 23:04 UTC (service_down ReadTimeout, эпизод 30.6 с, self-recovery): причина — плановая валидация библиотеки (23:04:11) насытила CPU, /health не ответил за 30 с. Рестарта не было (NRestarts=0), kernel OOM за 10 дней — ноль (oom_kill=0). MemoryHigh=8G/MemoryMax=10G применены 2026-10-08 через systemctl set-property — ежедневный OOM-цикл ffmpeg остановлен.
|
Known issues: add 2026-10-26 recurrence of /archive crawler storm (4th episode, named bots, self-recovered); proxy address .226 → .183.
...
2026-10-26 cpu_high #1081 reaction (navi rei): 4th recurrence of the /archive/*.zip crawler-storm class, now with named bots (Reflectionbot, ClaudeBot, SemrushBot); proxy address .226 → .183 corrected in Known issues.
|
| 2026-10-25 |
Minecraft stream mapping corrected to .160 and verified live; stale .218 mapping removed
...
Live verification 2026-10-25: the stale "minecraft → tcp://192.168.1.218" mapping in this doc was wrong; the live nginx stream config points at 192.168.1.160 and works end-to-end.
|
Public access verified working; stale dead-mapping note corrected
...
Live verification 2026-10-25: public access via minecraft.gnexus.space:25565 works; previous "no public proxy mapping" note was stale.
|
Minecraft stream mapping verified active; tunnel client corrected to ovpn VM with DNAT
...
Verified live 2026-10-25 by navi rei: minecraft.gnexus.space:25565 works end-to-end via ovpn VM DNAT and nginx stream on .183; tunnel client is the ovpn VM, not .183 itself.
|
| 2026-10-21 |
internal-proxy-vps: 2026-10-21 disk_high 90.3% cleanup executed (journal cap 200M, apt cache, old kernel, old nginx .gz), df 93%→76%, services healthy
...
Reaction session rei on GHard event 1ed8c8fd (disk_high / 90.3%, server id 33 internal-proxy). SSH triage: /var 2.3G (journal 768M, nginx logs 505M), apt cache 179M, old kernel 4.15.0-213. Cleanup: journal vacuum 768M→104M, apt-get clean, purge linux-image-4.15.0-213-generic + /lib/modules, 20 old nginx .gz removed. Persistent journald cap /etc/systemd/journald.conf.d/rei-disk-cap.conf (SystemMaxUse=200M, MaxRetentionSec=14day). Result: df 93%→76% (1.7G free), inodes 21%. Services verified: nginx active HTTPS 200, navi.service health 200, navi-postgres healthy, openvpn@client masked (pre-existing).
|
| 2026-10-19 |
nextcloud: серия флапа 19 окт ЗАКРЫТА — 65 эпизодов за 2.5 ч, тишина 67 мин, причина не установлена (gntodo 506)
...
recall re-check 12:06: тишина 67.1 мин >= 1 ч, серия затухла, закрытие
|
nextcloud: серия флапа — 65 эпизодов (08:22–10:59 UTC), после 10:59 пауза 35 мин, пробы чистые
...
recall re-check 11:34: эпизоды 62–65, серия на паузе, стоп-критерий тишина >=1 ч
|
nextcloud: серия флапа публичного пути — 61 эпизод за 2.5 ч (08:22–10:53 UTC), notify warning отправлен, механизм прежний (VPS 51.83.131.93)
...
recall re-check: серия разрослась до 61 эпизода, notify warning отправлен
|
external-vps: entrypoint 51.83.131.93 (OVH) verified; s-host 194.61.53.43 unreachable; 19 Oct path degradation documented
...
external-vps.md: entrypoint 51.83.131.93 (OVH) verified via DNS ×3 and TCP probes; s-host 194.61.53.43 unreachable (16/16 FAIL); 2026-10-19 path degradation section added (SYN drops home LAN → OVH, 12/16 FAIL, clean ICMP, clean internal path 16/16). Source: rei triage of nextcloud flapping storm 19 Oct.
|
nextcloud: серия 2026-10-19 продолжена — эпизоды 15–21 (09:00–09:13 UTC), все ConnectTimeout self-recovered, механизм прежний (флап публичного пути к VPS), VM здорова
...
Реакционная сессия rei: продолжение серии 2026-10-19 — эпизоды 15–21 (09:00–09:13 UTC), механизм прежний (флап публичного пути к VPS 51.83.131.93), VM и стек здоровы (LAN-проба 3/3 200, maintenance=false, v31.0.14.1)
|
nextcloud: серия флапа публичного пути продолжилась — 14 эпизодов (08:22–09:00 UTC), механизм прежний (VPS 51.83.131.93)
...
Реакционная сессия rei на событие #934 (08:56:17 UTC): серия флапа публичного пути продолжилась после 08:36 — эпизоды 7–14 (#921–#938, 08:38–09:00 UTC), все ConnectTimeout, self-recovered, интервалы 0.6–2.6 мин (все <4 мин). Верификация 08:59–09:00 UTC: публичные пробы 6/10 таймаутов, raw TCP к 51.83.131.93:443 4/10 FAIL, контроль 1.1.1.1 5/5 OK, LAN-проба VM 3/3 200 — механизм прежний (VPS), VM и стек здоровы. Коррекция вывода «серия оборвалась к 08:36:37» из commit 79afb62. Задача gntodo 506 актуальна.
|
Уточнение: задача владельцу создана в gntodo (id 506, проект GHard Monitor)
...
Уточнение: задача владельцу создана в gntodo (id 506, проект GHard Monitor) — в тексте серии 19 окт был указан только факт создания.
|
nextcloud: incident series 2026-10-19 08:22–08:36 UTC — public path flap to external VPS, not bserv
...
Reaction session rei on Synapse event 21148f3a (service_recovered nextcloud, panel #916, 08:29:08 UTC). Series: 6 down/recovered pairs 08:22:41–08:36:37 UTC, all ConnectTimeout ~30 s, all self-recovered, intervals 1.6–2.2 min. Triage: bserv healthy (RAM 47%, swap 5%, no OOM), VM/stack healthy (LAN probes 4/4 ~0.04 s), public path to external VPS 51.83.131.93:443 flapping (9/19 curl timeouts, raw TCP 4/10 FAIL, control host 1.1.1.1 clean in all rounds). Not a recurrence of the 11–12 Oct bserv swap-pressure series. External VPS SSH triage not possible in session (no confirmed access documented) — gntodo task created for owner.
|
| 2026-10-18 |
gnexus-home: recovery of 2026-10-18 22:28 crawler-storm incident confirmed (event #907 cpu_recovered, load back to baseline)
...
Reaction session rei delivered panel event #907 (cpu_recovered 22:30:38 UTC) for the 22:28 crawler-storm incident on gnexus-home; independently verified load decay 8.1→0.0 and appended recovery note to the incident section and incident history.
|
gnexus-home: инцидент load_high 2026-10-18 (краулер-шторм через ovpn_reserv) + адрес .183 в инвентаре
...
Инцидент load_high 5.98 на gnexus-home (hard-panel events #905/#906, 2026-10-18 22:28 UTC): краулер-шторм через внутренний прокси ovpn_reserv (192.168.1.183, MAC 52:54:00:10:2d:8e). Блок-правила RedirectMatch 404 в Apache vhost (бэкап .bak-rei-20261018), шторм затих ~22:32, load 0.17 к 22:34, gnexus.space → 200. Адрес .183 задокументирован в libvirt-vms.md и инвентаре VM.
|
libvirt-vms: ovpn_reserv также держит 192.168.1.183 (MAC 52:54:00:10:2d:8e, сверено 2026-10-18)
...
Адрес 192.168.1.183 (MAC 52:54:00:10:2d:8e) принадлежит домену ovpn_reserv — сверено по virsh dumpxml на гипервизоре 2026-10-18 при расследовании краулер-шторма на gnexus-home
|
gnexus-home: инцидент load_high 5.98 2026-10-18 (краулер-шторм через ovpn_reserv) + блок-правила Apache
...
Инцидент load_high 5.98 на gnexus-home 2026-10-18 22:28 UTC: краулер-шторм через ovpn_reserv (192.168.1.183), блок-правила в Apache; верифицировано по SSH и панели (hard-panel events #905/#906)
|