| 2026-10-31 |
gitbucket host doc: 8th archive-burst episode 2026-10-30 23:57 UTC (short burst, self-recovered ~7 min), GPTBot/1.4 crawl attribution, pending owner decision now 5 consecutive days
...
Episode 8 of the archive-burst class confirmed 2026-10-30 23:57 UTC (panel event #1136, diagnosed by navi rei via SSH + nginx logs): short burst of 20 archive/*.zip requests amid GPTBot crawl, peak load1 11.25, self-recovered ~7 min. Updates episode count, traffic-source note and pending owner decision (5 consecutive days).
|
gitbucket: 8th archive-crawler CPU episode (30.10 23:57 UTC), series 5 consecutive days
...
8th episode of the archive-crawler CPU class: 2026-10-30 23:57 UTC burst (cpu_high 91% #1136, load_high 2.11×cores #1137, cpu_recovered #1140), confirmed via read-only SSH journalctl on .156 — same /archive/*.zip signature, self-recovered in ~3.5 min. Series now 5 consecutive days (26–30 Oct).
|
| 2026-10-30 |
Document hpsa controller reset failure mode (DL380 G6) with 2026-10-30 incident
...
Reaction session: load_high on files (#36) and 5 other VMs traced to hpsa controller logical-volume resets on bserv (DL380 G6). Incident 2026-10-30 13:06-13:15 UTC, self-recovered. Documented signature, timeline, triage playbook.
|
| 2026-10-29 |
gitbucket: 7th archive-crawl episode (29.10, ~4.5 h sustained, self-recovered)
...
Reaction session rei on GHard event #1100 (cpu_high 90.5%, server 44 gitbucket, 2026-10-29 10:40:57 UTC). SSH triage confirmed the known archive-crawl class (7th episode since 2026-10-12) with a new signature: sustained ~4.5 h crawl 06:00–10:42 UTC, 5248 /archive/*.zip requests from 192.168.1.150 via public route, 29 Broken pipe/EofException errors, load decayed 5.07→0.66 by 10:48 UTC, self-recovered, no restart needed. Known behavior section updated with 7th episode, new signature, and owner recommendation (4 consecutive days of episodes).
|
| 2026-10-28 |
gitbucket: corrected archive-burst section — 6th episode of class, client attributed, proxy is .183 not .226
...
Correction after cross-checking memory of prior sessions: the 28.10 episode is the 6th of the /archive/*.zip class since 12.10; client already attributed (192.168.1.150 via ovpn VM SNAT, browser UA rotation; crawler UAs in 25.10 storm); actual nginx proxy is 192.168.1.183 (inventory .226 entry stale — .226 does not answer SSH/ICMP). Removed the incorrect "client IPs not identified" statement.
|
gitbucket: known behavior — load_high episodes from archive/*.zip download bursts (2026-10-26/27/28)
...
Diagnosed by navi rei during reaction to panel event #1097 (load_high 2.1x, server 44). journalctl evidence: bursts of GET /root/<repo>/archive/<path>/<sha>.zip through git.gnexus.space, clients abort (EofException/Broken pipe), GitBucket keeps building ZIPs -> java 325% CPU, load1 9.8. Self-recovered in ~90 s, HTTP 200 throughout, no restart needed. Triage recipe and pending owner decision (nginx rate-limit on .226) recorded.
|
| 2026-10-27 |
gitbucket.md: add 2026-10-27 17:50 UTC recurrence (5th /archive/*.zip cpu_high episode, self-recovered, crawler via ovpn VM .150)
...
Recurrence 2026-10-27 17:50 UTC (5th /archive/*.zip cpu_high episode, self-recovered, crawler via ovpn VM .150) — triage by navi rei during reaction session on panel event #1091.
|
| 2026-10-26 |
Known issues: add 2026-10-26 recurrence of /archive crawler storm (4th episode, named bots, self-recovered); proxy address .226 → .183.
...
2026-10-26 cpu_high #1081 reaction (navi rei): 4th recurrence of the /archive/*.zip crawler-storm class, now with named bots (Reflectionbot, ClaudeBot, SemrushBot); proxy address .226 → .183 corrected in Known issues.
|
| 2026-10-25 |
Minecraft stream mapping corrected to .160 and verified live; stale .218 mapping removed
...
Live verification 2026-10-25: the stale "minecraft → tcp://192.168.1.218" mapping in this doc was wrong; the live nginx stream config points at 192.168.1.160 and works end-to-end.
|
Public access verified working; stale dead-mapping note corrected
...
Live verification 2026-10-25: public access via minecraft.gnexus.space:25565 works; previous "no public proxy mapping" note was stale.
|
Minecraft stream mapping verified active; tunnel client corrected to ovpn VM with DNAT
...
Verified live 2026-10-25 by navi rei: minecraft.gnexus.space:25565 works end-to-end via ovpn VM DNAT and nginx stream on .183; tunnel client is the ovpn VM, not .183 itself.
|
| 2026-10-21 |
internal-proxy-vps: 2026-10-21 disk_high 90.3% cleanup executed (journal cap 200M, apt cache, old kernel, old nginx .gz), df 93%→76%, services healthy
...
Reaction session rei on GHard event 1ed8c8fd (disk_high / 90.3%, server id 33 internal-proxy). SSH triage: /var 2.3G (journal 768M, nginx logs 505M), apt cache 179M, old kernel 4.15.0-213. Cleanup: journal vacuum 768M→104M, apt-get clean, purge linux-image-4.15.0-213-generic + /lib/modules, 20 old nginx .gz removed. Persistent journald cap /etc/systemd/journald.conf.d/rei-disk-cap.conf (SystemMaxUse=200M, MaxRetentionSec=14day). Result: df 93%→76% (1.7G free), inodes 21%. Services verified: nginx active HTTPS 200, navi.service health 200, navi-postgres healthy, openvpn@client masked (pre-existing).
|
| 2026-10-19 |
nextcloud: серия флапа 19 окт ЗАКРЫТА — 65 эпизодов за 2.5 ч, тишина 67 мин, причина не установлена (gntodo 506)
...
recall re-check 12:06: тишина 67.1 мин >= 1 ч, серия затухла, закрытие
|
nextcloud: серия флапа — 65 эпизодов (08:22–10:59 UTC), после 10:59 пауза 35 мин, пробы чистые
...
recall re-check 11:34: эпизоды 62–65, серия на паузе, стоп-критерий тишина >=1 ч
|
nextcloud: серия флапа публичного пути — 61 эпизод за 2.5 ч (08:22–10:53 UTC), notify warning отправлен, механизм прежний (VPS 51.83.131.93)
...
recall re-check: серия разрослась до 61 эпизода, notify warning отправлен
|
external-vps: entrypoint 51.83.131.93 (OVH) verified; s-host 194.61.53.43 unreachable; 19 Oct path degradation documented
...
external-vps.md: entrypoint 51.83.131.93 (OVH) verified via DNS ×3 and TCP probes; s-host 194.61.53.43 unreachable (16/16 FAIL); 2026-10-19 path degradation section added (SYN drops home LAN → OVH, 12/16 FAIL, clean ICMP, clean internal path 16/16). Source: rei triage of nextcloud flapping storm 19 Oct.
|
nextcloud: серия 2026-10-19 продолжена — эпизоды 15–21 (09:00–09:13 UTC), все ConnectTimeout self-recovered, механизм прежний (флап публичного пути к VPS), VM здорова
...
Реакционная сессия rei: продолжение серии 2026-10-19 — эпизоды 15–21 (09:00–09:13 UTC), механизм прежний (флап публичного пути к VPS 51.83.131.93), VM и стек здоровы (LAN-проба 3/3 200, maintenance=false, v31.0.14.1)
|
nextcloud: серия флапа публичного пути продолжилась — 14 эпизодов (08:22–09:00 UTC), механизм прежний (VPS 51.83.131.93)
...
Реакционная сессия rei на событие #934 (08:56:17 UTC): серия флапа публичного пути продолжилась после 08:36 — эпизоды 7–14 (#921–#938, 08:38–09:00 UTC), все ConnectTimeout, self-recovered, интервалы 0.6–2.6 мин (все <4 мин). Верификация 08:59–09:00 UTC: публичные пробы 6/10 таймаутов, raw TCP к 51.83.131.93:443 4/10 FAIL, контроль 1.1.1.1 5/5 OK, LAN-проба VM 3/3 200 — механизм прежний (VPS), VM и стек здоровы. Коррекция вывода «серия оборвалась к 08:36:37» из commit 79afb62. Задача gntodo 506 актуальна.
|
Уточнение: задача владельцу создана в gntodo (id 506, проект GHard Monitor)
...
Уточнение: задача владельцу создана в gntodo (id 506, проект GHard Monitor) — в тексте серии 19 окт был указан только факт создания.
|
nextcloud: incident series 2026-10-19 08:22–08:36 UTC — public path flap to external VPS, not bserv
...
Reaction session rei on Synapse event 21148f3a (service_recovered nextcloud, panel #916, 08:29:08 UTC). Series: 6 down/recovered pairs 08:22:41–08:36:37 UTC, all ConnectTimeout ~30 s, all self-recovered, intervals 1.6–2.2 min. Triage: bserv healthy (RAM 47%, swap 5%, no OOM), VM/stack healthy (LAN probes 4/4 ~0.04 s), public path to external VPS 51.83.131.93:443 flapping (9/19 curl timeouts, raw TCP 4/10 FAIL, control host 1.1.1.1 clean in all rounds). Not a recurrence of the 11–12 Oct bserv swap-pressure series. External VPS SSH triage not possible in session (no confirmed access documented) — gntodo task created for owner.
|
| 2026-10-18 |
gnexus-home: recovery of 2026-10-18 22:28 crawler-storm incident confirmed (event #907 cpu_recovered, load back to baseline)
...
Reaction session rei delivered panel event #907 (cpu_recovered 22:30:38 UTC) for the 22:28 crawler-storm incident on gnexus-home; independently verified load decay 8.1→0.0 and appended recovery note to the incident section and incident history.
|
libvirt-vms: ovpn_reserv также держит 192.168.1.183 (MAC 52:54:00:10:2d:8e, сверено 2026-10-18)
...
Адрес 192.168.1.183 (MAC 52:54:00:10:2d:8e) принадлежит домену ovpn_reserv — сверено по virsh dumpxml на гипервизоре 2026-10-18 при расследовании краулер-шторма на gnexus-home
|
gnexus-home: инцидент load_high 5.98 2026-10-18 (краулер-шторм через ovpn_reserv) + блок-правила Apache
...
Инцидент load_high 5.98 на gnexus-home 2026-10-18 22:28 UTC: краулер-шторм через ovpn_reserv (192.168.1.183), блок-правила в Apache; верифицировано по SSH и панели (hard-panel events #905/#906)
|
gnexus-home: инцидент cpu_high 97.3% 18 Oct (скан-шторм /blog/.git через internal-proxy) + митигация RedirectMatch 404 в Apache
...
Инцидент cpu_high 97.3% (event #905, 2026-10-18 22:28 UTC): скан-шторм /blog/.git через internal-proxy на WordPress; применена митигация RedirectMatch 404 в 000-default.conf (бэкап .bak-rei-20261018), верифицировано по SSH и панели (CPU 97.3→1.9%).
|
| 2026-10-17 |
gnexus-home: док хоста + инвентарь (ram_high 2026-10-17, тюнинг RAM/swap/mysql/apache)
...
Reaction session rei на hard-panel event #903 (ram_high 90.81%, server 42 gnexus-home, 2026-10-17 18:37 UTC). Диагноз: baseline 76% (mysqld 373MB + apache prefork) + transient plocate-updatedb/check-new-release; OOM нет. Меры: swap 1G на /home (pri=1), mysql buffer_pool 64M + performance_schema OFF, apache prefork 2/2/4/20, диск / 96→86%. RAM теперь 47.3%, gnexus.space 200. Отчёт: workspace/incident_gnexus-home_ram_2026-10-17.md (на хосте navi).
|
internal-proxy-vps: LAN .183 закреплён netplan-статикой в госте (01-netcfg.yaml), DHCP-резервация pfSense не нужна по дизайну владельца
...
Владелец подтвердил дизайн: pfSense не выдаёт адреса из серверного диапазона, статика на стороне серверов. Проверено по SSH: netplan 01-netcfg.yaml на .183 уже статический (настроен утром), cloud-init в VM отсутствует; DHCP-резервация Kea не нужна.
|
tgclientmcp: blackout 2026-10-17 note; netplan static .171 restored (50-static.yaml, cloud-init disabled); TGCLIENT_SYNAPSE_URL rolled back to .172:8013; nginx upstream back to .171:8710
...
После блэкаута 2026-10-16/17 VM получила DHCP-адрес .24; по решению владельца возвращена на .171 через netplan-статику внутри гостя (cloud-init network config отключён), TGCLIENT_SYNAPSE_URL откатан на .172:8013, nginx-апстрим на .171:8710 — health 200, 2/2 аккаунта connected
|
gnsynapse: blackout 2026-10-17 note; netplan static .172 restored (50-static.yaml, cloud-init disabled); dependent configs rolled back to .172:8013
...
После блэкаута 2026-10-16/17 VM получила DHCP-адрес .29; по решению владельца возвращена на .172 через netplan-статику внутри гостя (cloud-init network config отключён), зависимые конфиги откатаны на .172:8013 — проверено health 200
|
LAN address .226 → .183 after 2026-10-16/17 blackout; added blackout recovery section; nginx upstreams for synapse/tgclientmcp restored to .172/.171
...
После блэкаута 2026-10-16/17 VM ovpn_reserv получила DHCP-адрес 192.168.1.183 вместо .226 (проверено SSH: enp1s0=192.168.1.183/24, enp7s0=192.168.105.181/24); зависимые конфиги обновлены, gnsynapse/tgclient_mcp возвращены на .172/.171 по просьбе владельца
|
| 2026-10-16 |
internal-proxy: добавлен второй эпизод disk_high 2026-10-16 (17:49→18:31 UTC, +1.07 GB temp nginx, самоосвобождение) — 3-й эпизод класса proxy_buffering; last_reviewed 2026-10-16
...
Реакция на GHard event f18217dc (disk_recovered #852, 18:31 UTC): второй эпизод disk_high за вечер на internal-proxy, идентичная сигнатура буферизации nginx (jellyfin vhost без proxy_buffering off) — класс инцидента подтверждён, документирую для истории триажа.
|