diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index c992f37..749041b 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -46,13 +46,14 @@ - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds) - **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances -## Known behavior: load_high episodes from archive downloads (2026-10-26/27/28) +## Known behavior: load_high episodes from archive downloads (class since 2026-10-12) Diagnosed 2026-10-28 21:33–21:37 UTC by navi rei (panel events #1083–#1097, -GHard server id 44). Three near-identical episodes in three days: - -- 2026-10-26 17:45 UTC (load1 2.35×), 2026-10-27 17:50 UTC (2.13×), 2026-10-28 21:33 UTC (2.1×, peak load1 9.8, java 325% CPU) -- Each episode: 60–90 s, self-recovered, no service restart, GitBucket HTTP 200 throughout. +GHard server id 44). The 2026-10-28 episode (21:33 UTC, load1 2.1×cores, peak +load1 9.8, java 325% CPU) is the **6th episode of the same class** since +2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, 26.10 17:44, +27.10 17:50, 28.10 21:33. Recent episodes: 60–90 s each, self-recovered, no +service restart, GitBucket HTTP 200 throughout. **Cause:** bursts of `GET /root//archive//.zip` (and a few `/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI @@ -63,16 +64,28 @@ building the ZIP server-side — several concurrent archive builds saturate all 4 vCPUs. Baseline between episodes: CPU 10–27%, load1 0.2–1.5. +**Traffic source (attributed in earlier sessions):** requests arrive via the +nginx reverse-proxy VM at **192.168.1.183** (note: inventory entry +"internal-proxy-vps .226" is stale — .226 does not answer SSH/ICMP; actual +proxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM, +SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10); crawler UAs +GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot +observed in the 25.10 storm. robots.txt is absent (GitBucket returns an HTML +error page for it). + **Triage recipe (next episode):** `journalctl -u gitbucket --since '' | grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this pattern; no restart needed, load decays within ~2 min after the burst ends. +For attribution: nginx access.log on 192.168.1.183 (creds +internal-proxy-vps-creds / bserv@192.168.1.130). -**Not applied (owner decision pending):** rate-limiting or blocking -`/archive/` at the nginx proxy (192.168.1.226), or a GitBucket plugin/config -change. Client IPs were not identified — SSH/ICMP to the proxy are filtered -and nginx access logs were not reachable during the incident; check -`/var/log/nginx/access.log` on .226 for `archive/` requests if attribution is -needed. +**Not applied (owner decision pending, 6th episode):** rate-limiting or +blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), +or a GitBucket plugin/config change. Escalation criterion: bursts ≥2/day or +non-decaying load. Related journals: +workspace/incident_gitbucket_cpu_2026-10-12.md, +workspace/incident_gitbucket_2026-10-25_load_high.md, +workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md. ## Access