diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index 8feaf13..ff62525 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -2,13 +2,13 @@ title: VM gitbucket (GitBucket + navi instance pilar) type: host ip: 192.168.1.156 -purpose: "GitBucket git hosting + navi agent instance (pilar, profile navi_code)" +purpose: "GitBucket git hosting + navi instance pilar (profile navi_code)" owner: gmikcon status: active confidence: "100" last_reviewed: 2026-10-31 review_interval: 90d -source_of_truth: ssh-host-and-owner-confirmed +source_of_truth: "ssh-host-and-owner-confirmed" --- # VM gitbucket (GitBucket + navi instance pilar) @@ -16,9 +16,9 @@ ## Overview KVM/QEMU VM on `hp-proliant-dl380-g6` (192.168.1.130). Primary role: GitBucket -git hosting (see [applications/gitbucket.md](../applications/gitbucket.md)). -Since 2026-09-19 the VM also hosts the navi-1 agent instance **pilar** -(profile `navi_code`), deployed by navi (rei) from the `deploy` branch. +git hosting (see [applications/gitbucket.md](../applications/gitbucket.md)). Since +2026-09-19 the VM also hosts the navi-1 agent instance **pilar** (profile +`navi_code`), deployed by navi (rei) from the `deploy` branch. ## Technical Details @@ -46,7 +46,7 @@ - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds) - **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances -## Known behavior: load_high episodes from archive downloads (class since 2026-10-12) +## Known behavior: load_high episodes from crawler bursts (class since 2026-10-12) Diagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100, GHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC, @@ -54,19 +54,6 @@ same class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, 26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40. -**2026-10-30 23:57 UTC (8th episode, panel #1136/#1137):** cpu_high 91% → -load_high 2.11×cores, peak load1 11.25 at 00:00:40, java up to 327% CPU; -~2600 archive/raw requests 23:50–00:04 from 192.168.1.150 via git.gnexus.space -(UA mix GPTBot/1.4 + Chrome 142–145 Win/Mac/Edge rotation — same signature as -27.10/29.10). Self-recovered: load1 1.01 by 00:06:08, zero archive requests -after 00:04. Journal: workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. -During the burst, nextcloud (separate VM, 192.168.1.152) had a 61-s -ConnectError blip (#1138, 23:58:58–23:59:59 UTC) — the **2nd such -coincidence**: an identical 61.3-s blip occurred at the 27.10 17:50 episode -start. Treat nextcloud blips during gitbucket archive episodes as expected -collateral (mechanism unclear, likely proxy/hypervisor-side contention; -nextcloud VM itself stays online), not an independent outage. - **New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl (06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests from 192.168.1.150 via the public route (journal volume per hour: 1413 lines @@ -78,7 +65,7 @@ Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, all 8 panel services up. -**Cause (confirmed across all 8 episodes):** bursts of +**Cause (confirmed across episodes 1–7):** bursts of `GET /root//archive//.zip` (and a few `/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI "download directory as ZIP" endpoint. Clients abort the download mid-stream (`EofException` / @@ -87,32 +74,93 @@ concurrent archive builds saturate all 4 vCPUs. Baseline between episodes: CPU 10–27%, load1 0.2–1.5. +**8th episode — 2026-10-30 23:58 UTC (pattern switch to issues/pulls enumeration):** +cpu_high 90.5% #1136 (23:57:40) → load_high 2.11×cores #1137 (23:58:40), peak +load1 8.54 at 00:01:10 UTC, decay to 0.89 by 00:07:30 (SSH-verified read-only, +ubuntu@192.168.1.156). NEW request pattern: NOT `/archive/*.zip` but +combinatorial enumeration `GET /root//issues?...` and `/pulls?...` with +filter permutations (labels/author/assigned/milestone/priority/state) across +repos root/yummspace, root/wwwcats etc., plus a few subdirectory archive-GETs +(`archive/webclient/.zip`). Volume: 11 611 requests to git.gnexus.space +in 10 min (23:5x, nginx access.log on 192.168.1.183) + 2682/min at 00:00. +Client: all remote_addr 192.168.1.150 (SNAT ovpn VM = external traffic via public +tunnel); UA browser rotation Chrome/Edge 142–145 Win/Mac (same masking as +29.10; Reflectionbot absent). gitbucket journal 23:50–00:10: 21 Request(GET) +lines, 102 error lines (EofException / Broken pipe, +RepositoryViewerController). Coincident: nextcloud ConnectError flap +23:58:58→23:59:23 (25.1 s) — 4th flap×burst coincidence (25.10 01:54, +26.10 17:44, 27.10 17:50, 30.10 23:58); counter-examples persist (28.10 and +29.10 bursts without flap; 27.10 02:52 flap without burst). Details: +workspace/incident_nextcloud_2026-10-30_connecterror.md (navi rei). +**CORRECTION (31.10):** ep8 did NOT self-extinguish ~00:09 — after the 00:01 +load peak the enumeration merely dipped below the CPU threshold (2682/min → +~440–480/min) and CONTINUED for the next ~3 h (hourly totals 00:xx 29 443, +01:xx 29 620, 02:xx 29 097, 03:xx 38 666 — nginx access.log on 192.168.1.183). + +**9th episode — 2026-10-31 03:30 UTC (CPU peak of the SAME continuous crawl):** +cpu_high 97.1% #1152 (03:30:51) → cpu_recovered 79.4% #1153 (03:31:51); panel +snapshot: java 333.7% CPU (≈3 concurrent ZIP builds on 4 vCPU), load +5.6/3.19/1.9, peak load1 7.05 at 03:34 then decaying. Trigger (gitbucket +journal, SSH-verified read-only): 4× `GET +/root/smart-home-server/archive/devices/.zip` at 03:30:28–03:30:54 +(3× sha 44250a7a…, 1× 8a945d0d…), 8× EofException/Broken pipe in the +03:15–03:45 window; NO restart (java etime 13d16h). Ep8+ep9 = ONE continuous +crawl ≥3.5 h (23:5x → ≥03:35 UTC), ~126.8k requests total — largest of the +series (~24× the 29.10 crawl). Rate: ~440–480/min 00:00–02:59, ~570/min from +03:00, ~2500/min 03:27–03:34 (escalation coinciding with the archive/devices +sub-crawl; still ~587 hits in the first 15 s of 03:35 — active at last +check). Client: 100% 192.168.1.150 (SNAT ovpn). UA: GPTBot/1.4 explicitly +declared (4136 hits in 03:xx, Mozilla-wrapped) + browser rotation Chrome/Edge +142–145 Win/Mac (~30k) — same masking as before, now with an honest bot UA in +the mix. Paths: continuous issues/pulls enumeration over all /root/* repos +(top: flow-task/pulls 1831, gnexus-creds/pulls 954, gnexus-auth-client-py/issues +931, music-loader/*, voice/issues, vue-indexer/issues…) + 4 archive/devices/*.zip +on smart-home-server (the ESP-credentials repo — note it contains device +config paths). Plain enumeration stays under the CPU threshold; CPU crosses +90% only when parallel archive-ZIP builds are added. Details: +workspace/incident_gitbucket_2026-10-31_ep9_crawl.md (navi rei). + **Traffic source (attributed in earlier sessions):** requests arrive via the nginx reverse-proxy VM at **192.168.1.183** (note: inventory entry "internal-proxy-vps .226" is stale — .226 does not answer SSH/ICMP; actual proxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM, -SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10); -crawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, -MJ12bot observed in the 25.10 storm. robots.txt is absent (GitBucket returns -an HTML error page for it). +SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10–31.10); +crawler +UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot +observed in the 25.10 storm; GPTBot/1.4 again declared on 31.10. robots.txt is +absent (GitBucket returns an HTML error page for it). **Triage recipe (next episode):** `journalctl -u gitbucket --since '' | -grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this -pattern; no restart needed, load decays within ~2 min after the burst ends -(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx -access.log on 192.168.1.183 (creds internal-proxy-vps-creds / -bserv@192.168.1.130). +grep -o 'Request(GET [^)]*)'` — if archive/*.zip OR issues/pulls-enumeration +bursts are present, it is this pattern (pattern switch observed 30.10); +no restart needed, load decays within ~2 min after the burst ends +(29.10: within ~7 min after the 4.5 h crawl ended; 30.10: 8.54 → 0.89 in ~6 min; +31.10: load1 7.05 → 4.82 within a minute after the archive sub-burst). +For attribution: nginx access.log on 192.168.1.183 (creds internal-proxy-vps-creds / +bserv@192.168.1.130). CAUTION: a sub-threshold crawl (~450/min) produces NO +panel events — to see whether a crawl is still active, check nginx access.log +rate on the proxy, not just gitbucket journal/panel. -**Not applied (owner decision pending, 8th episode):** rate-limiting or +**Not applied (owner decision pending, 9th episode):** rate-limiting or blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), -or a GitBucket plugin/config change. Escalation criterion "bursts ≥2/day" is -not formally exceeded (1 burst/day), but episodes now recur **5 consecutive -days** (26–30 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner -decide on proxy-level rate-limiting/robots.txt. Related journals: +or a GitBucket plugin/config change. Update (31.10, 9th episode): the crawler +switched from `/archive/*.zip` to combinatorial `/issues?`/`/pulls?` +enumeration AND runs as a continuous multi-hour crawl (~127k req in the +23:58–03:35 night, 29k/h for 3+ h), with GPTBot/1.4 declared — a +`/archive/`-only block is no longer sufficient. Recommended fix: **path-agnostic +per-client rate-limit on vhost git.gnexus.space** at the nginx proxy +(192.168.1.183), plus robots.txt and `/archive/` blocking for non-browser +UAs as additional measures. Blocking/rate-limiting now affects the owner's +own tooling (navigators use git.gnexus.space): apply a generous limit +(e.g. 30–60 req/min per client) rather than a hard block, or whitelist +internal IPs/UA tokens. Episodes recurred **5 consecutive days** (26–30 Oct) +and ep8–9 merged into one ≥3.5 h continuous crawl. Still awaiting owner +decision (9th episode in 19 days). Related journals: workspace/incident_gitbucket_cpu_2026-10-12.md, workspace/incident_gitbucket_2026-10-25_load_high.md, workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md, -workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. +workspace/incident_nextcloud_2026-10-30_connecterror.md, +workspace/incident_gitbucket_2026-10-31_ep9_crawl.md. ## Access