diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index 749041b..9528765 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -2,11 +2,11 @@ title: VM gitbucket (GitBucket + navi instance pilar) type: host ip: 192.168.1.156 -purpose: GitBucket git hosting + navi agent instance (pilar, profile navi_code) +purpose: "GitBucket git hosting + navi agent instance (pilar, profile navi_code)" owner: gmikcon status: active -confidence: 100 -last_reviewed: 2026-10-28 +confidence: "100" +last_reviewed: 2026-10-29 review_interval: 90d source_of_truth: ssh-host-and-owner-confirmed --- @@ -48,41 +48,54 @@ ## Known behavior: load_high episodes from archive downloads (class since 2026-10-12) -Diagnosed 2026-10-28 21:33–21:37 UTC by navi rei (panel events #1083–#1097, -GHard server id 44). The 2026-10-28 episode (21:33 UTC, load1 2.1×cores, peak -load1 9.8, java 325% CPU) is the **6th episode of the same class** since -2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, 26.10 17:44, -27.10 17:50, 28.10 21:33. Recent episodes: 60–90 s each, self-recovered, no -service restart, GitBucket HTTP 200 throughout. +Diagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100, +GHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC, +peak load1 5.07 at 10:41:27, java 181.6% CPU) is the **7th episode of the +same class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, +26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40. -**Cause:** bursts of `GET /root//archive//.zip` (and a few -`/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI -"download directory as ZIP" endpoint. Repos seen: smart-home-server, navi-1, -rpg, gnexus-ui-kit, vmk-ui-kit. Clients abort the download mid-stream -(`EofException` / `Broken pipe` in journalctl, logged as -`RepositoryViewerController - Catch unhandled error`), but GitBucket keeps -building the ZIP server-side — several concurrent archive builds saturate all -4 vCPUs. Baseline between episodes: CPU 10–27%, load1 0.2–1.5. +**New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl +(06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests +from 192.168.1.150 via the public route (journal volume per hour: 1413 lines +06:xx, 2891 08:xx, 1447 10:xx; 29 ERROR — Broken pipe / EofException / +NPE in `RepositoryViewerController.archiveRepository`). Repos: vmk-demo-bot +(incl. `node_modules`), navi-1 (incl. `webclient/dist`, `vendor/gnexus-ui-kit`), +smart-home-server, vue-indexer. Load decayed 5.07 → 0.66 by 10:48 UTC after +the crawl stopped (last archive request 10:42:10, zero since 10:44). +Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, +all 8 panel services up. + +**Cause (confirmed across all 7 episodes):** bursts of +`GET /root//archive//.zip` (and a few `/raw/...` file +downloads) through git.gnexus.space — GitBucket's web-UI "download directory +as ZIP" endpoint. Clients abort the download mid-stream (`EofException` / +`Broken pipe` in journalctl, logged as `RepositoryViewerController - Catch +unhandled error`), but GitBucket keeps building the ZIP server-side — several +concurrent archive builds saturate all 4 vCPUs. Baseline between episodes: +CPU 10–27%, load1 0.2–1.5. **Traffic source (attributed in earlier sessions):** requests arrive via the nginx reverse-proxy VM at **192.168.1.183** (note: inventory entry "internal-proxy-vps .226" is stale — .226 does not answer SSH/ICMP; actual proxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM, -SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10); crawler UAs -GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot +SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10); crawler +UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot observed in the 25.10 storm. robots.txt is absent (GitBucket returns an HTML error page for it). **Triage recipe (next episode):** `journalctl -u gitbucket --since '' | grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this -pattern; no restart needed, load decays within ~2 min after the burst ends. -For attribution: nginx access.log on 192.168.1.183 (creds -internal-proxy-vps-creds / bserv@192.168.1.130). +pattern; no restart needed, load decays within ~2 min after the burst ends +(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx +access.log on 192.168.1.183 (creds internal-proxy-vps-creds / +bserv@192.168.1.130). -**Not applied (owner decision pending, 6th episode):** rate-limiting or +**Not applied (owner decision pending, 7th episode):** rate-limiting or blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), -or a GitBucket plugin/config change. Escalation criterion: bursts ≥2/day or -non-decaying load. Related journals: +or a GitBucket plugin/config change. Escalation criterion "bursts ≥2/day" is +not formally exceeded (1 burst/day), but episodes now recur **4 consecutive +days** (26–29 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner +decide on proxy-level rate-limiting/robots.txt. Related journals: workspace/incident_gitbucket_cpu_2026-10-12.md, workspace/incident_gitbucket_2026-10-25_load_high.md, workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md.