diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index f63effc..c992f37 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -6,7 +6,7 @@ owner: gmikcon status: active confidence: 100 -last_reviewed: 2026-09-19 +last_reviewed: 2026-10-28 review_interval: 90d source_of_truth: ssh-host-and-owner-confirmed --- @@ -46,6 +46,34 @@ - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds) - **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances +## Known behavior: load_high episodes from archive downloads (2026-10-26/27/28) + +Diagnosed 2026-10-28 21:33–21:37 UTC by navi rei (panel events #1083–#1097, +GHard server id 44). Three near-identical episodes in three days: + +- 2026-10-26 17:45 UTC (load1 2.35×), 2026-10-27 17:50 UTC (2.13×), 2026-10-28 21:33 UTC (2.1×, peak load1 9.8, java 325% CPU) +- Each episode: 60–90 s, self-recovered, no service restart, GitBucket HTTP 200 throughout. + +**Cause:** bursts of `GET /root//archive//.zip` (and a few +`/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI +"download directory as ZIP" endpoint. Repos seen: smart-home-server, navi-1, +rpg, gnexus-ui-kit, vmk-ui-kit. Clients abort the download mid-stream +(`EofException` / `Broken pipe` in journalctl, logged as +`RepositoryViewerController - Catch unhandled error`), but GitBucket keeps +building the ZIP server-side — several concurrent archive builds saturate all +4 vCPUs. Baseline between episodes: CPU 10–27%, load1 0.2–1.5. + +**Triage recipe (next episode):** `journalctl -u gitbucket --since '' | +grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this +pattern; no restart needed, load decays within ~2 min after the burst ends. + +**Not applied (owner decision pending):** rate-limiting or blocking +`/archive/` at the nginx proxy (192.168.1.226), or a GitBucket plugin/config +change. Client IPs were not identified — SSH/ICMP to the proxy are filtered +and nginx access logs were not reachable during the incident; check +`/var/log/nginx/access.log` on .226 for `archive/` requests if attribution is +needed. + ## Access - SSH: `ubuntu@192.168.1.156`, credentials in gnexus-creds secret `bserv@192.168.1.130`; root SSH denied