Newer
Older
gnexus-book / 10-systems / hosts / gitbucket.md

title: VM gitbucket (GitBucket + navi instance pilar) type: host ip: 192.168.1.156 purpose: "GitBucket git hosting + navi instance pilar (profile navi_code)" owner: gmikcon status: active confidence: "100" last_reviewed: 2026-10-31 review_interval: 90d

source_of_truth: "ssh-host-and-owner-confirmed"

VM gitbucket (GitBucket + navi instance pilar)

Overview

KVM/QEMU VM on hp-proliant-dl380-g6 (192.168.1.130). Primary role: GitBucket git hosting (see applications/gitbucket.md). Since 2026-09-19 the VM also hosts the navi-1 agent instance pilar (profile navi_code), deployed by navi (rei) from the deploy branch.

Technical Details

  • IP: 192.168.1.156
  • OS: Ubuntu 18.04.6 LTS (Bionic Beaver), kernel 4.15.0-213-generic
  • Users: gitbucket (service owner, home = /home/ubuntu), navi (navi instance, home = /home/navi), ubuntu (admin; SSH/sudo credentials — see gnexus-creds secret bserv@192.168.1.130)
  • Resources: 4 vCPU, 8 GiB RAM, 100G raw disk /mnt/main/vm_main/gitbucket (p1 root 28G — 21% used; p5 data 71G — 6% used; observed 2026-09-19)
  • Hypervisor storage: /mnt/main 2.5T, 730G free (69%)
  • Instance name: pilar
  • Instance ID: c6962c3e-… (full value in /home/navi/navi-1/instance.json)
  • Repo: /home/navi/navi-1 (branch deploy, commit e276b1e)
  • Service: navi.service (systemd, enabled, Restart=always)
  • Server port: 8099 (0.0.0.0), health endpoint http://127.0.0.1:8099/health
  • Postgres: docker container navi-postgres (pgvector/pgvector:pg16) on 127.0.0.1:5433 — port shifted from the default 5432 because GitBucket's system PostgreSQL 10 occupies 5432; DATABASE_URL in .env points to 5433
  • Python: standalone CPython 3.12.14 at /home/navi/navi-1/.python/ (system python 3.6 is too old)
  • Swarm: .swarm-key copied from the hive host (rei, 192.168.1.168); HIVE_URL=http://192.168.1.168:8087; hive.announce_ok confirmed in journalctl; visible in peer list as pilar — 192.168.1.156:8099 [online]
  • Ollama backend: OLLAMA_HOST=https://ollama.com, model glm-5.3-flash:cloud; Ollama Cloud credential configured in .env (value stored in gnexus-creds secret "Ollama Cloud API key (navi swarm)")
  • Webclient/auth: NAVI_WEBCLIENT_ENABLED=false, NAVI_AUTH_ENABLED=false (default deploy shape)
  • Deploy notes:
    • docker-ce 24.0.2 + docker-compose-plugin v2.18.1 installed from download.docker.com (bionic repo); git/curl installed via apt
    • asyncpg 0.29.0 has no wheel for glibc 2.27 and clang is absent — built from source with CC=gcc CXX=g++ before re-running install.sh (script is idempotent)
    • user navi has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the navi user password matches the hypervisor bserv credential (gnexus-creds)
  • Known non-critical: /health embed.ok=false (embed backend 401) — same as other swarm instances

Known behavior: load_high episodes from crawler bursts (class since 2026-10-12)

Diagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100, GHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC, peak load1 5.07 at 10:41:27, java 181.6% CPU) is the 7th episode of the same class since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, 26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40.

New signature on 29.10: not a 60–90 s burst but a sustained ~4.5 h crawl (06:00–10:42 UTC): 5248 GET /root/<repo>/archive/<path>/<sha>.zip requests from 192.168.1.150 via the public route (journal volume per hour: 1413 lines 06:xx, 2891 08:xx, 1447 10:xx; 29 ERROR — Broken pipe / EofException / NPE in RepositoryViewerController.archiveRepository). Repos: vmk-demo-bot (incl. node_modules), navi-1 (incl. webclient/dist, vendor/gnexus-ui-kit), smart-home-server, vue-indexer. Load decayed 5.07 → 0.66 by 10:48 UTC after the crawl stopped (last archive request 10:42:10, zero since 10:44). Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, all 8 panel services up.

Cause (confirmed across episodes 1–7): bursts of GET /root/<repo>/archive/<path>/<sha>.zip (and a few /raw/... file downloads) through git.gnexus.space — GitBucket's web-UI "download directory as ZIP" endpoint. Clients abort the download mid-stream (EofException / Broken pipe in journalctl, logged as RepositoryViewerController - Catch unhandled error), but GitBucket keeps building the ZIP server-side — several concurrent archive builds saturate all 4 vCPUs. Baseline between episodes: CPU 10–27%, load1 0.2–1.5.

8th episode — 2026-10-30 23:58 UTC (pattern switch to issues/pulls enumeration): cpu_high 90.5% #1136 (23:57:40) → load_high 2.11×cores #1137 (23:58:40), peak load1 8.54 at 00:01:10 UTC, decay to 0.89 by 00:07:30 (SSH-verified read-only, ubuntu@192.168.1.156). NEW request pattern: NOT /archive/*.zip but combinatorial enumeration GET /root/<repo>/issues?... and /pulls?... with filter permutations (labels/author/assigned/milestone/priority/state) across repos root/yummspace, root/wwwcats etc., plus a few subdirectory archive-GETs (archive/webclient/<sha>.zip). Volume: 11 611 requests to git.gnexus.space in 10 min (23:5x, nginx access.log on 192.168.1.183) + 2682/min at 00:00. Client: all remote_addr 192.168.1.150 (SNAT ovpn VM = external traffic via public tunnel); UA browser rotation Chrome/Edge 142–145 Win/Mac (same masking as 29.10; Reflectionbot absent). gitbucket journal 23:50–00:10: 21 Request(GET) lines, 102 error lines (EofException / Broken pipe, RepositoryViewerController). Coincident: nextcloud ConnectError flap 23:58:58→23:59:23 (25.1 s) — 4th flap×burst coincidence (25.10 01:54, 26.10 17:44, 27.10 17:50, 30.10 23:58); counter-examples persist (28.10 and 29.10 bursts without flap; 27.10 02:52 flap without burst). Details: workspace/incident_nextcloud_2026-10-30_connecterror.md (navi rei). CORRECTION (31.10): ep8 did NOT self-extinguish ~00:09 — after the 00:01 load peak the enumeration merely dipped below the CPU threshold (2682/min → ~440–480/min) and CONTINUED for the next ~3 h (hourly totals 00:xx 29 443, 01:xx 29 620, 02:xx 29 097, 03:xx 38 666 — nginx access.log on 192.168.1.183).

9th episode — 2026-10-31 03:30 UTC (CPU peak of the SAME continuous crawl): cpu_high 97.1% #1152 (03:30:51) → cpu_recovered 79.4% #1153 (03:31:51); panel snapshot: java 333.7% CPU (≈3 concurrent ZIP builds on 4 vCPU), load 5.6/3.19/1.9, peak load1 7.05 at 03:34 then decaying. Trigger (gitbucket journal, SSH-verified read-only): 4× GET /root/smart-home-server/archive/devices/<sha>.zip at 03:30:28–03:30:54 (3× sha 44250a7a…, 1× 8a945d0d…), 8× EofException/Broken pipe in the 03:15–03:45 window; NO restart (java etime 13d16h). Ep8+ep9 = ONE continuous crawl ≥3.5 h (23:5x → ≥03:35 UTC), ~126.8k requests total — largest of the series (~24× the 29.10 crawl). Rate: ~440–480/min 00:00–02:59, ~570/min from 03:00, ~2500/min 03:27–03:34 (escalation coinciding with the archive/devices sub-crawl; still ~587 hits in the first 15 s of 03:35 — active at last check). Client: 100% 192.168.1.150 (SNAT ovpn). UA: GPTBot/1.4 explicitly declared (4136 hits in 03:xx, Mozilla-wrapped) + browser rotation Chrome/Edge 142–145 Win/Mac (~30k) — same masking as before, now with an honest bot UA in the mix. Paths: continuous issues/pulls enumeration over all /root/ repos (top: flow-task/pulls 1831, gnexus-creds/pulls 954, gnexus-auth-client-py/issues 931, music-loader/, voice/issues, vue-indexer/issues…) + 4 archive/devices/*.zip on smart-home-server (the ESP-credentials repo — note it contains device config paths). Plain enumeration stays under the CPU threshold; CPU crosses 90% only when parallel archive-ZIP builds are added. Details: workspace/incident_gitbucket_2026-10-31_ep9_crawl.md (navi rei).

Traffic source (attributed in earlier sessions): requests arrive via the nginx reverse-proxy VM at 192.168.1.183 (note: inventory entry "internal-proxy-vps .226" is stale — .226 does not answer SSH/ICMP; actual proxy is .183). Client seen in nginx access.log: 192.168.1.150 (ovpn VM, SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10–31.10); crawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot observed in the 25.10 storm; GPTBot/1.4 again declared on 31.10. robots.txt is absent (GitBucket returns an HTML error page for it).

Triage recipe (next episode): journalctl -u gitbucket --since '<start>' | grep -o 'Request(GET [^)]*)' — if archive/*.zip OR issues/pulls-enumeration bursts are present, it is this pattern (pattern switch observed 30.10); no restart needed, load decays within ~2 min after the burst ends (29.10: within ~7 min after the 4.5 h crawl ended; 30.10: 8.54 → 0.89 in ~6 min; 31.10: load1 7.05 → 4.82 within a minute after the archive sub-burst). For attribution: nginx access.log on 192.168.1.183 (creds internal-proxy-vps-creds / bserv@192.168.1.130). CAUTION: a sub-threshold crawl (~450/min) produces NO panel events — to see whether a crawl is still active, check nginx access.log rate on the proxy, not just gitbucket journal/panel.

Not applied (owner decision pending, 9th episode): rate-limiting or blocking /archive/ for bots + robots.txt at the nginx proxy (192.168.1.183), or a GitBucket plugin/config change. Update (31.10, 9th episode): the crawler switched from /archive/*.zip to combinatorial /issues?//pulls? enumeration AND runs as a continuous multi-hour crawl (~127k req in the 23:58–03:35 night, 29k/h for 3+ h), with GPTBot/1.4 declared — a /archive/-only block is no longer sufficient. Recommended fix: path-agnostic per-client rate-limit on vhost git.gnexus.space at the nginx proxy (192.168.1.183), plus robots.txt and /archive/ blocking for non-browser UAs as additional measures. Blocking/rate-limiting now affects the owner's own tooling (navigators use git.gnexus.space): apply a generous limit (e.g. 30–60 req/min per client) rather than a hard block, or whitelist internal IPs/UA tokens. Episodes recurred 5 consecutive days (26–30 Oct) and ep8–9 merged into one ≥3.5 h continuous crawl. Still awaiting owner decision (9th episode in 19 days). Related journals: workspace/incident_gitbucket_cpu_2026-10-12.md, workspace/incident_gitbucket_2026-10-25_load_high.md, workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md, workspace/incident_nextcloud_2026-10-30_connecterror.md, workspace/incident_gitbucket_2026-10-31_ep9_crawl.md.

Access

  • SSH: ubuntu@192.168.1.156, credentials in gnexus-creds secret bserv@192.168.1.130; root SSH denied
  • No QEMU guest agent on this VM; disk-level inspection via read-only loop mount on the hypervisor (losetup -f --show -r -P /mnt/main/vm_main/gitbucket, then mount -o ro,noload)

Listening ports (TCP, verified 2026-09-19)

  • 22 — sshd
  • 80 — GitBucket (java, gitbucket.war)
  • 5432 — system PostgreSQL 10 (GitBucket DB, localhost only)
  • 5433 — navi-postgres docker (localhost only)
  • 8099 — navi-server (pilar)

Notes

  • GitBucket java process has been running since Aug 13 with ~2.4G RSS; VM has 8G RAM — navi instance coexists comfortably.
  • Do not bind anything else to 5432/5433; both are taken.