diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index 9528765..a9ddfc0 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -6,8 +6,8 @@ owner: gmikcon status: active confidence: "100" -last_reviewed: 2026-10-29 -review_interval: 90d +last_reviewed: "2026-10-31" +review_interval: "90d" source_of_truth: ssh-host-and-owner-confirmed --- @@ -49,10 +49,22 @@ ## Known behavior: load_high episodes from archive downloads (class since 2026-10-12) Diagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100, -GHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC, -peak load1 5.07 at 10:41:27, java 181.6% CPU) is the **7th episode of the -same class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, -26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40. +GHard server id 44). Updated 2026-10-31 by navi rei (panel event #1140): +the 2026-10-30 23:57 episode is the **8th episode of the same class** +since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, +26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40, 30.10 23:57. + +**8th episode (30.10 23:57 UTC, panel #1136/#1137/#1140):** short burst +again — cpu_high 91.0% at 23:57:40, load_high 2.11×cores at 23:58:40 +(peak load1 8.44 at 23:58:40), cpu_recovered 66.6% at 00:01:10 +(~3.5 min total). 20 archive-requests in the 23:55–00:06 journal window +(navi-1/webclient sha 16b6e159 ×3 retries, smart-home-server/devices/relay), +EofException/Broken pipe at 00:00:34–00:00:48, last activity 00:00:48, +zero archive requests after 00:02. Coincident nextcloud flap #1138/#1139 +(23:58:59–00:00:00, 61 s, ConnectError). Verified 00:02:46 UTC read-only +SSH: GitBucket HTTP 200 in 0.043 s, java (PID 683) 43.5% CPU decaying, +load 2.46/4.46/2.53 decaying, RAM 75.6%, swap 0.9%. Self-recovered, +no restart, no intervention. **New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl (06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests @@ -65,7 +77,7 @@ Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, all 8 panel services up. -**Cause (confirmed across all 7 episodes):** bursts of +**Cause (confirmed across all 8 episodes):** bursts of `GET /root//archive//.zip` (and a few `/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI "download directory as ZIP" endpoint. Clients abort the download mid-stream (`EofException` / @@ -90,11 +102,11 @@ access.log on 192.168.1.183 (creds internal-proxy-vps-creds / bserv@192.168.1.130). -**Not applied (owner decision pending, 7th episode):** rate-limiting or +**Not applied (owner decision pending, 8th episode):** rate-limiting or blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), or a GitBucket plugin/config change. Escalation criterion "bursts ≥2/day" is -not formally exceeded (1 burst/day), but episodes now recur **4 consecutive -days** (26–29 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner +not formally exceeded (1 burst/day), but episodes now recur **5 consecutive +days** (26–30 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner decide on proxy-level rate-limiting/robots.txt. Related journals: workspace/incident_gitbucket_cpu_2026-10-12.md, workspace/incident_gitbucket_2026-10-25_load_high.md, diff --git a/90-maintenance/pending-changes/20261031000301-2cd25f67.json b/90-maintenance/pending-changes/20261031000301-2cd25f67.json new file mode 100644 index 0000000..70fec63 --- /dev/null +++ b/90-maintenance/pending-changes/20261031000301-2cd25f67.json @@ -0,0 +1,19 @@ +{ + "id": "20261031000301-2cd25f67", + "kind": "doc", + "target": "10-systems/hosts/gitbucket.md", + "summary": "gitbucket: 8th archive-crawler CPU episode (30.10 23:57 UTC), series 5 consecutive days", + "reason": "8th episode of the archive-crawler CPU class: 2026-10-30 23:57 UTC burst (cpu_high 91% #1136, load_high 2.11×cores #1137, cpu_recovered #1140), confirmed via read-only SSH journalctl on .156 — same /archive/*.zip signature, self-recovered in ~3.5 min. Series now 5 consecutive days (26–30 Oct).", + "payload": { + "content": "---\ntitle: VM gitbucket (GitBucket + navi instance pilar)\ntype: host\nip: 192.168.1.156\npurpose: \"GitBucket git hosting + navi agent instance (pilar, profile navi_code)\"\nowner: gmikcon\nstatus: active\nconfidence: \"100\"\nlast_reviewed: \"2026-10-31\"\nreview_interval: \"90d\"\nsource_of_truth: ssh-host-and-owner-confirmed\n---\n\n# VM gitbucket (GitBucket + navi instance pilar)\n\n## Overview\n\nKVM/QEMU VM on `hp-proliant-dl380-g6` (192.168.1.130). Primary role: GitBucket\ngit hosting (see [applications/gitbucket.md](../applications/gitbucket.md)).\nSince 2026-09-19 the VM also hosts the navi-1 agent instance **pilar**\n(profile `navi_code`), deployed by navi (rei) from the `deploy` branch.\n\n## Technical Details\n\n- **IP:** 192.168.1.156\n- **OS:** Ubuntu 18.04.6 LTS (Bionic Beaver), kernel 4.15.0-213-generic\n- **Users:** `gitbucket` (service owner, home = /home/ubuntu), `navi` (navi instance, home = /home/navi), `ubuntu` (admin; SSH/sudo credentials — see gnexus-creds secret `bserv@192.168.1.130`)\n- **Resources:** 4 vCPU, 8 GiB RAM, 100G raw disk `/mnt/main/vm_main/gitbucket` (p1 root 28G — 21% used; p5 data 71G — 6% used; observed 2026-09-19)\n- **Hypervisor storage:** /mnt/main 2.5T, 730G free (69%)\n\n## Navi instance pilar (deployed 2026-09-19)\n\n- **Instance name:** pilar\n- **Instance ID:** c6962c3e-… (full value in /home/navi/navi-1/instance.json)\n- **Repo:** /home/navi/navi-1 (branch `deploy`, commit e276b1e)\n- **Service:** navi.service (systemd, enabled, Restart=always)\n- **Server port:** 8099 (0.0.0.0), health endpoint http://127.0.0.1:8099/health\n- **Postgres:** docker container `navi-postgres` (pgvector/pgvector:pg16) on **127.0.0.1:5433** — port shifted from the default 5432 because GitBucket's system PostgreSQL 10 occupies 5432; DATABASE_URL in .env points to 5433\n- **Python:** standalone CPython 3.12.14 at /home/navi/navi-1/.python/ (system python 3.6 is too old)\n- **Swarm:** .swarm-key copied from the hive host (rei, 192.168.1.168); HIVE_URL=http://192.168.1.168:8087; hive.announce_ok confirmed in journalctl; visible in `peer list` as `pilar — 192.168.1.156:8099 [online]`\n- **Ollama backend:** OLLAMA_HOST=https://ollama.com, model glm-5.3-flash:cloud; Ollama Cloud credential configured in .env (value stored in gnexus-creds secret \"Ollama Cloud API key (navi swarm)\")\n- **Webclient/auth:** NAVI_WEBCLIENT_ENABLED=false, NAVI_AUTH_ENABLED=false (default deploy shape)\n- **Deploy notes:**\n - docker-ce 24.0.2 + docker-compose-plugin v2.18.1 installed from download.docker.com (bionic repo); git/curl installed via apt\n - asyncpg 0.29.0 has no wheel for glibc 2.27 and clang is absent — built from source with `CC=gcc CXX=g++` before re-running install.sh (script is idempotent)\n - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds)\n- **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances\n\n## Known behavior: load_high episodes from archive downloads (class since 2026-10-12)\n\nDiagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100,\nGHard server id 44). Updated 2026-10-31 by navi rei (panel event #1140):\nthe 2026-10-30 23:57 episode is the **8th episode of the same class**\nsince 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55,\n26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40, 30.10 23:57.\n\n**8th episode (30.10 23:57 UTC, panel #1136/#1137/#1140):** short burst\nagain — cpu_high 91.0% at 23:57:40, load_high 2.11×cores at 23:58:40\n(peak load1 8.44 at 23:58:40), cpu_recovered 66.6% at 00:01:10\n(~3.5 min total). 20 archive-requests in the 23:55–00:06 journal window\n(navi-1/webclient sha 16b6e159 ×3 retries, smart-home-server/devices/relay),\nEofException/Broken pipe at 00:00:34–00:00:48, last activity 00:00:48,\nzero archive requests after 00:02. Coincident nextcloud flap #1138/#1139\n(23:58:59–00:00:00, 61 s, ConnectError). Verified 00:02:46 UTC read-only\nSSH: GitBucket HTTP 200 in 0.043 s, java (PID 683) 43.5% CPU decaying,\nload 2.46/4.46/2.53 decaying, RAM 75.6%, swap 0.9%. Self-recovered,\nno restart, no intervention.\n\n**New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl\n(06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests\nfrom 192.168.1.150 via the public route (journal volume per hour: 1413 lines\n06:xx, 2891 08:xx, 1447 10:xx; 29 ERROR — Broken pipe / EofException /\nNPE in `RepositoryViewerController.archiveRepository`). Repos: vmk-demo-bot\n(incl. `node_modules`), navi-1 (incl. `webclient/dist`, `vendor/gnexus-ui-kit`),\nsmart-home-server, vue-indexer. Load decayed 5.07 → 0.66 by 10:48 UTC after\nthe crawl stopped (last archive request 10:42:10, zero since 10:44).\nSelf-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0,\nall 8 panel services up.\n\n**Cause (confirmed across all 8 episodes):** bursts of\n`GET /root//archive//.zip` (and a few `/raw/...` file\ndownloads) through git.gnexus.space — GitBucket's web-UI \"download directory\nas ZIP\" endpoint. Clients abort the download mid-stream (`EofException` /\n`Broken pipe` in journalctl, logged as `RepositoryViewerController - Catch\nunhandled error`), but GitBucket keeps building the ZIP server-side — several\nconcurrent archive builds saturate all 4 vCPUs. Baseline between episodes:\nCPU 10–27%, load1 0.2–1.5.\n\n**Traffic source (attributed in earlier sessions):** requests arrive via the\nnginx reverse-proxy VM at **192.168.1.183** (note: inventory entry\n\"internal-proxy-vps .226\" is stale — .226 does not answer SSH/ICMP; actual\nproxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM,\nSNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10); crawler\nUAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot\nobserved in the 25.10 storm. robots.txt is absent (GitBucket returns an HTML\nerror page for it).\n\n**Triage recipe (next episode):** `journalctl -u gitbucket --since '' |\ngrep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this\npattern; no restart needed, load decays within ~2 min after the burst ends\n(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx\naccess.log on 192.168.1.183 (creds internal-proxy-vps-creds /\nbserv@192.168.1.130).\n\n**Not applied (owner decision pending, 8th episode):** rate-limiting or\nblocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183),\nor a GitBucket plugin/config change. Escalation criterion \"bursts ≥2/day\" is\nnot formally exceeded (1 burst/day), but episodes now recur **5 consecutive\ndays** (26–30 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner\ndecide on proxy-level rate-limiting/robots.txt. Related journals:\nworkspace/incident_gitbucket_cpu_2026-10-12.md,\nworkspace/incident_gitbucket_2026-10-25_load_high.md,\nworkspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md.\n\n## Access\n\n- SSH: `ubuntu@192.168.1.156`, credentials in gnexus-creds secret `bserv@192.168.1.130`; root SSH denied\n- No QEMU guest agent on this VM; disk-level inspection via read-only loop mount on the hypervisor (`losetup -f --show -r -P /mnt/main/vm_main/gitbucket`, then `mount -o ro,noload`)\n\n## Listening ports (TCP, verified 2026-09-19)\n\n- 22 — sshd\n- 80 — GitBucket (java, gitbucket.war)\n- 5432 — system PostgreSQL 10 (GitBucket DB, localhost only)\n- 5433 — navi-postgres docker (localhost only)\n- 8099 — navi-server (pilar)\n\n## Notes\n\n- GitBucket java process has been running since Aug 13 with ~2.4G RSS; VM has 8G RAM — navi instance coexists comfortably.\n- Do not bind anything else to 5432/5433; both are taken." + }, + "status": "applied", + "created_at": "2026-10-31T00:03:01Z", + "updated_at": "2026-10-31T00:03:03Z", + "applied_at": "2026-10-31T00:03:03Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}