diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index a9ddfc0..0f69f39 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -6,8 +6,8 @@ owner: gmikcon status: active confidence: "100" -last_reviewed: "2026-10-31" -review_interval: "90d" +last_reviewed: 2026-10-31 +review_interval: 90d source_of_truth: ssh-host-and-owner-confirmed --- @@ -49,22 +49,10 @@ ## Known behavior: load_high episodes from archive downloads (class since 2026-10-12) Diagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100, -GHard server id 44). Updated 2026-10-31 by navi rei (panel event #1140): -the 2026-10-30 23:57 episode is the **8th episode of the same class** -since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, -26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40, 30.10 23:57. - -**8th episode (30.10 23:57 UTC, panel #1136/#1137/#1140):** short burst -again — cpu_high 91.0% at 23:57:40, load_high 2.11×cores at 23:58:40 -(peak load1 8.44 at 23:58:40), cpu_recovered 66.6% at 00:01:10 -(~3.5 min total). 20 archive-requests in the 23:55–00:06 journal window -(navi-1/webclient sha 16b6e159 ×3 retries, smart-home-server/devices/relay), -EofException/Broken pipe at 00:00:34–00:00:48, last activity 00:00:48, -zero archive requests after 00:02. Coincident nextcloud flap #1138/#1139 -(23:58:59–00:00:00, 61 s, ConnectError). Verified 00:02:46 UTC read-only -SSH: GitBucket HTTP 200 in 0.043 s, java (PID 683) 43.5% CPU decaying, -load 2.46/4.46/2.53 decaying, RAM 75.6%, swap 0.9%. Self-recovered, -no restart, no intervention. +GHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC, +peak load1 5.07 at 10:41:27, java 181.6% CPU) is the **7th episode of the +same class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, +26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40. **New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl (06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests @@ -77,6 +65,19 @@ Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, all 8 panel services up. +**Episode 8 (2026-10-30 23:57 UTC, diagnosed by navi rei, panel event #1136):** +short burst, not a crawl — 20 `GET /root//archive//.zip` +requests 23:45–00:00 UTC (navi-1 webclient + webclient/vendor/gnexus-ui-kit + +client/images, smart-home-server webclient/devices/relay/button/docs/ +webclient_legacy, rpg apps/game) amid a ~2600-request GPTBot/1.4 + +Chrome 142–145 Win/Mac/Edge rotation crawl on issues/pulls/blob/commits paths +(HTTP 200, small responses, no CPU impact). Peak CPU 99.6%, load1 11.25 at +00:00:40, java 327.6% CPU; last EofException/Broken pipe 00:00:48; +cpu_recovered #1140 00:01:10, load_recovered #1141 00:02:10; load1 1.30 by +00:04:26. Self-recovered ~7 min after the last archive request, no restart, +GitBucket HTTP 200 throughout. Journal: +workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. + **Cause (confirmed across all 8 episodes):** bursts of `GET /root//archive//.zip` (and a few `/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI "download directory @@ -90,17 +91,17 @@ nginx reverse-proxy VM at **192.168.1.183** (note: inventory entry "internal-proxy-vps .226" is stale — .226 does not answer SSH/ICMP; actual proxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM, -SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10); crawler -UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, MJ12bot -observed in the 25.10 storm. robots.txt is absent (GitBucket returns an HTML -error page for it). +SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10); +crawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, +MJ12bot observed in the 25.10 storm; GPTBot/1.4 dominant in the 30.10 crawl. +robots.txt is absent (GitBucket returns an HTML error page for it). **Triage recipe (next episode):** `journalctl -u gitbucket --since '' | grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this pattern; no restart needed, load decays within ~2 min after the burst ends -(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx -access.log on 192.168.1.183 (creds internal-proxy-vps-creds / -bserv@192.168.1.130). +(29.10: within ~7 min after the 4.5 h crawl ended; 30.10: ~7 min after the +burst ended). For attribution: nginx access.log on 192.168.1.183 (creds +internal-proxy-vps-creds / bserv@192.168.1.130). **Not applied (owner decision pending, 8th episode):** rate-limiting or blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), @@ -110,7 +111,8 @@ decide on proxy-level rate-limiting/robots.txt. Related journals: workspace/incident_gitbucket_cpu_2026-10-12.md, workspace/incident_gitbucket_2026-10-25_load_high.md, -workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md. +workspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md, +workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. ## Access diff --git a/90-maintenance/pending-changes/20261031000514-0af03550.json b/90-maintenance/pending-changes/20261031000514-0af03550.json new file mode 100644 index 0000000..7229b4c --- /dev/null +++ b/90-maintenance/pending-changes/20261031000514-0af03550.json @@ -0,0 +1,19 @@ +{ + "id": "20261031000514-0af03550", + "kind": "doc", + "target": "10-systems/hosts/gitbucket.md", + "summary": "gitbucket host doc: 8th archive-burst episode 2026-10-30 23:57 UTC (short burst, self-recovered ~7 min), GPTBot/1.4 crawl attribution, pending owner decision now 5 consecutive days", + "reason": "Episode 8 of the archive-burst class confirmed 2026-10-30 23:57 UTC (panel event #1136, diagnosed by navi rei via SSH + nginx logs): short burst of 20 archive/*.zip requests amid GPTBot crawl, peak load1 11.25, self-recovered ~7 min. Updates episode count, traffic-source note and pending owner decision (5 consecutive days).", + "payload": { + "content": "---\ntitle: VM gitbucket (GitBucket + navi instance pilar)\ntype: host\nip: 192.168.1.156\npurpose: \"GitBucket git hosting + navi agent instance (pilar, profile navi_code)\"\nowner: gmikcon\nstatus: active\nconfidence: \"100\"\nlast_reviewed: 2026-10-31\nreview_interval: 90d\nsource_of_truth: ssh-host-and-owner-confirmed\n---\n\n# VM gitbucket (GitBucket + navi instance pilar)\n\n## Overview\n\nKVM/QEMU VM on `hp-proliant-dl380-g6` (192.168.1.130). Primary role: GitBucket\ngit hosting (see [applications/gitbucket.md](../applications/gitbucket.md)).\nSince 2026-09-19 the VM also hosts the navi-1 agent instance **pilar**\n(profile `navi_code`), deployed by navi (rei) from the `deploy` branch.\n\n## Technical Details\n\n- **IP:** 192.168.1.156\n- **OS:** Ubuntu 18.04.6 LTS (Bionic Beaver), kernel 4.15.0-213-generic\n- **Users:** `gitbucket` (service owner, home = /home/ubuntu), `navi` (navi instance, home = /home/navi), `ubuntu` (admin; SSH/sudo credentials — see gnexus-creds secret `bserv@192.168.1.130`)\n- **Resources:** 4 vCPU, 8 GiB RAM, 100G raw disk `/mnt/main/vm_main/gitbucket` (p1 root 28G — 21% used; p5 data 71G — 6% used; observed 2026-09-19)\n- **Hypervisor storage:** /mnt/main 2.5T, 730G free (69%)\n\n## Navi instance pilar (deployed 2026-09-19)\n\n- **Instance name:** pilar\n- **Instance ID:** c6962c3e-… (full value in /home/navi/navi-1/instance.json)\n- **Repo:** /home/navi/navi-1 (branch `deploy`, commit e276b1e)\n- **Service:** navi.service (systemd, enabled, Restart=always)\n- **Server port:** 8099 (0.0.0.0), health endpoint http://127.0.0.1:8099/health\n- **Postgres:** docker container `navi-postgres` (pgvector/pgvector:pg16) on **127.0.0.1:5433** — port shifted from the default 5432 because GitBucket's system PostgreSQL 10 occupies 5432; DATABASE_URL in .env points to 5433\n- **Python:** standalone CPython 3.12.14 at /home/navi/navi-1/.python/ (system python 3.6 is too old)\n- **Swarm:** .swarm-key copied from the hive host (rei, 192.168.1.168); HIVE_URL=http://192.168.1.168:8087; hive.announce_ok confirmed in journalctl; visible in `peer list` as `pilar — 192.168.1.156:8099 [online]`\n- **Ollama backend:** OLLAMA_HOST=https://ollama.com, model glm-5.3-flash:cloud; Ollama Cloud credential configured in .env (value stored in gnexus-creds secret \"Ollama Cloud API key (navi swarm)\")\n- **Webclient/auth:** NAVI_WEBCLIENT_ENABLED=false, NAVI_AUTH_ENABLED=false (default deploy shape)\n- **Deploy notes:**\n - docker-ce 24.0.2 + docker-compose-plugin v2.18.1 installed from download.docker.com (bionic repo); git/curl installed via apt\n - asyncpg 0.29.0 has no wheel for glibc 2.27 and clang is absent — built from source with `CC=gcc CXX=g++` before re-running install.sh (script is idempotent)\n - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds)\n- **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances\n\n## Known behavior: load_high episodes from archive downloads (class since 2026-10-12)\n\nDiagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100,\nGHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC,\npeak load1 5.07 at 10:41:27, java 181.6% CPU) is the **7th episode of the\nsame class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55,\n26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40.\n\n**New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl\n(06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests\nfrom 192.168.1.150 via the public route (journal volume per hour: 1413 lines\n06:xx, 2891 08:xx, 1447 10:xx; 29 ERROR — Broken pipe / EofException /\nNPE in `RepositoryViewerController.archiveRepository`). Repos: vmk-demo-bot\n(incl. `node_modules`), navi-1 (incl. `webclient/dist`, `vendor/gnexus-ui-kit`),\nsmart-home-server, vue-indexer. Load decayed 5.07 → 0.66 by 10:48 UTC after\nthe crawl stopped (last archive request 10:42:10, zero since 10:44).\nSelf-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0,\nall 8 panel services up.\n\n**Episode 8 (2026-10-30 23:57 UTC, diagnosed by navi rei, panel event #1136):**\nshort burst, not a crawl — 20 `GET /root//archive//.zip`\nrequests 23:45–00:00 UTC (navi-1 webclient + webclient/vendor/gnexus-ui-kit +\nclient/images, smart-home-server webclient/devices/relay/button/docs/\nwebclient_legacy, rpg apps/game) amid a ~2600-request GPTBot/1.4 +\nChrome 142–145 Win/Mac/Edge rotation crawl on issues/pulls/blob/commits paths\n(HTTP 200, small responses, no CPU impact). Peak CPU 99.6%, load1 11.25 at\n00:00:40, java 327.6% CPU; last EofException/Broken pipe 00:00:48;\ncpu_recovered #1140 00:01:10, load_recovered #1141 00:02:10; load1 1.30 by\n00:04:26. Self-recovered ~7 min after the last archive request, no restart,\nGitBucket HTTP 200 throughout. Journal:\nworkspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md.\n\n**Cause (confirmed across all 8 episodes):** bursts of\n`GET /root//archive//.zip` (and a few `/raw/...` file\ndownloads) through git.gnexus.space — GitBucket's web-UI \"download directory\nas ZIP\" endpoint. Clients abort the download mid-stream (`EofException` /\n`Broken pipe` in journalctl, logged as `RepositoryViewerController - Catch\nunhandled error`), but GitBucket keeps building the ZIP server-side — several\nconcurrent archive builds saturate all 4 vCPUs. Baseline between episodes:\nCPU 10–27%, load1 0.2–1.5.\n\n**Traffic source (attributed in earlier sessions):** requests arrive via the\nnginx reverse-proxy VM at **192.168.1.183** (note: inventory entry\n\"internal-proxy-vps .226\" is stale — .226 does not answer SSH/ICMP; actual\nproxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM,\nSNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10);\ncrawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot,\nMJ12bot observed in the 25.10 storm; GPTBot/1.4 dominant in the 30.10 crawl.\nrobots.txt is absent (GitBucket returns an HTML error page for it).\n\n**Triage recipe (next episode):** `journalctl -u gitbucket --since '' |\ngrep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this\npattern; no restart needed, load decays within ~2 min after the burst ends\n(29.10: within ~7 min after the 4.5 h crawl ended; 30.10: ~7 min after the\nburst ended). For attribution: nginx access.log on 192.168.1.183 (creds\ninternal-proxy-vps-creds / bserv@192.168.1.130).\n\n**Not applied (owner decision pending, 8th episode):** rate-limiting or\nblocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183),\nor a GitBucket plugin/config change. Escalation criterion \"bursts ≥2/day\" is\nnot formally exceeded (1 burst/day), but episodes now recur **5 consecutive\ndays** (26–30 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner\ndecide on proxy-level rate-limiting/robots.txt. Related journals:\nworkspace/incident_gitbucket_cpu_2026-10-12.md,\nworkspace/incident_gitbucket_2026-10-25_load_high.md,\nworkspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md,\nworkspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md.\n\n## Access\n\n- SSH: `ubuntu@192.168.1.156`, credentials in gnexus-creds secret `bserv@192.168.1.130`; root SSH denied\n- No QEMU guest agent on this VM; disk-level inspection via read-only loop mount on the hypervisor (`losetup -f --show -r -P /mnt/main/vm_main/gitbucket`, then `mount -o ro,noload`)\n\n## Listening ports (TCP, verified 2026-09-19)\n\n- 22 — sshd\n- 80 — GitBucket (java, gitbucket.war)\n- 5432 — system PostgreSQL 10 (GitBucket DB, localhost only)\n- 5433 — navi-postgres docker (localhost only)\n- 8099 — navi-server (pilar)\n\n## Notes\n\n- GitBucket java process has been running since Aug 13 with ~2.4G RSS; VM has 8G RAM — navi instance coexists comfortably.\n- Do not bind anything else to 5432/5433; both are taken." + }, + "status": "applied", + "created_at": "2026-10-31T00:05:14Z", + "updated_at": "2026-10-31T00:05:15Z", + "applied_at": "2026-10-31T00:05:15Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}