diff --git a/10-systems/hosts/gitbucket.md b/10-systems/hosts/gitbucket.md index 0f69f39..8feaf13 100644 --- a/10-systems/hosts/gitbucket.md +++ b/10-systems/hosts/gitbucket.md @@ -54,6 +54,19 @@ same class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55, 26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40. +**2026-10-30 23:57 UTC (8th episode, panel #1136/#1137):** cpu_high 91% → +load_high 2.11×cores, peak load1 11.25 at 00:00:40, java up to 327% CPU; +~2600 archive/raw requests 23:50–00:04 from 192.168.1.150 via git.gnexus.space +(UA mix GPTBot/1.4 + Chrome 142–145 Win/Mac/Edge rotation — same signature as +27.10/29.10). Self-recovered: load1 1.01 by 00:06:08, zero archive requests +after 00:04. Journal: workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. +During the burst, nextcloud (separate VM, 192.168.1.152) had a 61-s +ConnectError blip (#1138, 23:58:58–23:59:59 UTC) — the **2nd such +coincidence**: an identical 61.3-s blip occurred at the 27.10 17:50 episode +start. Treat nextcloud blips during gitbucket archive episodes as expected +collateral (mechanism unclear, likely proxy/hypervisor-side contention; +nextcloud VM itself stays online), not an independent outage. + **New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl (06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests from 192.168.1.150 via the public route (journal volume per hour: 1413 lines @@ -65,19 +78,6 @@ Self-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0, all 8 panel services up. -**Episode 8 (2026-10-30 23:57 UTC, diagnosed by navi rei, panel event #1136):** -short burst, not a crawl — 20 `GET /root//archive//.zip` -requests 23:45–00:00 UTC (navi-1 webclient + webclient/vendor/gnexus-ui-kit + -client/images, smart-home-server webclient/devices/relay/button/docs/ -webclient_legacy, rpg apps/game) amid a ~2600-request GPTBot/1.4 + -Chrome 142–145 Win/Mac/Edge rotation crawl on issues/pulls/blob/commits paths -(HTTP 200, small responses, no CPU impact). Peak CPU 99.6%, load1 11.25 at -00:00:40, java 327.6% CPU; last EofException/Broken pipe 00:00:48; -cpu_recovered #1140 00:01:10, load_recovered #1141 00:02:10; load1 1.30 by -00:04:26. Self-recovered ~7 min after the last archive request, no restart, -GitBucket HTTP 200 throughout. Journal: -workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md. - **Cause (confirmed across all 8 episodes):** bursts of `GET /root//archive//.zip` (and a few `/raw/...` file downloads) through git.gnexus.space — GitBucket's web-UI "download directory @@ -93,15 +93,15 @@ proxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM, SNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10); crawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot, -MJ12bot observed in the 25.10 storm; GPTBot/1.4 dominant in the 30.10 crawl. -robots.txt is absent (GitBucket returns an HTML error page for it). +MJ12bot observed in the 25.10 storm. robots.txt is absent (GitBucket returns +an HTML error page for it). **Triage recipe (next episode):** `journalctl -u gitbucket --since '' | grep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this pattern; no restart needed, load decays within ~2 min after the burst ends -(29.10: within ~7 min after the 4.5 h crawl ended; 30.10: ~7 min after the -burst ended). For attribution: nginx access.log on 192.168.1.183 (creds -internal-proxy-vps-creds / bserv@192.168.1.130). +(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx +access.log on 192.168.1.183 (creds internal-proxy-vps-creds / +bserv@192.168.1.130). **Not applied (owner decision pending, 8th episode):** rate-limiting or blocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183), diff --git a/90-maintenance/pending-changes/20261031000721-1281bc72.json b/90-maintenance/pending-changes/20261031000721-1281bc72.json new file mode 100644 index 0000000..ef21d1d --- /dev/null +++ b/90-maintenance/pending-changes/20261031000721-1281bc72.json @@ -0,0 +1,19 @@ +{ + "id": "20261031000721-1281bc72", + "kind": "doc", + "target": "10-systems/hosts/gitbucket.md", + "summary": "gitbucket: 8th archive-burst episode (30.10 23:57 UTC), nextcloud 61-s blip correlation confirmed twice, pending-decision counter updated", + "reason": "8th archive-burst episode 2026-10-30 23:57 UTC (panel #1136/#1137, reaction events 63c2c371 + 34367587): added episode to the list, recorded the 2nd confirmed 61-s nextcloud blip coincidence (27.10 + 30.10), updated pending-decision counter to 5 consecutive days (26–30 Oct). Sources: SSH triage on 192.168.1.156, nginx access.log on 192.168.1.183, hard-panel service_incidents.", + "payload": { + "content": "---\ntitle: VM gitbucket (GitBucket + navi instance pilar)\ntype: host\nip: 192.168.1.156\npurpose: \"GitBucket git hosting + navi agent instance (pilar, profile navi_code)\"\nowner: gmikcon\nstatus: active\nconfidence: \"100\"\nlast_reviewed: 2026-10-31\nreview_interval: 90d\nsource_of_truth: ssh-host-and-owner-confirmed\n---\n\n# VM gitbucket (GitBucket + navi instance pilar)\n\n## Overview\n\nKVM/QEMU VM on `hp-proliant-dl380-g6` (192.168.1.130). Primary role: GitBucket\ngit hosting (see [applications/gitbucket.md](../applications/gitbucket.md)).\nSince 2026-09-19 the VM also hosts the navi-1 agent instance **pilar**\n(profile `navi_code`), deployed by navi (rei) from the `deploy` branch.\n\n## Technical Details\n\n- **IP:** 192.168.1.156\n- **OS:** Ubuntu 18.04.6 LTS (Bionic Beaver), kernel 4.15.0-213-generic\n- **Users:** `gitbucket` (service owner, home = /home/ubuntu), `navi` (navi instance, home = /home/navi), `ubuntu` (admin; SSH/sudo credentials — see gnexus-creds secret `bserv@192.168.1.130`)\n- **Resources:** 4 vCPU, 8 GiB RAM, 100G raw disk `/mnt/main/vm_main/gitbucket` (p1 root 28G — 21% used; p5 data 71G — 6% used; observed 2026-09-19)\n- **Hypervisor storage:** /mnt/main 2.5T, 730G free (69%)\n\n## Navi instance pilar (deployed 2026-09-19)\n\n- **Instance name:** pilar\n- **Instance ID:** c6962c3e-… (full value in /home/navi/navi-1/instance.json)\n- **Repo:** /home/navi/navi-1 (branch `deploy`, commit e276b1e)\n- **Service:** navi.service (systemd, enabled, Restart=always)\n- **Server port:** 8099 (0.0.0.0), health endpoint http://127.0.0.1:8099/health\n- **Postgres:** docker container `navi-postgres` (pgvector/pgvector:pg16) on **127.0.0.1:5433** — port shifted from the default 5432 because GitBucket's system PostgreSQL 10 occupies 5432; DATABASE_URL in .env points to 5433\n- **Python:** standalone CPython 3.12.14 at /home/navi/navi-1/.python/ (system python 3.6 is too old)\n- **Swarm:** .swarm-key copied from the hive host (rei, 192.168.1.168); HIVE_URL=http://192.168.1.168:8087; hive.announce_ok confirmed in journalctl; visible in `peer list` as `pilar — 192.168.1.156:8099 [online]`\n- **Ollama backend:** OLLAMA_HOST=https://ollama.com, model glm-5.3-flash:cloud; Ollama Cloud credential configured in .env (value stored in gnexus-creds secret \"Ollama Cloud API key (navi swarm)\")\n- **Webclient/auth:** NAVI_WEBCLIENT_ENABLED=false, NAVI_AUTH_ENABLED=false (default deploy shape)\n- **Deploy notes:**\n - docker-ce 24.0.2 + docker-compose-plugin v2.18.1 installed from download.docker.com (bionic repo); git/curl installed via apt\n - asyncpg 0.29.0 has no wheel for glibc 2.27 and clang is absent — built from source with `CC=gcc CXX=g++` before re-running install.sh (script is idempotent)\n - user `navi` has NOPASSWD sudoers (/etc/sudoers.d/navi) so install.sh's internal sudo works over non-interactive SSH; the `navi` user password matches the hypervisor bserv credential (gnexus-creds)\n- **Known non-critical:** /health embed.ok=false (embed backend 401) — same as other swarm instances\n\n## Known behavior: load_high episodes from archive downloads (class since 2026-10-12)\n\nDiagnosed 2026-10-29 10:41–10:48 UTC by navi rei (panel event #1100,\nGHard server id 44). The 2026-10-29 episode (cpu_high 90.5% at 10:40:57 UTC,\npeak load1 5.07 at 10:41:27, java 181.6% CPU) is the **7th episode of the\nsame class** since 2026-10-12: 12.10 crawler storm, 13.10 spike, 25.10 01:55,\n26.10 17:44, 27.10 17:50, 28.10 21:33, 29.10 10:40.\n\n**2026-10-30 23:57 UTC (8th episode, panel #1136/#1137):** cpu_high 91% →\nload_high 2.11×cores, peak load1 11.25 at 00:00:40, java up to 327% CPU;\n~2600 archive/raw requests 23:50–00:04 from 192.168.1.150 via git.gnexus.space\n(UA mix GPTBot/1.4 + Chrome 142–145 Win/Mac/Edge rotation — same signature as\n27.10/29.10). Self-recovered: load1 1.01 by 00:06:08, zero archive requests\nafter 00:04. Journal: workspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md.\nDuring the burst, nextcloud (separate VM, 192.168.1.152) had a 61-s\nConnectError blip (#1138, 23:58:58–23:59:59 UTC) — the **2nd such\ncoincidence**: an identical 61.3-s blip occurred at the 27.10 17:50 episode\nstart. Treat nextcloud blips during gitbucket archive episodes as expected\ncollateral (mechanism unclear, likely proxy/hypervisor-side contention;\nnextcloud VM itself stays online), not an independent outage.\n\n**New signature on 29.10:** not a 60–90 s burst but a sustained ~4.5 h crawl\n(06:00–10:42 UTC): 5248 `GET /root//archive//.zip` requests\nfrom 192.168.1.150 via the public route (journal volume per hour: 1413 lines\n06:xx, 2891 08:xx, 1447 10:xx; 29 ERROR — Broken pipe / EofException /\nNPE in `RepositoryViewerController.archiveRepository`). Repos: vmk-demo-bot\n(incl. `node_modules`), navi-1 (incl. `webclient/dist`, `vendor/gnexus-ui-kit`),\nsmart-home-server, vue-indexer. Load decayed 5.07 → 0.66 by 10:48 UTC after\nthe crawl stopped (last archive request 10:42:10, zero since 10:44).\nSelf-recovered, no restart, GitBucket HTTP 200 throughout, iowait 0,\nall 8 panel services up.\n\n**Cause (confirmed across all 8 episodes):** bursts of\n`GET /root//archive//.zip` (and a few `/raw/...` file\ndownloads) through git.gnexus.space — GitBucket's web-UI \"download directory\nas ZIP\" endpoint. Clients abort the download mid-stream (`EofException` /\n`Broken pipe` in journalctl, logged as `RepositoryViewerController - Catch\nunhandled error`), but GitBucket keeps building the ZIP server-side — several\nconcurrent archive builds saturate all 4 vCPUs. Baseline between episodes:\nCPU 10–27%, load1 0.2–1.5.\n\n**Traffic source (attributed in earlier sessions):** requests arrive via the\nnginx reverse-proxy VM at **192.168.1.183** (note: inventory entry\n\"internal-proxy-vps .226\" is stale — .226 does not answer SSH/ICMP; actual\nproxy is .183). Client seen in nginx access.log: **192.168.1.150** (ovpn VM,\nSNAT) with browser UA rotation Chrome 142–145 Win/Mac (27.10, 29.10, 30.10);\ncrawler UAs GPTBot, meta-externalagent, Reflectionbot, ClaudeBot, SemrushBot,\nMJ12bot observed in the 25.10 storm. robots.txt is absent (GitBucket returns\nan HTML error page for it).\n\n**Triage recipe (next episode):** `journalctl -u gitbucket --since '' |\ngrep -o 'Request(GET [^)]*)'` — if archive/*.zip bursts are present, it is this\npattern; no restart needed, load decays within ~2 min after the burst ends\n(29.10: within ~7 min after the 4.5 h crawl ended). For attribution: nginx\naccess.log on 192.168.1.183 (creds internal-proxy-vps-creds /\nbserv@192.168.1.130).\n\n**Not applied (owner decision pending, 8th episode):** rate-limiting or\nblocking `/archive/` for bots + robots.txt at the nginx proxy (192.168.1.183),\nor a GitBucket plugin/config change. Escalation criterion \"bursts ≥2/day\" is\nnot formally exceeded (1 burst/day), but episodes now recur **5 consecutive\ndays** (26–30 Oct) and the 29.10 episode lasted ~4.5 h — recommend the owner\ndecide on proxy-level rate-limiting/robots.txt. Related journals:\nworkspace/incident_gitbucket_cpu_2026-10-12.md,\nworkspace/incident_gitbucket_2026-10-25_load_high.md,\nworkspace/incident_gitbucket_cpu_2026-10-27_crawler_archives.md,\nworkspace/incident_gitbucket_cpu_2026-10-30_archive_burst.md.\n\n## Access\n\n- SSH: `ubuntu@192.168.1.156`, credentials in gnexus-creds secret `bserv@192.168.1.130`; root SSH denied\n- No QEMU guest agent on this VM; disk-level inspection via read-only loop mount on the hypervisor (`losetup -f --show -r -P /mnt/main/vm_main/gitbucket`, then `mount -o ro,noload`)\n\n## Listening ports (TCP, verified 2026-09-19)\n\n- 22 — sshd\n- 80 — GitBucket (java, gitbucket.war)\n- 5432 — system PostgreSQL 10 (GitBucket DB, localhost only)\n- 5433 — navi-postgres docker (localhost only)\n- 8099 — navi-server (pilar)\n\n## Notes\n\n- GitBucket java process has been running since Aug 13 with ~2.4G RSS; VM has 8G RAM — navi instance coexists comfortably.\n- Do not bind anything else to 5432/5433; both are taken." + }, + "status": "applied", + "created_at": "2026-10-31T00:07:21Z", + "updated_at": "2026-10-31T00:07:22Z", + "applied_at": "2026-10-31T00:07:22Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}