diff --git a/10-systems/hosts/bugtrail.md b/10-systems/hosts/bugtrail.md index f71fbb6..27fb40b 100644 --- a/10-systems/hosts/bugtrail.md +++ b/10-systems/hosts/bugtrail.md @@ -7,7 +7,7 @@ status: active confidence: "100" last_reviewed: "2026-09-18" -review_interval: "90d" +review_interval: 90d source_of_truth: ssh-host-and-owner-confirmed --- @@ -83,10 +83,10 @@ failed (anyio CancelledError), tool_loader.file_error for gmail.py (ModuleNotFoundError: html2text), startup.embed_unavailable (embed backend 401 — embed.ok=false in /health). Core server and hive announce are healthy. - -## Power-loss behaviour (observed 2026-11-04) -- Infra blackout 2026-11-04 ~04:05→09:54 UTC (whole home-LAN stand powered off, ~5 h 49 m). VM booted 09:54:26 UTC. -- On boot everything self-started: docker daemon (enabled, ActiveEnter 09:56:14 UTC), the bugtrail compose stack and navi.service (tsukiko) — no manual intervention required. -- Health lag: `/api/healthz` unreachable 09:58:46→10:09:28 UTC (connection refused — containers still starting, ~14 min after boot; the server container runs alembic migrations on start), then a short http 502 window from 10:09:58 (~65 s: caddy already listening, server:8000 not yet). Green from ~10:11 UTC. -- Verified 10:11 UTC by rei: 5 consecutive probes → HTTP 200, 4–12 ms, {"status":"ok"}; idempotent `docker compose -f docker-compose.prod.yml up -d` ran exit 0 with no changes. navi.service healthy. -- Lesson: after a power outage expect ~10–15 min before BugTrail health goes green; hard-panel will show down(refused) → 502 → up in that window. Do not force-rebuild. Incident report (on navi host): workspace/incident_infra_blackout_2026-11-04.md. +- 2026-11-04 (rei, verified): after the mass ACPI shutdown of the home infra + (04:07→09:54 UTC) the BugTrail stack did NOT come up immediately — docker + daemon started later than the host; containers were created ~10:11 UTC, + alembic migrations re-applied cleanly, healthz 200 (LAN + public) verified. + GOTCHA: after a host reboot expect `/api/healthz` to return 502 (caddy up, + FastAPI backend down) for ~15–20 min while docker daemon/containers start — + wait before escalating; `docker compose up -d` is idempotent and safe. diff --git a/90-maintenance/pending-changes/20261104101850-86146ee4.json b/90-maintenance/pending-changes/20261104101850-86146ee4.json new file mode 100644 index 0000000..01a4c96 --- /dev/null +++ b/90-maintenance/pending-changes/20261104101850-86146ee4.json @@ -0,0 +1,19 @@ +{ + "id": "20261104101850-86146ee4", + "kind": "doc", + "target": "10-systems/hosts/bugtrail.md", + "summary": "Добавлен gotcha в bugtrail.md: после ребута хоста docker-стек поднимается ~15–20 мин, healthz может отдавать 502 — ждать перед эскалацией", + "reason": "Инцидент 2026-11-04: после массового ребута инфры healthz bugtrail отдавал 502 ~15–20 мин (docker daemon стартует позже хоста); verified rei — стек поднялся сам, gotcha задокументирован, чтобы следующая реакция не эскалировала преждевременно.", + "payload": { + "content": "---\ntitle: VPS bugtrail\ntype: host\nip: 192.168.1.177\npurpose: Navi agent instance (tsukiko) + BugTrail web service in home LAN\nowner: gmikcon\nstatus: active\nconfidence: \"100\"\nlast_reviewed: \"2026-09-18\"\nreview_interval: 90d\nsource_of_truth: ssh-host-and-owner-confirmed\n---\n\n# VPS bugtrail\n\n## Overview\nHome LAN VPS (KVM/QEMU VM). Hosts the navi-1 agent instance **tsukiko**\n(deployed 2026-09-17 from the `deploy` branch of\nhttps://git.gnexus.space/git/root/navi-1.git) and the **BugTrail** web\nservice (deployed 2026-09-18 from\nhttps://git.gnexus.space/git/root/bugtrail.git).\n\n## Technical Details\n- **Local IP:** 192.168.1.177 (primary)\n- **Secondary IP:** 192.168.1.198/24 on the same interface (metric 100, purpose unknown — clarify with owner)\n- **OS:** Ubuntu 22.04.5 LTS (Jammy Jellyfish)\n- **Kernel:** 5.15.0-191-generic\n- **User:** bugtrail\n- **Hostname:** ubuntu (verified via SSH 2026-09-17)\n- **Credentials:** gnexus-creds secret \"VPS bugtrail SSH (192.168.1.177)\" (id d654f5aa-412b-4fe0-87ad-89cc640375bb); sudo uses the same password\n- **Platform:** KVM/QEMU VM (MAC 52:54:00:38:86:8c), 2 vCPU, 3.8 GiB RAM, 31G LVM root disk\n- **Note:** user `bugtrail` is NOT in the docker group — docker commands require sudo\n\n## Navi instance (deployed 2026-09-17)\n- **Instance name:** tsukiko\n- **Instance ID:** c9decfb7-39d1-4bf0-b3b3-c2c0d8d1e6fc\n- **Repo:** /home/bugtrail/navi-1 (branch `deploy`, commit e276b1e)\n- **Service:** navi.service (systemd, enabled, restarts on failure)\n- **Server port:** 8099 (0.0.0.0), health endpoint http://127.0.0.1:8099/health\n- **Postgres:** docker container, 127.0.0.1:5432\n- **Python:** standalone CPython 3.12.14 at /home/bugtrail/navi-1/.python/\n- **Swarm key:** /home/bugtrail/navi-1/.swarm-key (mode 600; value not documented)\n- **Hive:** HIVE_URL=http://192.168.1.168:8087 — announce_ok confirmed in journalctl\n- **Ollama backend:** OLLAMA_HOST=https://ollama.com, model glm-5.3-flash:cloud, API key set in .env (value not documented)\n- **Webclient/auth:** NAVI_WEBCLIENT_ENABLED=false, NAVI_AUTH_ENABLED=false\n- **Install note:** deploy/install.sh calls bare `sudo` (no -S); run it as\n `echo | sudo -S bash deploy/install.sh` over non-interactive SSH.\n The script is idempotent.\n\n## BugTrail service (deployed 2026-09-18)\n- **Repo:** https://git.gnexus.space/git/root/bugtrail.git → /home/bugtrail/bugtrail (main@dfe9226)\n- **What it is:** bug-report handoff service (tester → developer): page-annotation notes, screenshots, FastAPI backend + Vue 3 panel + Chrome/Firefox MV3 extension\n- **Stack:** docker compose (`docker-compose.prod.yml`): `bugtrail-postgres-1` (postgres:16-alpine), `bugtrail-server-1` (FastAPI, python:3.13-slim; alembic migrations auto-apply on container start), `bugtrail-caddy-1` (caddy:2-alpine — static panel + reverse proxy `/api` → server:8000)\n- **Public URL:** https://bugtrail.gnexus.space — TLS terminates on internal proxy 192.168.1.226 with a SEPARATE Let's Encrypt certificate (CN=bugtrail.gnexus.space, expires 2026-12-17); caddy serves plain HTTP on port 80 (`SITE_ADDRESS=:80`)\n- **Port:** 80 (caddy via docker-proxy, all interfaces)\n- **Health:** GET /api/healthz → {\"status\":\"ok\"}\n- **Config:** `.env` next to docker-compose.prod.yml — POSTGRES_PASSWORD (48-hex, chmod 600, value not documented), HTTP_PORT=80, HTTPS_PORT=443, SITE_ADDRESS=:80\n- **Restart policy:** `restart: unless-stopped` on all three services (added 2026-09-18)\n- **Data / backup points:** docker volume `bugtrail_postgres_data` + `./data` (uploaded screenshots)\n- **Build notes:** Node.js v22.23.2 installed via NodeSource (was absent on host); `make` is NOT installed — deploy steps were run manually: `npm ci`, build packages/extension + packages/web, `sudo docker compose -f docker-compose.prod.yml up -d --build`\n- **Deployed:** 2026-09-18 by navi (rei), verified end-to-end (healthz + panel via HTTPS through public entrypoint)\n\n## Network\n- `enp1s0` UP — 192.168.1.177/24 (primary), 192.168.1.198/24 (secondary, metric 100), link-local fe80::5054:ff:fe38:868c/64\n- `docker0` UP — 172.17.0.1/16 (navi-postgres attached); bugtrail compose network `bugtrail_default`\n\n## Listening ports (TCP, verified 2026-09-18)\n- 22 — sshd (all interfaces)\n- 53 — systemd-resolved (localhost only)\n- 80 — bugtrail caddy via docker-proxy (all interfaces)\n- 43225 — containerd API (localhost only)\n- 8099 — navi-server (all interfaces)\n- 5432 — postgres via docker-proxy (127.0.0.1 only, navi-postgres)\n\n## Running services (2026-09-18)\n- navi.service — navi-server (tsukiko instance)\n- docker compose stack `bugtrail`: postgres + server + caddy (restart: unless-stopped)\n- Standard Ubuntu daemons: cron, rsyslog, snapd, systemd-* suite\n\n## Notes\n- Secondary IP .198 purpose still to be confirmed by the owner.\n- Known non-critical warnings in navi journal: MCP server 'navi_ui' connect\n failed (anyio CancelledError), tool_loader.file_error for gmail.py\n (ModuleNotFoundError: html2text), startup.embed_unavailable (embed backend\n 401 — embed.ok=false in /health). Core server and hive announce are healthy.\n- 2026-11-04 (rei, verified): after the mass ACPI shutdown of the home infra\n (04:07→09:54 UTC) the BugTrail stack did NOT come up immediately — docker\n daemon started later than the host; containers were created ~10:11 UTC,\n alembic migrations re-applied cleanly, healthz 200 (LAN + public) verified.\n GOTCHA: after a host reboot expect `/api/healthz` to return 502 (caddy up,\n FastAPI backend down) for ~15–20 min while docker daemon/containers start —\n wait before escalating; `docker compose up -d` is idempotent and safe." + }, + "status": "applied", + "created_at": "2026-11-04T10:18:50Z", + "updated_at": "2026-11-04T10:18:52Z", + "applied_at": "2026-11-04T10:18:52Z", + "validation": { + "status": "ok", + "issue_count": 0, + "issues": [] + } +}