diff --git a/00-overview/postgres-migration-roadmap.md b/00-overview/postgres-migration-roadmap.md index 452358e..98bb3bf 100644 --- a/00-overview/postgres-migration-roadmap.md +++ b/00-overview/postgres-migration-roadmap.md @@ -501,6 +501,10 @@ | 2026-11-19 | Review fixes: data/audit | Secret scan covers every document version; actor user IDs stored in changesets and audit; concurrent changeset apply serialized | | 2026-11-19 | Review fixes: config/runtime | `GNEXUS_BOOK_REPO_ROOT` honored by Settings; webhook database work moved off the event loop; stale README limitation removed | | 2026-11-19 | Review fixes: tests | Full suite: 88 passed, 1 known stale-freshness failure; file/DB parity 0 | +| 2026-11-19 | Phase 8: backups and drills | Daily off-host backup timer, retention, checksum and failure alerts; restore-drill script and optional monthly timer | +| 2026-11-19 | Phase 8: health/monitoring | Healthcheck script, public `/health` contract, Synapse alert plumbing; hard-panel registration and Synapse source remain owner actions | +| 2026-11-19 | Phase 8: deploy/docs | `deploy.sh` applies migrations and refreshes units; operations and decommission runbooks; service page updated | +| 2026-11-19 | Phase 8: decommission | Blocked on production cutover and owner sign-off; file backend retained for rollback | Notes: @@ -537,5 +541,10 @@ tests. Production deployment must set `GNEXUS_BOOK_MCP_ALLOWED_HOSTS` to include the public MCP hostname and `GNEXUS_BOOK_MCP_ALLOWED_ORIGINS` to include its HTTPS origin, because FastMCP rejects unknown Host headers with `421`. -- The next phase is Phase 8: operations hardening, monitoring, documentation, and - decommissioning the file write path after the production cutover. +- Phase 8 operations are implemented locally: scheduled backups, restore drill, + healthcheck and Synapse alert plumbing, deploy migrations and runbooks. The actual + hard-panel probe and Synapse source/alert route still require owner credentials. +- Phase 8 decommission is not executed. The file backend, Git adapter and original + repository remain until the production cutover has run for the agreed stability + window and the owner signs off + [decommission-file-backend.md](../30-runbooks/decommission-file-backend.md). diff --git a/10-systems/applications/gnexus-book.md b/10-systems/applications/gnexus-book.md index 799a8cf..6597d75 100644 --- a/10-systems/applications/gnexus-book.md +++ b/10-systems/applications/gnexus-book.md @@ -1,7 +1,7 @@ --- owner: gmikcon status: active -last_reviewed: 2026-05-10 +last_reviewed: 2026-11-19 review_interval: 30d confidence: high source_of_truth: repository @@ -9,31 +9,73 @@ # Gnexus Book -Gnexus Book is the canonical infrastructure knowledge base and maintenance API for the personal digital and server infrastructure. +Gnexus Book is the canonical infrastructure knowledge base and maintenance API for the +personal digital and server infrastructure. -## Repository +## Access + +- UI and REST: gnexus-auth SSO when authentication is configured; otherwise the + local-development installation runs open. +- Agents: personal `mcp_*` keys over streamable HTTP at `/mcp-protocol/`. +- Service-to-service key provisioning: `/internal/keys`, protected by a service secret, + not exposed through nginx. +- Secrets are references only; no raw credentials are documented here. + +## Repository and storage - Local path: `/home/gmikcon/Projects/gnexus-book`. -- Source of truth: Markdown, YAML inventory, JSON Schema, and Git history. +- Canonical store: PostgreSQL (`gnexus_book`), with immutable document and inventory + versions grouped into changesets. +- Git remains the code repository and the rollback source until the file backend is + decommissioned. -## Runtime Components +## Runtime components - Backend: FastAPI under `server/`. -- Agent interface: MCP server under `server/app/mcp_server.py`. -- UI: Vue/Vite under `ui/`. -- UI dependency: `gnexus-ui-kit`. +- MCP: `server/app/mcp_http.py`, streamable HTTP at `/mcp-protocol/`. +- UI: Vue/Vite under `ui/`, using `gnexus-ui-kit`. +- Auth: gnexus-auth SSO plus per-user service keys. -## Current Local Endpoints +## Local endpoints - Backend API: `http://127.0.0.1:3005/`. - API docs: `http://127.0.0.1:3005/api-docs`. - UI: `http://127.0.0.1:3006/`. -- LAN UI: `http://192.168.1.75:3006/`. +- MCP: `http://127.0.0.1:8001/mcp-protocol/`. -## Maintenance Model +Production URLs and the public MCP hostname are set at deployment. -Agents should prefer structured inventory changes, validation, pending changes, and local commits through the maintenance API. +## Deployment -For agent-facing MCP usage rules, see [../../90-maintenance/agent-mcp-usage.md](../../90-maintenance/agent-mcp-usage.md). +- `scripts/install.sh` installs dependencies and user units. +- `scripts/deploy.sh` pulls, builds, applies database migrations when the `db` + backend is active, refreshes units and restarts backend, MCP and frontend. +- Runtime configuration: `server/.env` (`chmod 600`, gitignored) and gnexus-creds. +- systemd user units: backend, frontend, MCP, daily backup, optional monthly restore + drill. -No raw credentials are documented here. +## Backups and recovery + +- Daily off-host `pg_dump` through `gnexus-book-backup.timer`; retention is configured + by `GNEXUS_BOOK_BACKUP_KEEP`. +- Restore procedure and periodic drill: `30-runbooks/postgres-operations.md`. +- Backup failures and failed health probes alert through Gnexus Synapse when a + `syn_*` source key is configured. + +## Monitoring + +- `GET /health` follows the platform contract (`status`, `checks.db`, service, + version, timestamp); it is public and contains no personal data. +- hard-panel probes the same endpoint; alerts are routed through Synapse. +- Local check: `scripts/healthcheck.sh`. + +## Maintenance model + +Agents should prefer structured inventory changes, validation, draft changesets and +apply/publish through the maintenance API or MCP. For MCP rules see +[../../90-maintenance/agent-mcp-usage.md](../../90-maintenance/agent-mcp-usage.md). + +The PostgreSQL cutover is complete locally. Production cutover and the removal of the +file write path remain separate deployment actions; see +[../../30-runbooks/cutover-postgres.md](../../30-runbooks/cutover-postgres.md) and +[../../30-runbooks/decommission-file-backend.md](../../30-runbooks/decommission-file-backend.md). diff --git a/30-runbooks/decommission-file-backend.md b/30-runbooks/decommission-file-backend.md new file mode 100644 index 0000000..eadefe2 --- /dev/null +++ b/30-runbooks/decommission-file-backend.md @@ -0,0 +1,57 @@ +--- +owner: gmikcon +status: draft +last_reviewed: 2026-11-19 +review_interval: 90d +confidence: high +source_of_truth: migration-runbook +--- + +# Decommission the file backend + +Remove the file write path and Git adapter after the production cutover is accepted. +**Not executed yet.** The file backend remains available for rollback and for tests +until the prerequisites below are met. + +## Prerequisites + +- Production has run on `GNEXUS_BOOK_STORAGE_BACKEND=db` for an agreed stability + window with no unresolved data incidents. +- The pre-cutover dump and a fresh post-cutover dump have both passed restore drills. +- `validate_repository` and `validate_database` are clean. +- The parity suite reports zero differences. +- The owner has signed off the decommission in writing. + +## Steps + +1. Archive the original Git repository: + - create a `git bundle` of all refs and a worktree archive; + - copy both off-host and verify SHA-256 and a restore. +2. Keep the last pre-decommission PostgreSQL dump off-host with the agreed retention. +3. Update defaults so tests and development no longer depend on the file backend. +4. Remove the file write path: + - `DocsRepository`, `InventoryRepository` write paths where no longer needed; + - `GitAdapter` commit path and `/commit` routing; + - file-specific pending changes (`90-maintenance/pending-changes/*.json`). +5. Retarget the test suite to the database backend and remove file-parity fixtures + that no longer have a reference implementation. +6. Remove unused configuration (`GNEXUS_BOOK_STORAGE_BACKEND=file` branches) and + dependencies. +7. Update the service page, README, CLAUDE.md and this runbook. +8. Run the full test suite, validation, parity checks that still make sense, and a + restore drill. + +## Rollback + +Until the Git archive and the pre-decommission dump are retired: + +- restore the service from the last dump if the database is damaged; +- restore the file backend from the Git archive if the database path must be + abandoned entirely. + +After both artifacts are retired, rollback is no longer possible. + +## Status + +Blocked on the production cutover and owner sign-off. No file-backend code has been +removed. diff --git a/30-runbooks/postgres-operations.md b/30-runbooks/postgres-operations.md new file mode 100644 index 0000000..c3c85d6 --- /dev/null +++ b/30-runbooks/postgres-operations.md @@ -0,0 +1,94 @@ +--- +owner: gmikcon +status: active +last_reviewed: 2026-11-19 +review_interval: 90d +confidence: high +source_of_truth: repository +--- + +# PostgreSQL operations + +Backups, restore drills, health checks and migrations for the database-backed +Gnexus Book. + +## Backups + +- Unit: `systemd/gnexus-book-backup.service`, scheduled daily by + `gnexus-book-backup.timer` (`Persistent=true`, randomized delay up to 15 minutes). +- Script: `scripts/backup-db.sh`. + - custom-format `pg_dump` to `GNEXUS_BOOK_BACKUP_DIR` (off-host by default); + - `pg_restore --list` sanity check and SHA-256 checksum; + - keeps `GNEXUS_BOOK_BACKUP_KEEP` dumps (default 14); + - refuses to run if the off-host mount is unavailable and no explicit directory + is configured; + - on failure, sends a Synapse alert when `SYNAPSE_URL` and `SYNAPSE_API_KEY` are + configured. +- Manual run: + + ```bash + set -a; . server/.env; set +a + ./scripts/backup-db.sh + ``` + +A backup that has never been restored does not count. Run the restore drill below. + +## Restore drill + +```bash +set -a; . server/.env; set +a +./scripts/restore-drill.sh # latest dump +./scripts/restore-drill.sh /path/to.dump # specific dump +``` + +The script creates a temporary database, restores the dump, runs database validation +and count checks, prints the result and drops the temporary database. It requires a +database role with `CREATEDB`. + +`gnexus-book-restore-drill.timer` is installed but not enabled by default. Enable it +after the first successful manual drill: + +```bash +systemctl --user enable --now gnexus-book-restore-drill.timer +``` + +The monthly timer uses the same script and alerts through Synapse on failure. + +## Health and monitoring + +- `GET /health` is public, cheap and returns `status`, `service`, `version`, `storage`, + `timestamp` and `checks.db`. It exposes no users, secrets or topology. +- Monitor URL for hard-panel: `GNEXUS_BOOK_HEALTH_URL` (default + `http://127.0.0.1:8000/health`). +- Local probe: + + ```bash + ./scripts/healthcheck.sh + ``` + + It fails on a non-`ok` status or a failed `checks.db`, and sends a Synapse alert when + configured. +- Register the probe in hard-panel and create the alert route in Synapse. The book + service does not own notification channels; Synapse routing decides recipients. + +## Migrations and deploys + +`scripts/deploy.sh` sources `server/.env`; when +`GNEXUS_BOOK_STORAGE_BACKEND=db`, it runs `alembic upgrade head` before restarting the +backend and MCP services. It also refreshes the systemd user units. + +## Secrets + +- Runtime values live in `server/.env` (`chmod 600`, gitignored) and in gnexus-creds. +- The Synapse source key (`SYNAPSE_API_KEY`) is a `syn_*` key, never a user key. +- Never commit database URLs, OAuth client secrets, webhook secrets or API keys. + +## Quick checks + +```bash +systemctl --user status gnexus-book-backend.service gnexus-book-mcp.service +systemctl --user list-timers 'gnexus-book-*' +./scripts/healthcheck.sh +psql -h 127.0.0.1 -p 5433 -U gmikcon -d gnexus_book -c \ + "select count(*) from documents;" +``` diff --git a/CLAUDE.md b/CLAUDE.md index 66ddfd9..e137e85 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -73,8 +73,17 @@ # Local stack with database storage and gnexus-auth (loads server/.env) ../scripts/dev-server.sh ../scripts/dev-mcp.sh + +# Operations +../scripts/backup-db.sh +../scripts/restore-drill.sh +GNEXUS_BOOK_HEALTH_URL=http://127.0.0.1:3005/health ../scripts/healthcheck.sh ``` +`gnexus-book-backup.timer` runs the off-host dump daily; the restore-drill timer is +installed but disabled until a manual drill passes. Synapse alerts use +`SYNAPSE_URL`/`SYNAPSE_API_KEY`. See `30-runbooks/postgres-operations.md`. + `server/.env` is gitignored and holds the local database URL, OAuth client, webhook secret and internal key secret; systemd loads it through `EnvironmentFile`. Local SSO details are in `30-runbooks/local-auth-testing.md`. diff --git a/README.md b/README.md index 55d9942..55f93e9 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,8 @@ - **Backend** (FastAPI): `systemctl --user status gnexus-book-backend.service` — listens on `0.0.0.0:8000` - **Frontend** (nginx): served on port `80` from `/var/www/gnexus-book` -- **MCP Server** (SSE): `systemctl --user status gnexus-book-mcp.service` — listens on `0.0.0.0:8001`, endpoint `/sse` +- **MCP Server** (streamable HTTP): `systemctl --user status gnexus-book-mcp.service` — listens on `0.0.0.0:8001`, endpoint `/mcp-protocol/` +- **Database**: PostgreSQL; daily off-host backups run through `gnexus-book-backup.timer` ([operations](30-runbooks/postgres-operations.md)) ### Update / redeploy diff --git a/server/README.md b/server/README.md index 846673a..52f49ad 100644 --- a/server/README.md +++ b/server/README.md @@ -71,6 +71,25 @@ `EnvironmentFile`, and the Git checkout remains untouched as the rollback source. See `../30-runbooks/cutover-postgres.md` for the full procedure. +### Operations + +- `scripts/backup-db.sh` writes an off-host custom dump, verifies it, applies + retention and alerts through Synapse on failure when `SYNAPSE_URL` / + `SYNAPSE_API_KEY` are configured. +- `gnexus-book-backup.timer` runs the backup daily; the monthly + `gnexus-book-restore-drill.timer` is installed but disabled until a manual drill + succeeds. +- `scripts/restore-drill.sh [dump]` restores into a temporary database, validates it + and drops it. +- `scripts/healthcheck.sh` probes `GNEXUS_BOOK_HEALTH_URL` (default + `http://127.0.0.1:8000/health`) and checks `status` and `checks.db`. +- `scripts/notify-synapse.sh` sends a Synapse event envelope; it is used by the backup, + restore-drill and healthcheck failure paths. +- `deploy.sh` sources `server/.env`, runs `alembic upgrade head` when the `db` backend + is active, refreshes systemd units and restarts backend, MCP and frontend. + +Full procedure: `../30-runbooks/postgres-operations.md`. + ## Authentication (gnexus-auth) Authentication is disabled unless the gnexus-auth client is configured. When the