diff --git a/00-overview/postgres-migration-roadmap.md b/00-overview/postgres-migration-roadmap.md index 3dec662..be3e24f 100644 --- a/00-overview/postgres-migration-roadmap.md +++ b/00-overview/postgres-migration-roadmap.md @@ -468,14 +468,18 @@ | 2026-11-19 | Phase 1: PostgreSQL foundation | Dev cluster on `127.0.0.1:5433`; SQLAlchemy models; Alembic initial migration; database-backed read repositories; importer v0 | | 2026-11-19 | Phase 1: import and parity | 29 documents, 84 inventory items, 12 inventory types, 13 schemas imported; parity check reports 0 differences | | 2026-11-19 | Phase 1: backups | `scripts/backup-db.sh` produced a custom-format dump on the off-host NFS; `scripts/restore-db.sh` restored it into a temporary database with matching counts | +| 2026-11-19 | Phase 2: versioned writes | Database-backed draft/apply changesets; immutable document and inventory versions; transactional validation with rollback; audit entries | +| 2026-11-19 | Phase 2: history, diff, restore | API and MCP expose `document_history`, `document_version`, `document_diff`, and `restore_document_version`; restore creates a new version | +| 2026-11-19 | Phase 2: validation | `GET /validate` and the MCP validation tool validate the database in `db` mode | +| 2026-11-19 | Phase 2: tests | Full suite: 56 passed, 1 known stale-freshness failure | Notes: -- The default backend remains `file`. Set `GNEXUS_BOOK_STORAGE_BACKEND=db` to serve - reads from PostgreSQL. -- Document writes are blocked in database mode (the API and MCP return a clear error) - and still use the file backend until Phase 2. Validation and freshness still read - files until Phase 3. +- The default backend remains `file`. Set `GNEXUS_BOOK_STORAGE_BACKEND=db` to use + PostgreSQL for reads and writes. +- In database mode, applying a changeset publishes it; `commit_changes` is a no-op and + `POST /commit` is blocked. Freshness still reads files until Phase 3. - The date-dependent freshness test still fails because 25 documents are past their review interval. This is the recorded baseline, not a regression. -- The next phase is Phase 2: versioned documents and changesets. +- The next phase is Phase 3: inventory, relationships, search, and freshness at + database parity. diff --git a/CLAUDE.md b/CLAUDE.md index b530928..97c4c63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -95,7 +95,7 @@ - **`validation.py`** — the full-repo gate used before apply/commit: schemas parse as Draft 2020-12, every inventory file validates against its schema (`SCHEMA_BY_INVENTORY`), Markdown has required frontmatter, ids are unique per inventory file, and a regex scan detects raw secret assignments/private-key blocks. The separate inventory `docs`-link validator (`_validate_inventory_doc_links`) is intentionally disabled; freshness still reports those missing targets. - **`pending_changes.py`** — the write path. A proposal is a JSON file in `90-maintenance/pending-changes/`; `apply()` writes a Markdown doc or patches/creates an inventory item, runs whole-repo validation, and restores the previous file content if validation fails. Only `doc` and `inventory-item` kinds can be applied; full-file inventory replacement is not implemented. - **`git_adapter.py`** — status/diff/commit. Commit requires an explicit file list, rejects absolute/traversal/denied paths, runs validation first, stages only the named files, and creates a local commit only (never pushes). -- **`storage/`** — the PostgreSQL layer introduced for the migration. `models.py` defines the schema (documents/versions/blobs, inventory items/versions, changesets, users, audit), `db_repositories.py` implements the same read interfaces as the file repositories, `importer.py` copies the current repository into the database, `parity.py` compares both backends, and `cli.py` exposes `import`, `parity`, and `counts`. `factory.py` chooses the backend from `GNEXUS_BOOK_STORAGE_BACKEND`. Alembic lives in `server/alembic/`. Reads can use either backend; in `db` mode write endpoints/tools are blocked until Phase 2, and validation/freshness still read files until Phase 3. +- **`storage/`** — the PostgreSQL layer introduced for the migration. `models.py` defines the schema (documents/versions/blobs, inventory items/versions, changesets, users, audit), `db_repositories.py` implements the read interfaces, `pending_changes_db.py` implements draft/apply changesets transactionally, `documents.py` provides history/version/diff/restore, `validation.py` validates the database, `importer.py` copies the repository into PostgreSQL, `parity.py` compares both backends, and `cli.py` exposes `import`, `parity`, and `counts`. `factory.py` selects docs/inventory/pending repositories from `GNEXUS_BOOK_STORAGE_BACKEND`. Alembic lives in `server/alembic/`. In `db` mode all knowledge writes go through changesets; `commit_changes` is a no-op. Freshness still reads files until Phase 3. ### MCP server @@ -120,6 +120,6 @@ Tests run from `server/` with `pythonpath = ["."]`. Most tests read the real repository, so they assert on real inventory/docs content; fixture-based tests use `Settings(tmp_path)` and copy schemas from the real repo. -Database tests use `GNEXUS_BOOK_TEST_DATABASE_URL` (default `postgresql+psycopg://gmikcon@127.0.0.1:5433/gnexus_book_test`) and skip when PostgreSQL is unavailable. `tests/test_migrations.py` applies the Alembic migration to a temporary database. +Database tests use `GNEXUS_BOOK_TEST_DATABASE_URL` (default `postgresql+psycopg://gmikcon@127.0.0.1:5433/gnexus_book_test`) and skip when PostgreSQL is unavailable. `tests/test_db_storage.py` covers read parity, `tests/test_db_changesets.py` covers versioning/changesets/history/diff/restore, and `tests/test_migrations.py` applies the Alembic migration to a temporary database. `tests/test_docs_repository.py::test_freshness_report_is_clean_for_knowledge_docs` is date-dependent: it fails once documents pass their `review_interval`. When it fails, refresh `last_reviewed` only for docs whose content was actually reviewed, or treat the failure as genuine staleness. diff --git a/server/README.md b/server/README.md index cc0b97a..45de8a2 100644 --- a/server/README.md +++ b/server/README.md @@ -14,7 +14,7 @@ uvicorn app.main:app --reload --host 127.0.0.1 --port 8080 ``` -## PostgreSQL storage (Phase 1) +## PostgreSQL storage The read path can be served from PostgreSQL instead of files: @@ -36,9 +36,13 @@ database read results and exits non-zero on a difference. - `scripts/backup-db.sh` writes a custom-format dump to the off-host NFS by default; `scripts/restore-db.sh` requires `GNEXUS_BOOK_RESTORE_YES=1` before replacing data. -- Phase 1 scope: documents, inventory, and relationships are read from PostgreSQL. - Document writes are blocked in database mode until Phase 2; validation and freshness - still read files until Phase 3. +- Documents and inventory are read and written through PostgreSQL. Every write creates + immutable versions grouped into a changeset; applying a changeset publishes it. + History, version reads, diff, and restore are available in database mode. +- Validation uses the database when the backend is `db`. Freshness still reads files + until Phase 3. +- In database mode `POST /commit` is blocked: applying a changeset is the publish step. + `commit_changes` in MCP is a documented no-op. - Database tests use `GNEXUS_BOOK_TEST_DATABASE_URL` (default `postgresql+psycopg://gmikcon@127.0.0.1:5433/gnexus_book_test`) and are skipped when PostgreSQL is unavailable. @@ -79,6 +83,10 @@ - `apply_pending_change(change_id)` - `commit_changes(summary, files, details?)` - `update_doc(path, content, summary, reason?)` +- `document_history(path)` +- `document_version(path, version_no)` +- `document_diff(path, from_version?, to_version?)` +- `restore_document_version(path, version_no, summary?)` Agent usage policy: @@ -107,6 +115,10 @@ - `GET /health` - `GET /docs` - `GET /docs/read?path=...` +- `GET /documents/{path}/history` +- `GET /documents/{path}/versions/{version_no}` +- `GET /documents/{path}/diff?from=&to=` +- `POST /documents/{path}/restore/{version_no}?summary=...` - `GET /search?q=...` - `GET /inventory` - `GET /inventory/{type}` @@ -127,12 +139,14 @@ ## Current Limitations -- `POST /changes` can create pending change records under `90-maintenance/pending-changes/`. -- `POST /changes/{id}/apply` can apply `kind=doc` and `kind=inventory-item` changes after validation. +- `POST /changes` creates pending change records in the file backend or changesets in the database backend. +- `POST /changes/{id}/apply` applies `kind=doc` and `kind=inventory-item` changes transactionally after validation. +- Database mode publishes on apply; `POST /commit` is only available with the file backend. - No authentication yet. -- `POST /commit` creates a local Git commit only. It does not push. +- File-backend `POST /commit` creates a local Git commit only. It does not push. - Commit requests must provide an explicit file list. -- Validation uses JSON Schema 2020-12 for inventory files. +- Validation uses JSON Schema 2020-12 for inventory records in both backends. +- Freshness reports still read the file backend until Phase 3. ## Validation