From d3cdbdf3e27850089dfe48b476f4f940b9d18efa Mon Sep 17 00:00:00 2001 From: yukkop Date: Thu, 30 Apr 2026 22:12:18 +0000 Subject: [PATCH] docs(`postgres-hooks`): document hectic bundle + responsibility split Add lib/hook/sql/README.md describing bundle layout, apply order, Nix API (self.lib.hectic.*), shell helper contract, and the steps for adding a new SQL file. Rewrite db-tool README's hectic section: drop stale PG_HECTIC_INHERITANCE / HECTIC_INHERITANCE_SQL env vars, add HECTIC_DOTENV_FILE, document the postgres-init / migrator init / database hydrate responsibility split. --- lib/hook/sql/README.md | 101 ++++++++++++++++++++++++++++++++++++++ package/db-tool/README.md | 75 +++++++++++++++++----------- 2 files changed, 147 insertions(+), 29 deletions(-) create mode 100644 lib/hook/sql/README.md diff --git a/lib/hook/sql/README.md b/lib/hook/sql/README.md new file mode 100644 index 0000000..229ca94 --- /dev/null +++ b/lib/hook/sql/README.md @@ -0,0 +1,101 @@ +# hectic SQL bundle + +Single source of truth for every object created in the `hectic` PostgreSQL +schema. Consumed by: + +- `package/migrator` — applies the bundle on `migrator init` (mandatory). +- `package/db-tool` — applies the bundle in `database hydrate` (default; opt + out with `--no-hook`). +- External consumers (e.g. `proxydoe`) — invoke `psql -f` directly against the + paths exposed via `self.lib.hectic.*.path`. + +## Layout + +| File | Purpose | +| --- | --- | +| `HECTIC_VERSION` | Single version string for the whole bundle (e.g. `0.1.0`). Read via `lib.fileContents`. | +| `hectic-version.sql` | Templated. Creates `hectic.version`, inserts the current `versionString`, raises on mismatch. | +| `hectic-secret.sql` | Creates `hectic.secret`, `hectic.load_secrets_from_env(text)`, `hectic.get_secret(text)`. | +| `hectic-migration.sql` | Creates the `hectic.migration` table and supporting domains/triggers used by `migrator`. | +| `hectic-inheritance.sql` | Creates `hectic.created_at`, `hectic.updated_at`, `hectic.immutable` parent tables and the DDL event triggers that enforce inheritance, attach `BEFORE UPDATE` triggers, and block DML on immutable tables outside `migration_mode`. | + +`hectic-version.sql` is templated at Nix evaluation time: `@HECTIC_VERSION@` +is substituted with the contents of `HECTIC_VERSION`. All other files are +applied verbatim. + +## Apply order + +The bundle MUST be applied in this order (enforced by +`apply-hectic-bundle.sh`): + +1. `hectic-version.sql` — version check first; aborts the rest on mismatch. +2. `hectic-secret.sql` +3. `hectic-migration.sql` +4. `hectic-inheritance.sql` + +Re-applying the bundle is idempotent — every CREATE uses +`IF NOT EXISTS` / `CREATE OR REPLACE`, and the version check accepts a row +that already matches. + +## Nix API (`self.lib.hectic`) + +```nix +self.lib.hectic = { + versionString; # e.g. "0.1.0" + version = { sql; }; # templated + secret = { sql; path; }; + migration = { sql; path; }; + inheritance = { sql; path; }; + applyBundleScript; # ./hook/apply-hectic-bundle.sh +}; +``` + +`.sql` is the file contents as a string. `.path` is the Nix store path of the +verbatim source (only available on non-templated entries; consumers needing a +materialized version of `version.sql` must do +`pkgs.runCommand "hectic-version.sql" { text = self.lib.hectic.version.sql; passAsFile = ["text"]; } ''cp "$textPath" "$out"''`). + +## Shell helper (`apply-hectic-bundle.sh`) + +`lib/hook/apply-hectic-bundle.sh` is a dash-compatible helper sourced by both +`migrator` and `db-tool`. Public entry point: + +```sh +apply_hectic_bundle [] +``` + +- `` — full PostgreSQL connection string. +- `` — optional. When present, after applying the bundle the + helper invokes `hectic.load_secrets_from_env()` inside a + dollar-quoted (`$ps_env$`) string so secret values cannot terminate the + literal. + +Required environment (paths to the SQL files): + +- `HECTIC_VERSION_SQL` +- `HECTIC_SECRET_SQL` +- `HECTIC_MIGRATION_SQL` +- `HECTIC_INHERITANCE_SQL` + +`migrator` and `db-tool` set these via Nix at build time. External consumers +typically invoke `psql -f` against the paths directly instead of sourcing the +helper. + +## Adding a new SQL file + +1. Add `lib/hook/sql/hectic-.sql`. +2. Wire it into `lib/default.nix` under `lib.hectic.`. +3. Inject `HECTIC__SQL` in both `package/migrator/default.nix` and + `package/db-tool/default.nix`. +4. Append a `psql -f "$HECTIC__SQL"` step to + `lib/hook/apply-hectic-bundle.sh` in the correct order. +5. Bump `HECTIC_VERSION` if the new content changes existing semantics. +6. Update tests in `test/package/migrator/test/postgresql/init-hectic-bundle/` + and `test/package/db-tool/test/postgresql/hydrate-hook/`. + +## Versioning + +`HECTIC_VERSION` is a single global version for the bundle, not per-file. +Bump it on any breaking change to the schema. `hectic-version.sql` raises an +exception when the database row diverges from the bundle version, forcing a +deliberate migration before the rest of the bundle runs. diff --git a/package/db-tool/README.md b/package/db-tool/README.md index 71f8d3d..3f74185 100644 --- a/package/db-tool/README.md +++ b/package/db-tool/README.md @@ -32,8 +32,7 @@ These variables must be set for `db-tool` to function. | `PG_CONF_FILE` | (unset) | Path to a `postgresql.conf` file. When set, replaces the script-generated config entirely on fresh init. `port` and `unix_socket_directories` are still appended at runtime (always overridden). When set, `PG_DISABLE_LOGGING` and `PG_SHARED_PRELOAD_LIBRARIES` are ignored. | | `PG_SHARED_PRELOAD_LIBRARIES` | `pg_cron` | Comma-separated `shared_preload_libraries` value. Set to empty string to disable. Ignored when `PG_CONF_FILE` is set. | | `PG_DISABLE_LOGGING` | `0` | Set to `1` to disable PostgreSQL logging collector. Ignored when `PG_CONF_FILE` is set. | -| `PG_HECTIC_INHERITANCE` | `1` | Apply the [`hectic` inheritance bundle](#hectic-inheritance-bundle) to the target database after init. Set to `0` to disable. | -| `HECTIC_INHERITANCE_SQL` | (auto) | Override path to the SQL file applied by `PG_HECTIC_INHERITANCE=1`. Defaults to the SQL shipped with `postgres-init`. | +| `HECTIC_DOTENV_FILE` | (unset) | Optional dotenv file. When set and readable, `database hydrate` passes its contents to `hectic.load_secrets_from_env(...)` after applying the bundle. Falls back to `${LOCAL_DIR}/.env.${ENVIRONMENT}` when unset. | | `PATCH_LOG` | (stdout) | Path to log the output of database patches. | | `HYDRATE_LOG` | (stdout) | Path to log the output of database hydration. | @@ -109,24 +108,36 @@ To use `db-tool` in a Nix development shell, add the following to your `flake.ni } ``` -## hectic Inheritance Bundle +## hectic Bundle -`pkgs.hectic.hectic-inheritance` ships a SQL artifact that bootstraps a `hectic` -schema with three parent tables and DDL event triggers: +`db-tool` and `migrator` apply a single bundle of SQL files that bootstrap the +`hectic` schema. The bundle lives in +[`lib/hook/sql/`](../../lib/hook/sql/README.md) — see that README for full +contract, file layout, and the `self.lib.hectic.*` Nix API. + +The bundle creates: + +- `hectic.version` — single version row for the entire hectic system. + Mismatch between database and bundle raises an exception. +- `hectic.secret` + `hectic.load_secrets_from_env(text)` + + `hectic.get_secret(text)` — encrypted secret storage and dotenv loader. +- `hectic.migration` — table consumed by `migrator`. +- `hectic.created_at` / `hectic.updated_at` / `hectic.immutable` parent tables + and the DDL event triggers that enforce inheritance, attach + `BEFORE UPDATE` triggers, and block DML on immutable tables outside + `migration_mode`. + +Inheritance details: - `hectic.created_at(created_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — every user table must `INHERITS (hectic.created_at)`. The event trigger - `hectic_enforce_created_at_inheritance` raises an exception on `CREATE TABLE` - otherwise. -- `hectic.updated_at(updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — optional. - Any table that inherits from it automatically gets a `BEFORE UPDATE FOR EACH - ROW` trigger calling `hectic.set_updated_at()` attached by - `hectic_attach_updated_at_trigger`. -- `hectic.immutable()` — pure marker. Tables inheriting it are blocked from - `INSERT`/`UPDATE`/`DELETE`/`TRUNCATE` outside migration mode by triggers - attached by `hectic_attach_immutable_triggers`. Useful for reference data - that must only change via migrations. To allow DML inside a migration, wrap - it in a transaction: + `hectic_enforce_created_at_inheritance` raises on `CREATE TABLE` otherwise. +- `hectic.updated_at(updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — + optional. Any inheriting table automatically gets a + `BEFORE UPDATE FOR EACH ROW` trigger calling `hectic.set_updated_at()`. +- `hectic.immutable()` — pure marker. Inheriting tables are blocked from + `INSERT`/`UPDATE`/`DELETE`/`TRUNCATE` outside migration mode. To allow DML + inside a migration, wrap it in a transaction: ```sql BEGIN; @@ -148,6 +159,16 @@ Per-database opt-out for additional schemas via the ALTER DATABASE mydb SET hectic.inheritance_extra_excluded_schemas = 'legacy,etl'; ``` +### Responsibility split + +| Component | Applies bundle? | +| --- | --- | +| `postgres-init` | **No.** Pure PostgreSQL provisioner — starts a vanilla cluster, nothing more. | +| `migrator init` | **Yes, mandatory.** The bundle is a hard prerequisite for `hectic.migration`. | +| `database hydrate` | **Yes, by default.** Re-applied on every hydrate. Skip with `--no-hook`. After applying the bundle, hydrate also calls `hectic.load_secrets_from_env()` if `HECTIC_DOTENV_FILE` (or `${LOCAL_DIR}/.env.${ENVIRONMENT}`) is readable. | + +The bundle is idempotent — repeated application is safe. + ### `db-tool diff` and immutable tables `database diff` already includes immutable tables in its schema-level @@ -158,25 +179,21 @@ diff of the rows of every table inheriting `hectic.immutable`. Drift in "frozen" reference data therefore surfaces in the same pager view as schema drift, and the subcommand exits non-zero when either differs. -### Apply via `postgres-init` +### Apply manually via `psql` -Applied automatically. Set `PG_HECTIC_INHERITANCE=0` to opt out. - -### Apply via `migrator` or any psql pipeline +For external consumers (e.g. NixOS modules) bypass the helper and call `psql` +directly against the paths exposed by `self.lib.hectic.*.path`: ```nix -# in your devshell -shellHook = '' - export HECTIC_INHERITANCE_SQL=${pkgs.hectic.hectic-inheritance}/share/hectic/hectic-inheritance.sql +services.postgresql.initialScript = pkgs.writeText "hectic-init.sql" '' + \i ${self.lib.hectic.secret.path} + \i ${self.lib.hectic.migration.path} + \i ${self.lib.hectic.inheritance.path} ''; ``` -```sh -psql "$DB_URL" -v ON_ERROR_STOP=1 -f "$HECTIC_INHERITANCE_SQL" -``` - -The SQL is also exposed via `self.lib.hecticInheritance.sql` (string) and -`self.lib.hecticInheritance.path` (Nix path) for inline pipelines. +The version file (`self.lib.hectic.version`) is templated and only exposes +`.sql` (a string). Materialize it with `pkgs.writeText` if a path is needed. ## Exit Codes