hectic-lab/util.nix
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(postgres-hooks): document hectic bundle + responsibility split

Browse Source 
Add lib/hook/sql/README.md describing bundle layout, apply order, Nix API
(self.lib.hectic.*), shell helper contract, and the steps for adding a new
SQL file. Rewrite db-tool README's hectic section: drop stale
PG_HECTIC_INHERITANCE / HECTIC_INHERITANCE_SQL env vars, add
HECTIC_DOTENV_FILE, document the postgres-init / migrator init / database
hydrate responsibility split.
This commit is contained in:
yukkop
2026-04-30 22:12:18 +00:00
parent 1a209f6960
commit d3cdbdf3e2
2 changed files with 147 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

101
lib/hook/sql/README.md Normal file
View File

@@ -0,0 +1,101 @@
# hectic SQL bundle
Single source of truth for every object created in the `hectic` PostgreSQL
schema. Consumed by:
- `package/migrator` — applies the bundle on `migrator init` (mandatory).
- `package/db-tool` — applies the bundle in `database hydrate` (default; opt
out with `--no-hook`).
- External consumers (e.g. `proxydoe`) — invoke `psql -f` directly against the
paths exposed via `self.lib.hectic.*.path`.
## Layout
| File | Purpose |
| --- | --- |
| `HECTIC_VERSION` | Single version string for the whole bundle (e.g. `0.1.0`). Read via `lib.fileContents`. |
| `hectic-version.sql` | Templated. Creates `hectic.version`, inserts the current `versionString`, raises on mismatch. |
| `hectic-secret.sql` | Creates `hectic.secret`, `hectic.load_secrets_from_env(text)`, `hectic.get_secret(text)`. |
| `hectic-migration.sql` | Creates the `hectic.migration` table and supporting domains/triggers used by `migrator`. |
| `hectic-inheritance.sql` | Creates `hectic.created_at`, `hectic.updated_at`, `hectic.immutable` parent tables and the DDL event triggers that enforce inheritance, attach `BEFORE UPDATE` triggers, and block DML on immutable tables outside `migration_mode`. |
`hectic-version.sql` is templated at Nix evaluation time: `@HECTIC_VERSION@`
is substituted with the contents of `HECTIC_VERSION`. All other files are
applied verbatim.
## Apply order
The bundle MUST be applied in this order (enforced by
`apply-hectic-bundle.sh`):
1. `hectic-version.sql` — version check first; aborts the rest on mismatch.
2. `hectic-secret.sql`
3. `hectic-migration.sql`
4. `hectic-inheritance.sql`
Re-applying the bundle is idempotent — every CREATE uses
`IF NOT EXISTS` / `CREATE OR REPLACE`, and the version check accepts a row
that already matches.
## Nix API (`self.lib.hectic`)
```nix
self.lib.hectic = {
versionString; # e.g. "0.1.0"
version = { sql; }; # templated
secret = { sql; path; };
migration = { sql; path; };
inheritance = { sql; path; };
applyBundleScript; # ./hook/apply-hectic-bundle.sh
};
```
`.sql` is the file contents as a string. `.path` is the Nix store path of the
verbatim source (only available on non-templated entries; consumers needing a
materialized version of `version.sql` must do
`pkgs.runCommand "hectic-version.sql" { text = self.lib.hectic.version.sql; passAsFile = ["text"]; } ''cp "$textPath" "$out"''`).
## Shell helper (`apply-hectic-bundle.sh`)
`lib/hook/apply-hectic-bundle.sh` is a dash-compatible helper sourced by both
`migrator` and `db-tool`. Public entry point:
```sh
apply_hectic_bundle <PGURL> [<DOTENV_CONTENT>]
```
- `<PGURL>` — full PostgreSQL connection string.
- `<DOTENV_CONTENT>` — optional. When present, after applying the bundle the
helper invokes `hectic.load_secrets_from_env(<dotenv>)` inside a
dollar-quoted (`$ps_env$`) string so secret values cannot terminate the
literal.
Required environment (paths to the SQL files):
- `HECTIC_VERSION_SQL`
- `HECTIC_SECRET_SQL`
- `HECTIC_MIGRATION_SQL`
- `HECTIC_INHERITANCE_SQL`
`migrator` and `db-tool` set these via Nix at build time. External consumers
typically invoke `psql -f` against the paths directly instead of sourcing the
helper.
## Adding a new SQL file
1. Add `lib/hook/sql/hectic-<name>.sql`.
2. Wire it into `lib/default.nix` under `lib.hectic.<name>`.
3. Inject `HECTIC_<NAME>_SQL` in both `package/migrator/default.nix` and
`package/db-tool/default.nix`.
4. Append a `psql -f "$HECTIC_<NAME>_SQL"` step to
`lib/hook/apply-hectic-bundle.sh` in the correct order.
5. Bump `HECTIC_VERSION` if the new content changes existing semantics.
6. Update tests in `test/package/migrator/test/postgresql/init-hectic-bundle/`
and `test/package/db-tool/test/postgresql/hydrate-hook/`.
## Versioning
`HECTIC_VERSION` is a single global version for the bundle, not per-file.
Bump it on any breaking change to the schema. `hectic-version.sql` raises an
exception when the database row diverges from the bundle version, forcing a
deliberate migration before the rest of the bundle runs.

75
package/db-tool/README.md
View File

@@ -32,8 +32,7 @@ These variables must be set for `db-tool` to function.
| `PG_CONF_FILE` | (unset) | Path to a `postgresql.conf` file. When set, replaces the script-generated config entirely on fresh init. `port` and `unix_socket_directories` are still appended at runtime (always overridden). When set, `PG_DISABLE_LOGGING` and `PG_SHARED_PRELOAD_LIBRARIES` are ignored. | | `PG_CONF_FILE` | (unset) | Path to a `postgresql.conf` file. When set, replaces the script-generated config entirely on fresh init. `port` and `unix_socket_directories` are still appended at runtime (always overridden). When set, `PG_DISABLE_LOGGING` and `PG_SHARED_PRELOAD_LIBRARIES` are ignored. |
| `PG_SHARED_PRELOAD_LIBRARIES` | `pg_cron` | Comma-separated `shared_preload_libraries` value. Set to empty string to disable. Ignored when `PG_CONF_FILE` is set. | | `PG_SHARED_PRELOAD_LIBRARIES` | `pg_cron` | Comma-separated `shared_preload_libraries` value. Set to empty string to disable. Ignored when `PG_CONF_FILE` is set. |
| `PG_DISABLE_LOGGING` | `0` | Set to `1` to disable PostgreSQL logging collector. Ignored when `PG_CONF_FILE` is set. | | `PG_DISABLE_LOGGING` | `0` | Set to `1` to disable PostgreSQL logging collector. Ignored when `PG_CONF_FILE` is set. |
| `PG_HECTIC_INHERITANCE` | `1` | Apply the [`hectic` inheritance bundle](#hectic-inheritance-bundle) to the target database after init. Set to `0` to disable. | | `HECTIC_DOTENV_FILE` | (unset) | Optional dotenv file. When set and readable, `database hydrate` passes its contents to `hectic.load_secrets_from_env(...)` after applying the bundle. Falls back to `${LOCAL_DIR}/.env.${ENVIRONMENT}` when unset. |
| `HECTIC_INHERITANCE_SQL` | (auto) | Override path to the SQL file applied by `PG_HECTIC_INHERITANCE=1`. Defaults to the SQL shipped with `postgres-init`. |
| `PATCH_LOG` | (stdout) | Path to log the output of database patches. | | `PATCH_LOG` | (stdout) | Path to log the output of database patches. |
| `HYDRATE_LOG` | (stdout) | Path to log the output of database hydration. | | `HYDRATE_LOG` | (stdout) | Path to log the output of database hydration. |
@@ -109,24 +108,36 @@ To use `db-tool` in a Nix development shell, add the following to your `flake.ni
} }
``` ```
## hectic Inheritance Bundle ## hectic Bundle
`pkgs.hectic.hectic-inheritance` ships a SQL artifact that bootstraps a `hectic` `db-tool` and `migrator` apply a single bundle of SQL files that bootstrap the
schema with three parent tables and DDL event triggers: `hectic` schema. The bundle lives in
[`lib/hook/sql/`](../../lib/hook/sql/README.md) — see that README for full
contract, file layout, and the `self.lib.hectic.*` Nix API.
The bundle creates:
- `hectic.version` — single version row for the entire hectic system.
Mismatch between database and bundle raises an exception.
- `hectic.secret` + `hectic.load_secrets_from_env(text)` +
`hectic.get_secret(text)` — encrypted secret storage and dotenv loader.
- `hectic.migration` — table consumed by `migrator`.
- `hectic.created_at` / `hectic.updated_at` / `hectic.immutable` parent tables
and the DDL event triggers that enforce inheritance, attach
`BEFORE UPDATE` triggers, and block DML on immutable tables outside
`migration_mode`.
Inheritance details:
- `hectic.created_at(created_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — every - `hectic.created_at(created_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — every
user table must `INHERITS (hectic.created_at)`. The event trigger user table must `INHERITS (hectic.created_at)`. The event trigger
`hectic_enforce_created_at_inheritance` raises an exception on `CREATE TABLE` `hectic_enforce_created_at_inheritance` raises on `CREATE TABLE` otherwise.
otherwise. - `hectic.updated_at(updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` —
- `hectic.updated_at(updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW())` — optional. optional. Any inheriting table automatically gets a
Any table that inherits from it automatically gets a `BEFORE UPDATE FOR EACH `BEFORE UPDATE FOR EACH ROW` trigger calling `hectic.set_updated_at()`.
ROW` trigger calling `hectic.set_updated_at()` attached by - `hectic.immutable()` — pure marker. Inheriting tables are blocked from
`hectic_attach_updated_at_trigger`. `INSERT`/`UPDATE`/`DELETE`/`TRUNCATE` outside migration mode. To allow DML
- `hectic.immutable()` — pure marker. Tables inheriting it are blocked from inside a migration, wrap it in a transaction:
`INSERT`/`UPDATE`/`DELETE`/`TRUNCATE` outside migration mode by triggers
attached by `hectic_attach_immutable_triggers`. Useful for reference data
that must only change via migrations. To allow DML inside a migration, wrap
it in a transaction:
```sql ```sql
BEGIN; BEGIN;
@@ -148,6 +159,16 @@ Per-database opt-out for additional schemas via the
ALTER DATABASE mydb SET hectic.inheritance_extra_excluded_schemas = 'legacy,etl'; ALTER DATABASE mydb SET hectic.inheritance_extra_excluded_schemas = 'legacy,etl';
``` ```
### Responsibility split
| Component | Applies bundle? |
| --- | --- |
| `postgres-init` | **No.** Pure PostgreSQL provisioner — starts a vanilla cluster, nothing more. |
| `migrator init` | **Yes, mandatory.** The bundle is a hard prerequisite for `hectic.migration`. |
| `database hydrate` | **Yes, by default.** Re-applied on every hydrate. Skip with `--no-hook`. After applying the bundle, hydrate also calls `hectic.load_secrets_from_env(<dotenv>)` if `HECTIC_DOTENV_FILE` (or `${LOCAL_DIR}/.env.${ENVIRONMENT}`) is readable. |
The bundle is idempotent — repeated application is safe.
### `db-tool diff` and immutable tables ### `db-tool diff` and immutable tables
`database diff` already includes immutable tables in its schema-level `database diff` already includes immutable tables in its schema-level
@@ -158,25 +179,21 @@ diff of the rows of every table inheriting `hectic.immutable`. Drift in
"frozen" reference data therefore surfaces in the same pager view as schema "frozen" reference data therefore surfaces in the same pager view as schema
drift, and the subcommand exits non-zero when either differs. drift, and the subcommand exits non-zero when either differs.
### Apply via `postgres-init` ### Apply manually via `psql`
Applied automatically. Set `PG_HECTIC_INHERITANCE=0` to opt out. For external consumers (e.g. NixOS modules) bypass the helper and call `psql`
directly against the paths exposed by `self.lib.hectic.*.path`:
### Apply via `migrator` or any psql pipeline
```nix ```nix
# in your devshell services.postgresql.initialScript = pkgs.writeText "hectic-init.sql" ''
shellHook = '' \i ${self.lib.hectic.secret.path}
export HECTIC_INHERITANCE_SQL=${pkgs.hectic.hectic-inheritance}/share/hectic/hectic-inheritance.sql \i ${self.lib.hectic.migration.path}
\i ${self.lib.hectic.inheritance.path}
''; '';
``` ```
```sh The version file (`self.lib.hectic.version`) is templated and only exposes
psql "$DB_URL" -v ON_ERROR_STOP=1 -f "$HECTIC_INHERITANCE_SQL" `.sql` (a string). Materialize it with `pkgs.writeText` if a path is needed.
```
The SQL is also exposed via `self.lib.hecticInheritance.sql` (string) and
`self.lib.hecticInheritance.path` (Nix path) for inline pipelines.
## Exit Codes ## Exit Codes