163 lines
8.6 KiB
Markdown
163 lines
8.6 KiB
Markdown
# Attic repack migration helper
|
|
|
|
Local-only operator tool for safe resumable Attic cache repack/migration. Parent automation starts old/new services, supplies secrets, seeds spool, and runs this CLI.
|
|
|
|
## Deployment layout
|
|
|
|
### Operational state — 2026-09-10
|
|
|
|
The primary `/hectic` endpoint now serves `nix-cache-hectic-lab` through
|
|
`atticd-repacked`; the cutover was applied with NixOS `switch`. The original
|
|
bucket/database remain preserved and readable at `/previous/hectic`.
|
|
`/next/hectic` is an alias for the new backend. Existing public keys and CI
|
|
tokens remain valid, and the primary endpoint is writable again.
|
|
|
|
The migrated inventory contains 1343 paths and 1195 unique NAR hashes. The
|
|
independent inventory comparison and aggregate full-read receipts are recorded
|
|
in `/var/lib/attic-repack/verification-receipt.json`. Transient S3 504/read errors
|
|
required retries; this is data-integrity evidence, not a claim that Hetzner's
|
|
read availability is fixed.
|
|
|
|
All migration, verification, seeding, and watcher jobs have been stopped for
|
|
user-controlled load testing. Do not automatically restart bulk verification.
|
|
The transport-fix generation is
|
|
`/nix/store/mjx3gzrh03vq22j5nmrzwa56kfhdn702-nixos-system-hectic-lab-25.11.20260526.25f5383`
|
|
with the HTTP/1.1 S3 transport and upload-proxy timeout correction. Check
|
|
`/run/current-system` to confirm the active deployment.
|
|
The pinned rollback generation remains at
|
|
`/var/lib/attic-repack/rollback-system`; backups remain private under
|
|
`/var/lib/attic-repack/backups` and include the cache signing key.
|
|
|
|
- Original backend: `atticd`, port 8081, `/var/lib/atticd/server.db`, bucket
|
|
`cache-hectic-lab` in HEL1.
|
|
- During the write freeze and after cutover the original backend runs in
|
|
`api-server` mode, without its garbage collector, to preserve the comparison
|
|
dataset. Public write methods remain blocked by nginx.
|
|
- Repacked backend: `atticd-repacked`, port 8082,
|
|
`/var/lib/atticd-repacked/server.db`, bucket `nix-cache-hectic-lab` in HEL1.
|
|
- Both use the same `hectic` signing key and existing JWT verification secret;
|
|
clients do not need a new trusted public key or token.
|
|
- New chunk settings: threshold/minimum 1 MiB, average 2 MiB, maximum 4 MiB.
|
|
- `https://cache.hectic-lab.com/next/hectic` selects the new backend.
|
|
- `https://cache.hectic-lab.com/previous/hectic` selects the original backend;
|
|
nginx permits GET/HEAD only there.
|
|
- `repackedActive` in `nixos/system/hectic-lab/attic.nix` selects which backend
|
|
owns the original `/hectic` URL. Keep it false until all cutover gates pass.
|
|
|
|
## Cutover and rollback gates
|
|
|
|
1. Finish all migration partitions, then run an unfiltered migration/delta pass.
|
|
2. Confirm no CI writers remain. Set `migrationWriteFreeze = true` while
|
|
`repackedActive = false`, apply the small NixOS change, and briefly stop the
|
|
original Attic to drain/cancel any prior in-flight writes.
|
|
3. Take a SQLite backup with SQLite's backup API, not a raw live-file copy.
|
|
Keep backups and manifests under private `/var/lib/attic-repack`; the SQLite
|
|
backup includes the cache's private signing key.
|
|
4. Restart the original backend for reads only, refresh the complete inventory,
|
|
migrate any final delta, then run unfiltered `verify` across all 1343 old
|
|
paths. Its exit status must be zero with zero exhausted payload verification
|
|
failures; independently compare old/new store-path, NAR hash, size and
|
|
metadata inventories from the databases. `status` alone is not a cutover
|
|
certificate.
|
|
5. Pin the old/staging NixOS generation as a GC root, set `repackedActive = true`,
|
|
build, inspect dry activation, and switch. `/hectic` now reaches the new
|
|
backend; old data and `/previous/hectic` remain available.
|
|
6. Test public reads, signatures, and an authenticated upload at the original
|
|
URL. Do not remove the old bucket or database as part of this procedure.
|
|
|
|
Rollback reapplies the pinned staging generation. The new backend and its data
|
|
must remain preserved: paths first uploaded after cutover may exist only there.
|
|
When editing the flags manually, clear `migrationWriteFreeze` explicitly if
|
|
writes to the original backend are intended after rollback.
|
|
|
|
## Throughput comparison
|
|
|
|
Compare the same store-path hashes at `/next/hectic` and `/previous/hectic` with
|
|
the same request concurrency. For example, fetch
|
|
`https://cache.hectic-lab.com/next/hectic/nar/<store-path-hash>.nar` with
|
|
`curl --fail --location --output /dev/null --write-out 'bytes=%{size_download} seconds=%{time_total}\n'`.
|
|
Do not print effective redirect URLs: S3 redirects contain temporary signatures.
|
|
Compare wall time and error rate as well as bytes/second because compressed sizes
|
|
can differ after rechunking. Do not use a build with source fallback as a pure
|
|
cache throughput measurement. The two endpoints share the VPS and nginx, so run
|
|
the comparison sequentially or account for shared-resource contention.
|
|
|
|
## Spool/state convention
|
|
|
|
Default state dir: `/var/lib/attic-repack` (`0700`). Raw NAR spool path:
|
|
|
|
```text
|
|
/var/lib/attic-repack/raw/{sha256hex}.nar
|
|
```
|
|
|
|
Parent may seed this file directly. Tool always verifies SHA-256 and byte length before upload. Checkpoints live under `checkpoints/{store_path_hash}.json` and contain no keypair/token. Forced payload verification records a receipt with `payload_verified_at`, `payload_verify_attempts`, `payload_sha256`, and `payload_bytes` only after a complete successful read.
|
|
|
|
## Commands
|
|
|
|
```sh
|
|
attic-repack init \
|
|
--old-db file:/var/lib/atticd/server.db?mode=ro \
|
|
--old-url http://127.0.0.1:8081 \
|
|
--new-url http://127.0.0.1:8082 \
|
|
--host cache.hectic-lab.com \
|
|
--atticadm /run/current-system/sw/bin/atticadm \
|
|
--server-config /etc/atticd/server.toml
|
|
|
|
attic-repack inventory --state-dir /var/lib/attic-repack > inventory.json
|
|
attic-repack status --state-dir /var/lib/attic-repack
|
|
attic-repack migrate --state-dir /var/lib/attic-repack --workers 2 --limit 20
|
|
attic-repack verify --state-dir /var/lib/attic-repack --workers 2
|
|
```
|
|
|
|
`ATTIC_MIGRATION_TOKEN` may be set for manual/tests. Otherwise token is minted in memory with `atticadm make-token` for hectic pull/push/create-cache/configure-cache. Token/keypair are never printed.
|
|
|
|
## Inventory JSON
|
|
|
|
`inventory` writes `attic-repack-inventory-v1`:
|
|
|
|
```json
|
|
{
|
|
"format": "attic-repack-inventory-v1",
|
|
"cache": "hectic",
|
|
"spool_dir": "/var/lib/attic-repack/raw",
|
|
"raw_nar_filename": "{sha256hex}.nar",
|
|
"records": [
|
|
{"nar_hash":"sha256:...","nar_size":123,"store_path":"/nix/store/...","metadata_fingerprint":"..."}
|
|
]
|
|
}
|
|
```
|
|
|
|
Records also include upload metadata: `store_path_hash`, `references`, `system`, `deriver`, `sigs`, `ca`.
|
|
|
|
## Safety
|
|
|
|
- Checkpoints and `status` are progress information, not a final cutover proof.
|
|
After stopping old writers and taking a consistent snapshot, run an unfiltered
|
|
`verify` (no `--paths-file` or `--limit`) to reread every new NAR and reconcile
|
|
all paths, metadata, hashes, and sizes before switching the primary endpoint.
|
|
Do not skip files, change expected hashes, or relax server/client timeouts to
|
|
pass this gate.
|
|
- A local store path can differ from the historical cached NAR. Such a local
|
|
copy is rejected and recovered from the original S3 chunks instead.
|
|
- Old DB is opened readonly; old SQL NAR/chunk tables are never copied.
|
|
- Missing local raw NARs are reconstructed from old S3 chunkrefs with per-object retries and chunk/full hash checks.
|
|
- Upload uses Attic `PUT /_api/v1/upload-path` with JSON preamble plus raw uncompressed NAR.
|
|
- New cache verification compares immutable metadata against old rendered narinfo
|
|
and reads/decompresses one payload per verified path invocation. Payload reads
|
|
make up to three fresh attempts for transport HTTP 408/429/5xx and truncated
|
|
body/decompressor EOF failures only. Each attempt follows a new GET/redirect,
|
|
starts SHA-256 and byte counts from zero, closes failed readers, and fails
|
|
immediately on hash mismatch, full-size mismatch, oversized payload, missing
|
|
URL, unsupported compression, or HTTP 4xx other than 408/429.
|
|
- A receipt with retries proves the path was fully read and matched integrity; it
|
|
does not prove the storage provider is healthy. Treat retry events as provider
|
|
health signals separate from cutover correctness.
|
|
- Authenticated HTTP is refused unless URL host is loopback.
|
|
|
|
## Local build/test
|
|
|
|
```sh
|
|
nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; in pkgs.callPackage ./infra/attic-migration {}"
|
|
nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; p = pkgs.callPackage ./infra/attic-migration {}; in p.passthru.tests.unittest"
|
|
```
|