hearth/infra/attic-migration

Attic repack migration helper

Local-only operator tool for safe resumable Attic cache repack/migration. Parent automation starts old/new services, supplies secrets, seeds spool, and runs this CLI.

Deployment layout

• Original backend: atticd, port 8081, /var/lib/atticd/server.db, bucket cache-hectic-lab in HEL1.
• During the write freeze and after cutover the original backend runs in api-server mode, without its garbage collector, to preserve the comparison dataset. Public write methods remain blocked by nginx.
• Repacked backend: atticd-repacked, port 8082, /var/lib/atticd-repacked/server.db, bucket nix-cache-hectic-lab in HEL1.
• Both use the same hectic signing key and existing JWT verification secret; clients do not need a new trusted public key or token.
• New chunk settings: threshold/minimum 1 MiB, average 2 MiB, maximum 4 MiB.
• https://cache.hectic-lab.com/next/hectic selects the new backend.
• https://cache.hectic-lab.com/previous/hectic selects the original backend; nginx permits GET/HEAD only there.
• repackedActive in nixos/system/hectic-lab/attic.nix selects which backend owns the original /hectic URL. Keep it false until all cutover gates pass.

Cutover and rollback gates

1. Finish all migration partitions, then run an unfiltered migration/delta pass.
2. Confirm no CI writers remain. Set migrationWriteFreeze = true while repackedActive = false, apply the small NixOS change, and briefly stop the original Attic to drain/cancel any prior in-flight writes.
3. Take a SQLite backup with SQLite's backup API, not a raw live-file copy. Keep backups and manifests under private /var/lib/attic-repack; the SQLite backup includes the cache's private signing key.
4. Restart the original backend for reads only, refresh the complete inventory, migrate any final delta, then run unfiltered verify. Its exit status must be zero; independently compare old/new store-path, NAR hash, size and metadata inventories. status alone is not a cutover certificate.
5. Pin the old/staging NixOS generation as a GC root, set repackedActive = true, build, inspect dry activation, and switch. /hectic now reaches the new backend; old data and /previous/hectic remain available.
6. Test public reads, signatures, and an authenticated upload at the original URL. Do not remove the old bucket or database as part of this procedure.

Rollback reapplies the pinned staging generation. The new backend and its data must remain preserved: paths first uploaded after cutover may exist only there. When editing the flags manually, clear migrationWriteFreeze explicitly if writes to the original backend are intended after rollback.

Throughput comparison

Compare the same store-path hashes at /next/hectic and /previous/hectic with the same request concurrency. For example, fetch https://cache.hectic-lab.com/next/hectic/nar/<store-path-hash>.nar with curl --fail --location --output /dev/null --write-out 'bytes=%{size_download} seconds=%{time_total}\n'. Do not print effective redirect URLs: S3 redirects contain temporary signatures. Compare wall time and error rate as well as bytes/second because compressed sizes can differ after rechunking. Do not use a build with source fallback as a pure cache throughput measurement. The two endpoints share the VPS and nginx, so run the comparison sequentially or account for shared-resource contention.

Spool/state convention

Default state dir: /var/lib/attic-repack (0700). Raw NAR spool path:

/var/lib/attic-repack/raw/{sha256hex}.nar

Parent may seed this file directly. Tool always verifies SHA-256 and byte length before upload. Checkpoints live under checkpoints/{store_path_hash}.json and contain no keypair/token.

Commands

attic-repack init \
  --old-db file:/var/lib/atticd/server.db?mode=ro \
  --old-url http://127.0.0.1:8081 \
  --new-url http://127.0.0.1:8082 \
  --host cache.hectic-lab.com \
  --atticadm /run/current-system/sw/bin/atticadm \
  --server-config /etc/atticd/server.toml

attic-repack inventory --state-dir /var/lib/attic-repack > inventory.json
attic-repack status --state-dir /var/lib/attic-repack
attic-repack migrate --state-dir /var/lib/attic-repack --workers 2 --limit 20
attic-repack verify --state-dir /var/lib/attic-repack --workers 2

ATTIC_MIGRATION_TOKEN may be set for manual/tests. Otherwise token is minted in memory with atticadm make-token for hectic pull/push/create-cache/configure-cache. Token/keypair are never printed.

Inventory JSON

inventory writes attic-repack-inventory-v1:

{
  "format": "attic-repack-inventory-v1",
  "cache": "hectic",
  "spool_dir": "/var/lib/attic-repack/raw",
  "raw_nar_filename": "{sha256hex}.nar",
  "records": [
    {"nar_hash":"sha256:...","nar_size":123,"store_path":"/nix/store/...","metadata_fingerprint":"..."}
  ]
}

Records also include upload metadata: store_path_hash, references, system, deriver, sigs, ca.

Safety

• Checkpoints and status are progress information, not a final cutover proof. After stopping old writers and taking a consistent snapshot, run an unfiltered verify (no --paths-file or --limit) to reread every new NAR and reconcile all paths, metadata, hashes, and sizes before switching the primary endpoint.
• A local store path can differ from the historical cached NAR. Such a local copy is rejected and recovered from the original S3 chunks instead.
• Old DB is opened readonly; old SQL NAR/chunk tables are never copied.
• Missing local raw NARs are reconstructed from old S3 chunkrefs with per-object retries and chunk/full hash checks.
• Upload uses Attic PUT /_api/v1/upload-path with JSON preamble plus raw uncompressed NAR.
• New cache verification compares immutable metadata against old rendered narinfo and reads/decompresses one payload per verified path invocation.
• Authenticated HTTP is refused unless URL host is loopback.

Local build/test

nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; in pkgs.callPackage ./infra/attic-migration {}"
nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; p = pkgs.callPackage ./infra/attic-migration {}; in p.passthru.tests.unittest"