hearth/infra/attic-migration/README.md

Attic repack migration helper

Local-only operator tool for safe resumable Attic cache repack/migration. Parent automation starts old/new services, supplies secrets, seeds spool, and runs this CLI.

Deployment layout

Operational state — 2026-09-10

The primary /hectic endpoint now serves nix-cache-hectic-lab through atticd-repacked; the cutover was applied with NixOS switch. The original bucket/database remain preserved and readable at /previous/hectic. /next/hectic is an alias for the new backend. Existing public keys and CI tokens remain valid, and the primary endpoint is writable again.

The migrated inventory contains 1343 paths and 1195 unique NAR hashes. The independent inventory comparison and aggregate full-read receipts are recorded in /var/lib/attic-repack/verification-receipt.json. Transient S3 504/read errors required retries; this is data-integrity evidence, not a claim that Hetzner's read availability is fixed.

All migration, verification, seeding, and watcher jobs have been stopped for user-controlled load testing. Do not automatically restart bulk verification. The transport-fix generation is /nix/store/mjx3gzrh03vq22j5nmrzwa56kfhdn702-nixos-system-hectic-lab-25.11.20260526.25f5383 with the HTTP/1.1 S3 transport and upload-proxy timeout correction. Check /run/current-system to confirm the active deployment. The pinned rollback generation remains at /var/lib/attic-repack/rollback-system; backups remain private under /var/lib/attic-repack/backups and include the cache signing key.

• Original backend: atticd, port 8081, /var/lib/atticd/server.db, bucket cache-hectic-lab in HEL1.
• During the write freeze and after cutover the original backend runs in api-server mode, without its garbage collector, to preserve the comparison dataset. Public write methods remain blocked by nginx.
• Repacked backend: atticd-repacked, port 8082, /var/lib/atticd-repacked/server.db, bucket nix-cache-hectic-lab in HEL1.
• Both use the same hectic signing key and existing JWT verification secret; clients do not need a new trusted public key or token.
• New chunk settings: threshold/minimum 1 MiB, average 2 MiB, maximum 4 MiB.
• https://cache.hectic-lab.com/next/hectic selects the new backend.
• https://cache.hectic-lab.com/previous/hectic selects the original backend; nginx permits GET/HEAD only there.
• repackedActive in nixos/system/hectic-lab/attic.nix selects which backend owns the original /hectic URL. Keep it false until all cutover gates pass.

Cutover and rollback gates

1. Finish all migration partitions, then run an unfiltered migration/delta pass.
2. Confirm no CI writers remain. Set migrationWriteFreeze = true while repackedActive = false, apply the small NixOS change, and briefly stop the original Attic to drain/cancel any prior in-flight writes.
3. Take a SQLite backup with SQLite's backup API, not a raw live-file copy. Keep backups and manifests under private /var/lib/attic-repack; the SQLite backup includes the cache's private signing key.
4. Restart the original backend for reads only, refresh the complete inventory, migrate any final delta, then run unfiltered verify across all 1343 old paths. Its exit status must be zero with zero exhausted payload verification failures; independently compare old/new store-path, NAR hash, size and metadata inventories from the databases. status alone is not a cutover certificate.
5. Pin the old/staging NixOS generation as a GC root, set repackedActive = true, build, inspect dry activation, and switch. /hectic now reaches the new backend; old data and /previous/hectic remain available.
6. Test public reads, signatures, and an authenticated upload at the original URL. Do not remove the old bucket or database as part of this procedure.

Rollback reapplies the pinned staging generation. The new backend and its data must remain preserved: paths first uploaded after cutover may exist only there. When editing the flags manually, clear migrationWriteFreeze explicitly if writes to the original backend are intended after rollback.

Throughput comparison

Compare the same store-path hashes at /next/hectic and /previous/hectic with the same request concurrency. For example, fetch https://cache.hectic-lab.com/next/hectic/nar/<store-path-hash>.nar with curl --fail --location --output /dev/null --write-out 'bytes=%{size_download} seconds=%{time_total}\n'. Do not print effective redirect URLs: S3 redirects contain temporary signatures. Compare wall time and error rate as well as bytes/second because compressed sizes can differ after rechunking. Do not use a build with source fallback as a pure cache throughput measurement. The two endpoints share the VPS and nginx, so run the comparison sequentially or account for shared-resource contention.

Spool/state convention

Default state dir: /var/lib/attic-repack (0700). Raw NAR spool path:

/var/lib/attic-repack/raw/{sha256hex}.nar

Parent may seed this file directly. Tool always verifies SHA-256 and byte length before upload. Checkpoints live under checkpoints/{store_path_hash}.json and contain no keypair/token. Forced payload verification records a receipt with payload_verified_at, payload_verify_attempts, payload_sha256, and payload_bytes only after a complete successful read.

Commands

attic-repack init \
  --old-db file:/var/lib/atticd/server.db?mode=ro \
  --old-url http://127.0.0.1:8081 \
  --new-url http://127.0.0.1:8082 \
  --host cache.hectic-lab.com \
  --atticadm /run/current-system/sw/bin/atticadm \
  --server-config /etc/atticd/server.toml

attic-repack inventory --state-dir /var/lib/attic-repack > inventory.json
attic-repack status --state-dir /var/lib/attic-repack
attic-repack migrate --state-dir /var/lib/attic-repack --workers 2 --limit 20
attic-repack verify --state-dir /var/lib/attic-repack --workers 2

ATTIC_MIGRATION_TOKEN may be set for manual/tests. Otherwise token is minted in memory with atticadm make-token for hectic pull/push/create-cache/configure-cache. Token/keypair are never printed.

Inventory JSON

inventory writes attic-repack-inventory-v1:

{
  "format": "attic-repack-inventory-v1",
  "cache": "hectic",
  "spool_dir": "/var/lib/attic-repack/raw",
  "raw_nar_filename": "{sha256hex}.nar",
  "records": [
    {"nar_hash":"sha256:...","nar_size":123,"store_path":"/nix/store/...","metadata_fingerprint":"..."}
  ]
}

Records also include upload metadata: store_path_hash, references, system, deriver, sigs, ca.

Safety

• Checkpoints and status are progress information, not a final cutover proof. After stopping old writers and taking a consistent snapshot, run an unfiltered verify (no --paths-file or --limit) to reread every new NAR and reconcile all paths, metadata, hashes, and sizes before switching the primary endpoint. Do not skip files, change expected hashes, or relax server/client timeouts to pass this gate.
• A local store path can differ from the historical cached NAR. Such a local copy is rejected and recovered from the original S3 chunks instead.
• Old DB is opened readonly; old SQL NAR/chunk tables are never copied.
• Missing local raw NARs are reconstructed from old S3 chunkrefs with per-object retries and chunk/full hash checks.
• Upload uses Attic PUT /_api/v1/upload-path with JSON preamble plus raw uncompressed NAR.
• New cache verification compares immutable metadata against old rendered narinfo and reads/decompresses one payload per verified path invocation. Payload reads make up to three fresh attempts for transport HTTP 408/429/5xx and truncated body/decompressor EOF failures only. Each attempt follows a new GET/redirect, starts SHA-256 and byte counts from zero, closes failed readers, and fails immediately on hash mismatch, full-size mismatch, oversized payload, missing URL, unsupported compression, or HTTP 4xx other than 408/429.
• A receipt with retries proves the path was fully read and matched integrity; it does not prove the storage provider is healthy. Treat retry events as provider health signals separate from cutover correctness.
• Authenticated HTTP is refused unless URL host is loopback.

Local build/test

nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; in pkgs.callPackage ./infra/attic-migration {}"
nix build --option eval-cache false --impure --expr "let flake = builtins.getFlake \"git+file://$PWD\"; pkgs = import flake.inputs.nixpkgs { system = builtins.currentSystem; }; p = pkgs.callPackage ./infra/attic-migration {}; in p.passthru.tests.unittest"