hearth/infra/gitea-runners/runbook.md

# Gitea Runner Infrastructure Runbook

## Scope

This directory is the repo-owned boundary for the Gitea Actions runner pool.
The controller is the active on-demand path; Kubernetes manifests and the
Nix-capable image are retained for manual rollback and maintenance.

The target service is `https://gitea.hectic-lab.com` for the Gitea organization
`hectic-lab`. The pool is trusted-only. "Ephemeral" means each controller VM
and workflow job is disposable; the Kubernetes StatefulSet is rollback-only.

## Immutable decisions

- Infrastructure is managed with OpenTofu command examples only, using the
  `tofu` CLI.
- Cloud provider is Hetzner; cluster bootstrap uses kube-hetzner.
- Remote state uses the S3 backend bucket `gitea-runner-hectic-lab`.
- Runner implementation is the non-Enterprise `gitea/runner`.
- Runner registration uses a Gitea organization-scoped token for `hectic-lab`.
- Runtime token delivery is SOPS-backed and mounted into the runner pod as a
  file read through `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`; plaintext token
  environment variables are not the contract.
- Kubernetes runner lifecycle uses a StatefulSet with one PVC per pod for
  `/data`, including `/data/.runner`.
- Container builds run through privileged rootful DinD inside trusted runner
  pods; host Docker socket mounting is not an implementation path.
- `ubuntu-latest` and `nix` are controller-managed on-demand aliases for
  `gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels.
- First scope is trusted internal workflows only, with no untrusted fork or PR
  workflow support.
- On-demand allocation is handled by the repo-owned controller; Kubernetes is
  not an active autoscaling path.

## Lifecycle boundaries

- `infra/gitea-runners/opentofu/`: OpenTofu stack for the S3 backend
  contract, Hetzner provider configuration, and kube-hetzner module wiring.
- `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount,
  StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work.
- `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes
  rollback image; active on-demand Nix image is selected by Hetzner image ID.
- `infra/gitea-runners/runbook.md`: this contract plus later operational
  commands, rollback notes, and acceptance evidence references.

## Guardrails

- Enterprise ARC/actions-runner-controller are rejected alternatives and must
  not be implemented here. Do not add ARC custom resources, controller install
  instructions, or GitHub Actions ARC assumptions.
- Untrusted fork/PR workflows are out of first scope; privileged DinD is only
  acceptable for trusted internal jobs.
- The persistent StatefulSet is rollback-only and defaults to zero replicas;
  normal jobs use controller-managed on-demand VMs.
- No actual secrets are committed: no kubeconfig, runner token, Hetzner token,
  S3 credentials, decrypted SOPS files, or SOPS age keys.
- OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea
  runner token; Kubernetes receives the token as a mounted file secret instead.
- Do not use `localhost` or `127.0.0.1` as the Gitea URL inside job containers;
  jobs must reach the public HTTPS service.

## Initial acceptance commands

Run from the repository root:

```sh
test -d infra/gitea-runners/opentofu && test -d infra/gitea-runners/k8s && test -d infra/gitea-runners/image
test -f infra/gitea-runners/runbook.md
grep -n "OpenTofu\|kube-hetzner\|StatefulSet\|DinD\|SOPS\|trusted" infra/gitea-runners/runbook.md
grep -R "[E]nterprise ARC\|[a]ctions-runner-controller" infra/gitea-runners
grep -R "[t]erraform " infra/gitea-runners || true
grep -R "[D]ECISION NEEDED" infra/gitea-runners || true
```

Expected outcomes: the directory and file checks exit 0; the architecture-term
grep shows this contract; ARC references appear only in the rejected-alternative
guardrail above; there are no forbidden CLI command examples and no unresolved
decision placeholders.

## Downstream placeholders

- Task 2: add OpenTofu backend/provider files and verify S3 state safety.
- Task 3: add SOPS secret contract and runtime token delivery details.
- Task 4: define or package the Nix-capable runner image for the `nix` label.
- Task 5+: provision kube-hetzner, add Kubernetes resources, verify workflows,
  and document cleanup, rollback, and scaling operations.

## Runner lifecycle cleanup

All lifecycle commands are scoped to the runner namespace:

```sh
kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
```

The scheduled cleanup manifest is dry-run only. It lists the StatefulSet, active
runner pods, runner PVCs, and PVCs whose expected StatefulSet pod is absent. It
does not delete pods, PVCs, Docker data, or Gitea runner registrations.

Run the same inventory on demand without waiting for the schedule:

```sh
kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run
```

The cleanup job template has `ttlSecondsAfterFinished: 3600`, so completed
manual dry-run jobs are garbage-collected by Kubernetes instead of requiring an
operator to remove finished jobs manually.

If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching
pod loses `/data/.runner`. That runner identity must then be deregistered from
Gitea or the replacement pod must be allowed to re-register intentionally with
the current organization runner token. Do not delete an active runner PVC as a
normal cleanup step.

Non-UI Gitea registration reconciliation uses the Gitea API with a separate
admin token. Store that token outside this repository and pass it as a file; do
not print it:

```sh
kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run
```

Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run pod
has completed and its logs have been collected. Do not keep this admin token in
the runner namespace longer than the reconciliation window.

Only remove a stale Gitea runner registration after the corresponding pod/PVC
was intentionally deleted or `/data/.runner` was intentionally reset. Prefer a
Gitea CLI/API deletion from the Gitea server or an admin workstation; manual UI
cleanup is a fallback, not the only path. Record the removed runner name and the
Kubernetes PVC/pod deletion that made it stale.

After the dry-run list identifies a stale registration and the PVC/pod deletion
has been recorded, remove that exact Gitea runner by id through the API:

```sh
runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"
```

Do not run the delete command for a runner that still has an active
`gitea-runner-*` pod or a retained `data-gitea-runner-*` PVC unless that PVC is
being intentionally reset for re-registration.

## Docker-in-Docker storage cleanup

Docker layers live inside each DinD sidecar at `/var/lib/docker`, backed by the
pod-local `docker-graph` `emptyDir`; the host Docker socket is not used. Always
list disk usage before pruning, and run the command only against the `docker`
container in runner pods in `gitea-runners`. Because this storage is pod-local,
loop over pods for pool-wide cleanup:

```sh
for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done
```

For one pod, replace the StatefulSet target with the pod name:

```sh
kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system df
kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system prune --all --force --filter until=24h
```

Do not run host-level Docker cleanup commands and do not mount or prune a host
Docker socket. If a pod is deleted, its `emptyDir` Docker graph is removed by
Kubernetes; the `/data` PVC remains and still controls runner identity.

## Token rotation

Rotate the Gitea organization runner token without printing decrypted values:

```sh
sops sus/gitea-runners.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -
kubectl -n gitea-runners rollout restart statefulset/gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true
```

The `rollout restart` command above is the controlled restart path for this
StatefulSet. Observe the rollout and each ordinal until all replacement pods are
Ready; do not delete runner pods directly as part of normal token rotation:

```sh
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-0 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-1 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-2 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-3 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-4 --timeout=5m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
```

Verification must confirm the token file mount remains present while the token
value never appears in logs or evidence:

```sh
kubectl -n gitea-runners describe pod gitea-runner-0 | grep -n '/runner-secrets\|gitea-runner-token'
kubectl -n gitea-runners logs pod/gitea-runner-0 -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true
```

## Deploy and status

These commands are executable only when the external inputs are available:

- `TF_VAR_hcloud_token`
- `TF_VAR_ssh_public_key`
- `TF_VAR_ssh_private_key`
- S3 backend credentials and endpoint access
- a matching SOPS age identity for `sus/gitea-runners.yaml`
- `kubectl` access to the target cluster
- a valid `GCR_NIX_IMAGE_ID` for controller-managed Nix jobs

If any input is missing, stop before `tofu apply`. Do not guess values or reuse
stale kubeconfig files.

Before production Kubernetes apply or rollout, satisfy both manifest gates:

1. Create or update the `gitea-runner-token` Secret from SOPS. The active
   Kustomize overlay intentionally does not include a placeholder Secret, but
   the StatefulSet still mounts `secretName: gitea-runner-token` as
   `/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`.
2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the
   controller; Nix image readiness is governed by `GCR_NIX_IMAGE_ID`.

Use the same SOPS materialization pattern as token rotation before applying the
Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a
target namespace; the full overlay remains gated on the Secret.

```sh
kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -
```

Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command
above succeeds. The persistent pool ConfigMap must retain empty labels.

```sh
tofu -chdir=infra/gitea-runners/opentofu init
tofu -chdir=infra/gitea-runners/opentofu validate
tofu -chdir=infra/gitea-runners/opentofu plan -out=.sisyphus/evidence/task-12-deploy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-deploy.plan
export KUBECONFIG="$(tofu -chdir=infra/gitea-runners/opentofu output -raw kubeconfig_path)"
kubectl config current-context
kubectl get nodes -o wide
kubectl get sc
kubectl apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200
```

Expected status after deploy:

- `kubectl config current-context` names the runner cluster context.
- `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready.
- `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs.
- `kubectl -n gitea-runners get statefulset gitea-runner` shows 0 desired and 0 ready replicas.
- `kubectl -n gitea-runners get pvc` shows no active runner PVCs; retained PVCs are rollback-only.
- The controller host reports healthy and owns runner registrations; no persistent runner claims `ubuntu-latest` or `nix`.

## Legacy rollback pool scaling (manual only)

Persistent-pool scaling is not part of normal operation. Use only after restoring
its labels and disabling the zero-idle controller as described in `Rollback`.

```sh
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide

# Run only workflows supported by restored persistent labels.

kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
```

After scaling back down, inspect the cleanup dry-run and deregister any stale
runner registrations only for pods or PVCs that were intentionally removed.

## Cleanup and stale runner deregistration

Use the dry-run cleanup job to list the StatefulSet, active pods, PVCs, and any
PVC candidates whose pod is gone. It must not delete active resources.

```sh
kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run
```

Pool-wide DinD storage checks and cleanup:

```sh
for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done
```

If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching
pod loses `/data/.runner`. Deregister that runner from Gitea, or let the
replacement pod re-register intentionally with the current organization token.
Never delete an active runner PVC as routine cleanup.

Non-UI Gitea registration reconciliation uses an admin token stored outside this
repository:

```sh
kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run
```

Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run
pod has completed and its logs have been collected.

After the dry-run list identifies a stale registration and the PVC or pod
deletion has been recorded, remove that exact Gitea runner by id through the
API:

```sh
runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"
```

Do not run the delete command for a runner that still has an active
`gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is
being intentionally reset for re-registration.

## Legacy Kubernetes application rollback

Rollback the app layer only. Do not use this section to destroy the cluster.

```sh
kubectl -n gitea-runners rollout history statefulset/gitea-runner
kubectl -n gitea-runners rollout undo statefulset/gitea-runner --to-revision=<known-good-revision>
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200
```

If a manifest rollback is needed, reapply the repo overlay after checking out the
known-good revision, then re-run the rollout checks:

```sh
kubectl -n gitea-runners apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
```

## Full cluster teardown

This destroys Hetzner resources owned by the kube-hetzner stack, including the
`gitea-runners` cluster nodes, the `control-plane` node pool, the
`runner-workers` node pool, the cluster network, load balancer resources,
firewall objects, and any attached Hetzner CSI volumes still managed by the
stack. Do not run teardown unless the destruction is intentional.

```sh
tofu -chdir=infra/gitea-runners/opentofu plan -destroy -out=.sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-destroy.plan
```

## Partial OpenTofu apply recovery

If `tofu apply` fails after creating some resources, do not destroy blindly.
First reconcile state and inspect what the stack thinks exists:

```sh
tofu -chdir=infra/gitea-runners/opentofu plan -refresh-only -out=.sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu state list
```

Then rerun the normal plan path. Use `-target` only as a last resort when a
single resource is stuck and the drift is understood.

## S3 backend recovery

If backend init or state access fails, first verify the bucket and versioning
outside OpenTofu, then reconfigure the backend:

```sh
nix run nixpkgs#awscli2 -- s3api head-bucket --bucket gitea-runner-hectic-lab
nix run nixpkgs#awscli2 -- s3api get-bucket-versioning --bucket gitea-runner-hectic-lab
tofu -chdir=infra/gitea-runners/opentofu init -reconfigure
tofu -chdir=infra/gitea-runners/opentofu plan
```

If the backend reports a stale lock, confirm no `tofu` process is active, then
use `tofu force-unlock <LOCK_ID>` with the lock id from the error. Never force
unlock a live plan or apply.

## Gitea outage troubleshooting

Use the public HTTPS service, not `localhost` or `127.0.0.1` inside job
containers.

```sh
kubectl -n gitea-runners run gitea-outage-probe --rm --restart=Never --image=curlimages/curl:8.10.1 -- curl -fsS https://gitea.hectic-lab.com/api/healthz
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -E 'connection refused|timeout|tls|certificate|temporary failure' || true
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50
```

If Gitea is down, keep the existing StatefulSet and PVCs intact. Do not delete
`/data/.runner` just because the service is unavailable. Once Gitea returns,
repeat the token rotation or re-registration path if a pod restarted while the
service was unavailable and lost its runner identity.

## Release checklist

Do not release unless the following evidence files exist and are readable:

- `.sisyphus/evidence/task-5-cluster-plan.txt`
- `.sisyphus/evidence/task-5-secret-plan-scan.txt`
- `.sisyphus/evidence/task-9-deploy.txt`
- `.sisyphus/evidence/task-9-secret-mount.txt`
- `.sisyphus/evidence/task-10-ubuntu-workflow.txt`
- `.sisyphus/evidence/task-10-nix-workflow.txt`
- `.sisyphus/evidence/task-11-scale.txt`
- `.sisyphus/evidence/task-11-restart-cleanup.txt`

If any evidence file is missing, stop and collect it before treating the runbook
as complete.

## Ephemeral VM runner cutover

This section governs replacing the legacy persistent K8s runner pool with the
ephemeral-VM controller (`package/gitea-runner-controller`) on this host.
The K8s pool above remains rollback-only until cutover is explicitly accepted.

### Operator gates (all three required before enable)

1. Secrets — add to `sus/gitea-runners.yaml` under
   `gitea/hectic-lab/controller/*`:

   ```sh
   sops sus/gitea-runners.yaml
   # add keys:
   #   gitea:
   #     hectic-lab:
   #       controller:
   #         hcloud-token:    <Hetzner API token, VM create/destroy scope>
   #         webhook-secret:  <random 32+ bytes; also set as Gitea webhook secret>
   #         admin-token:     <Gitea token with admin:runner scope for stale cleanup>
   ```

   The org registration token key
   `gitea/hectic-lab/org-runner-registration-token` already exists and is reused.

2. Base image — build the MicroOS snapshot and record its id:

   ```sh
   nix develop .#gitea-runners -c gitea-runners-build-microos-snapshots x86
   hcloud image list --selector '' -o json | jq '.[] | select(.type=="snapshot")'
   ```

3. DNS — A record `runners.hectic-lab.com -> 128.140.75.58` (ACME needs it).

### Enable

```sh
# nixos/system/hectic-lab/hectic-lab.nix: resolve the FIXME block
hectic.services.gitea-runner-controller = {
  enable  = true;
  imageId = "<snapshot-id-from-gate-2>";
};
nixos-rebuild --target root@128.140.75.58 switch
systemctl status gitea-runner-controller.service gitea-runner-webhook.service
```

### Register the Gitea webhook

Org-level (preferred) or per-repo, on `https://gitea.hectic-lab.com`:

- URL: `https://runners.hectic-lab.com/`
- Method: `POST`, content type: JSON
- Secret: value of `gitea/hectic-lab/controller/webhook-secret`
- Trigger events: `Workflow jobs` only (`workflow_job`)

### Long CUDA/Magma deployment time budgets

The `deploy-neuro` workflow uses these nested limits:

| Layer | Limit |
| --- | --- |
| Wrapped build/deploy command | 6 hours (`WITH_ATTIC_BUILD_TIMEOUT=21600`) |
| Final cache drain | 1 hour (`WITH_ATTIC_DRAIN_TIMEOUT=3600`) |
| Workflow job | 435 minutes, including 15 minutes of setup/cleanup margin |
| `gross-nix-x86-perf` runner | 480 minutes |
| `gross-nix-x86-highmem` runner | 480 minutes |
| Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours |
| VM hard lifetime from allocation | 480 minutes; no destruction grace |

`ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for
long-running Nix deployments. Deploy the controller and Gitea watchdog settings
before dispatching the longer workflow. Already
allocated VMs retain the TTL and runner configuration assigned when they were
created; updating the controller does not extend a running job.

`gross-nix-x86-highmem` is an explicit costly high-memory escape hatch backed
only by Hetzner CCX53 in `nbg1`, `fsn1`, or `hel1`; it may fall back by region
only, never to a lower-RAM server type. Current Hetzner public pricing for
Germany/Finland CCX53 is 0.8550 EUR/hour excluding IPv4, so one 480-minute
allocation reserves 6.84 EUR against the controller budget before VM creation.

These are maximum lifetimes. Failed, cancelled, skipped, and unbootstrapped jobs
still trigger immediate VM teardown; failed jobs retain pre-destroy diagnostics.
After a successful job, its bootstrapped VM stays running until next hourly
boundary measured from original VM creation, capped by profile TTL. Same-repo,
same-label queued work can atomically claim that idle VM. Reuse preserves
original Hetzner labels and runner name, creates no server, fetches no new
registration token, and makes no second budget reservation. Idle VMs are still
billed: controller deletes them at slot expiry and never relies on stopping a
server to avoid charges.

Reuse retains runner host filesystem and registration identity. It is therefore
restricted to same repository and exact label inside this trusted-only pool;
allowed repositories must not run untrusted fork or pull-request code. A failed
or otherwise non-successful job is never reused.

Active and idle VMs are deleted at profile hard TTL without grace. Idle reuse is
allowed only when at least one configured reconcile interval remains before both
slot expiry and hard TTL. Controller budget reservation still uses full label TTL
on initial creation, so a long-running label reserves more of existing monthly
budget. Do not raise that budget or disable timeout safeguards just to bypass a
refused allocation.

After changing any timeout, verify the complete chain rather than only
`timeout-minutes`; a shorter wrapper, runner, server watchdog, or VM TTL wins.

### Pre-flight verification (before first real job)

```sh
curl -fsS https://runners.hectic-lab.com/ -o /dev/null -w '%{http_code}\n' # any 4xx from handler = reachable
journalctl -u gitea-runner-webhook -n 20 --no-pager
hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed")] | length' # expect 0
```

Zero managed VMs is expected after retained billing slots expire. Immediately
after successful work, one managed VM per retained profile may remain until its
recorded hourly boundary.

### End-to-end acceptance (Task 9)

Trigger `.gitea/workflows/runner-nix-smoke.yaml` via workflow_dispatch, then:

```sh
watch_labels() { hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed") | {id,name,labels}]'; }
watch_labels                                   # exactly one VM while queued/running
journalctl -f -u gitea-runner-controller       # vm-created / vm-destroyed events
watch_labels                                   # VM may remain until next hourly boundary
curl -fsS -H "Authorization: token $ADMIN" \
  https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners \
  | jq '[.entries[] | select(.name | startswith("gcr-"))] | length'  # may remain during retained slot
```

Failure paths to verify identically: duplicate delivery (send same webhook twice
via Gitea UI "Test delivery"), cancelled job, unknown-label job.

### Rollback

K8s rollback pool now defaults to deleted state:

- `infra/gitea-runners/k8s/statefulset.yaml` keeps `replicas: 0`
- old kube-hetzner nodes may be deleted to preserve zero idle cost
- PVCs and IaC remain for manual rollback only

Re-enable sequence:

```sh
# 1. edit nixos/system/hectic-lab/hectic-lab.nix and set
#    services.gitea-runner-controller.enable = false, then rebuild:
nixos-rebuild --target root@128.140.75.58 switch

# 2. reprovision old kube-hetzner nodes when they were deleted:
tofu -chdir=infra/gitea-runners/opentofu apply

# 3. while controller is disabled, destroy every surviving managed VM and
#    verify no gcr-* runner registration remains online:
hcloud server list -o json \
  | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
  | xargs -r -n1 hcloud server delete

# 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the
#    ConfigMap labels, then re-enable K8s runner pool. The legacy image does not
#    provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image
#    and label mapping are restored:
kubectl -n gitea-runners edit configmap/gitea-runner-config
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
```

Any managed VM or `gcr-*` registration found after step 3 must be removed before
restoring persistent labels; otherwise both pools can claim the same job.