hearth/infra/gitea-runners/runbook.md

# Gitea Runner Infrastructure Runbook

## Scope

This directory is the repo-owned boundary for the first Gitea Actions runner
pool. Task 1 only establishes the scaffold and immutable decision contract;
downstream tasks will add OpenTofu backend/provider files, Kubernetes manifests,
and a Nix-capable runner image under the existing subdirectories.

The target service is `https://gitea.hectic-lab.com` for the Gitea organization
`hectic-lab`. The first pool is fixed-size and trusted-only. "Ephemeral" means
workflow job containers are ephemeral, while each runner pod keeps its runner
identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.

## Immutable decisions

- Infrastructure is managed with OpenTofu command examples only, using the
  `tofu` CLI.
- Cloud provider is Hetzner; cluster bootstrap uses kube-hetzner.
- Remote state uses the S3 backend bucket `gitea-runner-hectic-lab`.
- Runner implementation is the non-Enterprise `gitea/runner`.
- Runner registration uses a Gitea organization-scoped token for `hectic-lab`.
- Runtime token delivery is SOPS-backed and mounted into the runner pod as a
  file read through `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`; plaintext token
  environment variables are not the contract.
- Kubernetes runner lifecycle uses a StatefulSet with one PVC per pod for
  `/data`, including `/data/.runner`.
- Container builds run through privileged rootful DinD inside trusted runner
  pods; host Docker socket mounting is not an implementation path.
- The active runner label is `ubuntu-latest`. The `nix` label is not live until
  the Nix-capable image has been pushed and a concrete registry-reported digest
  is added to the runner ConfigMap.
- First scope is trusted internal workflows only, with no untrusted fork or PR
  workflow support.
- First scope has no autoscaling, no KEDA, and no dynamic runner controller.

## Lifecycle boundaries

- `infra/gitea-runners/opentofu/`: downstream OpenTofu stack for the S3 backend
  contract, Hetzner provider configuration, and kube-hetzner module wiring.
- `infra/gitea-runners/k8s/`: downstream namespace, ConfigMap, Secret mount,
  StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work.
- `infra/gitea-runners/image/`: downstream notes or sources for the runner image
  handoff; package or flake output changes are outside Task 1.
- `infra/gitea-runners/runbook.md`: this contract plus later operational
  commands, rollback notes, and acceptance evidence references.

## Guardrails

- Enterprise ARC/actions-runner-controller are rejected alternatives and must
  not be implemented here. Do not add ARC custom resources, controller install
  instructions, or GitHub Actions ARC assumptions.
- Untrusted fork/PR workflows are out of first scope; privileged DinD is only
  acceptable for trusted internal jobs.
- Autoscaling/KEDA is out of first scope; start with a fixed-size StatefulSet
  runner pool.
- No actual secrets are committed: no kubeconfig, runner token, Hetzner token,
  S3 credentials, decrypted SOPS files, or SOPS age keys.
- OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea
  runner token; Kubernetes receives the token as a mounted file secret instead.
- Do not use `localhost` or `127.0.0.1` as the Gitea URL inside job containers;
  jobs must reach the public HTTPS service.

## Initial acceptance commands

Run from the repository root:

```sh
test -d infra/gitea-runners/opentofu && test -d infra/gitea-runners/k8s && test -d infra/gitea-runners/image
test -f infra/gitea-runners/runbook.md
grep -n "OpenTofu\|kube-hetzner\|StatefulSet\|DinD\|SOPS\|trusted" infra/gitea-runners/runbook.md
grep -R "[E]nterprise ARC\|[a]ctions-runner-controller" infra/gitea-runners
grep -R "[t]erraform " infra/gitea-runners || true
grep -R "[D]ECISION NEEDED" infra/gitea-runners || true
```

Expected outcomes: the directory and file checks exit 0; the architecture-term
grep shows this contract; ARC references appear only in the rejected-alternative
guardrail above; there are no forbidden CLI command examples and no unresolved
decision placeholders.

## Downstream placeholders

- Task 2: add OpenTofu backend/provider files and verify S3 state safety.
- Task 3: add SOPS secret contract and runtime token delivery details.
- Task 4: define or package the Nix-capable runner image for the `nix` label.
- Task 5+: provision kube-hetzner, add Kubernetes resources, verify workflows,
  and document cleanup, rollback, and scaling operations.

## Runner lifecycle cleanup

All lifecycle commands are scoped to the runner namespace:

```sh
kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
```

The scheduled cleanup manifest is dry-run only. It lists the StatefulSet, active
runner pods, runner PVCs, and PVCs whose expected StatefulSet pod is absent. It
does not delete pods, PVCs, Docker data, or Gitea runner registrations.

Run the same inventory on demand without waiting for the schedule:

```sh
kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run
```

The cleanup job template has `ttlSecondsAfterFinished: 3600`, so completed
manual dry-run jobs are garbage-collected by Kubernetes instead of requiring an
operator to remove finished jobs manually.

If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching
pod loses `/data/.runner`. That runner identity must then be deregistered from
Gitea or the replacement pod must be allowed to re-register intentionally with
the current organization runner token. Do not delete an active runner PVC as a
normal cleanup step.

Non-UI Gitea registration reconciliation uses the Gitea API with a separate
admin token. Store that token outside this repository and pass it as a file; do
not print it:

```sh
kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run
```

Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run pod
has completed and its logs have been collected. Do not keep this admin token in
the runner namespace longer than the reconciliation window.

Only remove a stale Gitea runner registration after the corresponding pod/PVC
was intentionally deleted or `/data/.runner` was intentionally reset. Prefer a
Gitea CLI/API deletion from the Gitea server or an admin workstation; manual UI
cleanup is a fallback, not the only path. Record the removed runner name and the
Kubernetes PVC/pod deletion that made it stale.

After the dry-run list identifies a stale registration and the PVC/pod deletion
has been recorded, remove that exact Gitea runner by id through the API:

```sh
runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"
```

Do not run the delete command for a runner that still has an active
`gitea-runner-*` pod or a retained `data-gitea-runner-*` PVC unless that PVC is
being intentionally reset for re-registration.

## Docker-in-Docker storage cleanup

Docker layers live inside each DinD sidecar at `/var/lib/docker`, backed by the
pod-local `docker-graph` `emptyDir`; the host Docker socket is not used. Always
list disk usage before pruning, and run the command only against the `docker`
container in runner pods in `gitea-runners`. Because this storage is pod-local,
loop over pods for pool-wide cleanup:

```sh
for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done
```

For one pod, replace the StatefulSet target with the pod name:

```sh
kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system df
kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system prune --all --force --filter until=24h
```

Do not run host-level Docker cleanup commands and do not mount or prune a host
Docker socket. If a pod is deleted, its `emptyDir` Docker graph is removed by
Kubernetes; the `/data` PVC remains and still controls runner identity.

## Token rotation

Rotate the Gitea organization runner token without printing decrypted values:

```sh
sops sus/gitea-runners.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -
kubectl -n gitea-runners rollout restart statefulset/gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true
```

The `rollout restart` command above is the controlled restart path for this
StatefulSet. Observe the rollout and each ordinal until all replacement pods are
Ready; do not delete runner pods directly as part of normal token rotation:

```sh
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-0 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-1 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-2 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-3 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-4 --timeout=5m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
```

Verification must confirm the token file mount remains present while the token
value never appears in logs or evidence:

```sh
kubectl -n gitea-runners describe pod gitea-runner-0 | grep -n '/runner-secrets\|gitea-runner-token'
kubectl -n gitea-runners logs pod/gitea-runner-0 -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true
```

## Deploy and status

These commands are executable only when the external inputs are available:

- `TF_VAR_hcloud_token`
- `TF_VAR_ssh_public_key`
- `TF_VAR_ssh_private_key`
- S3 backend credentials and endpoint access
- a matching SOPS age identity for `sus/gitea-runners.yaml`
- `kubectl` access to the target cluster
- a concrete digest for the pushed Nix-capable runner image, if enabling the
  `nix` label

If any input is missing, stop before `tofu apply`. Do not guess values or reuse
stale kubeconfig files.

Before production Kubernetes apply or rollout, satisfy both manifest gates:

1. Create or update the `gitea-runner-token` Secret from SOPS. The active
   Kustomize overlay intentionally does not include a placeholder Secret, but
   the StatefulSet still mounts `secretName: gitea-runner-token` as
   `/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`.
2. Keep the active ConfigMap on `ubuntu-latest` only unless the Nix-capable
   image has been pushed successfully. Enable the `nix` label only by adding a
   digest-pinned `docker://` mapping with the exact registry-reported sha256
   digest from that push.

Use the same SOPS materialization pattern as token rotation before applying the
Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a
target namespace; the full overlay remains gated on the Secret and digest
decisions:

```sh
kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -
```

Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command
above succeeds. Do not claim or enable the `nix` runner label until the image
publication step has produced the concrete digest.

```sh
tofu -chdir=infra/gitea-runners/opentofu init
tofu -chdir=infra/gitea-runners/opentofu validate
tofu -chdir=infra/gitea-runners/opentofu plan -out=.sisyphus/evidence/task-12-deploy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-deploy.plan
export KUBECONFIG="$(tofu -chdir=infra/gitea-runners/opentofu output -raw kubeconfig_path)"
kubectl config current-context
kubectl get nodes -o wide
kubectl get sc
kubectl apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200
```

Expected status after deploy:

- `kubectl config current-context` names the runner cluster context.
- `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready.
- `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs.
- `kubectl -n gitea-runners get statefulset gitea-runner` shows 5 desired and 5 ready replicas.
- `kubectl -n gitea-runners get pvc` shows 5 Bound PVCs.
- `kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200` shows the runner daemon started and no token value.

## Scale 5 to 10 to 5

Scaling is a temporary capacity exercise, not the steady-state setting. Scale up,
wait for readiness, run the concurrent smoke jobs, then scale back down to 5.

```sh
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide

# Run the concurrent smoke workflows now.

kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
```

After scaling back down, inspect the cleanup dry-run and deregister any stale
runner registrations only for pods or PVCs that were intentionally removed.

## Cleanup and stale runner deregistration

Use the dry-run cleanup job to list the StatefulSet, active pods, PVCs, and any
PVC candidates whose pod is gone. It must not delete active resources.

```sh
kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run
```

Pool-wide DinD storage checks and cleanup:

```sh
for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done
```

If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching
pod loses `/data/.runner`. Deregister that runner from Gitea, or let the
replacement pod re-register intentionally with the current organization token.
Never delete an active runner PVC as routine cleanup.

Non-UI Gitea registration reconciliation uses an admin token stored outside this
repository:

```sh
kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run
```

Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run
pod has completed and its logs have been collected.

After the dry-run list identifies a stale registration and the PVC or pod
deletion has been recorded, remove that exact Gitea runner by id through the
API:

```sh
runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"
```

Do not run the delete command for a runner that still has an active
`gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is
being intentionally reset for re-registration.

## Application rollback

Rollback the app layer only. Do not use this section to destroy the cluster.

```sh
kubectl -n gitea-runners rollout history statefulset/gitea-runner
kubectl -n gitea-runners rollout undo statefulset/gitea-runner --to-revision=<known-good-revision>
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200
```

If a manifest rollback is needed, reapply the repo overlay after checking out the
known-good revision, then re-run the rollout checks:

```sh
kubectl -n gitea-runners apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
```

## Full cluster teardown

This destroys Hetzner resources owned by the kube-hetzner stack, including the
`gitea-runners` cluster nodes, the `control-plane` node pool, the
`runner-workers` node pool, the cluster network, load balancer resources,
firewall objects, and any attached Hetzner CSI volumes still managed by the
stack. Do not run teardown unless the destruction is intentional.

```sh
tofu -chdir=infra/gitea-runners/opentofu plan -destroy -out=.sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-destroy.plan
```

## Partial OpenTofu apply recovery

If `tofu apply` fails after creating some resources, do not destroy blindly.
First reconcile state and inspect what the stack thinks exists:

```sh
tofu -chdir=infra/gitea-runners/opentofu plan -refresh-only -out=.sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu state list
```

Then rerun the normal plan path. Use `-target` only as a last resort when a
single resource is stuck and the drift is understood.

## S3 backend recovery

If backend init or state access fails, first verify the bucket and versioning
outside OpenTofu, then reconfigure the backend:

```sh
nix run nixpkgs#awscli2 -- s3api head-bucket --bucket gitea-runner-hectic-lab
nix run nixpkgs#awscli2 -- s3api get-bucket-versioning --bucket gitea-runner-hectic-lab
tofu -chdir=infra/gitea-runners/opentofu init -reconfigure
tofu -chdir=infra/gitea-runners/opentofu plan
```

If the backend reports a stale lock, confirm no `tofu` process is active, then
use `tofu force-unlock <LOCK_ID>` with the lock id from the error. Never force
unlock a live plan or apply.

## Gitea outage troubleshooting

Use the public HTTPS service, not `localhost` or `127.0.0.1` inside job
containers.

```sh
kubectl -n gitea-runners run gitea-outage-probe --rm --restart=Never --image=curlimages/curl:8.10.1 -- curl -fsS https://gitea.hectic-lab.com/api/healthz
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -E 'connection refused|timeout|tls|certificate|temporary failure' || true
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50
```

If Gitea is down, keep the existing StatefulSet and PVCs intact. Do not delete
`/data/.runner` just because the service is unavailable. Once Gitea returns,
repeat the token rotation or re-registration path if a pod restarted while the
service was unavailable and lost its runner identity.

## Release checklist

Do not release unless the following evidence files exist and are readable:

- `.sisyphus/evidence/task-5-cluster-plan.txt`
- `.sisyphus/evidence/task-5-secret-plan-scan.txt`
- `.sisyphus/evidence/task-9-deploy.txt`
- `.sisyphus/evidence/task-9-secret-mount.txt`
- `.sisyphus/evidence/task-10-ubuntu-workflow.txt`
- `.sisyphus/evidence/task-10-nix-workflow.txt`
- `.sisyphus/evidence/task-11-scale.txt`
- `.sisyphus/evidence/task-11-restart-cleanup.txt`

If any evidence file is missing, stop and collect it before treating the runbook
as complete.