hinterland/hearth
2
2
Fork 1
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
ee8a33c3b04e46fcdf46355a4cc6c73105e43103
hearth/infra/gitea-runners/runbook.md
T
yukkop ee8a33c3b0
runner ubuntu smoke / ubuntu-latest label smoke (push) Successful in 0s
Details
runner nix smoke / nix label and flake smoke (push) Failing after 7s
Details
feat: move all workers to zero-idle
2026-09-10 22:30:58 +00:00

30 KiB
Raw Blame History

Gitea Runner Infrastructure Runbook

Scope

This directory is the repo-owned boundary for the Gitea Actions runner pool. The controller is the active zero-idle path; Kubernetes manifests and the Nix-capable image are retained for manual rollback and maintenance.

The target service is https://gitea.hectic-lab.com for the Gitea organization hectic-lab. The pool is trusted-only. "Ephemeral" means each controller VM and workflow job is disposable; the Kubernetes StatefulSet is rollback-only.

Immutable decisions

  • Infrastructure is managed with OpenTofu command examples only, using the tofu CLI.
  • Cloud provider is Hetzner; cluster bootstrap uses kube-hetzner.
  • Remote state uses the S3 backend bucket gitea-runner-hectic-lab.
  • Runner implementation is the non-Enterprise gitea/runner.
  • Runner registration uses a Gitea organization-scoped token for hectic-lab.
  • Runtime token delivery is SOPS-backed and mounted into the runner pod as a file read through GITEA_RUNNER_REGISTRATION_TOKEN_FILE; plaintext token environment variables are not the contract.
  • Kubernetes runner lifecycle uses a StatefulSet with one PVC per pod for /data, including /data/.runner.
  • Container builds run through privileged rootful DinD inside trusted runner pods; host Docker socket mounting is not an implementation path.
  • ubuntu-latest and nix are controller-managed zero-idle aliases for gross-x86 and gross-nix-x86; the Kubernetes pool has no active labels.
  • First scope is trusted internal workflows only, with no untrusted fork or PR workflow support.
  • Zero-idle allocation is handled by the repo-owned controller; Kubernetes is not an active autoscaling path.

Lifecycle boundaries

  • infra/gitea-runners/opentofu/: OpenTofu stack for the S3 backend contract, Hetzner provider configuration, and kube-hetzner module wiring.
  • infra/gitea-runners/k8s/: rollback-only namespace, ConfigMap, Secret mount, StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work.
  • infra/gitea-runners/image/: notes and handoff for the optional Kubernetes rollback image; active zero-idle Nix image is selected by Hetzner image ID.
  • infra/gitea-runners/runbook.md: this contract plus later operational commands, rollback notes, and acceptance evidence references.

Guardrails

  • Enterprise ARC/actions-runner-controller are rejected alternatives and must not be implemented here. Do not add ARC custom resources, controller install instructions, or GitHub Actions ARC assumptions.
  • Untrusted fork/PR workflows are out of first scope; privileged DinD is only acceptable for trusted internal jobs.
  • The persistent StatefulSet is rollback-only and defaults to zero replicas; normal jobs use controller-managed zero-idle VMs.
  • No actual secrets are committed: no kubeconfig, runner token, Hetzner token, S3 credentials, decrypted SOPS files, or SOPS age keys.
  • OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea runner token; Kubernetes receives the token as a mounted file secret instead.
  • Do not use localhost or 127.0.0.1 as the Gitea URL inside job containers; jobs must reach the public HTTPS service.

Initial acceptance commands

Run from the repository root:

test -d infra/gitea-runners/opentofu && test -d infra/gitea-runners/k8s && test -d infra/gitea-runners/image
test -f infra/gitea-runners/runbook.md
grep -n "OpenTofu\|kube-hetzner\|StatefulSet\|DinD\|SOPS\|trusted" infra/gitea-runners/runbook.md
grep -R "[E]nterprise ARC\|[a]ctions-runner-controller" infra/gitea-runners
grep -R "[t]erraform " infra/gitea-runners || true
grep -R "[D]ECISION NEEDED" infra/gitea-runners || true

Expected outcomes: the directory and file checks exit 0; the architecture-term grep shows this contract; ARC references appear only in the rejected-alternative guardrail above; there are no forbidden CLI command examples and no unresolved decision placeholders.

Downstream placeholders

  • Task 2: add OpenTofu backend/provider files and verify S3 state safety.
  • Task 3: add SOPS secret contract and runtime token delivery details.
  • Task 4: define or package the Nix-capable runner image for the nix label.
  • Task 5+: provision kube-hetzner, add Kubernetes resources, verify workflows, and document cleanup, rollback, and scaling operations.

Runner lifecycle cleanup

All lifecycle commands are scoped to the runner namespace:

kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide

The scheduled cleanup manifest is dry-run only. It lists the StatefulSet, active runner pods, runner PVCs, and PVCs whose expected StatefulSet pod is absent. It does not delete pods, PVCs, Docker data, or Gitea runner registrations.

Run the same inventory on demand without waiting for the schedule:

kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run

The cleanup job template has ttlSecondsAfterFinished: 3600, so completed manual dry-run jobs are garbage-collected by Kubernetes instead of requiring an operator to remove finished jobs manually.

If a PVC such as data-gitea-runner-3 is intentionally deleted, the matching pod loses /data/.runner. That runner identity must then be deregistered from Gitea or the replacement pod must be allowed to re-register intentionally with the current organization runner token. Do not delete an active runner PVC as a normal cleanup step.

Non-UI Gitea registration reconciliation uses the Gitea API with a separate admin token. Store that token outside this repository and pass it as a file; do not print it:

kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run

Delete the temporary gitea-runner-admin-token Secret only after the dry-run pod has completed and its logs have been collected. Do not keep this admin token in the runner namespace longer than the reconciliation window.

Only remove a stale Gitea runner registration after the corresponding pod/PVC was intentionally deleted or /data/.runner was intentionally reset. Prefer a Gitea CLI/API deletion from the Gitea server or an admin workstation; manual UI cleanup is a fallback, not the only path. Record the removed runner name and the Kubernetes PVC/pod deletion that made it stale.

After the dry-run list identifies a stale registration and the PVC/pod deletion has been recorded, remove that exact Gitea runner by id through the API:

runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"

Do not run the delete command for a runner that still has an active gitea-runner-* pod or a retained data-gitea-runner-* PVC unless that PVC is being intentionally reset for re-registration.

Docker-in-Docker storage cleanup

Docker layers live inside each DinD sidecar at /var/lib/docker, backed by the pod-local docker-graph emptyDir; the host Docker socket is not used. Always list disk usage before pruning, and run the command only against the docker container in runner pods in gitea-runners. Because this storage is pod-local, loop over pods for pool-wide cleanup:

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

For one pod, replace the StatefulSet target with the pod name:

kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system df
kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system prune --all --force --filter until=24h

Do not run host-level Docker cleanup commands and do not mount or prune a host Docker socket. If a pod is deleted, its emptyDir Docker graph is removed by Kubernetes; the /data PVC remains and still controls runner identity.

Token rotation

Rotate the Gitea organization runner token without printing decrypted values:

sops sus/gitea-runners.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -
kubectl -n gitea-runners rollout restart statefulset/gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true

The rollout restart command above is the controlled restart path for this StatefulSet. Observe the rollout and each ordinal until all replacement pods are Ready; do not delete runner pods directly as part of normal token rotation:

kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-0 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-1 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-2 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-3 --timeout=5m
kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-4 --timeout=5m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide

Verification must confirm the token file mount remains present while the token value never appears in logs or evidence:

kubectl -n gitea-runners describe pod gitea-runner-0 | grep -n '/runner-secrets\|gitea-runner-token'
kubectl -n gitea-runners logs pod/gitea-runner-0 -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true

Deploy and status

These commands are executable only when the external inputs are available:

  • TF_VAR_hcloud_token
  • TF_VAR_ssh_public_key
  • TF_VAR_ssh_private_key
  • S3 backend credentials and endpoint access
  • a matching SOPS age identity for sus/gitea-runners.yaml
  • kubectl access to the target cluster
  • a valid GCR_NIX_IMAGE_ID for controller-managed Nix jobs

If any input is missing, stop before tofu apply. Do not guess values or reuse stale kubeconfig files.

Before production Kubernetes apply or rollout, satisfy both manifest gates:

  1. Create or update the gitea-runner-token Secret from SOPS. The active Kustomize overlay intentionally does not include a placeholder Secret, but the StatefulSet still mounts secretName: gitea-runner-token as /runner-secrets/token for GITEA_RUNNER_REGISTRATION_TOKEN_FILE.
  2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the controller; Nix image readiness is governed by GCR_NIX_IMAGE_ID.

Use the same SOPS materialization pattern as token rotation before applying the Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a target namespace; the full overlay remains gated on the Secret.

kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
umask 077
token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX)
trap 'rm -f "$token_file"' EXIT
sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file"
kubectl -n gitea-runners create secret generic gitea-runner-token \
  --from-file=token="$token_file" \
  --dry-run=client \
  -o yaml | kubectl -n gitea-runners apply -f -

Do not run kubectl apply -k infra/gitea-runners/k8s until the Secret command above succeeds. The persistent pool ConfigMap must retain empty labels.

tofu -chdir=infra/gitea-runners/opentofu init
tofu -chdir=infra/gitea-runners/opentofu validate
tofu -chdir=infra/gitea-runners/opentofu plan -out=.sisyphus/evidence/task-12-deploy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-deploy.plan
export KUBECONFIG="$(tofu -chdir=infra/gitea-runners/opentofu output -raw kubeconfig_path)"
kubectl config current-context
kubectl get nodes -o wide
kubectl get sc
kubectl apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners get statefulset gitea-runner
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200

Expected status after deploy:

  • kubectl config current-context names the runner cluster context.
  • kubectl get nodes -o wide shows all expected Hetzner nodes Ready.
  • kubectl get sc shows the Hetzner CSI storage class used by runner PVCs.
  • kubectl -n gitea-runners get statefulset gitea-runner shows 0 desired and 0 ready replicas.
  • kubectl -n gitea-runners get pvc shows no active runner PVCs; retained PVCs are rollback-only.
  • The controller host reports healthy and owns runner registrations; no persistent runner claims ubuntu-latest or nix.

Legacy rollback pool scaling (manual only)

Persistent-pool scaling is not part of normal operation. Use only after restoring its labels and disabling the zero-idle controller as described in Rollback.

kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide

# Run only workflows supported by restored persistent labels.

kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide

After scaling back down, inspect the cleanup dry-run and deregister any stale runner registrations only for pods or PVCs that were intentionally removed.

Cleanup and stale runner deregistration

Use the dry-run cleanup job to list the StatefulSet, active pods, PVCs, and any PVC candidates whose pod is gone. It must not delete active resources.

kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run
kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m
kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run

Pool-wide DinD storage checks and cleanup:

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h
done

for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do
  kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df
done

If a PVC such as data-gitea-runner-3 is intentionally deleted, the matching pod loses /data/.runner. Deregister that runner from Gitea, or let the replacement pod re-register intentionally with the current organization token. Never delete an active runner PVC as routine cleanup.

Non-UI Gitea registration reconciliation uses an admin token stored outside this repository:

kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token
kubectl -n gitea-runners run gitea-runner-registration-dry-run \
  --restart=Never \
  --image=curlimages/curl:8.10.1 \
  --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}'
kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run

Delete the temporary gitea-runner-admin-token Secret only after the dry-run pod has completed and its logs have been collected.

After the dry-run list identifies a stale registration and the PVC or pod deletion has been recorded, remove that exact Gitea runner by id through the API:

runner_id='REPLACE_WITH_STALE_RUNNER_ID'
umask 077
curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX)
trap 'rm -f "$curl_config"' EXIT
{
  printf 'request = "DELETE"\n'
  printf 'header = "Authorization: token '
  cat /secure/path/gitea-admin-token
  printf '"\n'
  printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id"
} > "$curl_config"
curl -fsS --config "$curl_config"

Do not run the delete command for a runner that still has an active gitea-runner-* pod or retained data-gitea-runner-* PVC unless that PVC is being intentionally reset for re-registration.

Legacy Kubernetes application rollback

Rollback the app layer only. Do not use this section to destroy the cluster.

kubectl -n gitea-runners rollout history statefulset/gitea-runner
kubectl -n gitea-runners rollout undo statefulset/gitea-runner --to-revision=<known-good-revision>
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200

If a manifest rollback is needed, reapply the repo overlay after checking out the known-good revision, then re-run the rollout checks:

kubectl -n gitea-runners apply -k infra/gitea-runners/k8s
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m

Full cluster teardown

This destroys Hetzner resources owned by the kube-hetzner stack, including the gitea-runners cluster nodes, the control-plane node pool, the runner-workers node pool, the cluster network, load balancer resources, firewall objects, and any attached Hetzner CSI volumes still managed by the stack. Do not run teardown unless the destruction is intentional.

tofu -chdir=infra/gitea-runners/opentofu plan -destroy -out=.sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-destroy.plan
tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-destroy.plan

Partial OpenTofu apply recovery

If tofu apply fails after creating some resources, do not destroy blindly. First reconcile state and inspect what the stack thinks exists:

tofu -chdir=infra/gitea-runners/opentofu plan -refresh-only -out=.sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-refresh.plan
tofu -chdir=infra/gitea-runners/opentofu state list

Then rerun the normal plan path. Use -target only as a last resort when a single resource is stuck and the drift is understood.

S3 backend recovery

If backend init or state access fails, first verify the bucket and versioning outside OpenTofu, then reconfigure the backend:

nix run nixpkgs#awscli2 -- s3api head-bucket --bucket gitea-runner-hectic-lab
nix run nixpkgs#awscli2 -- s3api get-bucket-versioning --bucket gitea-runner-hectic-lab
tofu -chdir=infra/gitea-runners/opentofu init -reconfigure
tofu -chdir=infra/gitea-runners/opentofu plan

If the backend reports a stale lock, confirm no tofu process is active, then use tofu force-unlock <LOCK_ID> with the lock id from the error. Never force unlock a live plan or apply.

Gitea outage troubleshooting

Use the public HTTPS service, not localhost or 127.0.0.1 inside job containers.

kubectl -n gitea-runners run gitea-outage-probe --rm --restart=Never --image=curlimages/curl:8.10.1 -- curl -fsS https://gitea.hectic-lab.com/api/healthz
kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -E 'connection refused|timeout|tls|certificate|temporary failure' || true
kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50

If Gitea is down, keep the existing StatefulSet and PVCs intact. Do not delete /data/.runner just because the service is unavailable. Once Gitea returns, repeat the token rotation or re-registration path if a pod restarted while the service was unavailable and lost its runner identity.

Release checklist

Do not release unless the following evidence files exist and are readable:

  • .sisyphus/evidence/task-5-cluster-plan.txt
  • .sisyphus/evidence/task-5-secret-plan-scan.txt
  • .sisyphus/evidence/task-9-deploy.txt
  • .sisyphus/evidence/task-9-secret-mount.txt
  • .sisyphus/evidence/task-10-ubuntu-workflow.txt
  • .sisyphus/evidence/task-10-nix-workflow.txt
  • .sisyphus/evidence/task-11-scale.txt
  • .sisyphus/evidence/task-11-restart-cleanup.txt

If any evidence file is missing, stop and collect it before treating the runbook as complete.

Ephemeral VM runner cutover

This section governs replacing the legacy persistent K8s runner pool with the ephemeral-VM controller (package/gitea-runner-controller) on this host. The K8s pool above remains rollback-only until cutover is explicitly accepted.

Operator gates (all three required before enable)

  1. Secrets — add to sus/gitea-runners.yaml under gitea/hectic-lab/controller/*:

    sops sus/gitea-runners.yaml
# add keys:
#   gitea:
#     hectic-lab:
#       controller:
#         hcloud-token:    <Hetzner API token, VM create/destroy scope>
#         webhook-secret:  <random 32+ bytes; also set as Gitea webhook secret>
#         admin-token:     <Gitea token with admin:runner scope for stale cleanup>

    The org registration token key gitea/hectic-lab/org-runner-registration-token already exists and is reused.

  2. Base image — build the MicroOS snapshot and record its id:

    nix develop .#gitea-runners -c gitea-runners-build-microos-snapshots x86
hcloud image list --selector '' -o json | jq '.[] | select(.type=="snapshot")'

  3. DNS — A record runners.hectic-lab.com -> 128.140.75.58 (ACME needs it).

Enable

# nixos/system/hectic-lab/hectic-lab.nix: resolve the FIXME block
hectic.services.gitea-runner-controller = {
  enable  = true;
  imageId = "<snapshot-id-from-gate-2>";
};
nixos-rebuild --target root@128.140.75.58 switch
systemctl status gitea-runner-controller.service gitea-runner-webhook.service

Register the Gitea webhook

Org-level (preferred) or per-repo, on https://gitea.hectic-lab.com:

  • URL: https://runners.hectic-lab.com/
  • Method: POST, content type: JSON
  • Secret: value of gitea/hectic-lab/controller/webhook-secret
  • Trigger events: Workflow jobs only (workflow_job)

Long CUDA/Magma deployment time budgets

The deploy-neuro workflow uses these nested limits:

Layer Limit
Wrapped build/deploy command 6 hours (WITH_ATTIC_BUILD_TIMEOUT=21600)
Final cache drain 1 hour (WITH_ATTIC_DRAIN_TIMEOUT=3600)
Workflow job 435 minutes, including 15 minutes of setup/cleanup margin
gross-nix-x86-perf runner 480 minutes
gross-nix-x86-highmem runner 480 minutes
Gitea actions.ENDLESS_TASK_TIMEOUT 8 hours
VM hard lifetime from allocation 480 minutes plus 10-minute controller grace

ubuntu-latest keeps a 180-minute limit; nix uses a 480-minute limit for long-running Nix deployments. Deploy the controller and Gitea watchdog settings before dispatching the longer workflow. Already allocated VMs retain the TTL and runner configuration assigned when they were created; updating the controller does not extend a running job.

gross-nix-x86-highmem is an explicit costly high-memory escape hatch backed only by Hetzner CCX53 in nbg1, fsn1, or hel1; it may fall back by region only, never to a lower-RAM server type. Current Hetzner public pricing for Germany/Finland CCX53 is 0.8550 EUR/hour excluding IPv4, so one 480-minute allocation reserves 6.84 EUR against the controller budget before VM creation.

These are maximum lifetimes: terminal jobs still trigger immediate VM teardown. The controller's budget reservation uses the full label TTL, so a long-running label reserves more of the existing monthly budget. Do not raise that budget or disable timeout safeguards just to bypass a refused allocation.

After changing any timeout, verify the complete chain rather than only timeout-minutes; a shorter wrapper, runner, server watchdog, or VM TTL wins.

Pre-flight verification (before first real job)

curl -fsS https://runners.hectic-lab.com/ -o /dev/null -w '%{http_code}\n' # any 4xx from handler = reachable
journalctl -u gitea-runner-webhook -n 20 --no-pager
hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed")] | length' # expect 0

Zero managed VMs at idle is the steady-state assertion.

End-to-end acceptance (Task 9)

Trigger .gitea/workflows/runner-nix-smoke.yaml via workflow_dispatch, then:

watch_labels() { hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed") | {id,name,labels}]'; }
watch_labels                                   # exactly one VM while queued/running
journalctl -f -u gitea-runner-controller       # vm-created / vm-destroyed events
watch_labels                                   # expect [] after completion
curl -fsS -H "Authorization: token $ADMIN" \
  https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners \
  | jq '[.entries[] | select(.name | startswith("gcr-"))] | length'  # expect 0

Failure paths to verify identically: duplicate delivery (send same webhook twice via Gitea UI "Test delivery"), cancelled job, unknown-label job.

Rollback

K8s rollback pool now defaults to deleted state:

  • infra/gitea-runners/k8s/statefulset.yaml keeps replicas: 0
  • old kube-hetzner nodes may be deleted to preserve zero idle cost
  • PVCs and IaC remain for manual rollback only

Re-enable sequence:

# 1. edit nixos/system/hectic-lab/hectic-lab.nix and set
#    services.gitea-runner-controller.enable = false, then rebuild:
nixos-rebuild --target root@128.140.75.58 switch

# 2. reprovision old kube-hetzner nodes when they were deleted:
tofu -chdir=infra/gitea-runners/opentofu apply

# 3. while controller is disabled, destroy every surviving managed VM and
#    verify no gcr-* runner registration remains online:
hcloud server list -o json \
  | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
  | xargs -r -n1 hcloud server delete

# 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the
#    ConfigMap labels, then re-enable K8s runner pool. The legacy image does not
#    provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image
#    and label mapping are restored:
kubectl -n gitea-runners edit configmap/gitea-runner-config
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m

Any managed VM or gcr-* registration found after step 3 must be removed before restoring persistent labels; otherwise both pools can claim the same job.

Reference in New Issue View Git Blame Copy Permalink