# Gitea Runner Infrastructure Runbook ## Scope This directory is the repo-owned boundary for the Gitea Actions runner pool. The controller is the active on-demand path; Kubernetes manifests and the Nix-capable image are retained for manual rollback and maintenance. The target service is `https://gitea.hectic-lab.com` for the Gitea organization `hectic-lab`. The pool is trusted-only. "Ephemeral" means each controller VM and workflow job is disposable; the Kubernetes StatefulSet is rollback-only. ## Immutable decisions - Infrastructure is managed with OpenTofu command examples only, using the `tofu` CLI. - Cloud provider is Hetzner; cluster bootstrap uses kube-hetzner. - Remote state uses the S3 backend bucket `gitea-runner-hectic-lab`. - Runner implementation is the non-Enterprise `gitea/runner`. - Runner registration uses a Gitea organization-scoped token for `hectic-lab`. - Runtime token delivery is SOPS-backed and mounted into the runner pod as a file read through `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`; plaintext token environment variables are not the contract. - Kubernetes runner lifecycle uses a StatefulSet with one PVC per pod for `/data`, including `/data/.runner`. - Container builds run through privileged rootful DinD inside trusted runner pods; host Docker socket mounting is not an implementation path. - `ubuntu-latest` and `nix` are controller-managed on-demand aliases for `gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels. - First scope is trusted internal workflows only, with no untrusted fork or PR workflow support. - On-demand allocation is handled by the repo-owned controller; Kubernetes is not an active autoscaling path. ## Lifecycle boundaries - `infra/gitea-runners/opentofu/`: OpenTofu stack for the S3 backend contract, Hetzner provider configuration, and kube-hetzner module wiring. - `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount, StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work. - `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes rollback image; active on-demand Nix image is selected by Hetzner image ID. - `infra/gitea-runners/runbook.md`: this contract plus later operational commands, rollback notes, and acceptance evidence references. ## Guardrails - Enterprise ARC/actions-runner-controller are rejected alternatives and must not be implemented here. Do not add ARC custom resources, controller install instructions, or GitHub Actions ARC assumptions. - Untrusted fork/PR workflows are out of first scope; privileged DinD is only acceptable for trusted internal jobs. - The persistent StatefulSet is rollback-only and defaults to zero replicas; normal jobs use controller-managed on-demand VMs. - No actual secrets are committed: no kubeconfig, runner token, Hetzner token, S3 credentials, decrypted SOPS files, or SOPS age keys. - OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea runner token; Kubernetes receives the token as a mounted file secret instead. - Do not use `localhost` or `127.0.0.1` as the Gitea URL inside job containers; jobs must reach the public HTTPS service. ## Initial acceptance commands Run from the repository root: ```sh test -d infra/gitea-runners/opentofu && test -d infra/gitea-runners/k8s && test -d infra/gitea-runners/image test -f infra/gitea-runners/runbook.md grep -n "OpenTofu\|kube-hetzner\|StatefulSet\|DinD\|SOPS\|trusted" infra/gitea-runners/runbook.md grep -R "[E]nterprise ARC\|[a]ctions-runner-controller" infra/gitea-runners grep -R "[t]erraform " infra/gitea-runners || true grep -R "[D]ECISION NEEDED" infra/gitea-runners || true ``` Expected outcomes: the directory and file checks exit 0; the architecture-term grep shows this contract; ARC references appear only in the rejected-alternative guardrail above; there are no forbidden CLI command examples and no unresolved decision placeholders. ## Downstream placeholders - Task 2: add OpenTofu backend/provider files and verify S3 state safety. - Task 3: add SOPS secret contract and runtime token delivery details. - Task 4: define or package the Nix-capable runner image for the `nix` label. - Task 5+: provision kube-hetzner, add Kubernetes resources, verify workflows, and document cleanup, rollback, and scaling operations. ## Runner lifecycle cleanup All lifecycle commands are scoped to the runner namespace: ```sh kubectl -n gitea-runners get statefulset gitea-runner kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide ``` The scheduled cleanup manifest is dry-run only. It lists the StatefulSet, active runner pods, runner PVCs, and PVCs whose expected StatefulSet pod is absent. It does not delete pods, PVCs, Docker data, or Gitea runner registrations. Run the same inventory on demand without waiting for the schedule: ```sh kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run ``` The cleanup job template has `ttlSecondsAfterFinished: 3600`, so completed manual dry-run jobs are garbage-collected by Kubernetes instead of requiring an operator to remove finished jobs manually. If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching pod loses `/data/.runner`. That runner identity must then be deregistered from Gitea or the replacement pod must be allowed to re-register intentionally with the current organization runner token. Do not delete an active runner PVC as a normal cleanup step. Non-UI Gitea registration reconciliation uses the Gitea API with a separate admin token. Store that token outside this repository and pass it as a file; do not print it: ```sh kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token kubectl -n gitea-runners run gitea-runner-registration-dry-run \ --restart=Never \ --image=curlimages/curl:8.10.1 \ --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}' kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run ``` Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run pod has completed and its logs have been collected. Do not keep this admin token in the runner namespace longer than the reconciliation window. Only remove a stale Gitea runner registration after the corresponding pod/PVC was intentionally deleted or `/data/.runner` was intentionally reset. Prefer a Gitea CLI/API deletion from the Gitea server or an admin workstation; manual UI cleanup is a fallback, not the only path. Record the removed runner name and the Kubernetes PVC/pod deletion that made it stale. After the dry-run list identifies a stale registration and the PVC/pod deletion has been recorded, remove that exact Gitea runner by id through the API: ```sh runner_id='REPLACE_WITH_STALE_RUNNER_ID' umask 077 curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX) trap 'rm -f "$curl_config"' EXIT { printf 'request = "DELETE"\n' printf 'header = "Authorization: token ' cat /secure/path/gitea-admin-token printf '"\n' printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id" } > "$curl_config" curl -fsS --config "$curl_config" ``` Do not run the delete command for a runner that still has an active `gitea-runner-*` pod or a retained `data-gitea-runner-*` PVC unless that PVC is being intentionally reset for re-registration. ## Docker-in-Docker storage cleanup Docker layers live inside each DinD sidecar at `/var/lib/docker`, backed by the pod-local `docker-graph` `emptyDir`; the host Docker socket is not used. Always list disk usage before pruning, and run the command only against the `docker` container in runner pods in `gitea-runners`. Because this storage is pod-local, loop over pods for pool-wide cleanup: ```sh for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df done for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h done for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df done ``` For one pod, replace the StatefulSet target with the pod name: ```sh kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system df kubectl -n gitea-runners exec pod/gitea-runner-0 -c docker -- docker system prune --all --force --filter until=24h ``` Do not run host-level Docker cleanup commands and do not mount or prune a host Docker socket. If a pod is deleted, its `emptyDir` Docker graph is removed by Kubernetes; the `/data` PVC remains and still controls runner identity. ## Token rotation Rotate the Gitea organization runner token without printing decrypted values: ```sh sops sus/gitea-runners.yaml umask 077 token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX) trap 'rm -f "$token_file"' EXIT sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file" kubectl -n gitea-runners create secret generic gitea-runner-token \ --from-file=token="$token_file" \ --dry-run=client \ -o yaml | kubectl -n gitea-runners apply -f - kubectl -n gitea-runners rollout restart statefulset/gitea-runner kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true ``` The `rollout restart` command above is the controlled restart path for this StatefulSet. Observe the rollout and each ordinal until all replacement pods are Ready; do not delete runner pods directly as part of normal token rotation: ```sh kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-0 --timeout=5m kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-1 --timeout=5m kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-2 --timeout=5m kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-3 --timeout=5m kubectl -n gitea-runners wait --for=condition=Ready pod/gitea-runner-4 --timeout=5m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide ``` Verification must confirm the token file mount remains present while the token value never appears in logs or evidence: ```sh kubectl -n gitea-runners describe pod gitea-runner-0 | grep -n '/runner-secrets\|gitea-runner-token' kubectl -n gitea-runners logs pod/gitea-runner-0 -c runner --tail=200 | grep -Eq 'token|GITEA_RUNNER_REGISTRATION_TOKEN' && exit 1 || true ``` ## Deploy and status These commands are executable only when the external inputs are available: - `TF_VAR_hcloud_token` - `TF_VAR_ssh_public_key` - `TF_VAR_ssh_private_key` - S3 backend credentials and endpoint access - a matching SOPS age identity for `sus/gitea-runners.yaml` - `kubectl` access to the target cluster - a valid `GCR_NIX_IMAGE_ID` for controller-managed Nix jobs If any input is missing, stop before `tofu apply`. Do not guess values or reuse stale kubeconfig files. Before production Kubernetes apply or rollout, satisfy both manifest gates: 1. Create or update the `gitea-runner-token` Secret from SOPS. The active Kustomize overlay intentionally does not include a placeholder Secret, but the StatefulSet still mounts `secretName: gitea-runner-token` as `/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`. 2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the controller; Nix image readiness is governed by `GCR_NIX_IMAGE_ID`. Use the same SOPS materialization pattern as token rotation before applying the Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a target namespace; the full overlay remains gated on the Secret. ```sh kubectl apply -f infra/gitea-runners/k8s/namespace.yaml umask 077 token_file=$(mktemp /tmp/gitea-runner-token.XXXXXX) trap 'rm -f "$token_file"' EXIT sops -d --extract '["gitea"]["hectic-lab"]["org-runner-registration-token"]' sus/gitea-runners.yaml > "$token_file" kubectl -n gitea-runners create secret generic gitea-runner-token \ --from-file=token="$token_file" \ --dry-run=client \ -o yaml | kubectl -n gitea-runners apply -f - ``` Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command above succeeds. The persistent pool ConfigMap must retain empty labels. ```sh tofu -chdir=infra/gitea-runners/opentofu init tofu -chdir=infra/gitea-runners/opentofu validate tofu -chdir=infra/gitea-runners/opentofu plan -out=.sisyphus/evidence/task-12-deploy.plan tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-deploy.plan export KUBECONFIG="$(tofu -chdir=infra/gitea-runners/opentofu output -raw kubeconfig_path)" kubectl config current-context kubectl get nodes -o wide kubectl get sc kubectl apply -k infra/gitea-runners/k8s kubectl -n gitea-runners get statefulset gitea-runner kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50 kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 ``` Expected status after deploy: - `kubectl config current-context` names the runner cluster context. - `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready. - `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs. - `kubectl -n gitea-runners get statefulset gitea-runner` shows 0 desired and 0 ready replicas. - `kubectl -n gitea-runners get pvc` shows no active runner PVCs; retained PVCs are rollback-only. - The controller host reports healthy and owns runner registrations; no persistent runner claims `ubuntu-latest` or `nix`. ## Legacy rollback pool scaling (manual only) Persistent-pool scaling is not part of normal operation. Use only after restoring its labels and disabling the zero-idle controller as described in `Rollback`. ```sh kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide # Run only workflows supported by restored persistent labels. kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide ``` After scaling back down, inspect the cleanup dry-run and deregister any stale runner registrations only for pods or PVCs that were intentionally removed. ## Cleanup and stale runner deregistration Use the dry-run cleanup job to list the StatefulSet, active pods, PVCs, and any PVC candidates whose pod is gone. It must not delete active resources. ```sh kubectl -n gitea-runners create job gitea-runner-cleanup-dry-run-manual --from=cronjob/gitea-runner-cleanup-dry-run kubectl -n gitea-runners wait --for=condition=complete job/gitea-runner-cleanup-dry-run-manual --timeout=2m kubectl -n gitea-runners logs job/gitea-runner-cleanup-dry-run-manual -c cleanup-dry-run ``` Pool-wide DinD storage checks and cleanup: ```sh for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df done for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system prune --all --force --filter until=24h done for pod in $(kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'); do kubectl -n gitea-runners exec "pod/${pod}" -c docker -- docker system df done ``` If a PVC such as `data-gitea-runner-3` is intentionally deleted, the matching pod loses `/data/.runner`. Deregister that runner from Gitea, or let the replacement pod re-register intentionally with the current organization token. Never delete an active runner PVC as routine cleanup. Non-UI Gitea registration reconciliation uses an admin token stored outside this repository: ```sh kubectl -n gitea-runners create secret generic gitea-runner-admin-token --from-file=token=/secure/path/gitea-admin-token kubectl -n gitea-runners run gitea-runner-registration-dry-run \ --restart=Never \ --image=curlimages/curl:8.10.1 \ --overrides='{"spec":{"containers":[{"name":"gitea-runner-registration-dry-run","image":"curlimages/curl:8.10.1","command":["/bin/sh","-ec","umask 077; cfg=$(mktemp); trap '\''rm -f \"$cfg\"'\'' EXIT; { printf '\''header = \"Authorization: token '\''; cat /admin-token/token; printf '\''\"\\n'\''; printf '\''url = \"https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners\"\\n'\''; } > \"$cfg\"; curl -fsS --config \"$cfg\""],"volumeMounts":[{"name":"admin-token","mountPath":"/admin-token","readOnly":true}]}],"volumes":[{"name":"admin-token","secret":{"secretName":"gitea-runner-admin-token","defaultMode":256}}]}}' kubectl -n gitea-runners logs pod/gitea-runner-registration-dry-run ``` Delete the temporary `gitea-runner-admin-token` Secret only after the dry-run pod has completed and its logs have been collected. After the dry-run list identifies a stale registration and the PVC or pod deletion has been recorded, remove that exact Gitea runner by id through the API: ```sh runner_id='REPLACE_WITH_STALE_RUNNER_ID' umask 077 curl_config=$(mktemp /tmp/gitea-runner-admin-curl.XXXXXX) trap 'rm -f "$curl_config"' EXIT { printf 'request = "DELETE"\n' printf 'header = "Authorization: token ' cat /secure/path/gitea-admin-token printf '"\n' printf 'url = "https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners/%s"\n' "$runner_id" } > "$curl_config" curl -fsS --config "$curl_config" ``` Do not run the delete command for a runner that still has an active `gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is being intentionally reset for re-registration. ## Legacy Kubernetes application rollback Rollback the app layer only. Do not use this section to destroy the cluster. ```sh kubectl -n gitea-runners rollout history statefulset/gitea-runner kubectl -n gitea-runners rollout undo statefulset/gitea-runner --to-revision= kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 ``` If a manifest rollback is needed, reapply the repo overlay after checking out the known-good revision, then re-run the rollout checks: ```sh kubectl -n gitea-runners apply -k infra/gitea-runners/k8s kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m ``` ## Full cluster teardown This destroys Hetzner resources owned by the kube-hetzner stack, including the `gitea-runners` cluster nodes, the `control-plane` node pool, the `runner-workers` node pool, the cluster network, load balancer resources, firewall objects, and any attached Hetzner CSI volumes still managed by the stack. Do not run teardown unless the destruction is intentional. ```sh tofu -chdir=infra/gitea-runners/opentofu plan -destroy -out=.sisyphus/evidence/task-12-destroy.plan tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-destroy.plan tofu -chdir=infra/gitea-runners/opentofu apply .sisyphus/evidence/task-12-destroy.plan ``` ## Partial OpenTofu apply recovery If `tofu apply` fails after creating some resources, do not destroy blindly. First reconcile state and inspect what the stack thinks exists: ```sh tofu -chdir=infra/gitea-runners/opentofu plan -refresh-only -out=.sisyphus/evidence/task-12-refresh.plan tofu -chdir=infra/gitea-runners/opentofu show -no-color .sisyphus/evidence/task-12-refresh.plan tofu -chdir=infra/gitea-runners/opentofu state list ``` Then rerun the normal plan path. Use `-target` only as a last resort when a single resource is stuck and the drift is understood. ## S3 backend recovery If backend init or state access fails, first verify the bucket and versioning outside OpenTofu, then reconfigure the backend: ```sh nix run nixpkgs#awscli2 -- s3api head-bucket --bucket gitea-runner-hectic-lab nix run nixpkgs#awscli2 -- s3api get-bucket-versioning --bucket gitea-runner-hectic-lab tofu -chdir=infra/gitea-runners/opentofu init -reconfigure tofu -chdir=infra/gitea-runners/opentofu plan ``` If the backend reports a stale lock, confirm no `tofu` process is active, then use `tofu force-unlock ` with the lock id from the error. Never force unlock a live plan or apply. ## Gitea outage troubleshooting Use the public HTTPS service, not `localhost` or `127.0.0.1` inside job containers. ```sh kubectl -n gitea-runners run gitea-outage-probe --rm --restart=Never --image=curlimages/curl:8.10.1 -- curl -fsS https://gitea.hectic-lab.com/api/healthz kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200 | grep -E 'connection refused|timeout|tls|certificate|temporary failure' || true kubectl -n gitea-runners get events --sort-by=.lastTimestamp | tail -n 50 ``` If Gitea is down, keep the existing StatefulSet and PVCs intact. Do not delete `/data/.runner` just because the service is unavailable. Once Gitea returns, repeat the token rotation or re-registration path if a pod restarted while the service was unavailable and lost its runner identity. ## Release checklist Do not release unless the following evidence files exist and are readable: - `.sisyphus/evidence/task-5-cluster-plan.txt` - `.sisyphus/evidence/task-5-secret-plan-scan.txt` - `.sisyphus/evidence/task-9-deploy.txt` - `.sisyphus/evidence/task-9-secret-mount.txt` - `.sisyphus/evidence/task-10-ubuntu-workflow.txt` - `.sisyphus/evidence/task-10-nix-workflow.txt` - `.sisyphus/evidence/task-11-scale.txt` - `.sisyphus/evidence/task-11-restart-cleanup.txt` If any evidence file is missing, stop and collect it before treating the runbook as complete. ## Ephemeral VM runner cutover This section governs replacing the legacy persistent K8s runner pool with the ephemeral-VM controller (`package/gitea-runner-controller`) on this host. The K8s pool above remains rollback-only until cutover is explicitly accepted. ### Operator gates (all three required before enable) 1. Secrets — add to `sus/gitea-runners.yaml` under `gitea/hectic-lab/controller/*`: ```sh sops sus/gitea-runners.yaml # add keys: # gitea: # hectic-lab: # controller: # hcloud-token: # webhook-secret: # admin-token: ``` The org registration token key `gitea/hectic-lab/org-runner-registration-token` already exists and is reused. 2. Base image — build the MicroOS snapshot and record its id: ```sh nix develop .#gitea-runners -c gitea-runners-build-microos-snapshots x86 hcloud image list --selector '' -o json | jq '.[] | select(.type=="snapshot")' ``` 3. DNS — A record `runners.hectic-lab.com -> 128.140.75.58` (ACME needs it). ### Enable ```sh # nixos/system/hectic-lab/hectic-lab.nix: resolve the FIXME block hectic.services.gitea-runner-controller = { enable = true; imageId = ""; }; nixos-rebuild --target root@128.140.75.58 switch systemctl status gitea-runner-controller.service gitea-runner-webhook.service ``` ### Register the Gitea webhook Org-level (preferred) or per-repo, on `https://gitea.hectic-lab.com`: - URL: `https://runners.hectic-lab.com/` - Method: `POST`, content type: JSON - Secret: value of `gitea/hectic-lab/controller/webhook-secret` - Trigger events: `Workflow jobs` only (`workflow_job`) ### Long CUDA/Magma deployment time budgets The `deploy-neuro` workflow uses these nested limits: | Layer | Limit | | --- | --- | | Wrapped build/deploy command | 6 hours (`WITH_ATTIC_BUILD_TIMEOUT=21600`) | | Final cache drain | 1 hour (`WITH_ATTIC_DRAIN_TIMEOUT=3600`) | | Workflow job | 435 minutes, including 15 minutes of setup/cleanup margin | | `gross-nix-x86-perf` runner | 480 minutes | | `gross-nix-x86-highmem` runner | 480 minutes | | Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours | | VM hard lifetime from allocation | 480 minutes; no destruction grace | `ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for long-running Nix deployments. Deploy the controller and Gitea watchdog settings before dispatching the longer workflow. Already allocated VMs retain the TTL and runner configuration assigned when they were created; updating the controller does not extend a running job. `gross-nix-x86-highmem` is an explicit costly high-memory escape hatch backed only by Hetzner CCX53 in `nbg1`, `fsn1`, or `hel1`; it may fall back by region only, never to a lower-RAM server type. Current Hetzner public pricing for Germany/Finland CCX53 is 0.8550 EUR/hour excluding IPv4, so one 480-minute allocation reserves 6.84 EUR against the controller budget before VM creation. These are maximum lifetimes. Failed, cancelled, skipped, and unbootstrapped jobs still trigger immediate VM teardown; failed jobs retain pre-destroy diagnostics. After a successful job, its bootstrapped VM stays running until next hourly boundary measured from original VM creation, capped by profile TTL. Same-repo, same-label queued work can atomically claim that idle VM. Reuse preserves original Hetzner labels and runner name, creates no server, fetches no new registration token, and makes no second budget reservation. Idle VMs are still billed: controller deletes them at slot expiry and never relies on stopping a server to avoid charges. Reuse retains runner host filesystem and registration identity. It is therefore restricted to same repository and exact label inside this trusted-only pool; allowed repositories must not run untrusted fork or pull-request code. A failed or otherwise non-successful job is never reused. Active and idle VMs are deleted at profile hard TTL without grace. Idle reuse is allowed only when at least one configured reconcile interval remains before both slot expiry and hard TTL. Controller budget reservation still uses full label TTL on initial creation, so a long-running label reserves more of existing monthly budget. Do not raise that budget or disable timeout safeguards just to bypass a refused allocation. After changing any timeout, verify the complete chain rather than only `timeout-minutes`; a shorter wrapper, runner, server watchdog, or VM TTL wins. ### Pre-flight verification (before first real job) ```sh curl -fsS https://runners.hectic-lab.com/ -o /dev/null -w '%{http_code}\n' # any 4xx from handler = reachable journalctl -u gitea-runner-webhook -n 20 --no-pager hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed")] | length' # expect 0 ``` Zero managed VMs is expected after retained billing slots expire. Immediately after successful work, one managed VM per retained profile may remain until its recorded hourly boundary. ### End-to-end acceptance (Task 9) Trigger `.gitea/workflows/runner-nix-smoke.yaml` via workflow_dispatch, then: ```sh watch_labels() { hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed") | {id,name,labels}]'; } watch_labels # exactly one VM while queued/running journalctl -f -u gitea-runner-controller # vm-created / vm-destroyed events watch_labels # VM may remain until next hourly boundary curl -fsS -H "Authorization: token $ADMIN" \ https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners \ | jq '[.entries[] | select(.name | startswith("gcr-"))] | length' # may remain during retained slot ``` Failure paths to verify identically: duplicate delivery (send same webhook twice via Gitea UI "Test delivery"), cancelled job, unknown-label job. ### Rollback K8s rollback pool now defaults to deleted state: - `infra/gitea-runners/k8s/statefulset.yaml` keeps `replicas: 0` - old kube-hetzner nodes may be deleted to preserve zero idle cost - PVCs and IaC remain for manual rollback only Re-enable sequence: ```sh # 1. edit nixos/system/hectic-lab/hectic-lab.nix and set # services.gitea-runner-controller.enable = false, then rebuild: nixos-rebuild --target root@128.140.75.58 switch # 2. reprovision old kube-hetzner nodes when they were deleted: tofu -chdir=infra/gitea-runners/opentofu apply # 3. while controller is disabled, destroy every surviving managed VM and # verify no gcr-* runner registration remains online: hcloud server list -o json \ | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \ | xargs -r -n1 hcloud server delete # 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the # ConfigMap labels, then re-enable K8s runner pool. The legacy image does not # provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image # and label mapping are restored: kubectl -n gitea-runners edit configmap/gitea-runner-config kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m ``` Any managed VM or `gcr-*` registration found after step 3 must be removed before restoring persistent labels; otherwise both pools can claim the same job.