diff --git a/devshell/gitea-runners.nix b/devshell/gitea-runners.nix index 8179ed26..2e614b07 100644 --- a/devshell/gitea-runners.nix +++ b/devshell/gitea-runners.nix @@ -224,18 +224,15 @@ Environment expected before real deploy/apply: S3 backend credentials and endpoint access a matching SOPS age identity for sus/gitea-runners.yaml kubectl access to the target cluster - a concrete registry digest for the pushed Nix-capable runner image if enabling - the nix label + a valid Hetzner Nix image ID in the controller host configuration OpenTofu validation gate: tofu version tofu -chdir=infra/gitea-runners/opentofu validate -Nix image build/publish/digest gate: +Nix image build/publish gate: nix build .#gitea-runner-nix-image - publish the archive, then pin the registry-reported digest in the runner label - mapping - nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256: + publish/import image, then set nixImageId / GCR_NIX_IMAGE_ID to its Hetzner ID SOPS token Secret creation gate: kubectl apply -f infra/gitea-runners/k8s/namespace.yaml @@ -272,7 +269,7 @@ Verification commands: Main blockers and gates: do not run tofu apply without all external inputs do not apply the k8s overlay until the gitea-runner-token Secret exists - do not enable the nix label until the image has been published with a concrete digest + do not dispatch nix jobs until nixImageId / GCR_NIX_IMAGE_ID is valid do not print, load, or require secrets on shell entry EOF ''; diff --git a/docs/README.md b/docs/README.md index ccbcfa8b..786e87a6 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,10 +4,10 @@ ## Gitea runner labels -Common labels for zero-idle runners: +Common labels for controller-managed zero-idle runners: -- `ubuntu-latest` — persistent Ubuntu 24.04 `cx23` worker -- `nix` — persistent Nix-capable Ubuntu `cx23` worker +- `ubuntu-latest` — zero-idle alias for `gross-x86` +- `nix` — zero-idle Nix alias with 480-minute TTL - `gross-x86` — x86 fallback chain `cx53` / `cx43` / `cx33` - `gross-arm` — ARM fallback chain `cax41` / `cax31` / `cax21` - `gross-x86-perf` — x86 performance chain `cx53` / `cpx62` / `cpx52` @@ -15,12 +15,13 @@ Common labels for zero-idle runners: - `gross-nix-x86` — `gross-x86` + Nix bootstrap - `gross-nix-arm` — `gross-arm` + Nix bootstrap - `gross-nix-x86-perf` — `gross-x86-perf` + Nix bootstrap +- `gross-nix-x86-highmem` — CCX53-only Nix runner, 480-minute TTL - `gross-nix-mixed-econ` — `gross-mixed-econ` + Nix bootstrap Region order for fallback: `nbg1`, then `fsn1`, then `hel1`. -`nix` and `ubuntu-latest` are persistent workers; only `gross-*` labels use -zero-idle ephemeral VMs. +The legacy Kubernetes persistent pool is disabled (`replicas: 0`) and has no +registered labels. All listed labels are handled by the zero-idle controller. Operational details: `infra/gitea-runners/runbook.md` and `package/gitea-runner-controller/decide.sh`. diff --git a/infra/gitea-runners/image/README.md b/infra/gitea-runners/image/README.md index e12a064e..207aa954 100644 --- a/infra/gitea-runners/image/README.md +++ b/infra/gitea-runners/image/README.md @@ -24,43 +24,30 @@ Preferred registry: gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image ``` -Publish the archive without adding secrets to the image layers, then use the -registry-reported digest as the only final `nix` label image reference: +Publish the archive without adding secrets to the image layers. Controller-owned +zero-idle runners select this image through `nixImageId` / `GCR_NIX_IMAGE_ID`; +they do not use a Gitea label-to-container-image mapping: ```text -nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256: +nixImageId = ""; ``` -The `2026-06-07` tag may be pushed as a human-readable companion tag, but the -runner label mapping must use the `@sha256:` reference above. Keep -`ubuntu-latest` on the `gitea/runner` default image unless a later runner -configuration task explicitly changes it. Only the `nix` label should select -this custom image. +The `2026-06-07` tag may be pushed as a human-readable companion tag. The +legacy Kubernetes rollback pool is currently disabled and has no labels. +If it is restored, its Nix-capable image must be configured separately and +digest-pinned before enabling a `nix` label. -If the Gitea container registry is unavailable, select a private registry that -is reachable from the runner Kubernetes cluster and requires authentication that -can be provided through Kubernetes image-pull secrets. Record the selected -registry and replace the host in the same digest-pinned form: - -```text -nix:docker:////gitea-runner-nix-image@sha256: -``` - -Do not fall back to `latest` or a tag-only mapping. +Do not use a tag-only image for a restored Kubernetes rollback pool. ## Task 7 publication status Local build evidence is recorded in -`.sisyphus/evidence/task-7-image-digest.txt`. In this environment, Docker could -load and tag the image, but pushing to the preferred registry failed with -`unauthorized: reqPackageAccess`, so no registry digest was available to pin as a -concrete final mapping. Kubernetes pull smoke is recorded in +`.sisyphus/evidence/task-7-image-digest.txt`. Kubernetes pull smoke is recorded in `.sisyphus/evidence/task-7-image-pull.txt` and is blocked here because `kubectl` is not installed or not on `PATH`. -Once registry credentials are available, rerun the push, capture the -registry-reported digest, and replace `` in the mapping above -before Task 6/9 consumes the label configuration. +After importing the archive as a Hetzner image, record its image ID in the +controller host configuration before dispatching Nix jobs. ## Image contents diff --git a/infra/gitea-runners/k8s/runner-config.yaml b/infra/gitea-runners/k8s/runner-config.yaml index e7d1a877..eee2d861 100644 --- a/infra/gitea-runners/k8s/runner-config.yaml +++ b/infra/gitea-runners/k8s/runner-config.yaml @@ -19,10 +19,8 @@ data: insecure: false fetch_timeout: 5s fetch_interval: 2s - labels: - - ubuntu-latest - # The nix label is intentionally disabled until the runner image has a - # concrete registry-reported digest; see ../runbook.md before deploy. + # Persistent pool is disabled; controller owns all runner labels. + labels: [] cache: enabled: true diff --git a/infra/gitea-runners/opentofu/variables.tf b/infra/gitea-runners/opentofu/variables.tf index f1f3f318..e74fcc3c 100644 --- a/infra/gitea-runners/opentofu/variables.tf +++ b/infra/gitea-runners/opentofu/variables.tf @@ -45,7 +45,7 @@ variable "control_plane_server_type" { } variable "worker_server_type" { - description = "Default worker server type for the budget trusted DinD runner pool. One cpx22 worker keeps the idle baseline cheap; scale out later if job pressure requires it." + description = "Default worker server type for the budget trusted DinD cluster. One cpx22 worker keeps the cluster baseline cheap; scale out later if rollback capacity requires it." type = string default = "cpx22" } diff --git a/infra/gitea-runners/runbook.md b/infra/gitea-runners/runbook.md index 446a44f1..861134b9 100644 --- a/infra/gitea-runners/runbook.md +++ b/infra/gitea-runners/runbook.md @@ -2,15 +2,13 @@ ## Scope -This directory is the repo-owned boundary for the first Gitea Actions runner -pool. Task 1 only establishes the scaffold and immutable decision contract; -downstream tasks will add OpenTofu backend/provider files, Kubernetes manifests, -and a Nix-capable runner image under the existing subdirectories. +This directory is the repo-owned boundary for the Gitea Actions runner pool. +The controller is the active zero-idle path; Kubernetes manifests and the +Nix-capable image are retained for manual rollback and maintenance. The target service is `https://gitea.hectic-lab.com` for the Gitea organization -`hectic-lab`. The first pool is fixed-size and trusted-only. "Ephemeral" means -workflow job containers are ephemeral, while each runner pod keeps its runner -identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC. +`hectic-lab`. The pool is trusted-only. "Ephemeral" means each controller VM +and workflow job is disposable; the Kubernetes StatefulSet is rollback-only. ## Immutable decisions @@ -27,21 +25,21 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC. `/data`, including `/data/.runner`. - Container builds run through privileged rootful DinD inside trusted runner pods; host Docker socket mounting is not an implementation path. -- The active runner label is `ubuntu-latest`. The `nix` label is not live until - the Nix-capable image has been pushed and a concrete registry-reported digest - is added to the runner ConfigMap. +- `ubuntu-latest` and `nix` are controller-managed zero-idle aliases for + `gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels. - First scope is trusted internal workflows only, with no untrusted fork or PR workflow support. -- First scope has no autoscaling, no KEDA, and no dynamic runner controller. +- Zero-idle allocation is handled by the repo-owned controller; Kubernetes is + not an active autoscaling path. ## Lifecycle boundaries -- `infra/gitea-runners/opentofu/`: downstream OpenTofu stack for the S3 backend +- `infra/gitea-runners/opentofu/`: OpenTofu stack for the S3 backend contract, Hetzner provider configuration, and kube-hetzner module wiring. -- `infra/gitea-runners/k8s/`: downstream namespace, ConfigMap, Secret mount, +- `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount, StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work. -- `infra/gitea-runners/image/`: downstream notes or sources for the runner image - handoff; package or flake output changes are outside Task 1. +- `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes + rollback image; active zero-idle Nix image is selected by Hetzner image ID. - `infra/gitea-runners/runbook.md`: this contract plus later operational commands, rollback notes, and acceptance evidence references. @@ -52,8 +50,8 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC. instructions, or GitHub Actions ARC assumptions. - Untrusted fork/PR workflows are out of first scope; privileged DinD is only acceptable for trusted internal jobs. -- Autoscaling/KEDA is out of first scope; start with a fixed-size StatefulSet - runner pool. +- The persistent StatefulSet is rollback-only and defaults to zero replicas; + normal jobs use controller-managed zero-idle VMs. - No actual secrets are committed: no kubeconfig, runner token, Hetzner token, S3 credentials, decrypted SOPS files, or SOPS age keys. - OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea @@ -248,8 +246,7 @@ These commands are executable only when the external inputs are available: - S3 backend credentials and endpoint access - a matching SOPS age identity for `sus/gitea-runners.yaml` - `kubectl` access to the target cluster -- a concrete digest for the pushed Nix-capable runner image, if enabling the - `nix` label +- a valid `GCR_NIX_IMAGE_ID` for controller-managed Nix jobs If any input is missing, stop before `tofu apply`. Do not guess values or reuse stale kubeconfig files. @@ -260,15 +257,12 @@ Before production Kubernetes apply or rollout, satisfy both manifest gates: Kustomize overlay intentionally does not include a placeholder Secret, but the StatefulSet still mounts `secretName: gitea-runner-token` as `/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`. -2. Keep the active ConfigMap on `ubuntu-latest` only unless the Nix-capable - image has been pushed successfully. Enable the `nix` label only by adding a - digest-pinned `docker://` mapping with the exact registry-reported sha256 - digest from that push. +2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the + controller; Nix image readiness is governed by `GCR_NIX_IMAGE_ID`. Use the same SOPS materialization pattern as token rotation before applying the Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a -target namespace; the full overlay remains gated on the Secret and digest -decisions: +target namespace; the full overlay remains gated on the Secret. ```sh kubectl apply -f infra/gitea-runners/k8s/namespace.yaml @@ -283,8 +277,7 @@ kubectl -n gitea-runners create secret generic gitea-runner-token \ ``` Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command -above succeeds. Do not claim or enable the `nix` runner label until the image -publication step has produced the concrete digest. +above succeeds. The persistent pool ConfigMap must retain empty labels. ```sh tofu -chdir=infra/gitea-runners/opentofu init @@ -309,14 +302,14 @@ Expected status after deploy: - `kubectl config current-context` names the runner cluster context. - `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready. - `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs. -- `kubectl -n gitea-runners get statefulset gitea-runner` shows 5 desired and 5 ready replicas. -- `kubectl -n gitea-runners get pvc` shows 5 Bound PVCs. -- `kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200` shows the runner daemon started and no token value. +- `kubectl -n gitea-runners get statefulset gitea-runner` shows 0 desired and 0 ready replicas. +- `kubectl -n gitea-runners get pvc` shows no active runner PVCs; retained PVCs are rollback-only. +- The controller host reports healthy and owns runner registrations; no persistent runner claims `ubuntu-latest` or `nix`. -## Scale 5 to 10 to 5 +## Legacy rollback pool scaling (manual only) -Scaling is a temporary capacity exercise, not the steady-state setting. Scale up, -wait for readiness, run the concurrent smoke jobs, then scale back down to 5. +Persistent-pool scaling is not part of normal operation. Use only after restoring +its labels and disabling the zero-idle controller as described in `Rollback`. ```sh kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10 @@ -324,9 +317,9 @@ kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide -# Run the concurrent smoke workflows now. +# Run only workflows supported by restored persistent labels. -kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=1 +kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide @@ -405,7 +398,7 @@ Do not run the delete command for a runner that still has an active `gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is being intentionally reset for re-registration. -## Application rollback +## Legacy Kubernetes application rollback Rollback the app layer only. Do not use this section to destroy the cluster. @@ -504,7 +497,7 @@ as complete. ## Ephemeral VM runner cutover -This section governs replacing the fixed K8s runner pool with the +This section governs replacing the legacy persistent K8s runner pool with the ephemeral-VM controller (`package/gitea-runner-controller`) on this host. The K8s pool above remains rollback-only until cutover is explicitly accepted. @@ -571,8 +564,9 @@ The `deploy-neuro` workflow uses these nested limits: | Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours | | VM hard lifetime from allocation | 480 minutes plus 10-minute controller grace | -Other runner labels keep their existing 180-minute limits. Deploy the controller -and Gitea watchdog settings before dispatching the longer workflow. Already +`ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for +long-running Nix deployments. Deploy the controller and Gitea watchdog settings +before dispatching the longer workflow. Already allocated VMs retain the TTL and runner configuration assigned when they were created; updating the controller does not extend a running job. @@ -628,23 +622,27 @@ K8s rollback pool now defaults to deleted state: Re-enable sequence: ```sh -# 1. stop ephemeral path -sed -i 's/hectic.services.gitea-runner-controller = {.*}/\/* disabled *\//' \ - nixos/system/hectic-lab/hectic-lab.nix # or set enable = false +# 1. edit nixos/system/hectic-lab/hectic-lab.nix and set +# services.gitea-runner-controller.enable = false, then rebuild: nixos-rebuild --target root@128.140.75.58 switch # 2. reprovision old kube-hetzner nodes when they were deleted: tofu -chdir=infra/gitea-runners/opentofu apply -# 3. restore kubeconfig / cluster access, then re-enable K8s runner pool: +# 3. while controller is disabled, destroy every surviving managed VM and +# verify no gcr-* runner registration remains online: +hcloud server list -o json \ + | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \ + | xargs -r -n1 hcloud server delete + +# 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the +# ConfigMap labels, then re-enable K8s runner pool. The legacy image does not +# provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image +# and label mapping are restored: +kubectl -n gitea-runners edit configmap/gitea-runner-config kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m ``` -Any surviving ephemeral VMs after step 1 must be destroyed manually once: - -```sh -hcloud server list -o json \ - | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \ - | xargs -r -n1 hcloud server delete -``` +Any managed VM or `gcr-*` registration found after step 3 must be removed before +restoring persistent labels; otherwise both pools can claim the same job. diff --git a/package/gitea-runner-controller/decide.sh b/package/gitea-runner-controller/decide.sh index 14d6bf48..fb59ed5f 100644 --- a/package/gitea-runner-controller/decide.sh +++ b/package/gitea-runner-controller/decide.sh @@ -23,11 +23,12 @@ gcr_server_hourly_rate() { gcr_label_ttl() { case "$1" in - gross-x86) printf '180' ;; + ubuntu-latest|gross-x86) printf '180' ;; gross-arm) printf '180' ;; gross-x86-perf) printf '180' ;; gross-mixed-econ) printf '180' ;; - gross-nix-x86) printf '180' ;; + nix) printf '480' ;; + gross-nix-x86) printf '180' ;; gross-nix-arm) printf '180' ;; gross-nix-x86-perf) printf '480' ;; gross-nix-x86-highmem) printf '480' ;; @@ -40,7 +41,7 @@ gcr_label_ttl() { gcr_label_candidates() { label="$1" case "$label" in - gross-x86) + ubuntu-latest|gross-x86) printf '%s\n' \ 'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \ 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \ @@ -64,7 +65,7 @@ gcr_label_candidates() { 'cax41 nbg1 arm64' 'cax41 fsn1 arm64' 'cax41 hel1 arm64' \ 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' ;; - gross-nix-x86) + nix|gross-nix-x86) printf '%s\n' \ 'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \ 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \ diff --git a/test/package/gitea-runner-controller/test/highmem-label.sh b/test/package/gitea-runner-controller/test/highmem-label.sh index 18697254..6e99df39 100644 --- a/test/package/gitea-runner-controller/test/highmem-label.sh +++ b/test/package/gitea-runner-controller/test/highmem-label.sh @@ -8,8 +8,18 @@ set -eu gcr_state_init export GCR_ALLOWED_REPOS='hectic-lab/util.nix' +export GCR_IMAGE_ID='313131' export GCR_NIX_IMAGE_ID='424242' +test "$(gcr_label_ttl ubuntu-latest)" = '180' +test "$(gcr_label_ttl nix)" = '480' +test "$(gcr_decide ubuntu-latest hectic-lab/util.nix)" = 'cx53 180 0.032' +test "$(gcr_decide nix hectic-lab/util.nix)" = 'cx53 480 0.032' +test "$(gcr_label_candidates ubuntu-latest | head -n1)" = 'cx53 nbg1 amd64' +test "$(gcr_label_candidates nix | head -n1)" = 'cx53 nbg1 amd64' +test "$(gcr_image_id_for_arch amd64 ubuntu-latest)" = '313131' +test "$(gcr_image_id_for_arch amd64 nix)" = '424242' + profile="$(gcr_decide gross-nix-x86-highmem hectic-lab/util.nix)" test "$profile" = 'ccx53 480 0.8550' test "$(gcr_label_ttl gross-nix-x86-highmem)" = '480'