feat: move all workers to zero-idle

devshell/gitea-runners.nix

@@ -224,18 +224,15 @@ Environment expected before real deploy/apply:
   S3 backend credentials and endpoint access
   a matching SOPS age identity for sus/gitea-runners.yaml
   kubectl access to the target cluster
-  a concrete registry digest for the pushed Nix-capable runner image if enabling
+  a valid Hetzner Nix image ID in the controller host configuration
-  the nix label
 
 OpenTofu validation gate:
   tofu version
   tofu -chdir=infra/gitea-runners/opentofu validate
 
-Nix image build/publish/digest gate:
+Nix image build/publish gate:
   nix build .#gitea-runner-nix-image
-  publish the archive, then pin the registry-reported digest in the runner label
+  publish/import image, then set nixImageId / GCR_NIX_IMAGE_ID to its Hetzner ID
-  mapping
-  nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256:<registry-digest>
 
 SOPS token Secret creation gate:
   kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
@@ -272,7 +269,7 @@ Verification commands:
 Main blockers and gates:
   do not run tofu apply without all external inputs
   do not apply the k8s overlay until the gitea-runner-token Secret exists
-  do not enable the nix label until the image has been published with a concrete digest
+  do not dispatch nix jobs until nixImageId / GCR_NIX_IMAGE_ID is valid
   do not print, load, or require secrets on shell entry
 EOF
   '';

docs/README.md

@@ -4,10 +4,10 @@
 
 ## Gitea runner labels
 
-Common labels for zero-idle runners:
+Common labels for controller-managed zero-idle runners:
 
-- `ubuntu-latest` — persistent Ubuntu 24.04 `cx23` worker
+- `ubuntu-latest` — zero-idle alias for `gross-x86`
-- `nix` — persistent Nix-capable Ubuntu `cx23` worker
+- `nix` — zero-idle Nix alias with 480-minute TTL
 - `gross-x86` — x86 fallback chain `cx53` / `cx43` / `cx33`
 - `gross-arm` — ARM fallback chain `cax41` / `cax31` / `cax21`
 - `gross-x86-perf` — x86 performance chain `cx53` / `cpx62` / `cpx52`
@@ -15,12 +15,13 @@ Common labels for zero-idle runners:
 - `gross-nix-x86` — `gross-x86` + Nix bootstrap
 - `gross-nix-arm` — `gross-arm` + Nix bootstrap
 - `gross-nix-x86-perf` — `gross-x86-perf` + Nix bootstrap
+- `gross-nix-x86-highmem` — CCX53-only Nix runner, 480-minute TTL
 - `gross-nix-mixed-econ` — `gross-mixed-econ` + Nix bootstrap
 
 Region order for fallback: `nbg1`, then `fsn1`, then `hel1`.
 
-`nix` and `ubuntu-latest` are persistent workers; only `gross-*` labels use
+The legacy Kubernetes persistent pool is disabled (`replicas: 0`) and has no
-zero-idle ephemeral VMs.
+registered labels. All listed labels are handled by the zero-idle controller.
 
 Operational details: `infra/gitea-runners/runbook.md` and
 `package/gitea-runner-controller/decide.sh`.

infra/gitea-runners/image/README.md

@@ -24,43 +24,30 @@ Preferred registry:
 gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image
 ```
 
-Publish the archive without adding secrets to the image layers, then use the
+Publish the archive without adding secrets to the image layers. Controller-owned
-registry-reported digest as the only final `nix` label image reference:
+zero-idle runners select this image through `nixImageId` / `GCR_NIX_IMAGE_ID`;
+they do not use a Gitea label-to-container-image mapping:
 
 ```text
-nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256:<registry-digest>
+nixImageId = "<Hetzner-image-id>";
 ```
 
-The `2026-06-07` tag may be pushed as a human-readable companion tag, but the
+The `2026-06-07` tag may be pushed as a human-readable companion tag. The
-runner label mapping must use the `@sha256:` reference above. Keep
+legacy Kubernetes rollback pool is currently disabled and has no labels.
-`ubuntu-latest` on the `gitea/runner` default image unless a later runner
+If it is restored, its Nix-capable image must be configured separately and
-configuration task explicitly changes it. Only the `nix` label should select
+digest-pinned before enabling a `nix` label.
-this custom image.
 
-If the Gitea container registry is unavailable, select a private registry that
+Do not use a tag-only image for a restored Kubernetes rollback pool.
-is reachable from the runner Kubernetes cluster and requires authentication that
-can be provided through Kubernetes image-pull secrets. Record the selected
-registry and replace the host in the same digest-pinned form:
-
-```text
-nix:docker://<private-registry>/<namespace>/gitea-runner-nix-image@sha256:<registry-digest>
-```
-
-Do not fall back to `latest` or a tag-only mapping.
 
 ## Task 7 publication status
 
 Local build evidence is recorded in
-`.sisyphus/evidence/task-7-image-digest.txt`. In this environment, Docker could
+`.sisyphus/evidence/task-7-image-digest.txt`. Kubernetes pull smoke is recorded in
-load and tag the image, but pushing to the preferred registry failed with
-`unauthorized: reqPackageAccess`, so no registry digest was available to pin as a
-concrete final mapping. Kubernetes pull smoke is recorded in
 `.sisyphus/evidence/task-7-image-pull.txt` and is blocked here because `kubectl`
 is not installed or not on `PATH`.
 
-Once registry credentials are available, rerun the push, capture the
+After importing the archive as a Hetzner image, record its image ID in the
-registry-reported digest, and replace `<registry-digest>` in the mapping above
+controller host configuration before dispatching Nix jobs.
-before Task 6/9 consumes the label configuration.
 
 ## Image contents
 

infra/gitea-runners/k8s/runner-config.yaml

@@ -19,10 +19,8 @@ data:
       insecure: false
       fetch_timeout: 5s
       fetch_interval: 2s
-      labels:
+      # Persistent pool is disabled; controller owns all runner labels.
-        - ubuntu-latest
+      labels: []
-        # The nix label is intentionally disabled until the runner image has a
-        # concrete registry-reported digest; see ../runbook.md before deploy.
 
     cache:
       enabled: true

infra/gitea-runners/opentofu/variables.tf

@@ -45,7 +45,7 @@ variable "control_plane_server_type" {
 }
 
 variable "worker_server_type" {
-  description = "Default worker server type for the budget trusted DinD runner pool. One cpx22 worker keeps the idle baseline cheap; scale out later if job pressure requires it."
+  description = "Default worker server type for the budget trusted DinD cluster. One cpx22 worker keeps the cluster baseline cheap; scale out later if rollback capacity requires it."
   type        = string
   default     = "cpx22"
 }

infra/gitea-runners/runbook.md

@@ -2,15 +2,13 @@
 
 ## Scope
 
-This directory is the repo-owned boundary for the first Gitea Actions runner
+This directory is the repo-owned boundary for the Gitea Actions runner pool.
-pool. Task 1 only establishes the scaffold and immutable decision contract;
+The controller is the active zero-idle path; Kubernetes manifests and the
-downstream tasks will add OpenTofu backend/provider files, Kubernetes manifests,
+Nix-capable image are retained for manual rollback and maintenance.
-and a Nix-capable runner image under the existing subdirectories.
 
 The target service is `https://gitea.hectic-lab.com` for the Gitea organization
-`hectic-lab`. The first pool is fixed-size and trusted-only. "Ephemeral" means
+`hectic-lab`. The pool is trusted-only. "Ephemeral" means each controller VM
-workflow job containers are ephemeral, while each runner pod keeps its runner
+and workflow job is disposable; the Kubernetes StatefulSet is rollback-only.
-identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
 
 ## Immutable decisions
 
@@ -27,21 +25,21 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
   `/data`, including `/data/.runner`.
 - Container builds run through privileged rootful DinD inside trusted runner
   pods; host Docker socket mounting is not an implementation path.
-- The active runner label is `ubuntu-latest`. The `nix` label is not live until
+- `ubuntu-latest` and `nix` are controller-managed zero-idle aliases for
-  the Nix-capable image has been pushed and a concrete registry-reported digest
+  `gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels.
-  is added to the runner ConfigMap.
 - First scope is trusted internal workflows only, with no untrusted fork or PR
   workflow support.
-- First scope has no autoscaling, no KEDA, and no dynamic runner controller.
+- Zero-idle allocation is handled by the repo-owned controller; Kubernetes is
+  not an active autoscaling path.
 
 ## Lifecycle boundaries
 
-- `infra/gitea-runners/opentofu/`: downstream OpenTofu stack for the S3 backend
+- `infra/gitea-runners/opentofu/`: OpenTofu stack for the S3 backend
   contract, Hetzner provider configuration, and kube-hetzner module wiring.
-- `infra/gitea-runners/k8s/`: downstream namespace, ConfigMap, Secret mount,
+- `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount,
   StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work.
-- `infra/gitea-runners/image/`: downstream notes or sources for the runner image
+- `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes
-  handoff; package or flake output changes are outside Task 1.
+  rollback image; active zero-idle Nix image is selected by Hetzner image ID.
 - `infra/gitea-runners/runbook.md`: this contract plus later operational
   commands, rollback notes, and acceptance evidence references.
 
@@ -52,8 +50,8 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
   instructions, or GitHub Actions ARC assumptions.
 - Untrusted fork/PR workflows are out of first scope; privileged DinD is only
   acceptable for trusted internal jobs.
-- Autoscaling/KEDA is out of first scope; start with a fixed-size StatefulSet
+- The persistent StatefulSet is rollback-only and defaults to zero replicas;
-  runner pool.
+  normal jobs use controller-managed zero-idle VMs.
 - No actual secrets are committed: no kubeconfig, runner token, Hetzner token,
   S3 credentials, decrypted SOPS files, or SOPS age keys.
 - OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea
@@ -248,8 +246,7 @@ These commands are executable only when the external inputs are available:
 - S3 backend credentials and endpoint access
 - a matching SOPS age identity for `sus/gitea-runners.yaml`
 - `kubectl` access to the target cluster
-- a concrete digest for the pushed Nix-capable runner image, if enabling the
+- a valid `GCR_NIX_IMAGE_ID` for controller-managed Nix jobs
-  `nix` label
 
 If any input is missing, stop before `tofu apply`. Do not guess values or reuse
 stale kubeconfig files.
@@ -260,15 +257,12 @@ Before production Kubernetes apply or rollout, satisfy both manifest gates:
    Kustomize overlay intentionally does not include a placeholder Secret, but
    the StatefulSet still mounts `secretName: gitea-runner-token` as
    `/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`.
-2. Keep the active ConfigMap on `ubuntu-latest` only unless the Nix-capable
+2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the
-   image has been pushed successfully. Enable the `nix` label only by adding a
+   controller; Nix image readiness is governed by `GCR_NIX_IMAGE_ID`.
-   digest-pinned `docker://` mapping with the exact registry-reported sha256
-   digest from that push.
 
 Use the same SOPS materialization pattern as token rotation before applying the
 Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a
-target namespace; the full overlay remains gated on the Secret and digest
+target namespace; the full overlay remains gated on the Secret.
-decisions:
 
 ```sh
 kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
@@ -283,8 +277,7 @@ kubectl -n gitea-runners create secret generic gitea-runner-token \
 ```
 
 Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command
-above succeeds. Do not claim or enable the `nix` runner label until the image
+above succeeds. The persistent pool ConfigMap must retain empty labels.
-publication step has produced the concrete digest.
 
 ```sh
 tofu -chdir=infra/gitea-runners/opentofu init
@@ -309,14 +302,14 @@ Expected status after deploy:
 - `kubectl config current-context` names the runner cluster context.
 - `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready.
 - `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs.
-- `kubectl -n gitea-runners get statefulset gitea-runner` shows 5 desired and 5 ready replicas.
+- `kubectl -n gitea-runners get statefulset gitea-runner` shows 0 desired and 0 ready replicas.
-- `kubectl -n gitea-runners get pvc` shows 5 Bound PVCs.
+- `kubectl -n gitea-runners get pvc` shows no active runner PVCs; retained PVCs are rollback-only.
-- `kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200` shows the runner daemon started and no token value.
+- The controller host reports healthy and owns runner registrations; no persistent runner claims `ubuntu-latest` or `nix`.
 
-## Scale 5 to 10 to 5
+## Legacy rollback pool scaling (manual only)
 
-Scaling is a temporary capacity exercise, not the steady-state setting. Scale up,
+Persistent-pool scaling is not part of normal operation. Use only after restoring
-wait for readiness, run the concurrent smoke jobs, then scale back down to 5.
+its labels and disabling the zero-idle controller as described in `Rollback`.
 
 ```sh
 kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10
@@ -324,9 +317,9 @@ kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
 kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
 kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
 
-# Run the concurrent smoke workflows now.
+# Run only workflows supported by restored persistent labels.
 
-kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=1
+kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
 kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
 kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
@@ -405,7 +398,7 @@ Do not run the delete command for a runner that still has an active
 `gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is
 being intentionally reset for re-registration.
 
-## Application rollback
+## Legacy Kubernetes application rollback
 
 Rollback the app layer only. Do not use this section to destroy the cluster.
 
@@ -504,7 +497,7 @@ as complete.
 
 ## Ephemeral VM runner cutover
 
-This section governs replacing the fixed K8s runner pool with the
+This section governs replacing the legacy persistent K8s runner pool with the
 ephemeral-VM controller (`package/gitea-runner-controller`) on this host.
 The K8s pool above remains rollback-only until cutover is explicitly accepted.
 
@@ -571,8 +564,9 @@ The `deploy-neuro` workflow uses these nested limits:
 | Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours |
 | VM hard lifetime from allocation | 480 minutes plus 10-minute controller grace |
 
-Other runner labels keep their existing 180-minute limits. Deploy the controller
+`ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for
-and Gitea watchdog settings before dispatching the longer workflow. Already
+long-running Nix deployments. Deploy the controller and Gitea watchdog settings
+before dispatching the longer workflow. Already
 allocated VMs retain the TTL and runner configuration assigned when they were
 created; updating the controller does not extend a running job.
 
@@ -628,23 +622,27 @@ K8s rollback pool now defaults to deleted state:
 Re-enable sequence:
 
 ```sh
-# 1. stop ephemeral path
+# 1. edit nixos/system/hectic-lab/hectic-lab.nix and set
-sed -i 's/hectic.services.gitea-runner-controller = {.*}/\/* disabled *\//' \
+#    services.gitea-runner-controller.enable = false, then rebuild:
-  nixos/system/hectic-lab/hectic-lab.nix   # or set enable = false
 nixos-rebuild --target root@128.140.75.58 switch
 
 # 2. reprovision old kube-hetzner nodes when they were deleted:
 tofu -chdir=infra/gitea-runners/opentofu apply
 
-# 3. restore kubeconfig / cluster access, then re-enable K8s runner pool:
+# 3. while controller is disabled, destroy every surviving managed VM and
+#    verify no gcr-* runner registration remains online:
+hcloud server list -o json \
+  | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
+  | xargs -r -n1 hcloud server delete
+
+# 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the
+#    ConfigMap labels, then re-enable K8s runner pool. The legacy image does not
+#    provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image
+#    and label mapping are restored:
+kubectl -n gitea-runners edit configmap/gitea-runner-config
 kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
 kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
 ```
 
-Any surviving ephemeral VMs after step 1 must be destroyed manually once:
+Any managed VM or `gcr-*` registration found after step 3 must be removed before
-
+restoring persistent labels; otherwise both pools can claim the same job.
-```sh
-hcloud server list -o json \
-  | jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
-  | xargs -r -n1 hcloud server delete
-```

package/gitea-runner-controller/decide.sh

@@ -23,11 +23,12 @@ gcr_server_hourly_rate() {
 
 gcr_label_ttl() {
     case "$1" in
-        gross-x86)       printf '180' ;;
+        ubuntu-latest|gross-x86) printf '180' ;;
         gross-arm)       printf '180' ;;
         gross-x86-perf)  printf '180' ;;
         gross-mixed-econ) printf '180' ;;
-        gross-nix-x86) printf '180' ;;
+        nix)              printf '480' ;;
+        gross-nix-x86)    printf '180' ;;
         gross-nix-arm) printf '180' ;;
         gross-nix-x86-perf) printf '480' ;;
         gross-nix-x86-highmem) printf '480' ;;
@@ -40,7 +41,7 @@ gcr_label_ttl() {
 gcr_label_candidates() {
     label="$1"
     case "$label" in
-        gross-x86)
+        ubuntu-latest|gross-x86)
             printf '%s\n' \
                 'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \
                 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \
@@ -64,7 +65,7 @@ gcr_label_candidates() {
                 'cax41 nbg1 arm64' 'cax41 fsn1 arm64' 'cax41 hel1 arm64' \
                 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64'
             ;;
-        gross-nix-x86)
+        nix|gross-nix-x86)
             printf '%s\n' \
                 'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \
                 'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \

test/package/gitea-runner-controller/test/highmem-label.sh

@@ -8,8 +8,18 @@ set -eu
 
 gcr_state_init
 export GCR_ALLOWED_REPOS='hectic-lab/util.nix'
+export GCR_IMAGE_ID='313131'
 export GCR_NIX_IMAGE_ID='424242'
 
+test "$(gcr_label_ttl ubuntu-latest)" = '180'
+test "$(gcr_label_ttl nix)" = '480'
+test "$(gcr_decide ubuntu-latest hectic-lab/util.nix)" = 'cx53 180 0.032'
+test "$(gcr_decide nix hectic-lab/util.nix)" = 'cx53 480 0.032'
+test "$(gcr_label_candidates ubuntu-latest | head -n1)" = 'cx53 nbg1 amd64'
+test "$(gcr_label_candidates nix | head -n1)" = 'cx53 nbg1 amd64'
+test "$(gcr_image_id_for_arch amd64 ubuntu-latest)" = '313131'
+test "$(gcr_image_id_for_arch amd64 nix)" = '424242'
+
 profile="$(gcr_decide gross-nix-x86-highmem hectic-lab/util.nix)"
 test "$profile" = 'ccx53 480 0.8550'
 test "$(gcr_label_ttl gross-nix-x86-highmem)" = '480'