feat: move all workers to zero-idle
This commit is contained in:
@@ -224,18 +224,15 @@ Environment expected before real deploy/apply:
|
||||
S3 backend credentials and endpoint access
|
||||
a matching SOPS age identity for sus/gitea-runners.yaml
|
||||
kubectl access to the target cluster
|
||||
a concrete registry digest for the pushed Nix-capable runner image if enabling
|
||||
the nix label
|
||||
a valid Hetzner Nix image ID in the controller host configuration
|
||||
|
||||
OpenTofu validation gate:
|
||||
tofu version
|
||||
tofu -chdir=infra/gitea-runners/opentofu validate
|
||||
|
||||
Nix image build/publish/digest gate:
|
||||
Nix image build/publish gate:
|
||||
nix build .#gitea-runner-nix-image
|
||||
publish the archive, then pin the registry-reported digest in the runner label
|
||||
mapping
|
||||
nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256:<registry-digest>
|
||||
publish/import image, then set nixImageId / GCR_NIX_IMAGE_ID to its Hetzner ID
|
||||
|
||||
SOPS token Secret creation gate:
|
||||
kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
|
||||
@@ -272,7 +269,7 @@ Verification commands:
|
||||
Main blockers and gates:
|
||||
do not run tofu apply without all external inputs
|
||||
do not apply the k8s overlay until the gitea-runner-token Secret exists
|
||||
do not enable the nix label until the image has been published with a concrete digest
|
||||
do not dispatch nix jobs until nixImageId / GCR_NIX_IMAGE_ID is valid
|
||||
do not print, load, or require secrets on shell entry
|
||||
EOF
|
||||
'';
|
||||
|
||||
+6
-5
@@ -4,10 +4,10 @@
|
||||
|
||||
## Gitea runner labels
|
||||
|
||||
Common labels for zero-idle runners:
|
||||
Common labels for controller-managed zero-idle runners:
|
||||
|
||||
- `ubuntu-latest` — persistent Ubuntu 24.04 `cx23` worker
|
||||
- `nix` — persistent Nix-capable Ubuntu `cx23` worker
|
||||
- `ubuntu-latest` — zero-idle alias for `gross-x86`
|
||||
- `nix` — zero-idle Nix alias with 480-minute TTL
|
||||
- `gross-x86` — x86 fallback chain `cx53` / `cx43` / `cx33`
|
||||
- `gross-arm` — ARM fallback chain `cax41` / `cax31` / `cax21`
|
||||
- `gross-x86-perf` — x86 performance chain `cx53` / `cpx62` / `cpx52`
|
||||
@@ -15,12 +15,13 @@ Common labels for zero-idle runners:
|
||||
- `gross-nix-x86` — `gross-x86` + Nix bootstrap
|
||||
- `gross-nix-arm` — `gross-arm` + Nix bootstrap
|
||||
- `gross-nix-x86-perf` — `gross-x86-perf` + Nix bootstrap
|
||||
- `gross-nix-x86-highmem` — CCX53-only Nix runner, 480-minute TTL
|
||||
- `gross-nix-mixed-econ` — `gross-mixed-econ` + Nix bootstrap
|
||||
|
||||
Region order for fallback: `nbg1`, then `fsn1`, then `hel1`.
|
||||
|
||||
`nix` and `ubuntu-latest` are persistent workers; only `gross-*` labels use
|
||||
zero-idle ephemeral VMs.
|
||||
The legacy Kubernetes persistent pool is disabled (`replicas: 0`) and has no
|
||||
registered labels. All listed labels are handled by the zero-idle controller.
|
||||
|
||||
Operational details: `infra/gitea-runners/runbook.md` and
|
||||
`package/gitea-runner-controller/decide.sh`.
|
||||
|
||||
@@ -24,43 +24,30 @@ Preferred registry:
|
||||
gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image
|
||||
```
|
||||
|
||||
Publish the archive without adding secrets to the image layers, then use the
|
||||
registry-reported digest as the only final `nix` label image reference:
|
||||
Publish the archive without adding secrets to the image layers. Controller-owned
|
||||
zero-idle runners select this image through `nixImageId` / `GCR_NIX_IMAGE_ID`;
|
||||
they do not use a Gitea label-to-container-image mapping:
|
||||
|
||||
```text
|
||||
nix:docker://gitea.hectic-lab.com/hectic-lab/gitea-runner-nix-image@sha256:<registry-digest>
|
||||
nixImageId = "<Hetzner-image-id>";
|
||||
```
|
||||
|
||||
The `2026-06-07` tag may be pushed as a human-readable companion tag, but the
|
||||
runner label mapping must use the `@sha256:` reference above. Keep
|
||||
`ubuntu-latest` on the `gitea/runner` default image unless a later runner
|
||||
configuration task explicitly changes it. Only the `nix` label should select
|
||||
this custom image.
|
||||
The `2026-06-07` tag may be pushed as a human-readable companion tag. The
|
||||
legacy Kubernetes rollback pool is currently disabled and has no labels.
|
||||
If it is restored, its Nix-capable image must be configured separately and
|
||||
digest-pinned before enabling a `nix` label.
|
||||
|
||||
If the Gitea container registry is unavailable, select a private registry that
|
||||
is reachable from the runner Kubernetes cluster and requires authentication that
|
||||
can be provided through Kubernetes image-pull secrets. Record the selected
|
||||
registry and replace the host in the same digest-pinned form:
|
||||
|
||||
```text
|
||||
nix:docker://<private-registry>/<namespace>/gitea-runner-nix-image@sha256:<registry-digest>
|
||||
```
|
||||
|
||||
Do not fall back to `latest` or a tag-only mapping.
|
||||
Do not use a tag-only image for a restored Kubernetes rollback pool.
|
||||
|
||||
## Task 7 publication status
|
||||
|
||||
Local build evidence is recorded in
|
||||
`.sisyphus/evidence/task-7-image-digest.txt`. In this environment, Docker could
|
||||
load and tag the image, but pushing to the preferred registry failed with
|
||||
`unauthorized: reqPackageAccess`, so no registry digest was available to pin as a
|
||||
concrete final mapping. Kubernetes pull smoke is recorded in
|
||||
`.sisyphus/evidence/task-7-image-digest.txt`. Kubernetes pull smoke is recorded in
|
||||
`.sisyphus/evidence/task-7-image-pull.txt` and is blocked here because `kubectl`
|
||||
is not installed or not on `PATH`.
|
||||
|
||||
Once registry credentials are available, rerun the push, capture the
|
||||
registry-reported digest, and replace `<registry-digest>` in the mapping above
|
||||
before Task 6/9 consumes the label configuration.
|
||||
After importing the archive as a Hetzner image, record its image ID in the
|
||||
controller host configuration before dispatching Nix jobs.
|
||||
|
||||
## Image contents
|
||||
|
||||
|
||||
@@ -19,10 +19,8 @@ data:
|
||||
insecure: false
|
||||
fetch_timeout: 5s
|
||||
fetch_interval: 2s
|
||||
labels:
|
||||
- ubuntu-latest
|
||||
# The nix label is intentionally disabled until the runner image has a
|
||||
# concrete registry-reported digest; see ../runbook.md before deploy.
|
||||
# Persistent pool is disabled; controller owns all runner labels.
|
||||
labels: []
|
||||
|
||||
cache:
|
||||
enabled: true
|
||||
|
||||
@@ -45,7 +45,7 @@ variable "control_plane_server_type" {
|
||||
}
|
||||
|
||||
variable "worker_server_type" {
|
||||
description = "Default worker server type for the budget trusted DinD runner pool. One cpx22 worker keeps the idle baseline cheap; scale out later if job pressure requires it."
|
||||
description = "Default worker server type for the budget trusted DinD cluster. One cpx22 worker keeps the cluster baseline cheap; scale out later if rollback capacity requires it."
|
||||
type = string
|
||||
default = "cpx22"
|
||||
}
|
||||
|
||||
@@ -2,15 +2,13 @@
|
||||
|
||||
## Scope
|
||||
|
||||
This directory is the repo-owned boundary for the first Gitea Actions runner
|
||||
pool. Task 1 only establishes the scaffold and immutable decision contract;
|
||||
downstream tasks will add OpenTofu backend/provider files, Kubernetes manifests,
|
||||
and a Nix-capable runner image under the existing subdirectories.
|
||||
This directory is the repo-owned boundary for the Gitea Actions runner pool.
|
||||
The controller is the active zero-idle path; Kubernetes manifests and the
|
||||
Nix-capable image are retained for manual rollback and maintenance.
|
||||
|
||||
The target service is `https://gitea.hectic-lab.com` for the Gitea organization
|
||||
`hectic-lab`. The first pool is fixed-size and trusted-only. "Ephemeral" means
|
||||
workflow job containers are ephemeral, while each runner pod keeps its runner
|
||||
identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
|
||||
`hectic-lab`. The pool is trusted-only. "Ephemeral" means each controller VM
|
||||
and workflow job is disposable; the Kubernetes StatefulSet is rollback-only.
|
||||
|
||||
## Immutable decisions
|
||||
|
||||
@@ -27,21 +25,21 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
|
||||
`/data`, including `/data/.runner`.
|
||||
- Container builds run through privileged rootful DinD inside trusted runner
|
||||
pods; host Docker socket mounting is not an implementation path.
|
||||
- The active runner label is `ubuntu-latest`. The `nix` label is not live until
|
||||
the Nix-capable image has been pushed and a concrete registry-reported digest
|
||||
is added to the runner ConfigMap.
|
||||
- `ubuntu-latest` and `nix` are controller-managed zero-idle aliases for
|
||||
`gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels.
|
||||
- First scope is trusted internal workflows only, with no untrusted fork or PR
|
||||
workflow support.
|
||||
- First scope has no autoscaling, no KEDA, and no dynamic runner controller.
|
||||
- Zero-idle allocation is handled by the repo-owned controller; Kubernetes is
|
||||
not an active autoscaling path.
|
||||
|
||||
## Lifecycle boundaries
|
||||
|
||||
- `infra/gitea-runners/opentofu/`: downstream OpenTofu stack for the S3 backend
|
||||
- `infra/gitea-runners/opentofu/`: OpenTofu stack for the S3 backend
|
||||
contract, Hetzner provider configuration, and kube-hetzner module wiring.
|
||||
- `infra/gitea-runners/k8s/`: downstream namespace, ConfigMap, Secret mount,
|
||||
- `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount,
|
||||
StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work.
|
||||
- `infra/gitea-runners/image/`: downstream notes or sources for the runner image
|
||||
handoff; package or flake output changes are outside Task 1.
|
||||
- `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes
|
||||
rollback image; active zero-idle Nix image is selected by Hetzner image ID.
|
||||
- `infra/gitea-runners/runbook.md`: this contract plus later operational
|
||||
commands, rollback notes, and acceptance evidence references.
|
||||
|
||||
@@ -52,8 +50,8 @@ identity in per-pod `/data/.runner` storage backed by a StatefulSet PVC.
|
||||
instructions, or GitHub Actions ARC assumptions.
|
||||
- Untrusted fork/PR workflows are out of first scope; privileged DinD is only
|
||||
acceptable for trusted internal jobs.
|
||||
- Autoscaling/KEDA is out of first scope; start with a fixed-size StatefulSet
|
||||
runner pool.
|
||||
- The persistent StatefulSet is rollback-only and defaults to zero replicas;
|
||||
normal jobs use controller-managed zero-idle VMs.
|
||||
- No actual secrets are committed: no kubeconfig, runner token, Hetzner token,
|
||||
S3 credentials, decrypted SOPS files, or SOPS age keys.
|
||||
- OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea
|
||||
@@ -248,8 +246,7 @@ These commands are executable only when the external inputs are available:
|
||||
- S3 backend credentials and endpoint access
|
||||
- a matching SOPS age identity for `sus/gitea-runners.yaml`
|
||||
- `kubectl` access to the target cluster
|
||||
- a concrete digest for the pushed Nix-capable runner image, if enabling the
|
||||
`nix` label
|
||||
- a valid `GCR_NIX_IMAGE_ID` for controller-managed Nix jobs
|
||||
|
||||
If any input is missing, stop before `tofu apply`. Do not guess values or reuse
|
||||
stale kubeconfig files.
|
||||
@@ -260,15 +257,12 @@ Before production Kubernetes apply or rollout, satisfy both manifest gates:
|
||||
Kustomize overlay intentionally does not include a placeholder Secret, but
|
||||
the StatefulSet still mounts `secretName: gitea-runner-token` as
|
||||
`/runner-secrets/token` for `GITEA_RUNNER_REGISTRATION_TOKEN_FILE`.
|
||||
2. Keep the active ConfigMap on `ubuntu-latest` only unless the Nix-capable
|
||||
image has been pushed successfully. Enable the `nix` label only by adding a
|
||||
digest-pinned `docker://` mapping with the exact registry-reported sha256
|
||||
digest from that push.
|
||||
2. Keep the persistent-pool ConfigMap labels empty. Runner labels belong to the
|
||||
controller; Nix image readiness is governed by `GCR_NIX_IMAGE_ID`.
|
||||
|
||||
Use the same SOPS materialization pattern as token rotation before applying the
|
||||
Kubernetes overlay. Applying the namespace alone is allowed so the Secret has a
|
||||
target namespace; the full overlay remains gated on the Secret and digest
|
||||
decisions:
|
||||
target namespace; the full overlay remains gated on the Secret.
|
||||
|
||||
```sh
|
||||
kubectl apply -f infra/gitea-runners/k8s/namespace.yaml
|
||||
@@ -283,8 +277,7 @@ kubectl -n gitea-runners create secret generic gitea-runner-token \
|
||||
```
|
||||
|
||||
Do not run `kubectl apply -k infra/gitea-runners/k8s` until the Secret command
|
||||
above succeeds. Do not claim or enable the `nix` runner label until the image
|
||||
publication step has produced the concrete digest.
|
||||
above succeeds. The persistent pool ConfigMap must retain empty labels.
|
||||
|
||||
```sh
|
||||
tofu -chdir=infra/gitea-runners/opentofu init
|
||||
@@ -309,14 +302,14 @@ Expected status after deploy:
|
||||
- `kubectl config current-context` names the runner cluster context.
|
||||
- `kubectl get nodes -o wide` shows all expected Hetzner nodes Ready.
|
||||
- `kubectl get sc` shows the Hetzner CSI storage class used by runner PVCs.
|
||||
- `kubectl -n gitea-runners get statefulset gitea-runner` shows 5 desired and 5 ready replicas.
|
||||
- `kubectl -n gitea-runners get pvc` shows 5 Bound PVCs.
|
||||
- `kubectl -n gitea-runners logs statefulset/gitea-runner -c runner --tail=200` shows the runner daemon started and no token value.
|
||||
- `kubectl -n gitea-runners get statefulset gitea-runner` shows 0 desired and 0 ready replicas.
|
||||
- `kubectl -n gitea-runners get pvc` shows no active runner PVCs; retained PVCs are rollback-only.
|
||||
- The controller host reports healthy and owns runner registrations; no persistent runner claims `ubuntu-latest` or `nix`.
|
||||
|
||||
## Scale 5 to 10 to 5
|
||||
## Legacy rollback pool scaling (manual only)
|
||||
|
||||
Scaling is a temporary capacity exercise, not the steady-state setting. Scale up,
|
||||
wait for readiness, run the concurrent smoke jobs, then scale back down to 5.
|
||||
Persistent-pool scaling is not part of normal operation. Use only after restoring
|
||||
its labels and disabling the zero-idle controller as described in `Rollback`.
|
||||
|
||||
```sh
|
||||
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=10
|
||||
@@ -324,9 +317,9 @@ kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
|
||||
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
|
||||
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
|
||||
|
||||
# Run the concurrent smoke workflows now.
|
||||
# Run only workflows supported by restored persistent labels.
|
||||
|
||||
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=1
|
||||
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
|
||||
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
|
||||
kubectl -n gitea-runners get pods -l app.kubernetes.io/name=gitea-runner -o wide
|
||||
kubectl -n gitea-runners get pvc -l app.kubernetes.io/name=gitea-runner -o wide
|
||||
@@ -405,7 +398,7 @@ Do not run the delete command for a runner that still has an active
|
||||
`gitea-runner-*` pod or retained `data-gitea-runner-*` PVC unless that PVC is
|
||||
being intentionally reset for re-registration.
|
||||
|
||||
## Application rollback
|
||||
## Legacy Kubernetes application rollback
|
||||
|
||||
Rollback the app layer only. Do not use this section to destroy the cluster.
|
||||
|
||||
@@ -504,7 +497,7 @@ as complete.
|
||||
|
||||
## Ephemeral VM runner cutover
|
||||
|
||||
This section governs replacing the fixed K8s runner pool with the
|
||||
This section governs replacing the legacy persistent K8s runner pool with the
|
||||
ephemeral-VM controller (`package/gitea-runner-controller`) on this host.
|
||||
The K8s pool above remains rollback-only until cutover is explicitly accepted.
|
||||
|
||||
@@ -571,8 +564,9 @@ The `deploy-neuro` workflow uses these nested limits:
|
||||
| Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours |
|
||||
| VM hard lifetime from allocation | 480 minutes plus 10-minute controller grace |
|
||||
|
||||
Other runner labels keep their existing 180-minute limits. Deploy the controller
|
||||
and Gitea watchdog settings before dispatching the longer workflow. Already
|
||||
`ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for
|
||||
long-running Nix deployments. Deploy the controller and Gitea watchdog settings
|
||||
before dispatching the longer workflow. Already
|
||||
allocated VMs retain the TTL and runner configuration assigned when they were
|
||||
created; updating the controller does not extend a running job.
|
||||
|
||||
@@ -628,23 +622,27 @@ K8s rollback pool now defaults to deleted state:
|
||||
Re-enable sequence:
|
||||
|
||||
```sh
|
||||
# 1. stop ephemeral path
|
||||
sed -i 's/hectic.services.gitea-runner-controller = {.*}/\/* disabled *\//' \
|
||||
nixos/system/hectic-lab/hectic-lab.nix # or set enable = false
|
||||
# 1. edit nixos/system/hectic-lab/hectic-lab.nix and set
|
||||
# services.gitea-runner-controller.enable = false, then rebuild:
|
||||
nixos-rebuild --target root@128.140.75.58 switch
|
||||
|
||||
# 2. reprovision old kube-hetzner nodes when they were deleted:
|
||||
tofu -chdir=infra/gitea-runners/opentofu apply
|
||||
|
||||
# 3. restore kubeconfig / cluster access, then re-enable K8s runner pool:
|
||||
# 3. while controller is disabled, destroy every surviving managed VM and
|
||||
# verify no gcr-* runner registration remains online:
|
||||
hcloud server list -o json \
|
||||
| jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
|
||||
| xargs -r -n1 hcloud server delete
|
||||
|
||||
# 4. restore kubeconfig / cluster access, restore `ubuntu-latest` in the
|
||||
# ConfigMap labels, then re-enable K8s runner pool. The legacy image does not
|
||||
# provide `nix`; do not dispatch Nix workflows until a Nix-capable K8s image
|
||||
# and label mapping are restored:
|
||||
kubectl -n gitea-runners edit configmap/gitea-runner-config
|
||||
kubectl -n gitea-runners scale statefulset/gitea-runner --replicas=5
|
||||
kubectl -n gitea-runners rollout status statefulset/gitea-runner --timeout=10m
|
||||
```
|
||||
|
||||
Any surviving ephemeral VMs after step 1 must be destroyed manually once:
|
||||
|
||||
```sh
|
||||
hcloud server list -o json \
|
||||
| jq -r '.[] | select(.labels["gitea-runner-controller"]=="managed") | .id' \
|
||||
| xargs -r -n1 hcloud server delete
|
||||
```
|
||||
Any managed VM or `gcr-*` registration found after step 3 must be removed before
|
||||
restoring persistent labels; otherwise both pools can claim the same job.
|
||||
|
||||
@@ -23,10 +23,11 @@ gcr_server_hourly_rate() {
|
||||
|
||||
gcr_label_ttl() {
|
||||
case "$1" in
|
||||
gross-x86) printf '180' ;;
|
||||
ubuntu-latest|gross-x86) printf '180' ;;
|
||||
gross-arm) printf '180' ;;
|
||||
gross-x86-perf) printf '180' ;;
|
||||
gross-mixed-econ) printf '180' ;;
|
||||
nix) printf '480' ;;
|
||||
gross-nix-x86) printf '180' ;;
|
||||
gross-nix-arm) printf '180' ;;
|
||||
gross-nix-x86-perf) printf '480' ;;
|
||||
@@ -40,7 +41,7 @@ gcr_label_ttl() {
|
||||
gcr_label_candidates() {
|
||||
label="$1"
|
||||
case "$label" in
|
||||
gross-x86)
|
||||
ubuntu-latest|gross-x86)
|
||||
printf '%s\n' \
|
||||
'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \
|
||||
'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \
|
||||
@@ -64,7 +65,7 @@ gcr_label_candidates() {
|
||||
'cax41 nbg1 arm64' 'cax41 fsn1 arm64' 'cax41 hel1 arm64' \
|
||||
'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64'
|
||||
;;
|
||||
gross-nix-x86)
|
||||
nix|gross-nix-x86)
|
||||
printf '%s\n' \
|
||||
'cx53 nbg1 amd64' 'cx53 fsn1 amd64' 'cx53 hel1 amd64' \
|
||||
'cx43 nbg1 amd64' 'cx43 fsn1 amd64' 'cx43 hel1 amd64' \
|
||||
|
||||
@@ -8,8 +8,18 @@ set -eu
|
||||
|
||||
gcr_state_init
|
||||
export GCR_ALLOWED_REPOS='hectic-lab/util.nix'
|
||||
export GCR_IMAGE_ID='313131'
|
||||
export GCR_NIX_IMAGE_ID='424242'
|
||||
|
||||
test "$(gcr_label_ttl ubuntu-latest)" = '180'
|
||||
test "$(gcr_label_ttl nix)" = '480'
|
||||
test "$(gcr_decide ubuntu-latest hectic-lab/util.nix)" = 'cx53 180 0.032'
|
||||
test "$(gcr_decide nix hectic-lab/util.nix)" = 'cx53 480 0.032'
|
||||
test "$(gcr_label_candidates ubuntu-latest | head -n1)" = 'cx53 nbg1 amd64'
|
||||
test "$(gcr_label_candidates nix | head -n1)" = 'cx53 nbg1 amd64'
|
||||
test "$(gcr_image_id_for_arch amd64 ubuntu-latest)" = '313131'
|
||||
test "$(gcr_image_id_for_arch amd64 nix)" = '424242'
|
||||
|
||||
profile="$(gcr_decide gross-nix-x86-highmem hectic-lab/util.nix)"
|
||||
test "$profile" = 'ccx53 480 0.8550'
|
||||
test "$(gcr_label_ttl gross-nix-x86-highmem)" = '480'
|
||||
|
||||
Reference in New Issue
Block a user