diff --git a/docs/README.md b/docs/README.md index 786e87a6..4fbeb11c 100644 --- a/docs/README.md +++ b/docs/README.md @@ -4,10 +4,10 @@ ## Gitea runner labels -Common labels for controller-managed zero-idle runners: +Common labels for controller-managed on-demand runners: -- `ubuntu-latest` — zero-idle alias for `gross-x86` -- `nix` — zero-idle Nix alias with 480-minute TTL +- `ubuntu-latest` — on-demand alias for `gross-x86` +- `nix` — on-demand Nix alias with 480-minute TTL - `gross-x86` — x86 fallback chain `cx53` / `cx43` / `cx33` - `gross-arm` — ARM fallback chain `cax41` / `cax31` / `cax21` - `gross-x86-perf` — x86 performance chain `cx53` / `cpx62` / `cpx52` @@ -21,7 +21,10 @@ Common labels for controller-managed zero-idle runners: Region order for fallback: `nbg1`, then `fsn1`, then `hel1`. The legacy Kubernetes persistent pool is disabled (`replicas: 0`) and has no -registered labels. All listed labels are handled by the zero-idle controller. +registered labels. All listed labels are handled by the VM controller. After a +successful job, a bootstrapped VM remains running and idle until its next hourly +lifetime boundary, capped by label TTL. A queued job from same repository with +same label reuses it without another VM creation or budget reservation. Operational details: `infra/gitea-runners/runbook.md` and `package/gitea-runner-controller/decide.sh`. diff --git a/docs/attic-cache.md b/docs/attic-cache.md index e84820b5..8dc6df61 100644 --- a/docs/attic-cache.md +++ b/docs/attic-cache.md @@ -201,9 +201,9 @@ build/deploy command, batches of at most 8 paths, and 600 seconds per upload att upload deadline covers the **whole batch**, not each individual path. Its final drain is bounded at 1 hour; the 435-minute job budget leaves 15 minutes for setup and cleanup. The `gross-nix-x86-perf` runner limit and Gitea's endless-task -watchdog are 8 hours. The VM lifetime starts at allocation and includes the -controller's additional 10-minute grace. A prolonged cache outage can still -exhaust the drain before every queued path is uploaded. +watchdog are 8 hours. VM hard lifetime starts at allocation and has no controller +destruction grace. A prolonged cache outage can still exhaust the drain before +every queued path is uploaded. The build timeout covers the entire wrapped command, not each derivation. Completed outputs can be reused from the cache, but an interrupted CUDA/Magma diff --git a/infra/gitea-runners/runbook.md b/infra/gitea-runners/runbook.md index 861134b9..40eb210d 100644 --- a/infra/gitea-runners/runbook.md +++ b/infra/gitea-runners/runbook.md @@ -3,7 +3,7 @@ ## Scope This directory is the repo-owned boundary for the Gitea Actions runner pool. -The controller is the active zero-idle path; Kubernetes manifests and the +The controller is the active on-demand path; Kubernetes manifests and the Nix-capable image are retained for manual rollback and maintenance. The target service is `https://gitea.hectic-lab.com` for the Gitea organization @@ -25,11 +25,11 @@ and workflow job is disposable; the Kubernetes StatefulSet is rollback-only. `/data`, including `/data/.runner`. - Container builds run through privileged rootful DinD inside trusted runner pods; host Docker socket mounting is not an implementation path. -- `ubuntu-latest` and `nix` are controller-managed zero-idle aliases for +- `ubuntu-latest` and `nix` are controller-managed on-demand aliases for `gross-x86` and `gross-nix-x86`; the Kubernetes pool has no active labels. - First scope is trusted internal workflows only, with no untrusted fork or PR workflow support. -- Zero-idle allocation is handled by the repo-owned controller; Kubernetes is +- On-demand allocation is handled by the repo-owned controller; Kubernetes is not an active autoscaling path. ## Lifecycle boundaries @@ -39,7 +39,7 @@ and workflow job is disposable; the Kubernetes StatefulSet is rollback-only. - `infra/gitea-runners/k8s/`: rollback-only namespace, ConfigMap, Secret mount, StatefulSet, PVC, DinD sidecar, cleanup, and operational manifest work. - `infra/gitea-runners/image/`: notes and handoff for the optional Kubernetes - rollback image; active zero-idle Nix image is selected by Hetzner image ID. + rollback image; active on-demand Nix image is selected by Hetzner image ID. - `infra/gitea-runners/runbook.md`: this contract plus later operational commands, rollback notes, and acceptance evidence references. @@ -51,7 +51,7 @@ and workflow job is disposable; the Kubernetes StatefulSet is rollback-only. - Untrusted fork/PR workflows are out of first scope; privileged DinD is only acceptable for trusted internal jobs. - The persistent StatefulSet is rollback-only and defaults to zero replicas; - normal jobs use controller-managed zero-idle VMs. + normal jobs use controller-managed on-demand VMs. - No actual secrets are committed: no kubeconfig, runner token, Hetzner token, S3 credentials, decrypted SOPS files, or SOPS age keys. - OpenTofu must not manage plaintext Kubernetes Secrets containing the Gitea @@ -562,7 +562,7 @@ The `deploy-neuro` workflow uses these nested limits: | `gross-nix-x86-perf` runner | 480 minutes | | `gross-nix-x86-highmem` runner | 480 minutes | | Gitea `actions.ENDLESS_TASK_TIMEOUT` | 8 hours | -| VM hard lifetime from allocation | 480 minutes plus 10-minute controller grace | +| VM hard lifetime from allocation | 480 minutes; no destruction grace | `ubuntu-latest` keeps a 180-minute limit; `nix` uses a 480-minute limit for long-running Nix deployments. Deploy the controller and Gitea watchdog settings @@ -576,10 +576,27 @@ only, never to a lower-RAM server type. Current Hetzner public pricing for Germany/Finland CCX53 is 0.8550 EUR/hour excluding IPv4, so one 480-minute allocation reserves 6.84 EUR against the controller budget before VM creation. -These are maximum lifetimes: terminal jobs still trigger immediate VM teardown. -The controller's budget reservation uses the full label TTL, so a long-running -label reserves more of the existing monthly budget. Do not raise that budget or -disable timeout safeguards just to bypass a refused allocation. +These are maximum lifetimes. Failed, cancelled, skipped, and unbootstrapped jobs +still trigger immediate VM teardown; failed jobs retain pre-destroy diagnostics. +After a successful job, its bootstrapped VM stays running until next hourly +boundary measured from original VM creation, capped by profile TTL. Same-repo, +same-label queued work can atomically claim that idle VM. Reuse preserves +original Hetzner labels and runner name, creates no server, fetches no new +registration token, and makes no second budget reservation. Idle VMs are still +billed: controller deletes them at slot expiry and never relies on stopping a +server to avoid charges. + +Reuse retains runner host filesystem and registration identity. It is therefore +restricted to same repository and exact label inside this trusted-only pool; +allowed repositories must not run untrusted fork or pull-request code. A failed +or otherwise non-successful job is never reused. + +Active and idle VMs are deleted at profile hard TTL without grace. Idle reuse is +allowed only when at least one configured reconcile interval remains before both +slot expiry and hard TTL. Controller budget reservation still uses full label TTL +on initial creation, so a long-running label reserves more of existing monthly +budget. Do not raise that budget or disable timeout safeguards just to bypass a +refused allocation. After changing any timeout, verify the complete chain rather than only `timeout-minutes`; a shorter wrapper, runner, server watchdog, or VM TTL wins. @@ -592,7 +609,9 @@ journalctl -u gitea-runner-webhook -n 20 --no-pager hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed")] | length' # expect 0 ``` -Zero managed VMs at idle is the steady-state assertion. +Zero managed VMs is expected after retained billing slots expire. Immediately +after successful work, one managed VM per retained profile may remain until its +recorded hourly boundary. ### End-to-end acceptance (Task 9) @@ -602,10 +621,10 @@ Trigger `.gitea/workflows/runner-nix-smoke.yaml` via workflow_dispatch, then: watch_labels() { hcloud server list -o json | jq '[.[] | select(.labels["gitea-runner-controller"]=="managed") | {id,name,labels}]'; } watch_labels # exactly one VM while queued/running journalctl -f -u gitea-runner-controller # vm-created / vm-destroyed events -watch_labels # expect [] after completion +watch_labels # VM may remain until next hourly boundary curl -fsS -H "Authorization: token $ADMIN" \ https://gitea.hectic-lab.com/api/v1/orgs/hectic-lab/actions/runners \ - | jq '[.entries[] | select(.name | startswith("gcr-"))] | length' # expect 0 + | jq '[.entries[] | select(.name | startswith("gcr-"))] | length' # may remain during retained slot ``` Failure paths to verify identically: duplicate delivery (send same webhook twice diff --git a/nixos/module/hectic/service/gitea-runner-controller.nix b/nixos/module/hectic/service/gitea-runner-controller.nix index 08786a37..dc803188 100644 --- a/nixos/module/hectic/service/gitea-runner-controller.nix +++ b/nixos/module/hectic/service/gitea-runner-controller.nix @@ -92,12 +92,12 @@ in concurrencyCap = lib.mkOption { type = lib.types.int; default = 2; - description = "Maximum simultaneously running ephemeral VMs (global)."; + description = "Maximum simultaneously assigned runner VMs (global); retained idle VMs do not count."; }; perRepoCap = lib.mkOption { type = lib.types.int; default = 1; - description = "Maximum concurrent ephemeral VMs per repo."; + description = "Maximum concurrently assigned runner VMs per repo; retained idle VMs do not count."; }; reconcileIntervalSec = lib.mkOption { type = lib.types.int; diff --git a/package/gitea-runner-controller/controller.sh b/package/gitea-runner-controller/controller.sh index bfe03b5d..06fa2fde 100644 --- a/package/gitea-runner-controller/controller.sh +++ b/package/gitea-runner-controller/controller.sh @@ -3,13 +3,9 @@ # Owns: TTL sweep, orphan-VM sweep, deferred-job retry, stale-runner dereg, # startup convergence. Runs forever under systemd; webhook service is separate. -gcr_ttl_grace_sec() { - printf '%s' "$((10 * 60))" -} - gcr_record_age_sec() { created_at="$(gcr_record_field "$1" created_at)" - now="$(date -u '+%s')" + now="$(gcr_now_epoch)" case "$created_at" in ''|*[!0-9]*) printf '%s' 999999 ;; *) printf '%s' "$((now - created_at))" ;; @@ -20,16 +16,63 @@ gcr_sweep_ttl() { for f in $(gcr_active_records); do rec="$(cat "$f")" status="$(gcr_record_field "$rec" status)" - [ "$status" = "vm_active" ] || [ "$status" = "pending_vm" ] || continue - job_id="$(gcr_record_field "$rec" job_id)" attempt="$(gcr_record_field "$rec" run_attempt)" + + if [ "$status" = "idle_vm" ]; then + gcr_lock_acquire idle-pool || continue + rec="$(gcr_record_get "$job_id" "$attempt")" + if [ -z "$rec" ] || [ "$(gcr_record_field "$rec" status)" != "idle_vm" ]; then + gcr_lock_release idle-pool + continue + fi + if gcr_idle_record_unexpired "$rec"; then + gcr_lock_release idle-pool + continue + fi + + vm_id="$(gcr_record_field "$rec" vm_id)" + if gcr_record_vm_owned_elsewhere "$vm_id" "$job_id" "$attempt"; then + gcr_log warn --ns=sweep "removing superseded idle record job=$job_id vm=$vm_id" + gcr_record_del "$job_id" "$attempt" + gcr_lock_release idle-pool + continue + fi + gcr_log info --ns=sweep "idle slot expired job=$job_id vm=$vm_id" + if [ -n "$vm_id" ] && [ "$vm_id" != "null" ] && [ "$vm_id" != "0" ]; then + gcr_vm_destroy "$vm_id" || true + gcr_event "vm-destroyed" "$job_id" "{\"vm_id\":$vm_id,\"reason\":\"idle-expired\"}" + fi + gcr_record_del "$job_id" "$attempt" + gcr_lock_release idle-pool + continue + fi + + [ "$status" = "vm_active" ] || [ "$status" = "pending_vm" ] || continue ttl_min="$(gcr_record_field "$rec" ttl_min)" case "$ttl_min" in ''|*[!0-9]*) continue ;; esac - max_sec="$((ttl_min * 60 + $(gcr_ttl_grace_sec)))" + max_sec="$((ttl_min * 60))" age="$(gcr_record_age_sec "$rec")" - if [ "$age" -gt "$max_sec" ]; then + if [ "$age" -ge "$max_sec" ]; then + key="$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_acquire "$key" || continue + rec="$(gcr_record_get "$job_id" "$attempt")" + case "$(gcr_record_field "$rec" status)" in + pending_vm|vm_active) ;; + *) gcr_lock_release "$key"; continue ;; + esac + ttl_min="$(gcr_record_field "$rec" ttl_min)" + case "$ttl_min" in + ''|*[!0-9]*) gcr_lock_release "$key"; continue ;; + esac + max_sec="$((ttl_min * 60))" + age="$(gcr_record_age_sec "$rec")" + if [ "$age" -lt "$max_sec" ]; then + gcr_lock_release "$key" + continue + fi + vm_id="$(gcr_record_field "$rec" vm_id)" gcr_log warn --ns=sweep "TTL exceeded job=$job_id age=${age}s max=${max_sec}s" if [ -n "$vm_id" ] && [ "$vm_id" != "null" ] && [ "$vm_id" != "0" ]; then @@ -40,13 +83,21 @@ gcr_sweep_ttl() { fi gcr_event "job-ttl-expired" "$job_id" "{\"age\":$age}" gcr_record_del "$job_id" "$attempt" - gcr_lock_release "$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_release "$key" fi done } gcr_sweep_orphan_vms() { - vms_json="$(gcr_vm_list_managed)" || return 0 + gcr_lock_acquire admission || return 0 + vms_json="$(gcr_vm_list_managed)" || { + gcr_lock_release admission + return 0 + } + if ! gcr_lock_acquire idle-pool; then + gcr_lock_release admission + return 0 + fi count="$(printf '%s' "$vms_json" | jq 'length')" i=0 while [ "$i" -lt "$count" ]; do @@ -55,69 +106,125 @@ gcr_sweep_orphan_vms() { jid="$(printf '%s' "$vm" | jq -r '.labels["gcr.job-id"] // ""')" att="$(printf '%s' "$vm" | jq -r '.labels["gcr.run-attempt"] // ""')" - known="" - if [ -n "$jid" ] && [ -n "$att" ]; then - rec="$(gcr_record_get "$jid" "$att")" - [ -n "$rec" ] && known=1 - fi - - if [ -z "$known" ]; then + if ! gcr_record_exists_for_vm_id "$vm_id"; then gcr_log warn --ns=sweep "orphan VM $vm_id job=$jid attempt=$att -> destroy" gcr_vm_destroy "$vm_id" || true gcr_event "orphan-vm-destroyed" "${jid:-unknown}" "{\"vm_id\":$vm_id}" fi i=$((i + 1)) done + gcr_lock_release idle-pool + gcr_lock_release admission } gcr_alloc_deferred() { job_id="$1"; attempt="$2" + key="$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_acquire "$key" || return 0 rec="$(gcr_record_get "$job_id" "$attempt")" - [ -n "$rec" ] || return 0 - [ "$(gcr_record_field "$rec" status)" = "deferred" ] || return 0 + if [ -z "$rec" ] || [ "$(gcr_record_field "$rec" status)" != "deferred" ]; then + gcr_lock_release "$key" + return 0 + fi repo="$(gcr_record_field "$rec" repo)" label="$(gcr_record_field "$rec" label)" - state="$(gcr_gitea_job_state "$repo" "$job_id")" || return 0 + state="$(gcr_gitea_job_state "$repo" "$job_id")" || { + gcr_lock_release "$key" + return 0 + } case "$state" in completed:*) gcr_log info --ns=alloc "deferred job=$job_id already terminal ($state), dropping record" gcr_record_del "$job_id" "$attempt" + gcr_lock_release "$key" return 0 ;; esac - profile="$(gcr_label_profile "$label")" || return 0 + profile="$(gcr_label_profile "$label")" || { + gcr_lock_release "$key" + return 0 + } set -- $profile server_type="$1"; ttl_min="$2"; rate="$3" + gcr_lock_acquire admission || { + gcr_lock_release "$key" + return 0 + } active="$(gcr_count_active)" repo_active="$(gcr_count_active_repo "$repo")" - [ "$active" -ge "${GCR_CONCURRENCY_CAP:-2}" ] && return 0 - [ "$repo_active" -ge "${GCR_PER_REPO_CAP:-1}" ] && return 0 + if [ "$active" -ge "${GCR_CONCURRENCY_CAP:-2}" ] \ + || [ "$repo_active" -ge "${GCR_PER_REPO_CAP:-1}" ]; then + gcr_lock_release admission + gcr_lock_release "$key" + return 0 + fi - gcr_budget_add "$rate" "$ttl_min" || return 0 - reg_token="$(gcr_gitea_registration_token "$repo")" || return 0 + claim_status=0 + gcr_claim_idle "$job_id" "$attempt" "$repo" "$label" || claim_status="$?" + if [ "$claim_status" -eq 0 ]; then + reused="$(gcr_record_get "$job_id" "$attempt")" + vm_id="$(gcr_record_field "$reused" vm_id)" + if gcr_vm_runner_service "$vm_id" start \ + && gcr_gitea_runner_disabled "$repo" "$(gcr_record_field "$reused" vm_name)" false; then + reused="$(gcr_record_get "$job_id" "$attempt")" + reused="$(printf '%s' "$reused" | jq -c '.bootstrapped = true | del(.reused_vm)')" + gcr_record_put "$job_id" "$attempt" "$reused" + else + gcr_gitea_runner_disabled "$repo" "$(gcr_record_field "$reused" vm_name)" true || true + gcr_event "vm-reuse-start-failed" "$job_id" "{\"vm_id\":$vm_id,\"via\":\"deferred-retry\"}" + fi + gcr_lock_release admission + gcr_lock_release "$key" + gcr_event "vm-reused" "$job_id" "{\"vm_id\":$vm_id,\"label\":\"$label\",\"via\":\"deferred-retry\"}" + gcr_log info --ns=alloc "deferred job=$job_id reused vm=$vm_id" + return 0 + fi + if [ "$claim_status" -eq 2 ]; then + gcr_lock_release admission + gcr_lock_release "$key" + return 0 + fi - key="$(gcr_alloc_key "$job_id" "$attempt")" - gcr_lock_acquire "$key" || return 0 - - vm_name="gcr-${job_id}-${attempt}" - vm_id="$(gcr_vm_create "$vm_name" "$label" "$server_type" "$ttl_min" \ - "$reg_token" "$job_id" "$attempt" "$repo")" && [ -n "$vm_id" ] || { + gcr_budget_can_add "$rate" "$ttl_min" || { + gcr_lock_release admission + gcr_lock_release "$key" + return 0 + } + reg_token="$(gcr_gitea_registration_token "$repo")" || { + gcr_lock_release admission gcr_lock_release "$key" return 0 } + vm_name="gcr-${job_id}-${attempt}" + created_at="$(gcr_now_epoch)" + vm_id="$(gcr_vm_create "$vm_name" "$label" "$server_type" "$ttl_min" \ + "$reg_token" "$job_id" "$attempt" "$repo")" && [ -n "$vm_id" ] || { + gcr_lock_release admission + gcr_lock_release "$key" + return 0 + } + + gcr_budget_add "$rate" "$ttl_min" + rec="$(jq -n --arg j "$job_id" --arg a "$attempt" --arg r "$repo" \ - --arg l "$label" --arg t "$(date -u '+%s')" --arg v "$vm_id" \ + --arg l "$label" --arg t "$created_at" --arg v "$vm_id" \ --arg vn "$vm_name" --arg ttl "$ttl_min" \ '{job_id:$j, run_attempt:$a, repo:$r, label:$l, created_at:$t, ttl_min:($ttl|tonumber), vm_id:($v|tonumber), vm_name:$vn, bootstrapped:false, status:"pending_vm"}')" - gcr_record_put "$job_id" "$attempt" "$rec" + if ! gcr_record_put "$job_id" "$attempt" "$rec"; then + gcr_vm_destroy "$vm_id" || true + gcr_lock_release admission + gcr_lock_release "$key" + return 0 + fi + gcr_lock_release admission gcr_lock_release "$key" gcr_event "vm-created" "$job_id" "{\"vm_id\":$vm_id,\"label\":\"$label\",\"ttl_min\":$ttl_min,\"via\":\"deferred-retry\"}" gcr_log info --ns=alloc "deferred job=$job_id allocated vm=$vm_id" @@ -134,38 +241,56 @@ gcr_retry_deferred() { } gcr_sweep_stale_runners() { - runners="$(gcr_gitea_list_runners)" || return 0 - # Here-doc instead of pipe: dash runs pipe tails in a subshell, which - # would strand gcr_event/audit writes from the caller's perspective. - while read -r rid rname; do - [ -n "${rid:-}" ] || continue - case "$rname" in - gcr-*) ;; + gcr_lock_acquire idle-pool || return 0 + oldIFS="$IFS" + IFS=, + for allowed_repo in ${GCR_ALLOWED_REPOS:-}; do + IFS="$oldIFS" + case "$allowed_repo" in + */\*) + owner="${allowed_repo%/*}" + repos="$(gcr_gitea_list_org_repos "$owner")" || { + IFS=, + continue + } + ;; + */*) repos="$allowed_repo" ;; *) continue ;; esac + while read -r repo; do + [ -n "${repo:-}" ] || continue + gcr_repo_allowed "$repo" || continue + runners="$(gcr_gitea_list_runners "$repo")" || continue + # Here-doc instead of pipe: dash runs pipe tails in a subshell, which + # would strand gcr_event/audit writes from the caller's perspective. + while read -r rid rname; do + [ -n "${rid:-}" ] || continue + case "$rname" in + gcr-*) ;; + *) continue ;; + esac - # gcr--: alive iff a matching active/pending record exists. - rest="${rname#gcr-}" - jid="${rest%-*}" - att="${rest##*-}" - rec="" - case "$jid" in *[!0-9]*|"") rec="" ;; - *) case "$att" in *[!0-9]*|"") rec="" ;; - *) rec="$(gcr_record_get "$jid" "$att")" ;; - esac ;; - esac - - if [ -z "$rec" ]; then - gcr_log warn --ns=sweep "stale runner registration id=$rid name=$rname -> delete" - if gcr_gitea_delete_runner "$rid"; then - gcr_event "stale-runner-deleted" "${jid:-unknown}" "{\"runner_id\":$rid,\"name\":\"$rname\"}" - else - gcr_log error --ns=sweep "failed deleting runner id=$rid" + # Runner name stays tied to original VM across later job assignments. + rest="${rname#gcr-}" + jid="${rest%-*}" + if ! gcr_record_exists_for_vm_name "$rname"; then + gcr_log warn --ns=sweep "stale repo=$repo registration id=$rid name=$rname -> delete" + if gcr_gitea_delete_runner "$repo" "$rid"; then + gcr_event "stale-runner-deleted" "${jid:-unknown}" "{\"repo\":\"$repo\",\"runner_id\":$rid,\"name\":\"$rname\"}" + else + gcr_log error --ns=sweep "failed deleting repo=$repo runner id=$rid" + fi fi - fi - done <:". @@ -40,8 +58,35 @@ gcr_gitea_job_state() { } gcr_gitea_delete_runner() { - id="$1" + repo="$1"; id="$2" token="$(gcr_gitea_admin_token)" || return 1 + owner="${repo%%/*}" + name="${repo#*/}" curl -fsS -X DELETE -H "Authorization: token $token" \ - "$GCR_GITEA_URL/api/v1/orgs/hectic-lab/actions/runners/$id" + "$GCR_GITEA_URL/api/v1/repos/$owner/$name/actions/runners/$id" +} + +gcr_gitea_set_runner_disabled() { + repo="$1"; id="$2"; disabled="$3" + case "$disabled" in true|false) ;; *) return 1 ;; esac + token="$(gcr_gitea_admin_token)" || return 1 + owner="${repo%%/*}" + name="${repo#*/}" + curl -fsS -X PATCH -H "Authorization: token $token" \ + -H 'Content-Type: application/json' --data "{\"disabled\":$disabled}" \ + "$GCR_GITEA_URL/api/v1/repos/$owner/$name/actions/runners/$id" >/dev/null +} + +# gcr_gitea_runner_disabled REPO RUNNER_NAME true|false +gcr_gitea_runner_disabled() { + repo="$1"; runner_name="$2"; disabled="$3" + runners="$(gcr_gitea_list_runners "$repo")" || return 1 + while read -r id name; do + [ "$name" = "$runner_name" ] || continue + gcr_gitea_set_runner_disabled "$repo" "$id" "$disabled" + return "$?" + done < "$key_tmp" + printf '\n' >> "$key_tmp" + chmod 0600 "$key_tmp" + ssh_opts="-i $key_tmp -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=5 -o BatchMode=yes" + if timeout 30 ssh $ssh_opts "root@$ip" "systemctl $action gitea-runner.service"; then + rm -f "$key_tmp" + return 0 + fi + rm -f "$key_tmp" + return 1 +} + gcr_vm_collect_diagnostics() { vm_id="$1"; ip="$2"; job_id="$3"; reason="$4" @@ -321,7 +342,7 @@ STARTEOF chmod 0700 /usr/local/sbin/gcr-runner-start cat > /etc/systemd/system/gitea-runner.service </dev/null || return 1 + if mkdir "$gcr_takeover_dir" 2>/dev/null; then + printf '%s %s\n' "$$" "$(gcr_now_epoch)" > "$gcr_takeover_dir/owner" + rm -rf "$gcr_takeover_old" + return 0 + fi + rm -rf "$gcr_takeover_old" + return 1 +} + gcr_lock_acquire() { - mkdir "$(printf '%s/jobs/.lock.%s' "$GCR_STATE_DIR" "$1")" 2>/dev/null + gcr_lock_key="$1" + gcr_lock_dir="$(printf '%s/jobs/.lock.%s' "$GCR_STATE_DIR" "$gcr_lock_key")" + if mkdir "$gcr_lock_dir" 2>/dev/null; then + printf '%s %s\n' "$$" "$(gcr_now_epoch)" > "$gcr_lock_dir/owner" + return 0 + fi + + gcr_lock_owner="$(cat "$gcr_lock_dir/owner" 2>/dev/null || true)" + if [ -z "$gcr_lock_owner" ]; then + # A crash between mkdir and owner write leaves no PID. Give a live + # creator a short initialization window, then unblock hard TTL work. + gcr_lock_mtime="$(stat -c %Y "$gcr_lock_dir" 2>/dev/null || true)" + gcr_lock_now="$(gcr_now_epoch)" + case "$gcr_lock_mtime:$gcr_lock_now" in + *[!0-9:]*|:*|*::*|*:) return 1 ;; + esac + [ "$((gcr_lock_now - gcr_lock_mtime))" -ge 30 ] || return 1 + gcr_lock_takeover "$gcr_lock_dir" + return "$?" + fi + set -- $gcr_lock_owner + gcr_lock_pid="${1:-}" + case "$gcr_lock_pid" in ''|*[!0-9]*) return 1 ;; esac + if kill -0 "$gcr_lock_pid" 2>/dev/null; then + return 1 + fi + + # Dead PID means a process crash, not live contention. Atomically take + # over its directory; concurrent recovery cannot erase a new lock. + gcr_lock_takeover "$gcr_lock_dir" } gcr_lock_release() { @@ -46,7 +91,151 @@ gcr_record_del() { } gcr_record_field() { - printf '%s' "$1" | jq -r --arg f "$2" '.[$f] // ""' + printf '%s' "$1" | jq -r --arg f "$2" 'if has($f) then .[$f] else "" end' +} + +gcr_now_epoch() { + date -u '+%s' +} + +# Successful VMs remain reusable until next billing-hour boundary, but never +# beyond profile hard TTL. Prints updated idle record when retention is safe. +gcr_record_idle_json() { + gcr_idle_rec="$1" + [ "$(gcr_record_field "$gcr_idle_rec" bootstrapped)" = "true" ] || return 1 + + gcr_idle_vm_id="$(gcr_record_field "$gcr_idle_rec" vm_id)" + gcr_idle_created="$(gcr_record_field "$gcr_idle_rec" created_at)" + gcr_idle_ttl="$(gcr_record_field "$gcr_idle_rec" ttl_min)" + case "$gcr_idle_vm_id:$gcr_idle_created:$gcr_idle_ttl" in + *[!0-9:]*|0:*|:*|*::*|*:) return 1 ;; + esac + + gcr_idle_now="$(gcr_now_epoch)" + gcr_idle_hard_expires="$((gcr_idle_created + gcr_idle_ttl * 60))" + [ "$gcr_idle_now" -lt "$gcr_idle_hard_expires" ] || return 1 + + gcr_idle_age="$((gcr_idle_now - gcr_idle_created))" + [ "$gcr_idle_age" -ge 0 ] || gcr_idle_age=0 + gcr_idle_slots="$((gcr_idle_age / 3600))" + [ "$((gcr_idle_age % 3600))" -eq 0 ] || gcr_idle_slots=$((gcr_idle_slots + 1)) + [ "$gcr_idle_slots" -gt 0 ] || gcr_idle_slots=1 + gcr_idle_expires="$((gcr_idle_created + gcr_idle_slots * 3600))" + [ "$gcr_idle_expires" -le "$gcr_idle_hard_expires" ] \ + || gcr_idle_expires="$gcr_idle_hard_expires" + + printf '%s' "$gcr_idle_rec" | jq -c \ + --arg now "$gcr_idle_now" --arg expires "$gcr_idle_expires" \ + '.status = "idle_vm" + | .idle_since = ($now | tonumber) + | .idle_expires_at = ($expires | tonumber)' +} + +gcr_idle_record_unexpired() { + gcr_idle_rec="$1" + [ "$(gcr_record_field "$gcr_idle_rec" status)" = "idle_vm" ] || return 1 + [ "$(gcr_record_field "$gcr_idle_rec" bootstrapped)" = "true" ] || return 1 + + gcr_idle_now="$(gcr_now_epoch)" + gcr_idle_vm_id="$(gcr_record_field "$gcr_idle_rec" vm_id)" + gcr_idle_vm_name="$(gcr_record_field "$gcr_idle_rec" vm_name)" + gcr_idle_expires="$(gcr_record_field "$gcr_idle_rec" idle_expires_at)" + gcr_idle_created="$(gcr_record_field "$gcr_idle_rec" created_at)" + gcr_idle_ttl="$(gcr_record_field "$gcr_idle_rec" ttl_min)" + case "$gcr_idle_expires:$gcr_idle_created:$gcr_idle_ttl" in + *[!0-9:]*|:*|*::*|*:) return 1 ;; + esac + case "$gcr_idle_vm_id" in ''|0|*[!0-9]*) return 1 ;; esac + [ -n "$gcr_idle_vm_name" ] || return 1 + gcr_idle_hard_expires="$((gcr_idle_created + gcr_idle_ttl * 60))" + [ "$gcr_idle_now" -lt "$gcr_idle_expires" ] \ + && [ "$gcr_idle_now" -lt "$gcr_idle_hard_expires" ] +} + +gcr_idle_record_usable() { + gcr_idle_rec="$1" + gcr_idle_record_unexpired "$gcr_idle_rec" || return 1 + gcr_idle_min_remaining="${GCR_RECONCILE_INTERVAL_SEC:-60}" + case "$gcr_idle_min_remaining" in ''|*[!0-9]*) return 1 ;; esac + [ "$((gcr_idle_expires - gcr_idle_now))" -ge "$gcr_idle_min_remaining" ] \ + && [ "$((gcr_idle_hard_expires - gcr_idle_now))" -ge "$gcr_idle_min_remaining" ] +} + +# Caller must hold destination allocation lock. Global pool lock ensures one +# queued job claims an idle VM; destination write precedes source deletion so +# orphan/stale sweeps always see an owner during transfer. +gcr_claim_idle() { + gcr_claim_job="$1"; gcr_claim_attempt="$2" + gcr_claim_repo="$3"; gcr_claim_label="$4" + # Exit 2 means pool is busy; callers must defer instead of charging for a + # new VM without knowing whether matching paid capacity is available. + gcr_lock_acquire idle-pool || return 2 + + for gcr_claim_file in $(gcr_active_records); do + gcr_claim_rec="$(cat "$gcr_claim_file")" + [ "$(gcr_record_field "$gcr_claim_rec" status)" = "idle_vm" ] || continue + [ "$(gcr_record_field "$gcr_claim_rec" repo)" = "$gcr_claim_repo" ] || continue + [ "$(gcr_record_field "$gcr_claim_rec" label)" = "$gcr_claim_label" ] || continue + gcr_idle_record_usable "$gcr_claim_rec" || continue + + gcr_claim_old_job="$(gcr_record_field "$gcr_claim_rec" job_id)" + gcr_claim_old_attempt="$(gcr_record_field "$gcr_claim_rec" run_attempt)" + gcr_claim_vm_id="$(gcr_record_field "$gcr_claim_rec" vm_id)" + gcr_record_vm_owned_elsewhere "$gcr_claim_vm_id" \ + "$gcr_claim_old_job" "$gcr_claim_old_attempt" && continue + gcr_claim_now="$(gcr_now_epoch)" + gcr_claim_new="$(printf '%s' "$gcr_claim_rec" | jq -c \ + --arg job "$gcr_claim_job" --arg attempt "$gcr_claim_attempt" \ + --arg repo "$gcr_claim_repo" --arg label "$gcr_claim_label" \ + --arg now "$gcr_claim_now" \ + '.job_id = $job | .run_attempt = $attempt + | .repo = $repo | .label = $label | .status = "pending_vm" + | .bootstrapped = false + | .reused_vm = true + | .assigned_at = ($now | tonumber) + | del(.idle_since, .idle_expires_at)')" + if ! gcr_record_put "$gcr_claim_job" "$gcr_claim_attempt" "$gcr_claim_new"; then + gcr_lock_release idle-pool + return 1 + fi + gcr_record_del "$gcr_claim_old_job" "$gcr_claim_old_attempt" + gcr_lock_release idle-pool + return 0 + done + + gcr_lock_release idle-pool + return 1 +} + +gcr_record_exists_for_vm_id() { + gcr_lookup="$1" + for gcr_lookup_file in $(gcr_active_records); do + [ "$(gcr_record_field "$(cat "$gcr_lookup_file")" vm_id)" = "$gcr_lookup" ] \ + && return 0 + done + return 1 +} + +gcr_record_exists_for_vm_name() { + gcr_lookup="$1" + for gcr_lookup_file in $(gcr_active_records); do + [ "$(gcr_record_field "$(cat "$gcr_lookup_file")" vm_name)" = "$gcr_lookup" ] \ + && return 0 + done + return 1 +} + +gcr_record_vm_owned_elsewhere() { + gcr_lookup_vm="$1"; gcr_lookup_job="$2"; gcr_lookup_attempt="$3" + for gcr_lookup_file in $(gcr_active_records); do + gcr_lookup_rec="$(cat "$gcr_lookup_file")" + [ "$(gcr_record_field "$gcr_lookup_rec" vm_id)" = "$gcr_lookup_vm" ] || continue + if [ "$(gcr_record_field "$gcr_lookup_rec" job_id)" != "$gcr_lookup_job" ] \ + || [ "$(gcr_record_field "$gcr_lookup_rec" run_attempt)" != "$gcr_lookup_attempt" ]; then + return 0 + fi + done + return 1 } gcr_event() { @@ -55,8 +244,8 @@ gcr_event() { "$(printf '%s' "$3" | jq -Rs .)" >> "$GCR_STATE_DIR/events.jsonl" } -# Exit-code contract: 0 = recorded under budget, 1 = would exceed cap. -gcr_budget_add() { +# Exit-code contract: 0 = remains under budget, 1 = would exceed cap. +gcr_budget_can_add() { rate="$1"; ttl_min="$2" month="$(date -u '+%Y-%m')" file="$GCR_STATE_DIR/budget/$month" @@ -65,10 +254,20 @@ gcr_budget_add() { if awk -v p="$projected" -v b="${GCR_BUDGET_EUR_MONTHLY:-15}" 'BEGIN {exit !(p > b)}'; then return 1 fi + return 0 +} + +# Caller holds admission lock and has already checked gcr_budget_can_add. +gcr_budget_add() { + rate="$1"; ttl_min="$2" + month="$(date -u '+%Y-%m')" + file="$GCR_STATE_DIR/budget/$month" + current="$(cat "$file" 2>/dev/null || echo 0)" + projected="$(awk -v c="$current" -v r="$rate" -v t="$ttl_min" 'BEGIN {printf "%.4f", c + r * t / 60}')" printf '%s\n' "$projected" > "$file" return 0 } gcr_active_records() { - grep -El '"status"[[:space:]]*:[[:space:]]*"(pending_vm|vm_active|deferred)"' "$GCR_STATE_DIR"/jobs/*.json 2>/dev/null || true + grep -El '"status"[[:space:]]*:[[:space:]]*"(pending_vm|vm_active|idle_vm|deferred)"' "$GCR_STATE_DIR"/jobs/*.json 2>/dev/null || true } diff --git a/package/gitea-runner-controller/webhook.sh b/package/gitea-runner-controller/webhook.sh index 0d862a2c..8152a03c 100644 --- a/package/gitea-runner-controller/webhook.sh +++ b/package/gitea-runner-controller/webhook.sh @@ -94,6 +94,12 @@ gcr_alloc() { RESPONSE_CODE=204 return 0 fi + existing="$(gcr_record_get "$job_id" "$attempt")" + if [ -n "$existing" ]; then + gcr_lock_release "$key" + RESPONSE_CODE=204 + return 0 + fi if [ "$label_count" -ne 1 ]; then gcr_lock_release "$key" @@ -112,24 +118,74 @@ gcr_alloc() { set -- $profile server_type="$1"; ttl_min="$2"; rate="$3" + if ! gcr_lock_acquire admission; then + rec="$(jq -n --arg j "$job_id" --arg a "$attempt" --arg r "$repo" \ + --arg l "$label" --arg t "$(gcr_now_epoch)" \ + '{job_id:$j, run_attempt:$a, repo:$r, label:$l, + created_at:$t, ttl_min:null, vm_id:"", vm_name:"", + status:"deferred"}')" + gcr_record_put "$job_id" "$attempt" "$rec" + gcr_lock_release "$key" + gcr_event "deferred" "$job_id" "{\"reason\":\"admission-busy\"}" + RESPONSE_CODE=202; RESPONSE_BODY="deferred: admission busy" + return 0 + fi + active="$(gcr_count_active)" repo_active="$(gcr_count_active_repo "$repo")" if [ "$active" -ge "${GCR_CONCURRENCY_CAP:-2}" ] \ || [ "$repo_active" -ge "${GCR_PER_REPO_CAP:-1}" ]; then rec="$(jq -n --arg j "$job_id" --arg a "$attempt" --arg r "$repo" \ - --arg l "$label" --arg t "$(date -u '+%s')" \ + --arg l "$label" --arg t "$(gcr_now_epoch)" \ '{job_id:$j, run_attempt:$a, repo:$r, label:$l, created_at:$t, ttl_min:null, vm_id:"", vm_name:"", status:"deferred"}')" gcr_record_put "$job_id" "$attempt" "$rec" + gcr_lock_release admission gcr_lock_release "$key" gcr_event "deferred" "$job_id" "{\"active\":$active,\"repo_active\":$repo_active}" RESPONSE_CODE=202; RESPONSE_BODY="deferred: capacity" return 0 fi - if ! gcr_budget_add "$rate" "$ttl_min"; then + claim_status=0 + gcr_claim_idle "$job_id" "$attempt" "$repo" "$label" || claim_status="$?" + if [ "$claim_status" -eq 0 ]; then + reused="$(gcr_record_get "$job_id" "$attempt")" + vm_id="$(gcr_record_field "$reused" vm_id)" + vm_name="$(gcr_record_field "$reused" vm_name)" + if gcr_vm_runner_service "$vm_id" start \ + && gcr_gitea_runner_disabled "$repo" "$vm_name" false; then + reused="$(gcr_record_get "$job_id" "$attempt")" + reused="$(printf '%s' "$reused" | jq -c '.bootstrapped = true | del(.reused_vm)')" + gcr_record_put "$job_id" "$attempt" "$reused" + else + gcr_gitea_runner_disabled "$repo" "$vm_name" true || true + gcr_event "vm-reuse-start-failed" "$job_id" "{\"vm_id\":$vm_id}" + fi + gcr_lock_release admission + gcr_lock_release "$key" + gcr_event "vm-reused" "$job_id" "{\"vm_id\":$vm_id,\"label\":\"$label\"}" + RESPONSE_CODE=202; RESPONSE_BODY="reused $vm_name" + return 0 + fi + if [ "$claim_status" -eq 2 ]; then + rec="$(jq -n --arg j "$job_id" --arg a "$attempt" --arg r "$repo" \ + --arg l "$label" --arg t "$(gcr_now_epoch)" \ + '{job_id:$j, run_attempt:$a, repo:$r, label:$l, + created_at:$t, ttl_min:null, vm_id:"", vm_name:"", + status:"deferred"}')" + gcr_record_put "$job_id" "$attempt" "$rec" + gcr_lock_release admission + gcr_lock_release "$key" + gcr_event "deferred" "$job_id" "{\"reason\":\"idle-pool-busy\"}" + RESPONSE_CODE=202; RESPONSE_BODY="deferred: idle pool busy" + return 0 + fi + + if ! gcr_budget_can_add "$rate" "$ttl_min"; then gcr_record_del "$job_id" "$attempt" + gcr_lock_release admission gcr_lock_release "$key" gcr_event "budget-refused" "$job_id" "{\"rate\":$rate,\"ttl_min\":$ttl_min}" RESPONSE_CODE=202; RESPONSE_BODY="refused: monthly budget exhausted" @@ -138,6 +194,7 @@ gcr_alloc() { reg_token="$(gcr_gitea_registration_token "$repo")" || { gcr_record_del "$job_id" "$attempt" + gcr_lock_release admission gcr_lock_release "$key" gcr_event "token-error" "$job_id" "{}" RESPONSE_CODE=202; RESPONSE_BODY="registration token unavailable" @@ -145,22 +202,33 @@ gcr_alloc() { } vm_name="gcr-${job_id}-${attempt}" + created_at="$(gcr_now_epoch)" vm_id="$(gcr_vm_create "$vm_name" "$label" "$server_type" "$ttl_min" \ "$reg_token" "$job_id" "$attempt" "$repo")" || { gcr_record_del "$job_id" "$attempt" + gcr_lock_release admission gcr_lock_release "$key" gcr_event "vm-create-failed" "$job_id" "{}" RESPONSE_CODE=202; RESPONSE_BODY="VM creation failed" return 0 } + gcr_budget_add "$rate" "$ttl_min" + rec="$(jq -n --arg j "$job_id" --arg a "$attempt" --arg r "$repo" \ - --arg l "$label" --arg t "$(date -u '+%s')" --arg v "$vm_id" \ + --arg l "$label" --arg t "$created_at" --arg v "$vm_id" \ --arg vn "$vm_name" --arg ttl "$ttl_min" \ '{job_id:$j, run_attempt:$a, repo:$r, label:$l, created_at:$t, ttl_min:($ttl|tonumber), vm_id:($v|tonumber), vm_name:$vn, bootstrapped:false, status:"pending_vm"}')" - gcr_record_put "$job_id" "$attempt" "$rec" + if ! gcr_record_put "$job_id" "$attempt" "$rec"; then + gcr_vm_destroy "$vm_id" || true + gcr_lock_release admission + gcr_lock_release "$key" + RESPONSE_CODE=202; RESPONSE_BODY="VM state write failed" + return 0 + fi + gcr_lock_release admission gcr_lock_release "$key" gcr_event "vm-created" "$job_id" "{\"vm_id\":$vm_id,\"label\":\"$label\",\"ttl_min\":$ttl_min}" @@ -170,10 +238,48 @@ gcr_alloc() { gcr_deallocate() { job_id="$1"; attempt="$2"; new_status="$3" + key="$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_acquire "$key" || return 0 + rec="$(gcr_record_get "$job_id" "$attempt")" - [ -n "$rec" ] || return 0 + if [ -z "$rec" ]; then + gcr_lock_release "$key" + return 0 + fi + + case "$(gcr_record_field "$rec" status)" in + pending_vm|vm_active) ;; + *) + gcr_lock_release "$key" + return 0 + ;; + esac vm_id="$(gcr_record_field "$rec" vm_id)" + if [ "$new_status" = "completed:success" ] \ + && idle_rec="$(gcr_record_idle_json "$rec")"; then + if ! gcr_lock_acquire idle-pool; then + gcr_lock_release "$key" + return 0 + fi + runner_name="$(gcr_record_field "$rec" vm_name)" + if ! gcr_gitea_runner_disabled "$(gcr_record_field "$rec" repo)" "$runner_name" true \ + || ! gcr_vm_runner_service "$vm_id" stop; then + gcr_lock_release idle-pool + gcr_vm_destroy "$vm_id" || true + gcr_record_del "$job_id" "$attempt" + gcr_lock_release "$key" + gcr_event "vm-destroyed" "$job_id" "{\"vm_id\":$vm_id,\"reason\":\"idle-stop-failed\"}" + return 0 + fi + gcr_record_put "$job_id" "$attempt" "$idle_rec" + idle_expires="$(gcr_record_field "$idle_rec" idle_expires_at)" + gcr_lock_release idle-pool + gcr_lock_release "$key" + gcr_event "vm-idle" "$job_id" "{\"vm_id\":$vm_id,\"expires_at\":$idle_expires}" + return 0 + fi + if [ -n "$vm_id" ] && [ "$vm_id" != "null" ] && [ "$vm_id" != "0" ]; then case "$new_status" in completed:success|completed:cancelled|completed:skipped) ;; @@ -187,7 +293,21 @@ gcr_deallocate() { fi gcr_record_del "$job_id" "$attempt" - gcr_lock_release "$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_release "$key" +} + +gcr_mark_in_progress() { + job_id="$1"; attempt="$2" + key="$(gcr_alloc_key "$job_id" "$attempt")" + gcr_lock_acquire "$key" || return 0 + rec="$(gcr_record_get "$job_id" "$attempt")" + case "$(gcr_record_field "$rec" status)" in + pending_vm|vm_active) + rec="$(printf '%s' "$rec" | jq -c '.status = "vm_active"')" + gcr_record_put "$job_id" "$attempt" "$rec" + ;; + esac + gcr_lock_release "$key" } gcr_handle_webhook() { @@ -210,8 +330,11 @@ gcr_handle_webhook() { repo="$(printf '%s' "$gcr_body" | jq -r '.repository.full_name // ""')" labels_json="$(printf '%s' "$gcr_body" | jq -c '.workflow_job.labels // []')" - case "$action:$job_id" in - :*|"queued:"|*":0") gcr_respond 400 "malformed payload"; exit 0 ;; + [ -n "$action" ] || { gcr_respond 400 "malformed payload"; exit 0; } + case "$job_id:$attempt" in + *[!0-9:]*|:*|*::*|*:|0:*|*:0) + gcr_respond 400 "malformed payload"; exit 0 + ;; esac case "$action" in @@ -220,11 +343,7 @@ gcr_handle_webhook() { gcr_log info --ns=alloc "queued job=$job_id repo=$repo code=$RESPONSE_CODE $RESPONSE_BODY" ;; in_progress) - rec="$(gcr_record_get "$job_id" "$attempt")" - if [ -n "$rec" ]; then - rec="$(printf '%s' "$rec" | jq -c '.status = "vm_active"')" - gcr_record_put "$job_id" "$attempt" "$rec" - fi + gcr_mark_in_progress "$job_id" "$attempt" RESPONSE_CODE=204 ;; completed) diff --git a/test/package/gitea-runner-controller/test/highmem-label.sh b/test/package/gitea-runner-controller/test/highmem-label.sh index 6e99df39..e10e9a60 100644 --- a/test/package/gitea-runner-controller/test/highmem-label.sh +++ b/test/package/gitea-runner-controller/test/highmem-label.sh @@ -40,7 +40,7 @@ if printf '%s\n' "$candidates" | grep -Evq '^ccx53 (nbg1|fsn1|hel1) amd64$'; the fi export GCR_BUDGET_EUR_MONTHLY='6.83' -if gcr_budget_add 0.8550 480; then +if gcr_budget_can_add 0.8550 480; then printf 'highmem full-TTL reservation must obey budget cap\n' >&2 exit 1 fi diff --git a/test/package/gitea-runner-controller/test/idle-reuse.sh b/test/package/gitea-runner-controller/test/idle-reuse.sh new file mode 100644 index 00000000..efd53f07 --- /dev/null +++ b/test/package/gitea-runner-controller/test/idle-reuse.sh @@ -0,0 +1,306 @@ +#!/bin/dash +set -eu + +. "$LOG_SH" +. "$STATE_SH" +. "$DECIDE_SH" +. "$HCLOUD_SH" +. "$GITEA_SH" +. "$CONTROLLER_SH" +. "$WEBHOOK_SH" + +gcr_state_init +export GCR_ALLOWED_REPOS='hinterland/hearth' +export GCR_CONCURRENCY_CAP=2 +export GCR_PER_REPO_CAP=1 +NOW=2800 +gcr_now_epoch() { printf '%s' "$NOW"; } + +calls="$GCR_STATE_DIR/calls" +gcr_budget_add() { printf 'budget\n' >> "$calls"; } +gcr_gitea_registration_token() { printf 'token'; printf 'token\n' >> "$calls"; } +gcr_vm_create() { printf 'create\n' >> "$calls"; printf '99'; } +gcr_vm_destroy() { printf 'destroy vm=%s\n' "$1" >> "$calls"; } +gcr_vm_runner_service() { printf 'runner %s vm=%s\n' "$2" "$1" >> "$calls"; } +gcr_gitea_runner_disabled() { printf 'runner-disabled %s %s\n' "$2" "$3" >> "$calls"; } + +original='{"job_id":"301","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":1000,"ttl_min":480,"vm_id":71,"vm_name":"gcr-301-1","bootstrapped":true,"status":"vm_active"}' +gcr_record_put 301 1 "$original" +gcr_deallocate 301 1 completed:success +jq -e 'select(.status == "idle_vm" and .idle_expires_at == 4600)' \ + "$(gcr_record_path 301 1)" >/dev/null +idle_once="$(gcr_record_get 301 1)" +NOW=3000 +gcr_deallocate 301 1 completed:success +test "$(gcr_record_get 301 1)" = "$idle_once" +NOW=2800 + +# Exact elapsed hours keep current boundary instead of extending another hour. +exact='{"job_id":"300","run_attempt":"1","repo":"hinterland/hearth","label":"gross-x86","created_at":1000,"ttl_min":180,"vm_id":70,"vm_name":"gcr-300-1","bootstrapped":true,"status":"vm_active"}' +gcr_record_put 300 1 "$exact" +NOW=4600 +gcr_deallocate 300 1 completed:success +jq -e 'select(.status == "idle_vm" and .idle_expires_at == 4600)' \ + "$(gcr_record_path 300 1)" >/dev/null +gcr_record_del 300 1 +NOW=2800 +gcr_deallocate 301 1 completed:success +jq -e 'select(.status == "idle_vm" and .idle_expires_at == 4600)' \ + "$(gcr_record_path 301 1)" >/dev/null + +# Exact billing boundary must expire now, not roll into another paid hour. +NOW=4600 +boundary='{"job_id":"302","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":1000,"ttl_min":480,"vm_id":74,"vm_name":"gcr-302-1","bootstrapped":true,"status":"vm_active"}' +test "$(gcr_record_idle_json "$boundary" | jq -r '.idle_expires_at')" = 4600 +NOW=2800 + +gcr_alloc 302 1 hinterland/hearth '["gross-nix-x86-perf"]' +test "$RESPONSE_BODY" = 'reused gcr-301-1' +test ! -e "$(gcr_record_path 301 1)" +jq -e 'select(.job_id == "302" and .vm_id == 71 and + .vm_name == "gcr-301-1" and .status == "pending_vm" and + .created_at == 1000 and .assigned_at == 2800)' \ + "$(gcr_record_path 302 1)" >/dev/null +test "$(grep -Ec '^(budget|token|create)$' "$calls" || true)" = 0 + +# Start failure keeps Gitea runner disabled and record retryable. +retry_idle='{"job_id":"315","run_attempt":"1","repo":"hinterland/hearth","label":"gross-arm","created_at":1000,"ttl_min":180,"vm_id":79,"vm_name":"gcr-315-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":4600}' +gcr_record_put 315 1 "$retry_idle" +export GCR_PER_REPO_CAP=2 +FAIL_START=1 +gcr_vm_runner_service() { + printf 'runner %s vm=%s\n' "$2" "$1" >> "$calls" + [ "$2" = start ] && [ "$FAIL_START" = 1 ] && return 1 +} +gcr_alloc 316 1 hinterland/hearth '["gross-arm"]' +retry_rec="$(gcr_record_get 316 1)" +test "$(gcr_record_field "$retry_rec" bootstrapped)" = false +test "$(gcr_record_field "$retry_rec" reused_vm)" = true +grep -q '^runner-disabled gcr-315-1 true$' "$calls" +FAIL_START=0 +gcr_record_del 316 1 + +# Expired idle capacity is never claimed; normal allocation then charges once. +expired='{"job_id":"303","run_attempt":"1","repo":"hinterland/hearth","label":"gross-x86","created_at":1000,"ttl_min":180,"vm_id":72,"vm_name":"gcr-303-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":2800}' +gcr_record_put 303 1 "$expired" +NOW=2800 +export GCR_PER_REPO_CAP=2 +gcr_alloc 304 1 hinterland/hearth '["gross-x86"]' +test "$RESPONSE_BODY" = 'allocated gcr-304-1' +jq -e 'select(.vm_id == 99 and .status == "pending_vm")' \ + "$(gcr_record_path 304 1)" >/dev/null +test "$(grep -c '^budget$' "$calls")" = 1 +test "$(grep -c '^token$' "$calls")" = 1 +test "$(grep -c '^create$' "$calls")" = 1 + +gcr_sweep_ttl +grep -q 'destroy vm=72' "$calls" +test ! -e "$(gcr_record_path 303 1)" + +# Busy pool lock defers instead of racing into paid allocation. +export GCR_CONCURRENCY_CAP=3 +export GCR_PER_REPO_CAP=3 +gcr_lock_acquire idle-pool +gcr_alloc 305 1 hinterland/hearth '["gross-arm"]' +test "$RESPONSE_BODY" = 'deferred: idle pool busy' +test "$(gcr_record_field "$(gcr_record_get 305 1)" status)" = deferred +test "$(grep -c '^create$' "$calls")" = 1 +gcr_lock_release idle-pool + +# Claim needs at least one reconcile interval before slot and hard expiry. +export GCR_RECONCILE_INTERVAL_SEC=60 +near_expiry='{"job_id":"313","run_attempt":"1","repo":"hinterland/hearth","label":"gross-mixed-econ","created_at":1000,"ttl_min":180,"vm_id":78,"vm_name":"gcr-313-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":2860}' +gcr_record_put 313 1 "$near_expiry" +NOW=2801 +gcr_lock_acquire "$(gcr_alloc_key 314 1)" +if gcr_claim_idle 314 1 hinterland/hearth gross-mixed-econ; then + printf 'near-expiry VM must not be reused\n' >&2 + exit 1 +fi +gcr_lock_release "$(gcr_alloc_key 314 1)" +test -e "$(gcr_record_path 313 1)" +test ! -e "$(gcr_record_path 314 1)" +gcr_record_del 313 1 +NOW=2800 + +# Two concurrent claims transfer one VM once; original labels/name stay safe. +race_idle='{"job_id":"306","run_attempt":"1","repo":"hinterland/hearth","label":"gross-arm","created_at":1000,"ttl_min":180,"vm_id":73,"vm_name":"gcr-306-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":4600}' +gcr_record_put 306 1 "$race_idle" +claim_script="$GCR_STATE_DIR/claim.sh" +cat > "$claim_script" <<'EOF' +#!/bin/dash +set -eu +. "$STATE_SH" +gcr_now_epoch() { printf '2800'; } +key="$(gcr_alloc_key "$1" 1)" +gcr_lock_acquire "$key" +if gcr_claim_idle "$1" 1 hinterland/hearth gross-arm; then + printf 'reused\n' > "$GCR_STATE_DIR/result-$1" +else + printf 'missed\n' > "$GCR_STATE_DIR/result-$1" +fi +gcr_lock_release "$key" +EOF +dash "$claim_script" 307 & first=$! +dash "$claim_script" 308 & second=$! +wait "$first" +wait "$second" +reused_count=0 +for result in "$GCR_STATE_DIR"/result-*; do + [ "$(cat "$result")" = reused ] && reused_count=$((reused_count + 1)) +done +test "$reused_count" = 1 +test ! -e "$(gcr_record_path 306 1)" + +owner="" +for job in 307 308; do + rec="$(gcr_record_get "$job" 1)" + if [ -n "$rec" ]; then + test "$(gcr_record_field "$rec" vm_id)" = 73 + test "$(gcr_record_field "$rec" vm_name)" = gcr-306-1 + owner="$job" + fi +done +test -n "$owner" + +gcr_vm_list_managed() { + printf 'list\n' >> "$GCR_STATE_DIR/list-calls" + printf '[{"id":73,"labels":{"gcr.job-id":"306","gcr.run-attempt":"1"}}]' +} +gcr_gitea_list_runners() { printf '17 gcr-306-1\n'; } +gcr_gitea_delete_runner() { printf 'delete repo=%s runner=%s\n' "$1" "$2" >> "$calls"; } +before_destroy="$(grep -c '^destroy vm=73$' "$calls" || true)" +gcr_lock_acquire admission +gcr_sweep_orphan_vms +test ! -e "$GCR_STATE_DIR/list-calls" +gcr_lock_release admission +gcr_sweep_orphan_vms +test "$(wc -l < "$GCR_STATE_DIR/list-calls" | tr -d ' ')" = 1 +gcr_sweep_stale_runners +after_destroy="$(grep -c '^destroy vm=73$' "$calls" || true)" +test "$before_destroy" = "$after_destroy" +if grep -q '^delete repo=.* runner=17$' "$calls"; then + printf 'reused VM runner registration must not be swept as stale\n' >&2 + exit 1 +fi + +# Interrupted destination-first transfer leaves duplicate state, never a VM +# deletion: sweep drops superseded idle source and keeps active destination. +duplicate_idle='{"job_id":"309","run_attempt":"1","repo":"hinterland/hearth","label":"gross-arm","created_at":1000,"ttl_min":180,"vm_id":75,"vm_name":"gcr-309-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":2800}' +duplicate_active='{"job_id":"310","run_attempt":"1","repo":"hinterland/hearth","label":"gross-arm","created_at":1000,"ttl_min":180,"vm_id":75,"vm_name":"gcr-309-1","bootstrapped":true,"status":"pending_vm"}' +gcr_record_put 309 1 "$duplicate_idle" +gcr_record_put 310 1 "$duplicate_active" +gcr_sweep_ttl +test ! -e "$(gcr_record_path 309 1)" +test -e "$(gcr_record_path 310 1)" +if grep -q '^destroy vm=75$' "$calls"; then + printf 'superseded idle record must not destroy reassigned VM\n' >&2 + exit 1 +fi + +# Delayed old-job in_progress delivery cannot reactivate idle ownership. +late_idle='{"job_id":"311","run_attempt":"1","repo":"hinterland/hearth","label":"gross-arm","created_at":1000,"ttl_min":180,"vm_id":76,"vm_name":"gcr-311-1","bootstrapped":true,"status":"idle_vm","idle_since":2000,"idle_expires_at":4600}' +gcr_record_put 311 1 "$late_idle" +gcr_mark_in_progress 311 1 +test "$(gcr_record_field "$(gcr_record_get 311 1)" status)" = idle_vm + +# Crash-stranded locks are reclaimed by dead owner PID; live contention remains. +mkdir "$GCR_STATE_DIR/jobs/.lock.stale-test" +if gcr_lock_acquire stale-test; then + printf 'pre-existing lock must not be reclaimed\n' >&2 + exit 1 +fi +rm -rf "$GCR_STATE_DIR/jobs/.lock.stale-test" +mkdir "$GCR_STATE_DIR/jobs/.lock.crashed-test" +printf '999999 1\n' > "$GCR_STATE_DIR/jobs/.lock.crashed-test/owner" +gcr_lock_acquire crashed-test +gcr_lock_release crashed-test + +# Concurrent recovery of one dead lock admits exactly one owner. +mkdir "$GCR_STATE_DIR/jobs/.lock.crashed-race" +printf '999999 1\n' > "$GCR_STATE_DIR/jobs/.lock.crashed-race/owner" +reclaim_script="$GCR_STATE_DIR/reclaim.sh" +cat > "$reclaim_script" <<'EOF' +#!/bin/dash +set -eu +. "$STATE_SH" +if gcr_lock_acquire crashed-race; then + printf 'acquired\n' > "$GCR_STATE_DIR/reclaim-$1" + sleep 1 + gcr_lock_release crashed-race +else + printf 'busy\n' > "$GCR_STATE_DIR/reclaim-$1" +fi +EOF +dash "$reclaim_script" first & first=$! +dash "$reclaim_script" second & second=$! +wait "$first" +wait "$second" +test "$(grep -lc '^acquired$' "$GCR_STATE_DIR"/reclaim-* | wc -l | tr -d ' ')" = 1 +mkdir "$GCR_STATE_DIR/jobs/.lock.crashed-test" +printf '999999 1\n' > "$GCR_STATE_DIR/jobs/.lock.crashed-test/owner" +gcr_lock_acquire crashed-test +gcr_lock_release crashed-test + +for job in 307 308 310 311; do + gcr_record_del "$job" 1 +done +ttl_record='{"job_id":"312","run_attempt":"1","repo":"hinterland/hearth","label":"gross-x86","created_at":1000,"ttl_min":180,"vm_id":77,"vm_name":"gcr-312-1","bootstrapped":true,"status":"vm_active"}' +gcr_record_put 312 1 "$ttl_record" +gcr_vm_public_ip() { printf '192.0.2.%s' "$1"; } +gcr_vm_collect_diagnostics() { + printf 'diag vm=%s job=%s reason=%s\n' "$1" "$3" "$4" >> "$calls" +} +NOW=11799 +gcr_sweep_ttl +test -e "$(gcr_record_path 312 1)" +NOW=11800 +gcr_sweep_ttl +test ! -e "$(gcr_record_path 312 1)" +grep -q '^diag vm=77 job=312 reason=ttl$' "$calls" +grep -q '^destroy vm=77$' "$calls" + +# Admission lock serializes cap check and creation across webhook processes. +ADMISSION_STATE="$GCR_STATE_DIR/admission-state" +mkdir "$ADMISSION_STATE" +admission_script="$ADMISSION_STATE/allocate.sh" +cat > "$admission_script" <<'EOF' +#!/bin/dash +set -eu +. "$LOG_SH" +. "$STATE_SH" +. "$DECIDE_SH" +. "$WEBHOOK_SH" +gcr_now_epoch() { printf '2000'; } +gcr_budget_add() { printf 'budget %s\n' "$1" >> "$GCR_STATE_DIR/admission-calls"; } +gcr_gitea_registration_token() { printf token; } +gcr_vm_create() { + sleep 1 + printf 'create %s\n' "$6" >> "$GCR_STATE_DIR/admission-calls" + printf '%s' "$6" +} +gcr_vm_destroy() { :; } +gcr_state_init +gcr_alloc "$1" 1 hinterland/hearth '["gross-x86"]' +EOF +old_state="$GCR_STATE_DIR" +GCR_STATE_DIR="$ADMISSION_STATE" \ +GCR_ALLOWED_REPOS=hinterland/hearth \ +GCR_CONCURRENCY_CAP=1 \ +GCR_PER_REPO_CAP=1 \ +dash "$admission_script" 401 & first=$! +GCR_STATE_DIR="$ADMISSION_STATE" \ +GCR_ALLOWED_REPOS=hinterland/hearth \ +GCR_CONCURRENCY_CAP=1 \ +GCR_PER_REPO_CAP=1 \ +dash "$admission_script" 402 & second=$! +wait "$first" +wait "$second" +GCR_STATE_DIR="$ADMISSION_STATE" +test "$(grep -c '^create ' "$GCR_STATE_DIR/admission-calls")" = 1 +active_count="$(gcr_count_active)" +test "$active_count" = 1 +deferred_count="$(grep -El '"status"[[:space:]]*:[[:space:]]*"deferred"' \ + "$GCR_STATE_DIR"/jobs/*.json | wc -l | tr -d ' ')" +test "$deferred_count" = 1 +GCR_STATE_DIR="$old_state" diff --git a/test/package/gitea-runner-controller/test/reap-diagnostics.sh b/test/package/gitea-runner-controller/test/reap-diagnostics.sh index ef546ef6..3f3a753a 100644 --- a/test/package/gitea-runner-controller/test/reap-diagnostics.sh +++ b/test/package/gitea-runner-controller/test/reap-diagnostics.sh @@ -10,6 +10,7 @@ set -eu gcr_state_init calls="$GCR_STATE_DIR/calls" +gcr_now_epoch() { printf '1800'; } gcr_gitea_job_state() { case "$2" in @@ -31,14 +32,19 @@ gcr_vm_destroy() { printf 'destroy vm=%s\n' "$1" >> "$calls" } -record_success='{"job_id":"101","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"1","ttl_min":480,"vm_id":41,"vm_name":"gcr-101-1","bootstrapped":true,"status":"vm_active"}' +gcr_vm_runner_service() { + printf 'runner %s vm=%s\n' "$2" "$1" >> "$calls" +} + +gcr_gitea_runner_disabled() { :; } + +record_success='{"job_id":"101","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"0","ttl_min":480,"vm_id":41,"vm_name":"gcr-101-1","bootstrapped":true,"status":"vm_active"}' record_failure='{"job_id":"102","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"1","ttl_min":480,"vm_id":42,"vm_name":"gcr-102-1","bootstrapped":true,"status":"vm_active"}' gcr_record_put 101 1 "$record_success" gcr_record_put 102 1 "$record_failure" gcr_reap_finished_jobs -grep -q 'destroy vm=41' "$calls" grep -q 'destroy vm=42' "$calls" grep -q 'diag vm=42 ip=192.0.2.42 job=102 reason=completed:failure' "$calls" if grep -q 'diag vm=41' "$calls"; then @@ -46,7 +52,15 @@ if grep -q 'diag vm=41' "$calls"; then exit 1 fi -test ! -e "$(gcr_record_path 101 1)" +if grep -q 'destroy vm=41' "$calls"; then + printf 'successful job VM should remain idle until billing boundary\n' >&2 + exit 1 +fi +jq -e 'select(.status == "idle_vm" and .idle_expires_at == 3600)' \ + "$(gcr_record_path 101 1)" >/dev/null +idle_once="$(gcr_record_get 101 1)" +gcr_reap_finished_jobs +test "$(gcr_record_get 101 1)" = "$idle_once" test ! -e "$(gcr_record_path 102 1)" calls_ip_fail="$GCR_STATE_DIR/calls-ip-fail" diff --git a/test/package/gitea-runner-controller/test/webhook-diagnostics.sh b/test/package/gitea-runner-controller/test/webhook-diagnostics.sh index de06c7d5..d8407eb9 100644 --- a/test/package/gitea-runner-controller/test/webhook-diagnostics.sh +++ b/test/package/gitea-runner-controller/test/webhook-diagnostics.sh @@ -10,6 +10,7 @@ set -eu gcr_state_init calls="$GCR_STATE_DIR/calls" +gcr_now_epoch() { printf '1800'; } gcr_vm_public_ip() { printf '192.0.2.%s' "$1" @@ -23,7 +24,13 @@ gcr_vm_destroy() { printf 'destroy vm=%s\n' "$1" >> "$calls" } -record_success='{"job_id":"201","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"1","ttl_min":480,"vm_id":51,"vm_name":"gcr-201-1","bootstrapped":true,"status":"vm_active"}' +gcr_vm_runner_service() { + printf 'runner %s vm=%s\n' "$2" "$1" >> "$calls" +} + +gcr_gitea_runner_disabled() { :; } + +record_success='{"job_id":"201","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"0","ttl_min":480,"vm_id":51,"vm_name":"gcr-201-1","bootstrapped":true,"status":"vm_active"}' record_failure='{"job_id":"202","run_attempt":"1","repo":"hinterland/hearth","label":"gross-nix-x86-perf","created_at":"1","ttl_min":480,"vm_id":52,"vm_name":"gcr-202-1","bootstrapped":true,"status":"vm_active"}' gcr_record_put 201 1 "$record_success" @@ -31,7 +38,6 @@ gcr_record_put 202 1 "$record_failure" gcr_deallocate 201 1 completed:success gcr_deallocate 202 1 completed:failure -grep -q 'destroy vm=51' "$calls" grep -q 'destroy vm=52' "$calls" grep -q 'diag vm=52 ip=192.0.2.52 job=202 reason=completed:failure' "$calls" if grep -q 'diag vm=51' "$calls"; then @@ -39,7 +45,15 @@ if grep -q 'diag vm=51' "$calls"; then exit 1 fi -test ! -e "$(gcr_record_path 201 1)" +if grep -q 'destroy vm=51' "$calls"; then + printf 'successful webhook VM should remain idle until billing boundary\n' >&2 + exit 1 +fi +jq -e 'select(.status == "idle_vm" and .idle_expires_at == 3600)' \ + "$(gcr_record_path 201 1)" >/dev/null +idle_once="$(gcr_record_get 201 1)" +gcr_deallocate 201 1 completed:success +test "$(gcr_record_get 201 1)" = "$idle_once" test ! -e "$(gcr_record_path 202 1)" calls_ip_fail="$GCR_STATE_DIR/calls-ip-fail"