fix: world-of-sosal

2026-09-23 19:26:19 +00:00
parent c917e4908c
commit e444ea5936
5 changed files with 274 additions and 13 deletions
@@ -3,22 +3,24 @@
 `hectic.services."project-zomboid".backup` creates local backups without stopping
 or pausing the server. The default schedule is every 30 minutes. Each run:

-1. rsyncs `Zomboid/Saves/Multiplayer/<serverName>` and non-secret server
+1. sends the local RCON `save` command and waits for the configured save grace
+   period;
+2. rsyncs `Zomboid/Saves/Multiplayer/<serverName>` and non-secret server
   settings (`SandboxVars`, spawn-points, and spawn-regions) from
   `Zomboid/Server` into a private staging tree;
-2. waits five seconds and repeats the rsync to narrow the live-write window;
-3. publishes a timestamped `tar.zst` archive; and
-4. deletes local archives older than `backup.retentionDays`.
+3. waits five seconds and repeats the rsync to narrow the live-write window;
+4. publishes a timestamped `tar.zst` archive; and
+5. deletes local archives older than `backup.retentionDays`.

 The service lock prevents overlapping runs. Missing save or server-config paths
 skip the run through systemd `ConditionPathExists` checks.

 ## Consistency and secrets

-This is a best-effort, crash-consistent backup. It does not stop Project
-Zomboid and does not use an atomic filesystem snapshot. A backup taken during a
-busy save can therefore contain files from slightly different moments; the
-second rsync reduces but cannot remove this risk.
+This is a best-effort backup. It does not stop Project Zomboid and does not use
+an atomic filesystem snapshot. The RCON save command flushes the world before
+copying, and the second rsync narrows the remaining live-write window, but
+neither makes the filesystem copy an atomic snapshot.

 Archives do not include the generated server INI, `admin-password`,
 host-generated password files, or the S3 credentials file. The server INI is
@@ -41,12 +43,17 @@ systemctl status project-zomboid-backup.service
 journalctl -u project-zomboid-backup.service
 ```

+RCON is enabled on localhost port `27015`; the firewall does not expose this
+port. The password is generated at
+`/var/lib/project-zomboid/rcon-password` with mode `0600`. The server also uses
+`SaveWorldEveryMinutes=15` as a periodic persistence fallback.
+
 ## Optional S3 upload

 S3 upload is disabled by default. Enabling it requires `bucket`, `endpoint`,
 `region`, and an absolute runtime `credentialsFile` outside `/nix/store`. The
 endpoint must use HTTPS. systemd reads the environment file without executing
-it; keep it root-owned and mode `0400`:
+it; this host keeps it owned by `project-zomboid` with mode `0400`:

 ```sh
 AWS_ACCESS_KEY_ID=...
@@ -64,6 +71,18 @@ available; it remains the stronger recovery and cleanup control.
 Restoring must be done while the server is stopped so it cannot modify files
 during extraction:

+The versioned helper creates a fresh current-state backup, stops the timer and
+server, validates archive paths, restores the save, and starts both services:
+
+```sh
+sudo ./docs/project-zomboid-restore.sh \
+  /var/lib/project-zomboid/backups/archive/<archive>.tar.zst
+```
+
+It writes a rollback archive named
+`project-zomboid-<serverName>-pre-restore-<timestamp>.tar.zst` before changing
+the save.
+
 ```sh
 systemctl stop project-zomboid.service
 tar --zstd --no-same-owner --no-same-permissions \
@@ -0,0 +1,143 @@
+#!/bin/sh
+set -eu
+
+SERVER_NAME=${SERVER_NAME:-servertest}
+DATA_DIR=${DATA_DIR:-/var/lib/project-zomboid}
+ARCHIVE=${1:-}
+
+usage() {
+  printf '%s\n' "Usage: $0 /path/to/project-zomboid-${SERVER_NAME}-<timestamp>.tar.zst"
+  printf '%s\n' "Environment: SERVER_NAME, DATA_DIR"
+}
+
+if [ "$(id -u)" -ne 0 ]; then
+  printf '%s\n' 'Run as root.' >&2
+  exit 1
+fi
+
+if [ -z "$ARCHIVE" ]; then
+  usage >&2
+  exit 2
+fi
+
+if [ ! -r "$ARCHIVE" ]; then
+  printf 'Backup archive is not readable: %s\n' "$ARCHIVE" >&2
+  exit 1
+fi
+
+ARCHIVE_DIR="$DATA_DIR/backups/archive"
+SAVE_DIR="$DATA_DIR/Zomboid/Saves/Multiplayer/$SERVER_NAME"
+SERVER_DIR="$DATA_DIR/Zomboid/Server"
+TMP_LIST=$(mktemp)
+ROLLBACK_ARCHIVE=''
+SERVER_STOPPED=0
+RESTORE_SUCCEEDED=0
+
+cleanup() {
+  rm -f "$TMP_LIST"
+}
+
+on_exit() {
+  status=$?
+  if [ "$status" -ne 0 ] && [ "$SERVER_STOPPED" -eq 1 ] \
+    && [ "$RESTORE_SUCCEEDED" -eq 0 ] && [ -n "$ROLLBACK_ARCHIVE" ]; then
+    set +e
+    rm -rf "$SAVE_DIR"
+    rm -f \
+      "$SERVER_DIR/${SERVER_NAME}_SandboxVars.lua" \
+      "$SERVER_DIR/${SERVER_NAME}_spawnpoints.lua" \
+      "$SERVER_DIR/${SERVER_NAME}_spawnregions.lua"
+    tar --zstd --no-same-owner --no-same-permissions \
+      -xpf "$ROLLBACK_ARCHIVE" -C "$DATA_DIR"
+    chown -R project-zomboid:project-zomboid "$SAVE_DIR" "$SERVER_DIR"
+    systemctl start project-zomboid.service
+    systemctl start project-zomboid-backup.timer
+    printf '%s\n' "Restore failed; current state restored from $ROLLBACK_ARCHIVE" >&2
+  fi
+  cleanup
+  exit "$status"
+}
+trap on_exit EXIT
+
+if ! tar --zstd -tf "$ARCHIVE" >"$TMP_LIST"; then
+  printf 'Archive integrity check failed: %s\n' "$ARCHIVE" >&2
+  exit 1
+fi
+
+while IFS= read -r member; do
+  case "$member" in
+    Zomboid/*) ;;
+    *)
+      printf 'Unsafe archive member: %s\n' "$member" >&2
+      exit 1
+      ;;
+  esac
+  case "$member" in
+    /*|*../*)
+      printf 'Path traversal member: %s\n' "$member" >&2
+      exit 1
+      ;;
+  esac
+done <"$TMP_LIST"
+
+if ! systemctl start project-zomboid-backup.service; then
+  printf '%s\n' 'Could not create fresh backup of current state.' >&2
+  exit 1
+fi
+
+CURRENT_ARCHIVE=$(find "$ARCHIVE_DIR" -maxdepth 1 -type f \
+  -name "project-zomboid-$SERVER_NAME-*.tar.zst" \
+  -printf '%T@ %p\n' | sort -nr | awk 'NR == 1 {sub(/^[^ ]* /, ""); print}')
+
+if [ -z "$CURRENT_ARCHIVE" ]; then
+  printf '%s\n' 'Fresh current-state backup was not found.' >&2
+  exit 1
+fi
+
+stamp=$(date -u +%Y%m%dT%H%M%SZ)
+ROLLBACK_ARCHIVE="$ARCHIVE_DIR/project-zomboid-$SERVER_NAME-pre-restore-$stamp.tar.zst"
+cp --reflink=auto "$CURRENT_ARCHIVE" "$ROLLBACK_ARCHIVE" 2>/dev/null \
+  || cp "$CURRENT_ARCHIVE" "$ROLLBACK_ARCHIVE"
+chmod 0600 "$ROLLBACK_ARCHIVE"
+chown project-zomboid:project-zomboid "$ROLLBACK_ARCHIVE"
+
+systemctl stop project-zomboid-backup.timer
+systemctl stop project-zomboid.service
+SERVER_STOPPED=1
+
+if [ "$(systemctl show project-zomboid --property=ActiveState --value)" != inactive ]; then
+  printf '%s\n' 'Project Zomboid did not stop; refusing to restore.' >&2
+  exit 1
+fi
+
+rm -rf "$SAVE_DIR"
+rm -f \
+  "$SERVER_DIR/${SERVER_NAME}_SandboxVars.lua" \
+  "$SERVER_DIR/${SERVER_NAME}_spawnpoints.lua" \
+  "$SERVER_DIR/${SERVER_NAME}_spawnregions.lua"
+
+tar --zstd --no-same-owner --no-same-permissions \
+  -xpf "$ARCHIVE" -C "$DATA_DIR"
+chown -R project-zomboid:project-zomboid "$SAVE_DIR" "$SERVER_DIR"
+
+systemctl start project-zomboid.service
+started=0
+for _ in $(seq 1 90); do
+  if [ "$(systemctl show project-zomboid --property=ActiveState --value)" = active ] \
+    && [ "$(systemctl show project-zomboid --property=SubState --value)" = running ]; then
+    started=1
+    break
+  fi
+  sleep 2
+done
+
+if [ "$started" -ne 1 ]; then
+  printf 'Restore completed, but service did not become healthy. Rollback: %s\n' \
+    "$ROLLBACK_ARCHIVE" >&2
+  exit 1
+fi
+
+systemctl start project-zomboid-backup.timer
+RESTORE_SUCCEEDED=1
+printf 'Restore completed.\n'
+printf 'Rollback archive: %s\n' "$ROLLBACK_ARCHIVE"