hearth/docs/project-zomboid-backups.md

Project Zomboid backups

hectic.services."project-zomboid".backup creates local backups without stopping or pausing the server. The default schedule is every 30 minutes. Each run:

1. sends the local RCON save command and waits for the configured save grace period;
2. rsyncs Zomboid/Saves/Multiplayer/<serverName> and non-secret server settings (SandboxVars, spawn-points, and spawn-regions) from Zomboid/Server into a private staging tree;
3. waits five seconds and repeats the rsync to narrow the live-write window;
4. publishes a timestamped tar.zst archive; and
5. uploads it to S3 when enabled, then applies local retention.

With backup.retentionDays = 0, local timestamped archives are removed after a successful S3 upload (or after local creation when S3 is disabled). A failed upload leaves the current archive locally and the next run retries that archive before creating a new one. Restore archives named pre-restore are not managed by this cleanup.

The service lock prevents overlapping runs. Missing save or server-config paths skip the run through systemd ConditionPathExists checks.

Consistency and secrets

This is a best-effort backup. It does not stop Project Zomboid and does not use an atomic filesystem snapshot. The RCON save command flushes the world before copying, and the second rsync narrows the remaining live-write window, but neither makes the filesystem copy an atomic snapshot.

Archives do not include the generated server INI, admin-password, host-generated password files, or the S3 credentials file. The server INI is generated again during service startup; provision secret-backed values separately after a restore.

hectic-lab

hectic-lab runs the timer every 30 minutes and stores timestamped backups in S3 for 14 days. It keeps no regular timestamped backup archives locally:

/var/lib/project-zomboid/backups/archive/

The existing project-zomboid-restore.sh helper expects the fresh backup it creates to remain in this directory for rollback. Therefore, with local retention set to zero, do not use the helper until it is adapted for S3-only retention; temporarily configure positive local retention for a restore.

Check it with:

systemctl list-timers project-zomboid-backup.timer
systemctl status project-zomboid-backup.service
journalctl -u project-zomboid-backup.service

RCON is enabled on localhost port 27015; the firewall does not expose this port. The password is generated at /var/lib/project-zomboid/rcon-password with mode 0600. The server also uses SaveWorldEveryMinutes=15 as a periodic persistence fallback.

Optional S3 upload

S3 upload is disabled by default. Enabling it requires bucket, endpoint, region, and an absolute runtime credentialsFile outside /nix/store. The endpoint must use HTTPS. systemd reads the environment file without executing it; this host keeps it owned by project-zomboid with mode 0400:

AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...

Set backup.s3.prefix to choose the object-key prefix and backup.s3.remoteRetentionDays to prune old archives from that prefix. Remote deletion runs only after a successful upload and only matches this server's archive name prefix. Configure bucket lifecycle expiration/versioning too when available; it remains the stronger recovery and cleanup control.

Restore

Restoring must be done while the server is stopped so it cannot modify files during extraction:

On hectic-lab, regular timestamped archives are retained only in S3. With valid S3 credentials, download the chosen archive before restoring it:

aws s3 cp \
  s3://backup-hectic-lab/project-zomboid/project-zomboid-servertest-<timestamp>.tar.zst \
  /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
  --endpoint-url https://hel1.your-objectstorage.com \
  --region hel1

The versioned helper creates a fresh current-state backup, stops the timer and server, validates archive paths, restores the save, and starts both services:

sudo ./docs/project-zomboid-restore.sh \
  /var/lib/project-zomboid/backups/archive/<archive>.tar.zst

It writes a rollback archive named project-zomboid-<serverName>-pre-restore-<timestamp>.tar.zst before changing the save.

systemctl stop project-zomboid.service
tar --zstd --no-same-owner --no-same-permissions \
  -xf /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
  -C /var/lib/project-zomboid
chown -R project-zomboid:project-zomboid /var/lib/project-zomboid/Zomboid
systemctl start project-zomboid.service

Re-provision password files and secret-backed INI values before starting. Verify the restored save and server name before allowing players to reconnect.