# Project Zomboid backups

`hectic.services."project-zomboid".backup` creates local backups without stopping
or pausing the server. The default schedule is every 30 minutes. Each run:

1. sends the local RCON `save` command and waits for the configured save grace
   period;
2. rsyncs `Zomboid/Saves/Multiplayer/<serverName>` and non-secret server
   settings (`SandboxVars`, spawn-points, and spawn-regions) from
   `Zomboid/Server` into a private staging tree;
3. waits five seconds and repeats the rsync to narrow the live-write window;
4. publishes a timestamped `tar.zst` archive; and
5. uploads it to S3 when enabled, then applies local retention.

With `backup.retentionDays = 0`, local timestamped archives are removed after a
successful S3 upload (or after local creation when S3 is disabled). A failed
upload leaves the current archive locally and the next run retries that archive
before creating a new one. Restore archives named `pre-restore` are not managed
by this cleanup.

The service lock prevents overlapping runs. Missing save or server-config paths
skip the run through systemd `ConditionPathExists` checks.

## Consistency and secrets

This is a best-effort backup. It does not stop Project Zomboid and does not use
an atomic filesystem snapshot. The RCON save command flushes the world before
copying, and the second rsync narrows the remaining live-write window, but
neither makes the filesystem copy an atomic snapshot.

Archives do not include the generated server INI, `admin-password`,
host-generated password files, or the S3 credentials file. The server INI is
generated again during service startup; provision secret-backed values separately
after a restore.

## hectic-lab

hectic-lab runs the timer every 30 minutes and stores timestamped backups in
S3 for 14 days. It keeps no regular timestamped backup archives locally:

```text
/var/lib/project-zomboid/backups/archive/
```

The existing `project-zomboid-restore.sh` helper expects the fresh backup it
creates to remain in this directory for rollback. Therefore, with local
retention set to zero, do not use the helper until it is adapted for S3-only
retention; temporarily configure positive local retention for a restore.

Check it with:

```sh
systemctl list-timers project-zomboid-backup.timer
systemctl status project-zomboid-backup.service
journalctl -u project-zomboid-backup.service
```

RCON is enabled on localhost port `27015`; the firewall does not expose this
port. The password is generated at
`/var/lib/project-zomboid/rcon-password` with mode `0600`. The server also uses
`SaveWorldEveryMinutes=15` as a periodic persistence fallback.

## Optional S3 upload

S3 upload is disabled by default. Enabling it requires `bucket`, `endpoint`,
`region`, and an absolute runtime `credentialsFile` outside `/nix/store`. The
endpoint must use HTTPS. systemd reads the environment file without executing
it; this host keeps it owned by `project-zomboid` with mode `0400`:

```sh
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
```

Set `backup.s3.prefix` to choose the object-key prefix and
`backup.s3.remoteRetentionDays` to prune old archives from that prefix. Remote
deletion runs only after a successful upload and only matches this server's
archive name prefix. Configure bucket lifecycle expiration/versioning too when
available; it remains the stronger recovery and cleanup control.

## Restore

Restoring must be done while the server is stopped so it cannot modify files
during extraction:

On hectic-lab, regular timestamped archives are retained only in S3. With
valid S3 credentials, download the chosen archive before restoring it:

```sh
aws s3 cp \
  s3://backup-hectic-lab/project-zomboid/project-zomboid-servertest-<timestamp>.tar.zst \
  /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
  --endpoint-url https://hel1.your-objectstorage.com \
  --region hel1
```

The versioned helper creates a fresh current-state backup, stops the timer and
server, validates archive paths, restores the save, and starts both services:

```sh
sudo ./docs/project-zomboid-restore.sh \
  /var/lib/project-zomboid/backups/archive/<archive>.tar.zst
```

It writes a rollback archive named
`project-zomboid-<serverName>-pre-restore-<timestamp>.tar.zst` before changing
the save.

```sh
systemctl stop project-zomboid.service
tar --zstd --no-same-owner --no-same-permissions \
  -xf /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
  -C /var/lib/project-zomboid
chown -R project-zomboid:project-zomboid /var/lib/project-zomboid/Zomboid
systemctl start project-zomboid.service
```

Re-provision password files and secret-backed INI values before starting.
Verify the restored save and server name before allowing players to reconnect.