feat: zomboid backups

2026-09-22 17:15:18 +00:00
parent ef849b085f
commit c917e4908c
9 changed files with 578 additions and 49 deletions
@@ -1,6 +1,8 @@
 # Documentation

 - [Using the `hectic` Attic Cache](./attic-cache.md)
+- [Minecraft incident log](./minecraft-incidents.md)
+- [Project Zomboid backups](./project-zomboid-backups.md)

 ## Gitea runner labels

@@ -0,0 +1,101 @@
+# Minecraft incident log
+
+This file records only observed evidence, actions, and verification results.
+An entity appearing in a stack trace is a trigger-path observation, not a
+proven root cause.
+
+## 2026-09-19 — WorldOfSosal crashes in Sable block-change handling
+
+### Impact
+
+- `minecraft-server-wowMineMap.service` terminates while a player is online.
+- Public Minecraft endpoint is `store.hectic-lab.com:25568`.
+- Server is intentionally stopped after the latest crash to prevent repeated
+  crash-save cycles while recovery is investigated.
+
+### Observed evidence
+
+All crash reports contain `sable@2.0.5` in
+`LevelAccelerator.getBlockState`, followed by
+`ArrayIndexOutOfBoundsException` where the requested section index exceeds
+the world section array length of `24`.
+
+| UTC timestamp | Crash report | Observed trigger path | Exception |
+| --- | --- | --- | --- |
+| 18:47:03 | `crash-2026-09-19_18.47.03-server.txt` | `EnderMan$EndermanTakeBlockGoal.tick` | index `38` / length `24` |
+| 18:52:17 | `crash-2026-09-19_18.52.17-server.txt` | `GlowSquid.aiStep` → `RedStoneOreBlock.stepOn` | index `33` / length `24` |
+| 19:14:46 | `crash-2026-09-19_19.14.46-server.txt` | `Skeleton.tick` → `RedStoneOreBlock.stepOn` | index `34` / length `24` |
+
+Evidence locations on `neuro`:
+
+```text
+/srv/minecraft/wowMineMap/crash-reports/
+/srv/minecraft/wowMineMap/logs/latest.log
+```
+
+### Actions performed
+
+| UTC timestamp | Action | Result |
+| --- | --- | --- |
+| 17:51 | Archived current world before recovery | Archive checksum recorded |
+| 18:08 | Set `randomTickSpeed=0` | Server started, but later crashed from an entity block change |
+| 18:48 | Set `mobGriefing=false` | Prevented Enderman block pickup only; later crashes still occurred |
+| 18:54 | Archived post-crash world | Archive checksum recorded |
+| 19:00 | Moved Boss offline player NBT from `(3299.067, 142.630, 8613.742)` to `(3296, 500, 8608)` in `crafting_azeroth:azeroth` | Only `Pos` and `Dimension` changed; later crash still occurred |
+| after 19:14 crash | Stopped `minecraft-server-wowMineMap.service` | Prevented further automatic crash/restart saves |
+
+### Recovery artifacts
+
+```text
+/srv/minecraft/backups/wowMineMap-before-sable-recovery-20260919T175139Z.tar.zst
+/srv/minecraft/backups/wowMineMap-after-sable-crashes-20260919T185445Z.tar.zst
+/srv/minecraft/wowMineMap/world/playerdata/1c189af5-2713-3fa6-bcc4-893dfadedfa4.dat.before-relocation
+```
+
+### Conclusions supported by evidence
+
+- Public proxy and reverse tunnel are not the failure point: server-list ping
+  succeeded before later in-world crashes.
+- The failure is not limited to Endermen, random ticks, or one player
+  position.
+- Sable's block-change callback is present in every captured crash.
+
+### Not established
+
+- Exact corrupt chunk, block, or mod data.
+- Whether world data is corrupt, Sable itself is defective, or another mod is
+  supplying incompatible world state.
+- Whether deleting any chunk, region, or Sable state would be safe.
+
+### External research
+
+No exact upstream match was found for Sable `2.0.5` on NeoForge `1.21.1` with
+`LevelAccelerator.getBlockState` and a requested section index of `33`, `34`,
+or `38` against a section array of length `24`.
+
+Related but non-identical upstream reports:
+
+- [Sable #776](https://github.com/ryanhcode/sable/issues/776) documents an
+  `ArrayIndexOutOfBoundsException` associated with unusual dimension height
+  bounds. This is relevant to section-coordinate handling, but is an older
+  version and different stack trace.
+- [Sable #1087](https://github.com/ryanhcode/sable/issues/1087) documents a
+  `LevelAccelerator.getBlockState` recursion during block-shape processing.
+  The failure type differs.
+- [Sable #820](https://github.com/ryanhcode/sable/issues/820) documents a
+  ticking-entity block-change crash. The reported downgrade to `1.1.3` helped
+  that distinct recursive-update failure; it is not evidence for this crash.
+- [Sable #1223](https://github.com/ryanhcode/sable/issues/1223) documents a
+  different `ArrayIndexOutOfBoundsException` in voxel-neighborhood handling.
+  Its suggested Lithium setting only reduced crashes for some reporters and is
+  not a verified mitigation here.
+
+Sable `2.0.4` and `2.0.5` release notes mention other block or contraption
+crash fixes, but not this exception. No version upgrade or downgrade is
+currently evidence-backed as a production fix.
+
+### Next recovery step
+
+Use a disposable full-world copy to test a supported Sable/physics integration
+mitigation. Do not restart production, delete region files, or overwrite a
+backup until that test gives reproducible evidence.
@@ -0,0 +1,77 @@
+# Project Zomboid backups
+
+`hectic.services."project-zomboid".backup` creates local backups without stopping
+or pausing the server. The default schedule is every 30 minutes. Each run:
+
+1. rsyncs `Zomboid/Saves/Multiplayer/<serverName>` and non-secret server
+   settings (`SandboxVars`, spawn-points, and spawn-regions) from
+   `Zomboid/Server` into a private staging tree;
+2. waits five seconds and repeats the rsync to narrow the live-write window;
+3. publishes a timestamped `tar.zst` archive; and
+4. deletes local archives older than `backup.retentionDays`.
+
+The service lock prevents overlapping runs. Missing save or server-config paths
+skip the run through systemd `ConditionPathExists` checks.
+
+## Consistency and secrets
+
+This is a best-effort, crash-consistent backup. It does not stop Project
+Zomboid and does not use an atomic filesystem snapshot. A backup taken during a
+busy save can therefore contain files from slightly different moments; the
+second rsync reduces but cannot remove this risk.
+
+Archives do not include the generated server INI, `admin-password`,
+host-generated password files, or the S3 credentials file. The server INI is
+generated again during service startup; provision secret-backed values separately
+after a restore.
+
+## hectic-lab
+
+hectic-lab runs the timer every 30 minutes and keeps local archives for 14 days:
+
+```text
+/var/lib/project-zomboid/backups/archive/
+```
+
+Check it with:
+
+```sh
+systemctl list-timers project-zomboid-backup.timer
+systemctl status project-zomboid-backup.service
+journalctl -u project-zomboid-backup.service
+```
+
+## Optional S3 upload
+
+S3 upload is disabled by default. Enabling it requires `bucket`, `endpoint`,
+`region`, and an absolute runtime `credentialsFile` outside `/nix/store`. The
+endpoint must use HTTPS. systemd reads the environment file without executing
+it; keep it root-owned and mode `0400`:
+
+```sh
+AWS_ACCESS_KEY_ID=...
+AWS_SECRET_ACCESS_KEY=...
+```
+
+Set `backup.s3.prefix` to choose the object-key prefix and
+`backup.s3.remoteRetentionDays` to prune old archives from that prefix. Remote
+deletion runs only after a successful upload and only matches this server's
+archive name prefix. Configure bucket lifecycle expiration/versioning too when
+available; it remains the stronger recovery and cleanup control.
+
+## Restore
+
+Restoring must be done while the server is stopped so it cannot modify files
+during extraction:
+
+```sh
+systemctl stop project-zomboid.service
+tar --zstd --no-same-owner --no-same-permissions \
+  -xf /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
+  -C /var/lib/project-zomboid
+chown -R project-zomboid:project-zomboid /var/lib/project-zomboid/Zomboid
+systemctl start project-zomboid.service
+```
+
+Re-provision password files and secret-backed INI values before starting.
+Verify the restored save and server name before allowing players to reconnect.