feat: zomboid backups

.sops.yaml

@@ -6,7 +6,7 @@ keys:
   - &bfs-server        age15yzgmsvl3ku2w863h6gw2vpmw37m9aruv6xrj4fue6n2jpm7pyuqk9xjmj
   - &bfs-pol-server    age1fpytf05sg9n6ywpwkmn09lhpfvgtud9h75h76jhxha475zpnasqq952rpu
   - &bfs-new-server    age17yx98qk9gzgcf2q6zhhp05p6mmtrkgz66dvyk9gqclypvlr8rersxjy5v7
-  - &neuro-server      age15yzgmsvl3ku2w863h6gw2vpmw37m9aruv6xrj4fue6n2jpm7pyuqk9xjmj
+  - &neuro-server      age1ak7heljpr0pjr4m0rcwxgn3sp0jjxw03lxyf33r8lcemqh2u2sgqx0aplq
   - &games-server      age15yzgmsvl3ku2w863h6gw2vpmw37m9aruv6xrj4fue6n2jpm7pyuqk9xjmj
   - &hectic-lab-server age13h8twnwvgxn04l5ywtru89a6psw5d0uckr2eghxsjp88a5augvsstq5ard
   - &umbriel-bfs       age1jxntjca8q2vxvf2jaal4xyvm2ae6sh62fhv897694kuzawfrk5asj00zdt

docs/README.md

@@ -1,6 +1,8 @@
 # Documentation
 
 - [Using the `hectic` Attic Cache](./attic-cache.md)
+- [Minecraft incident log](./minecraft-incidents.md)
+- [Project Zomboid backups](./project-zomboid-backups.md)
 
 ## Gitea runner labels
 

docs/minecraft-incidents.md

@@ -0,0 +1,101 @@
+# Minecraft incident log
+
+This file records only observed evidence, actions, and verification results.
+An entity appearing in a stack trace is a trigger-path observation, not a
+proven root cause.
+
+## 2026-09-19 — WorldOfSosal crashes in Sable block-change handling
+
+### Impact
+
+- `minecraft-server-wowMineMap.service` terminates while a player is online.
+- Public Minecraft endpoint is `store.hectic-lab.com:25568`.
+- Server is intentionally stopped after the latest crash to prevent repeated
+  crash-save cycles while recovery is investigated.
+
+### Observed evidence
+
+All crash reports contain `sable@2.0.5` in
+`LevelAccelerator.getBlockState`, followed by
+`ArrayIndexOutOfBoundsException` where the requested section index exceeds
+the world section array length of `24`.
+
+| UTC timestamp | Crash report | Observed trigger path | Exception |
+| --- | --- | --- | --- |
+| 18:47:03 | `crash-2026-09-19_18.47.03-server.txt` | `EnderMan$EndermanTakeBlockGoal.tick` | index `38` / length `24` |
+| 18:52:17 | `crash-2026-09-19_18.52.17-server.txt` | `GlowSquid.aiStep` → `RedStoneOreBlock.stepOn` | index `33` / length `24` |
+| 19:14:46 | `crash-2026-09-19_19.14.46-server.txt` | `Skeleton.tick` → `RedStoneOreBlock.stepOn` | index `34` / length `24` |
+
+Evidence locations on `neuro`:
+
+```text
+/srv/minecraft/wowMineMap/crash-reports/
+/srv/minecraft/wowMineMap/logs/latest.log
+```
+
+### Actions performed
+
+| UTC timestamp | Action | Result |
+| --- | --- | --- |
+| 17:51 | Archived current world before recovery | Archive checksum recorded |
+| 18:08 | Set `randomTickSpeed=0` | Server started, but later crashed from an entity block change |
+| 18:48 | Set `mobGriefing=false` | Prevented Enderman block pickup only; later crashes still occurred |
+| 18:54 | Archived post-crash world | Archive checksum recorded |
+| 19:00 | Moved Boss offline player NBT from `(3299.067, 142.630, 8613.742)` to `(3296, 500, 8608)` in `crafting_azeroth:azeroth` | Only `Pos` and `Dimension` changed; later crash still occurred |
+| after 19:14 crash | Stopped `minecraft-server-wowMineMap.service` | Prevented further automatic crash/restart saves |
+
+### Recovery artifacts
+
+```text
+/srv/minecraft/backups/wowMineMap-before-sable-recovery-20260919T175139Z.tar.zst
+/srv/minecraft/backups/wowMineMap-after-sable-crashes-20260919T185445Z.tar.zst
+/srv/minecraft/wowMineMap/world/playerdata/1c189af5-2713-3fa6-bcc4-893dfadedfa4.dat.before-relocation
+```
+
+### Conclusions supported by evidence
+
+- Public proxy and reverse tunnel are not the failure point: server-list ping
+  succeeded before later in-world crashes.
+- The failure is not limited to Endermen, random ticks, or one player
+  position.
+- Sable's block-change callback is present in every captured crash.
+
+### Not established
+
+- Exact corrupt chunk, block, or mod data.
+- Whether world data is corrupt, Sable itself is defective, or another mod is
+  supplying incompatible world state.
+- Whether deleting any chunk, region, or Sable state would be safe.
+
+### External research
+
+No exact upstream match was found for Sable `2.0.5` on NeoForge `1.21.1` with
+`LevelAccelerator.getBlockState` and a requested section index of `33`, `34`,
+or `38` against a section array of length `24`.
+
+Related but non-identical upstream reports:
+
+- [Sable #776](https://github.com/ryanhcode/sable/issues/776) documents an
+  `ArrayIndexOutOfBoundsException` associated with unusual dimension height
+  bounds. This is relevant to section-coordinate handling, but is an older
+  version and different stack trace.
+- [Sable #1087](https://github.com/ryanhcode/sable/issues/1087) documents a
+  `LevelAccelerator.getBlockState` recursion during block-shape processing.
+  The failure type differs.
+- [Sable #820](https://github.com/ryanhcode/sable/issues/820) documents a
+  ticking-entity block-change crash. The reported downgrade to `1.1.3` helped
+  that distinct recursive-update failure; it is not evidence for this crash.
+- [Sable #1223](https://github.com/ryanhcode/sable/issues/1223) documents a
+  different `ArrayIndexOutOfBoundsException` in voxel-neighborhood handling.
+  Its suggested Lithium setting only reduced crashes for some reporters and is
+  not a verified mitigation here.
+
+Sable `2.0.4` and `2.0.5` release notes mention other block or contraption
+crash fixes, but not this exception. No version upgrade or downgrade is
+currently evidence-backed as a production fix.
+
+### Next recovery step
+
+Use a disposable full-world copy to test a supported Sable/physics integration
+mitigation. Do not restart production, delete region files, or overwrite a
+backup until that test gives reproducible evidence.

docs/project-zomboid-backups.md

@@ -0,0 +1,77 @@
+# Project Zomboid backups
+
+`hectic.services."project-zomboid".backup` creates local backups without stopping
+or pausing the server. The default schedule is every 30 minutes. Each run:
+
+1. rsyncs `Zomboid/Saves/Multiplayer/<serverName>` and non-secret server
+   settings (`SandboxVars`, spawn-points, and spawn-regions) from
+   `Zomboid/Server` into a private staging tree;
+2. waits five seconds and repeats the rsync to narrow the live-write window;
+3. publishes a timestamped `tar.zst` archive; and
+4. deletes local archives older than `backup.retentionDays`.
+
+The service lock prevents overlapping runs. Missing save or server-config paths
+skip the run through systemd `ConditionPathExists` checks.
+
+## Consistency and secrets
+
+This is a best-effort, crash-consistent backup. It does not stop Project
+Zomboid and does not use an atomic filesystem snapshot. A backup taken during a
+busy save can therefore contain files from slightly different moments; the
+second rsync reduces but cannot remove this risk.
+
+Archives do not include the generated server INI, `admin-password`,
+host-generated password files, or the S3 credentials file. The server INI is
+generated again during service startup; provision secret-backed values separately
+after a restore.
+
+## hectic-lab
+
+hectic-lab runs the timer every 30 minutes and keeps local archives for 14 days:
+
+```text
+/var/lib/project-zomboid/backups/archive/
+```
+
+Check it with:
+
+```sh
+systemctl list-timers project-zomboid-backup.timer
+systemctl status project-zomboid-backup.service
+journalctl -u project-zomboid-backup.service
+```
+
+## Optional S3 upload
+
+S3 upload is disabled by default. Enabling it requires `bucket`, `endpoint`,
+`region`, and an absolute runtime `credentialsFile` outside `/nix/store`. The
+endpoint must use HTTPS. systemd reads the environment file without executing
+it; keep it root-owned and mode `0400`:
+
+```sh
+AWS_ACCESS_KEY_ID=...
+AWS_SECRET_ACCESS_KEY=...
+```
+
+Set `backup.s3.prefix` to choose the object-key prefix and
+`backup.s3.remoteRetentionDays` to prune old archives from that prefix. Remote
+deletion runs only after a successful upload and only matches this server's
+archive name prefix. Configure bucket lifecycle expiration/versioning too when
+available; it remains the stronger recovery and cleanup control.
+
+## Restore
+
+Restoring must be done while the server is stopped so it cannot modify files
+during extraction:
+
+```sh
+systemctl stop project-zomboid.service
+tar --zstd --no-same-owner --no-same-permissions \
+  -xf /var/lib/project-zomboid/backups/archive/<archive>.tar.zst \
+  -C /var/lib/project-zomboid
+chown -R project-zomboid:project-zomboid /var/lib/project-zomboid/Zomboid
+systemctl start project-zomboid.service
+```
+
+Re-provision password files and secret-backed INI values before starting.
+Verify the restored save and server name before allowing players to reconnect.

nixos/module/hectic/service/project-zomboid.nix

@@ -37,6 +37,147 @@
   ) cfg.sandboxProperties;
   zomboidDir = "${cfg.dataDir}/Zomboid";
   adminPasswordFile = "${cfg.dataDir}/admin-password";
+  backupCfg = cfg.backup;
+  s3CredentialsFile = if backupCfg.s3.credentialsFile == null then "" else backupCfg.s3.credentialsFile;
+  s3Bucket = if backupCfg.s3.bucket == null then "" else backupCfg.s3.bucket;
+  s3Endpoint = if backupCfg.s3.endpoint == null then "" else backupCfg.s3.endpoint;
+  s3Region = if backupCfg.s3.region == null then "" else backupCfg.s3.region;
+  saveDir = "${zomboidDir}/Saves/Multiplayer/${cfg.serverName}";
+  serverConfigDir = "${zomboidDir}/Server";
+  backupScript = pkgs.writeShellScript "project-zomboid-backup" ''
+    set -eu
+
+    staging_dir=${lib.escapeShellArg backupCfg.stagingDir}
+    archive_dir=${lib.escapeShellArg backupCfg.archiveDir}
+    lock_file="$archive_dir/.backup.lock"
+
+    ${pkgs.coreutils}/bin/install -d -m 0700 \
+      "$staging_dir/Zomboid/Saves/Multiplayer/${cfg.serverName}" \
+      "$staging_dir/Zomboid/Server" \
+      "$archive_dir"
+
+    exec 9>"$lock_file"
+    if ! ${pkgs.util-linux}/bin/flock -n 9; then
+      ${pkgs.coreutils}/bin/printf '%s\n' 'Project Zomboid backup already running; skipping.' >&2
+      exit 0
+    fi
+
+    sync_staging() {
+      ${pkgs.rsync}/bin/rsync -a --delete \
+        ${lib.escapeShellArg "${saveDir}/"} \
+        "$staging_dir/Zomboid/Saves/Multiplayer/${cfg.serverName}/"
+      ${pkgs.rsync}/bin/rsync -a --delete --delete-excluded \
+        --include=${lib.escapeShellArg "/${cfg.serverName}_SandboxVars.lua"} \
+        --include=${lib.escapeShellArg "/${cfg.serverName}_spawnpoints.lua"} \
+        --include=${lib.escapeShellArg "/${cfg.serverName}_spawnregions.lua"} \
+        --exclude='*' \
+        ${lib.escapeShellArg "${serverConfigDir}/"} \
+        "$staging_dir/Zomboid/Server/"
+    }
+
+    # Second pass narrows, but cannot eliminate, live-save inconsistency.
+    sync_staging
+    ${pkgs.coreutils}/bin/sleep 5
+    sync_staging
+
+    timestamp="$(${pkgs.coreutils}/bin/date -u +%Y%m%dT%H%M%SZ)"
+    archive_name="project-zomboid-${cfg.serverName}-$timestamp.tar.zst"
+    archive_tmp="$archive_dir/.$archive_name.tmp"
+    archive="$archive_dir/$archive_name"
+    trap '${pkgs.coreutils}/bin/rm -f "$archive_tmp"' EXIT
+    ${pkgs.gnutar}/bin/tar \
+      --use-compress-program=${lib.escapeShellArg "${pkgs.zstd}/bin/zstd -T0"} \
+      -C "$staging_dir" -cf "$archive_tmp" Zomboid
+    ${pkgs.coreutils}/bin/chmod 0600 "$archive_tmp"
+    ${pkgs.coreutils}/bin/mv "$archive_tmp" "$archive"
+    trap - EXIT
+
+    ${pkgs.findutils}/bin/find "$archive_dir" -maxdepth 1 -type f \
+      -name ${lib.escapeShellArg "project-zomboid-${cfg.serverName}-*.tar.zst"} \
+      -mmin +${toString (backupCfg.retentionDays * 1440)} -delete
+
+    ${lib.optionalString backupCfg.s3.enable ''
+      if [ -z "''${AWS_ACCESS_KEY_ID:-}" ] || [ -z "''${AWS_SECRET_ACCESS_KEY:-}" ]; then
+        ${pkgs.coreutils}/bin/printf '%s\n' \
+          'AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY missing from Project Zomboid S3 credentials file.' >&2
+        exit 1
+      fi
+      s3_bucket=${lib.escapeShellArg s3Bucket}
+      s3_prefix=${lib.escapeShellArg backupCfg.s3.prefix}
+      s3_key="''${s3_prefix:+$s3_prefix/}$archive_name"
+      ${pkgs.awscli2}/bin/aws s3 cp "$archive" \
+        "s3://$s3_bucket/$s3_key" \
+        --endpoint-url ${lib.escapeShellArg s3Endpoint} \
+        --region ${lib.escapeShellArg s3Region} \
+        --cli-connect-timeout 30 \
+        --cli-read-timeout 300 \
+        --only-show-errors
+
+      remote_prefix="$s3_prefix"
+      if [ -n "$remote_prefix" ]; then
+        remote_prefix="$remote_prefix/"
+      fi
+      archive_prefix=${lib.escapeShellArg "project-zomboid-${cfg.serverName}-"}
+      remote_list="$staging_dir/.remote-objects.json"
+      remote_delete_dir="$staging_dir/.remote-delete"
+      ${pkgs.awscli2}/bin/aws s3api list-objects-v2 \
+        --bucket "$s3_bucket" \
+        --prefix "$remote_prefix" \
+        --endpoint-url ${lib.escapeShellArg s3Endpoint} \
+        --region ${lib.escapeShellArg s3Region} \
+        --output json > "$remote_list"
+      ${pkgs.python3}/bin/python3 - "$remote_list" "$remote_delete_dir" \
+        "$(( $(${pkgs.coreutils}/bin/date +%s) - ${toString (backupCfg.s3.remoteRetentionDays * 86400)} ))" \
+        "$remote_prefix$archive_prefix" <<'PY'
+import datetime
+import json
+import os
+import re
+import sys
+
+objects_path, delete_dir, cutoff, key_prefix = sys.argv[1:]
+cutoff = int(cutoff)
+archive_pattern = re.compile(
+    re.escape(key_prefix) + r"\d{8}T\d{6}Z\.tar\.zst\Z"
+)
+with open(objects_path, encoding="utf-8") as stream:
+    objects = json.load(stream).get("Contents", [])
+
+old_keys = []
+for item in objects:
+    key = item.get("Key", "")
+    if not archive_pattern.fullmatch(key):
+        continue
+    modified = datetime.datetime.fromisoformat(
+        item["LastModified"].replace("Z", "+00:00")
+    )
+    if int(modified.timestamp()) < cutoff:
+        old_keys.append(key)
+
+os.makedirs(delete_dir, exist_ok=True)
+for batch_number in range(0, len(old_keys), 1000):
+    batch = old_keys[batch_number:batch_number + 1000]
+    manifest_path = os.path.join(
+        delete_dir, f"batch-{batch_number // 1000:04d}.json"
+    )
+    with open(manifest_path, "w", encoding="utf-8") as stream:
+        json.dump(
+            {"Objects": [{"Key": key} for key in batch], "Quiet": True},
+            stream,
+        )
+PY
+      for remote_manifest in "$remote_delete_dir"/*.json; do
+        [ -f "$remote_manifest" ] || continue
+        ${pkgs.awscli2}/bin/aws s3api delete-objects \
+          --bucket "$s3_bucket" \
+          --delete "file://$remote_manifest" \
+          --endpoint-url ${lib.escapeShellArg s3Endpoint} \
+          --region ${lib.escapeShellArg s3Region} \
+          --only-show-errors
+      done
+      ${pkgs.coreutils}/bin/rm -rf "$remote_list" "$remote_delete_dir"
+    ''}
+  '';
   startScript = pkgs.writeShellScript "project-zomboid-start" ''
     admin_password=$(${pkgs.coreutils}/bin/cat ${lib.escapeShellArg adminPasswordFile})
     exec ${pkgs.steam-run}/bin/steam-run \
@@ -131,9 +272,120 @@ in {
       default = true;
       description = "Open the Project Zomboid UDP ports in the firewall.";
     };
+
+    backup = {
+      enable = lib.mkEnableOption "no-stop Project Zomboid backups";
+
+      onCalendar = lib.mkOption {
+        type = lib.types.str;
+        default = "*:0/30";
+        description = "systemd calendar expression controlling backup frequency.";
+      };
+
+      stagingDir = lib.mkOption {
+        type = lib.types.path;
+        default = "${cfg.dataDir}/backups/staging";
+        description = "Local directory containing the two-pass rsync staging tree.";
+      };
+
+      archiveDir = lib.mkOption {
+        type = lib.types.path;
+        default = "${cfg.dataDir}/backups/archive";
+        description = "Local directory containing timestamped tar.zst archives.";
+      };
+
+      retentionDays = lib.mkOption {
+        type = lib.types.ints.positive;
+        default = 14;
+        description = "Delete local archives older than this many days.";
+      };
+
+      s3 = {
+        enable = lib.mkEnableOption "uploading Project Zomboid backups to S3-compatible storage";
+
+        credentialsFile = lib.mkOption {
+          type = lib.types.nullOr lib.types.str;
+          default = null;
+          description = ''
+            Runtime env file containing AWS_ACCESS_KEY_ID and
+            AWS_SECRET_ACCESS_KEY. Required when S3 upload is enabled.
+          '';
+        };
+
+        bucket = lib.mkOption {
+          type = lib.types.nullOr lib.types.str;
+          default = null;
+          description = "S3 bucket receiving backup archives.";
+        };
+
+        endpoint = lib.mkOption {
+          type = lib.types.nullOr lib.types.str;
+          default = null;
+          description = "S3-compatible endpoint URL.";
+        };
+
+        region = lib.mkOption {
+          type = lib.types.nullOr lib.types.str;
+          default = null;
+          description = "S3 region passed to awscli2.";
+        };
+
+        prefix = lib.mkOption {
+          type = lib.types.str;
+          default = "project-zomboid";
+          description = "Optional object key prefix within the S3 bucket.";
+        };
+
+        remoteRetentionDays = lib.mkOption {
+          type = lib.types.ints.positive;
+          default = 14;
+          description = "Delete uploaded archives older than this many days.";
+        };
+      };
+    };
   };
 
   config = lib.mkIf cfg.enable {
+    assertions = [
+      {
+        assertion = !backupCfg.s3.enable || backupCfg.enable;
+        message = "hectic.services.project-zomboid.backup must be enabled before S3 upload.";
+      }
+      {
+        assertion = !backupCfg.s3.enable || backupCfg.s3.credentialsFile != null;
+        message = "hectic.services.project-zomboid.backup.s3.credentialsFile is required when S3 upload is enabled.";
+      }
+      {
+        assertion = !backupCfg.s3.enable || backupCfg.s3.bucket != null;
+        message = "hectic.services.project-zomboid.backup.s3.bucket is required when S3 upload is enabled.";
+      }
+      {
+        assertion = !backupCfg.s3.enable || backupCfg.s3.endpoint != null;
+        message = "hectic.services.project-zomboid.backup.s3.endpoint is required when S3 upload is enabled.";
+      }
+      {
+        assertion = !backupCfg.s3.enable || backupCfg.s3.region != null;
+        message = "hectic.services.project-zomboid.backup.s3.region is required when S3 upload is enabled.";
+      }
+      {
+        assertion =
+          !backupCfg.s3.enable
+          || backupCfg.s3.credentialsFile == null
+          || (
+            lib.hasPrefix "/" backupCfg.s3.credentialsFile
+            && !lib.hasPrefix "/nix/store/" backupCfg.s3.credentialsFile
+          );
+        message = "hectic.services.project-zomboid.backup.s3.credentialsFile must be a runtime path outside /nix/store.";
+      }
+      {
+        assertion =
+          !backupCfg.s3.enable
+          || backupCfg.s3.endpoint == null
+          || lib.hasPrefix "https://" backupCfg.s3.endpoint;
+        message = "hectic.services.project-zomboid.backup.s3.endpoint must use HTTPS.";
+      }
+    ];
+
     users.groups.project-zomboid = { };
     users.users.project-zomboid = {
       isSystemUser = true;
@@ -145,6 +397,11 @@ in {
     systemd.tmpfiles.rules = [
       "d ${cfg.dataDir} 0750 project-zomboid project-zomboid - -"
       "d ${cfg.installDir} 0750 project-zomboid project-zomboid - -"
+    ] ++ lib.optionals backupCfg.enable [
+      "d ${cfg.dataDir}/backups 0700 project-zomboid project-zomboid - -"
+      "Z ${cfg.dataDir}/backups 0700 project-zomboid project-zomboid - -"
+      "d ${backupCfg.stagingDir} 0700 project-zomboid project-zomboid - -"
+      "d ${backupCfg.archiveDir} 0700 project-zomboid project-zomboid - -"
     ];
 
     systemd.services.project-zomboid = {
@@ -186,6 +443,10 @@ in {
             ${pkgs.coreutils}/bin/printf '%s\n' '};';
           } > ${lib.escapeShellArg "${zomboidDir}/Server/${cfg.serverName}_SandboxVars.lua"}
         ''}
+        ${lib.optionalString (cfg.sandboxProperties == { }) ''
+          ${pkgs.coreutils}/bin/rm -f \
+            ${lib.escapeShellArg "${zomboidDir}/Server/${cfg.serverName}_SandboxVars.lua"}
+        ''}
       '';
 
       serviceConfig = {
@@ -205,6 +466,34 @@ in {
       };
     };
 
+    systemd.services.project-zomboid-backup = lib.mkIf backupCfg.enable {
+      description = "No-stop Project Zomboid backup";
+      after = [ "project-zomboid.service" ];
+      unitConfig.ConditionPathExists = [
+        saveDir
+        serverConfigDir
+      ];
+      serviceConfig = {
+        Type = "oneshot";
+        User = "project-zomboid";
+        Group = "project-zomboid";
+        ExecStart = backupScript;
+        TimeoutStartSec = "30min";
+        UMask = "0077";
+      } // lib.optionalAttrs backupCfg.s3.enable {
+        EnvironmentFile = s3CredentialsFile;
+      };
+    };
+
+    systemd.timers.project-zomboid-backup = lib.mkIf backupCfg.enable {
+      description = "Run Project Zomboid backup";
+      wantedBy = [ "timers.target" ];
+      timerConfig = {
+        OnCalendar = backupCfg.onCalendar;
+        Persistent = true;
+      };
+    };
+
     networking.firewall.allowedUDPPorts = lib.mkIf cfg.openFirewall [
       cfg.port
       cfg.udpPort

nixos/system/hectic-lab/hectic-lab.nix

@@ -106,8 +106,14 @@ in {
       memory = "3g";
       serverName = "servertest";
       serverPropertiesFile = /var/lib/project-zomboid/server-password.ini;
+      backup = {
+        enable = true;
+        onCalendar = "*:0/30";
+        retentionDays = 14;
+        s3.enable = false;
+      };
       serverProperties = {
-        Map = "vehicle_interior_arcadia75;Muldraugh, KY";
+        Map = "Muldraugh, KY";
         DoLuaChecksum = false;
         Public = true;
         AntiCheatSafety = 4;
@@ -127,54 +133,42 @@ in {
         AntiCheatNoClip = 4;
         AntiCheatServerCustomization = 4;
       };
+      workshopItems = [
+        "3676456221" # Lua Digital Watch Framework
+        "3600401184" # Realistic Temperature Mod
+      ];
+      mods = [
+        "\\LuaDigitalWatchUI"
+        "\\RC_RealisticColdMod"
+      ];
     sandboxProperties = {
       StartMonth = 12;
       StartDay = 1;
       WaterShut = 3;
-        WaterShutModifier = 60;
+      WaterShutModifier = 150;
       ElecShut = 3;
-        ElecShutModifier = 60;
+      ElecShutModifier = 150;
       MinutesPerPage = 0.5;
+      Zombies = 4;
+      ZombieConfig = {
+        PopulationMultiplier = 1.3;
+        PopulationStartMultiplier = 1.0;
+        PopulationPeakMultiplier = 1.0;
+        RespawnHours = 0.0;
+        RespawnUnseenHours = 0.0;
+        RespawnMultiplier = 0.0;
+        RedistributeHours = 0.0;
+      };
       ZombieLore = {
         Transmission = 4;
         Mortality = 7;
+        Speed = 2;
+        SprinterPercentage = 0;
+        Strength = 2;
+        Cognition = 2;
+        DoorOpeningPercentage = 10;
       };
     };
-      workshopItems = [
-        "3773972040" # Arcadia RV Interiors B42.20
-        "2210760610" # Cryogenic Winter +Easy/Hard Modes
-        "3676456221" # Lua Digital Watch Framework
-        "3600401184" # Realistic Temperature Mod
-        "3387824513" # Material Weight Reducer
-        "3413150945" # More Damaged Objects
-        "3512708849" # Shotgun Trajectory
-        "3401576145" # Firearm Models: Redux
-        "3401134276" # Vanilla Gear Expanded
-        "2956146279" # Rain Cleans Blood
-        "3693258802" # Tactical Hold
-        "3394588830" # Simple Flashlight on Belt
-        "2684285534" # Spongie's Clothing
-        "2812326159" # Spongie's Open Jackets
-      ];
-      mods = [
-        "\\ArcadiaRVInterior_B42_MP"
-        "\\ArcadiaRVInterior_B42_Vanilla"
-        "\\CryogenicWinter2NormalMode"
-        "\\LuaDigitalWatchUI"
-        "\\RC_RealisticColdMod"
-        "\\Material Weight Reducer"
-        "\\Ammunition Weight Reducer"
-        "\\MoreDamagedObjects"
-        "\\ShotgunTrajectory"
-        "\\FMR"
-        "\\VanillaGearExpanded"
-        "\\RainCleansBlood"
-        "\\TacHold Complete"
-        "\\LightOnBelt"
-        "\\SpnCloth"
-        "\\SpnOpenClothBase"
-        "\\SpnOpenCloth"
-      ];
   };
     services.p4d = {
       enable = true;

nixos/system/neuro/default.nix

@@ -19,6 +19,8 @@ in self.lib.nixpkgs-lib.nixosSystem {
       self.lib.cudaUnfreePredicate pkg || builtins.elem (self.lib.nixpkgs-lib.getName pkg) [
        "minecraft-server"
        "neoforge"
+       "steamcmd"
+       "steam-unwrapped"
 
        "nvidia-x11"
     ];

nixos/system/neuro/minecraft/wow-mine-map.nix

@@ -43,7 +43,7 @@
       } >> server.properties
     '';
     enable = true;
-    jvmOpts = "-Xmx8G -Xms2G";
+    jvmOpts = "-Xmx24G -Xms2G";
     # WorldOfSosal client and server use the same pinned NeoForge.
     package = pkgs.minecraftServers.neoforge-1_21_1.override (
       builtins.fromJSON (builtins.readFile ./neoforge-21.1.250.json)

nixos/system/neuro/neuro.nix

@@ -139,7 +139,7 @@ in {
   };
 
   services.nginx = {
-    enable = true;
+    enable = false;
     virtualHosts."bfs.band" = let
       site = pkgs.runCommand "bfs-band-site" {} ''
         mkdir -p $out
@@ -224,6 +224,70 @@ in {
     archetype.dev.enable  = true;
   };
 
+  hectic.services."project-zomboid" = {
+    enable = true;
+    memory = "8g";
+    serverName = "servertest";
+    serverPropertiesFile = /var/lib/project-zomboid/server-password.ini;
+    serverProperties = {
+      Map = "Muldraugh, KY";
+      DoLuaChecksum = false;
+      Public = true;
+      AntiCheatSafety = 4;
+      AntiCheatMovement = 4;
+      AntiCheatSpeed = 4;
+      AntiCheatHit = 4;
+      AntiCheatPacket = 4;
+      AntiCheatPacketException = 4;
+      AntiCheatPermission = 4;
+      AntiCheatXP = 4;
+      AntiCheatFire = 4;
+      AntiCheatSafeHouse = 4;
+      AntiCheatRecipe = 4;
+      AntiCheatPlayer = 4;
+      AntiCheatChecksum = 4;
+      AntiCheatItem = 4;
+      AntiCheatNoClip = 4;
+      AntiCheatServerCustomization = 4;
+    };
+    workshopItems = [
+      "3676456221" # Lua Digital Watch Framework
+      "3600401184" # Realistic Temperature Mod
+    ];
+    mods = [
+      "\\LuaDigitalWatchUI"
+      "\\RC_RealisticColdMod"
+    ];
+    sandboxProperties = {
+      Zombies = 6;
+      ZombieConfig = {
+        PopulationMultiplier = 0.0;
+        PopulationStartMultiplier = 0.0;
+        PopulationPeakMultiplier = 0.0;
+        RespawnHours = 0.0;
+        RespawnUnseenHours = 0.0;
+        RespawnMultiplier = 0.0;
+        RedistributeHours = 0.0;
+      };
+    };
+  };
+
+  systemd.services.project-zomboid.preStart = lib.mkBefore ''
+    password_file=${lib.escapeShellArg "/var/lib/project-zomboid/server-password"}
+    properties_file=${lib.escapeShellArg "/var/lib/project-zomboid/server-password.ini"}
+
+    if [ ! -s "$password_file" ] || ! ${pkgs.gnugrep}/bin/grep -Eq '^[0-9a-f]{48}$' "$password_file"; then
+      umask 077
+      ${pkgs.openssl}/bin/openssl rand -hex 24 > "$password_file"
+    fi
+    ${pkgs.coreutils}/bin/chmod 0600 "$password_file"
+
+    properties_file_tmp="$( ${pkgs.coreutils}/bin/mktemp "$(dirname "$properties_file")/.server-password.ini.XXXXXX")"
+    ${pkgs.coreutils}/bin/printf 'Password=%s\n' "$(<"$password_file")" > "$properties_file_tmp"
+    ${pkgs.coreutils}/bin/chmod 0600 "$properties_file_tmp"
+    ${pkgs.coreutils}/bin/mv "$properties_file_tmp" "$properties_file"
+  '';
+
   sops = {
     gnupg.sshKeyPaths         = [ ];
     age.sshKeyPaths           = [ "/etc/ssh/ssh_host_ed25519_key" ];