feat(kiosk-gene-desk): disko/impermanence rewrite + disaster recovery#693
Merged
Conversation
Adds a local aarch64-linux/x86_64-linux builder VM (Apple Virtualization framework via nix.linux-builder) so aarch64-linux builds - like kiosk-gene-desk's - don't have to go over the network to hetznix02. Started/stopped manually via a `linux-builder` wrapper script rather than always-on (KeepAlive/RunAtLoad forced false), since it's only needed occasionally and would otherwise keep 4 cores/6GB running permanently in the background. Its qcow2 disk persists across start/stop either way. Also adds nixos-anywhere to systemPackages, for remote NixOS installation over SSH.
… regression it introduced Replaces the old hand-rolled sd-image build (custom sfdisk/resize2fs/ mkfs.f2fs scripting, read-only-root.nix's ad hoc tmpfs overlay) with a declarative disko partition layout + impermanence-managed tmpfs root, installed via nixos-anywhere instead of manually flashing an image. Partitions are GPT with explicit priority, since the Pi's boot ROM always reads physical partition 1 for its own firmware regardless of partition labels. Hardware/bootloader support moves from nixos-hardware's raspberry-pi-4 bundle to nixos-raspberrypi, which provides vendor kernel/firmware plus the declarative boot.loader.raspberry-pi module disko needs (kept in sync on every switch, instead of nixpkgs' sd-image module's one-shot image population). sops secrets now bootstrap via nixos-anywhere's --extra-files (scripts/prep-install-bootstrap.sh derives the age key from a per-host SSH key already in modules/shared/secrets.yaml), since disko wipes /persist from scratch on every install and sops-nix has no other way to recover if its key is missing on first boot. age.keyFile points directly at /persist rather than the impermanence-bind-mounted alias - sops-nix's setupSecrets runs during initrd, earlier than impermanence can bind-mount an arbitrary path - and lives at a flat path with no shared ancestor with anything impermanence creates, since nixos-anywhere's extra-files copy (running as root, before nixos-install's own activation) would otherwise create that ancestor as root:root as a side effect, permanently out of impermanence's reach to fix (confirmed on hardware: even systemd-tmpfiles refuses to correct it after the fact, with a hardcoded "unsafe path transition" symlink-attack safety check). Switching to impermanence is also what caused chromium to start crashing on this kiosk (SIGTRAP on cage-tty1) - it never did on the old read-only- root.nix setup, which had no bind-mount-ownership mechanism for chromium to trip over in the first place. Its crash reporter (chrome_crashpad_handler) hard-requires being able to mkdir ~/.config/chromium/Crash Reports on every launch, and stale root:root ownership on impermanence-managed directories (created once, before ownership was fixed, and never re-owned since impermanence only sets ownership on directories it creates fresh) broke that outright - confirmed via coredumpctl and strace after an earlier, wrong hypothesis (EDID/KMS misconfiguration) turned out to have no effect on hardware. Fixed with systemd.tmpfiles.rules `z` rules forcing ownership every boot, and (for a second, independently-confirmed instance of the same race - $HOME itself, breaking home-manager and anything gene tried to create directly under it) an ExecStartPre hook on cage-tty1 and home-manager-gene that re-applies those rules immediately before each service starts, since systemd-tmpfiles-setup.service itself only runs once, early in boot, and doesn't get a second chance if whatever creates a given directory fresh loses the race against it.
…stence Restic backup/restore for the three things on this kiosk that would be a real hassle to reconstruct by hand after a card death, given it normally has no keyboard attached: hass-browser_mod's chromium device registration, atuin's sync-server login, and tailscale's node identity. Backed up from their real /persist-side paths directly rather than the impermanence bind-mounted aliases under $HOME. genebean.programs.kiosk-restic-full-restore (modules/genebean/nixos/ programs/) is the recovery side - a package taking restorePaths and stopServices as options rather than hardcoding this host, so any kiosk running the same cage+chromium(+tailscale) setup can reuse it. It restores by $(hostname) and the snapshot's own recorded paths, rather than duplicating services.restic.backups.daily.paths into an --include list, and deliberately doesn't default to "latest" - a boot-triggered catch-up backup can capture empty post-reinstall state and become "latest" before there's a chance to restore, silently making it the wrong snapshot. scripts/prep-install-bootstrap.sh now tags the last 5 snapshots for the host being reinstalled as pre-reinstall right before generating the bootstrap files, protecting them from that same --keep-daily 7 rule legitimately forgetting the real pre-reinstall snapshot as superseded within the same calendar day. Two prune-only jobs manage that tag's lifecycle: a per-host 45-day expiry (kiosk-gene-desk/persistence.nix) and a fleet-wide 1-year backstop (modules/shared/nixos/restic.nix) for anything that slips through - e.g. a host decommissioned before its own cleanup ran. --keep-tag pre-reinstall has to live in the shared module rather than per-host: none of these forget invocations are scoped by --host, so any host's prune run processes retention for every host's snapshots, and the exemption needs to be honored no matter whose prune job happens to run first. The shared restic module's default Persistent = true also fires a catch-up backup immediately on the same boot a job is first created, so every timerConfig here explicitly opts out of that - a missed day is a non-issue on hosts backed up daily. Also persists atuin's local history db and sync-server login/key across reboots (previously wiped every time, along with everything else on this host's tmpfs root), redirects restic's local index cache off the 512M tmpfs root (it holds the index for the entire shared repo, not just this host's data, and filled the tmpfs to 100% in testing), rotates the text console to match the GUI's own independent rotation (fbcon=rotate:3 - kiosk.sh's wlr-randr transform only affects the Wayland/cage output), and persists systemd-timesyncd's clock file so this Pi's clock (no RTC) has a real floor at boot instead of starting from the kernel/image build's fallback date until NTP catches up.
genebean
force-pushed
the
feat/kiosk-gene-desk-impermanence
branch
from
July 19, 2026 01:09
fc391ab to
029313f
Compare
genebean
changed the title
… kiosks Replaces the old image-flashing steps (obsolete now that kiosk-gene-desk runs on disko + impermanence, not a flashed system image) and the "not something I have properly documented" placeholder for everything else, with the actual process: build the pre-decompressed installer image, flash it to USB (not the SD card), boot with a display attached and the SD card removed (the installer sets a random root password on boot, and a Pi's boot order can otherwise prefer the SD card over USB), seed the sops/timesync/pre-reinstall-tag bootstrap files, install via nixos-anywhere, and restore state via kiosk-restic-full-restore using the specific pre-reinstall-tagged snapshot rather than "latest".
genebean
enabled auto-merge
July 19, 2026 02:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
chrome_crashpad_handlercouldn't create its crash-report directory under stale root-owned persisted paths).genebean.programs.kiosk-restic-full-restore) is generalized for reuse on other kiosks, not hardcoded to this host.mightymac: a local aarch64-linux/x86_64-linux builder VM andnixos-anywhere, both needed to build/deploy this rewrite.Test plan
nixos-anywhere+scripts/prep-install-bootstrap.sh, confirmed clean bootkiosk-restic-full-restorerestore cycle validated end-to-end on hardware (multiple times, including the pre-reinstall snapshot-tagging safeguard)nix flake checkand fullnixosConfigurations.kiosk-gene-deskbuild pass