diff --git a/.gitignore b/.gitignore index a61b0969..b9955725 100644 --- a/.gitignore +++ b/.gitignore @@ -11,6 +11,9 @@ # From running nixos-rebuild build-vm *.qcow2 +# From scripts/prep-install-bootstrap.sh - contains a decrypted age key +/nixos-anywhere-extras* + .claude/* !.claude/skills/ !.claude/skills/** diff --git a/README.md b/README.md index 0441c14d..d90db1fc 100644 --- a/README.md +++ b/README.md @@ -7,8 +7,7 @@ This repo is a Nix flake that manages most of my setup on macOS and fully manage - [Historical bits](#historical-bits) - [Host Bootstrapping](#host-bootstrapping) - [Replacements](#replacements) - - [Image-based Systems](#image-based-systems) - - [Other Systems](#other-systems) + - [Raspberry Pi kiosks (`kiosk-gene-desk`, and any future host built the same way)](#raspberry-pi-kiosks-kiosk-gene-desk-and-any-future-host-built-the-same-way) - [Net-new Hosts](#net-new-hosts) - [Adding a new macOS host](#adding-a-new-macos-host) - [Extras steps not done by Nix and/or Homebrew and/or mas](#extras-steps-not-done-by-nix-andor-homebrew-andor-mas) @@ -62,32 +61,89 @@ This repo historically contained my dot files. Historically symlinked files on W Sometimes hosts, or their storage, need replacing... sepcially ones that run on SD cards like `kiosk-gene-desk`. When that time comes, here is how to get it back up and running. -#### Image-based Systems +#### Raspberry Pi kiosks (`kiosk-gene-desk`, and any future host built the same way) + +These hosts use `disko` for partitioning, `impermanence` for a tmpfs root +(wiped every boot, with an explicit persist allowlist), and +`nixos-anywhere` for install - not a flashed system image. `disko` wipes +the target disk from scratch every time, so this is the same process for +a first-time install and a full replacement (dead SD card, etc). + +1. **Build the installer image** (on mightymac, or anywhere with the + flake): `nix build .#rpi4-installer`. This is a generic + `nixos-raspberrypi` installer image, pre-decompressed so Raspberry Pi + Imager can flash it directly - it silently corrupts the card if you + hand it the raw `.img.zst` instead, since Imager doesn't decompress + zstd itself. `result` is the flashable `.img` file. + +2. **Flash it to a USB drive**, not the SD card - Raspberry Pi Imager, + "Use custom image", pick `result`. + +3. **Connect a display and boot the Pi from that USB drive with the SD + card removed.** Both matter: `nixos-anywhere`/`disko` can't + repartition a disk the system is currently running from, and a Pi's + boot order can prefer the SD card over USB if one's already inserted - + booting from USB with no SD card present avoids that ambiguity + entirely. A display is required too: the installer sets a random root + password on boot and prints it on screen along with its IP - read both + off the display. You'll need the password again for `nixos-anywhere` + in step 5. Once it's up, insert the SD card - it's just a second, + currently-empty disk to the running installer at this point, safe to + insert at any time after boot. + +4. **Seed the bootstrap files.** `disko` wipes `/persist` from scratch, + so the sops age key has to exist on the target *before* + `nixos-install`'s activation runs, or `sops-install-secrets` fails on + first boot with no way to recover short of a manual SSH patch. From + the repo root: -1. install image -2. boot with wired connection -3. ssh in but don’t use known hosts file -4. restore user and host ssh keys -5. run `mkdir -p ~/.config/sops/age && ssh-to-age -private-key -i ~/.ssh/id_ed25519 > ~/.config/sops/age/keys.txt && ssh-to-age -i ~/.ssh/id_ed25519.pub > ~/.config/sops/age/pub-keys.txt` -6. reboot -7. ssh in as normal -8. run these commands: + ```bash + scripts/prep-install-bootstrap.sh + ``` + + This derives the age key from a per-host SSH key already in + `modules/shared/secrets.yaml`, seeds a clock file so the fresh boot's + NTP sync starts from a real floor instead of the image build's + fallback date (no RTC on these Pis), and tags the last 5 restic + snapshots already in the backup repo for that hostname as + `pre-reinstall`, so they survive the fresh install's own backup + rotation - this talks to the repo directly (mightymac is already a + valid recipient for the restic secrets), so it works regardless of + whether the host being replaced is still reachable. Confirm the + derived age recipient printed at the end matches the host's entry in + `.sops.yaml` before continuing. + +5. **Install:** ```bash - mkdir repos - cd repos - git clone git@github.com:genebean/dots - cd dots - nix-auth login - nix flake update private-flake # needed so private bits are cached properly - nixup + nixos-anywhere --flake ~/repos/dots# \ + --extra-files ./nixos-anywhere-extras- root@ ``` -9. reboot + It'll prompt for the root password from step 3 to make its initial SSH + connection, then reboots into the real system when done. -#### Other Systems +6. **Restore state.** SSH in once it's back up, then: + + ```bash + sudo kiosk-restic-full-restore # lists available snapshots + sudo kiosk-restic-full-restore # restores from a specific one + ``` -Yeah.... this is not something I have properly documented. Best guess: install like a net-new host but then restore keys and such like on an image based system. Supplement that with restores from restic backups. + Use the specific `pre-reinstall`-tagged snapshot ID from step 4, not + whatever's newest - a boot-triggered catch-up backup on the fresh + install can capture empty state and become "latest" before you get a + chance to restore, so the tool never guesses for you. This restores + Home Assistant's `hass-browser_mod` chromium registration, atuin's + sync-server login, and the tailscale node identity, and stops/starts + the affected services itself. + +7. Verify: chromium shows the kiosk page (browser_mod registered in Home + Assistant), `atuin status` shows a recent sync, `tailscale status` + shows the expected node name (if it comes up suffixed like + `-1`, the old pre-replacement node is probably still listed + as active in the Tailscale admin console under the plain name - remove + it there). ### Net-new Hosts diff --git a/flake.lock b/flake.lock index 6353737f..48a99948 100644 --- a/flake.lock +++ b/flake.lock @@ -1,5 +1,21 @@ { "nodes": { + "argononed": { + "flake": false, + "locked": { + "lastModified": 1729566243, + "narHash": "sha256-DPNI0Dpk5aym3Baf5UbEe5GENDrSmmXVdriRSWE+rgk=", + "owner": "nvmd", + "repo": "argononed", + "rev": "16dbee54d49b66d5654d228d1061246b440ef7cf", + "type": "github" + }, + "original": { + "owner": "nvmd", + "repo": "argononed", + "type": "github" + } + }, "blobs": { "flake": false, "locked": { @@ -132,6 +148,28 @@ "type": "github" } }, + "disko_2": { + "inputs": { + "nixpkgs": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1781152676, + "narHash": "sha256-RxWs5ND31KzTG7wvMM+PMfUjyNpmIEr999lqNARaM5o=", + "owner": "nix-community", + "repo": "disko", + "rev": "ff8702b4de27f72b4c78573dfb89ec74e36abdf1", + "type": "github" + }, + "original": { + "owner": "nix-community", + "ref": "master", + "repo": "disko", + "type": "github" + } + }, "fenix": { "inputs": { "nixpkgs": [ @@ -209,6 +247,21 @@ } }, "flake-compat_3": { + "locked": { + "lastModified": 1767039857, + "narHash": "sha256-vNpUSpF5Nuw8xvDLj2KCwwksIbjua2LZCqhV1LNRDns=", + "owner": "edolstra", + "repo": "flake-compat", + "rev": "5edf11c44bc78a0d334f6334cdaf7d60d732daab", + "type": "github" + }, + "original": { + "owner": "edolstra", + "repo": "flake-compat", + "type": "github" + } + }, + "flake-compat_4": { "flake": false, "locked": { "lastModified": 1767039857, @@ -464,6 +517,29 @@ "type": "github" } }, + "impermanence": { + "inputs": { + "home-manager": [ + "home-manager" + ], + "nixpkgs": [ + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1769548169, + "narHash": "sha256-03+JxvzmfwRu+5JafM0DLbxgHttOQZkUtDWBmeUkN8Y=", + "owner": "nix-community", + "repo": "impermanence", + "rev": "7b1d382faf603b6d264f58627330f9faa5cba149", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "impermanence", + "type": "github" + } + }, "nix-auth": { "inputs": { "flake-parts": "flake-parts", @@ -579,6 +655,27 @@ "type": "github" } }, + "nix-vm-test": { + "inputs": { + "nixpkgs": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1783265394, + "narHash": "sha256-cq4YfNFGYzp0NItZP8tEC7xUI8OSgY4fj75AU/NSaPM=", + "owner": "numtide", + "repo": "nix-vm-test", + "rev": "1a587212d2ac8b669c6c32499015f996506b6ba5", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "nix-vm-test", + "type": "github" + } + }, "nix-wrapper-modules": { "inputs": { "nixpkgs": [ @@ -600,6 +697,31 @@ "type": "github" } }, + "nixos-anywhere": { + "inputs": { + "disko": "disko_2", + "nix-vm-test": "nix-vm-test", + "nixos-images": "nixos-images", + "nixos-stable": "nixos-stable", + "nixpkgs": [ + "nixpkgs" + ], + "treefmt-nix": "treefmt-nix_2" + }, + "locked": { + "lastModified": 1784099966, + "narHash": "sha256-h+q9j0ivHhll0sXIPCuJJ+Ivcqh7QDL8eulVUl+CqTQ=", + "owner": "nix-community", + "repo": "nixos-anywhere", + "rev": "7239104f1a38546b999cd817658407d80f56e7db", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "nixos-anywhere", + "type": "github" + } + }, "nixos-cosmic": { "inputs": { "flake-compat": "flake-compat_2", @@ -644,6 +766,94 @@ "type": "github" } }, + "nixos-images": { + "inputs": { + "nixos-stable": [ + "nixos-anywhere", + "nixos-stable" + ], + "nixos-unstable": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1783593136, + "narHash": "sha256-zy5an02BdZ65OgVKdRkz2TpbdBrsW+uQD7AA2wLuiTM=", + "owner": "nix-community", + "repo": "nixos-images", + "rev": "803f28511c7d5f39f2537c342122fd94b8e1d519", + "type": "github" + }, + "original": { + "owner": "nix-community", + "repo": "nixos-images", + "type": "github" + } + }, + "nixos-images_2": { + "inputs": { + "nixos-stable": [ + "nixos-raspberrypi", + "nixpkgs" + ], + "nixos-unstable": [ + "nixos-raspberrypi", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1747747741, + "narHash": "sha256-LUOH27unNWbGTvZFitHonraNx0JF/55h30r9WxqrznM=", + "owner": "nvmd", + "repo": "nixos-images", + "rev": "cbbd6db325775096680b65e2a32fb6187c09bbb4", + "type": "github" + }, + "original": { + "owner": "nvmd", + "ref": "sdimage-installer", + "repo": "nixos-images", + "type": "github" + } + }, + "nixos-raspberrypi": { + "inputs": { + "argononed": "argononed", + "flake-compat": "flake-compat_3", + "nixos-images": "nixos-images_2", + "nixpkgs": "nixpkgs_4" + }, + "locked": { + "lastModified": 1783956693, + "narHash": "sha256-3qkLbPLBaSzeLsg30WuMV7R8Xg7mFstqYfcjnMt2UcE=", + "owner": "nvmd", + "repo": "nixos-raspberrypi", + "rev": "cd985a201b8da917676b10125d97a20fed9736db", + "type": "github" + }, + "original": { + "owner": "nvmd", + "repo": "nixos-raspberrypi", + "type": "github" + } + }, + "nixos-stable": { + "locked": { + "lastModified": 1784011430, + "narHash": "sha256-lDebytrYdd47IBLwvNOD+6AGeoqZ78CIKlp70hzW280=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "8eeec934ae0dbeca3d7868c059568a65c08b2fc3", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-26.05", + "repo": "nixpkgs", + "type": "github" + } + }, "nixpkgs": { "locked": { "lastModified": 1742889210, @@ -721,6 +931,22 @@ } }, "nixpkgs_4": { + "locked": { + "lastModified": 1783703440, + "narHash": "sha256-O3/YajjWo001VUIgD8BwaRdSNLUFe7nZ1qV5TwhRBcw=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "8f0500b9660505dc3cb647775fe9a978a74b5283", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-26.05", + "repo": "nixpkgs", + "type": "github" + } + }, + "nixpkgs_5": { "locked": { "lastModified": 1784011430, "narHash": "sha256-lDebytrYdd47IBLwvNOD+6AGeoqZ78CIKlp70hzW280=", @@ -841,13 +1067,16 @@ "genebean-omp-themes": "genebean-omp-themes", "hermes-social-summerizer": "hermes-social-summerizer", "home-manager": "home-manager", + "impermanence": "impermanence", "nix-auth": "nix-auth", "nix-darwin": "nix-darwin", "nix-flatpak": "nix-flatpak", "nix-homebrew": "nix-homebrew", + "nixos-anywhere": "nixos-anywhere", "nixos-cosmic": "nixos-cosmic", "nixos-hardware": "nixos-hardware", - "nixpkgs": "nixpkgs_4", + "nixos-raspberrypi": "nixos-raspberrypi", + "nixpkgs": "nixpkgs_5", "nixpkgs-unstable": "nixpkgs-unstable", "plasma-manager": "plasma-manager", "private-flake": "private-flake", @@ -916,7 +1145,7 @@ "simple-nixos-mailserver": { "inputs": { "blobs": "blobs", - "flake-compat": "flake-compat_3", + "flake-compat": "flake-compat_4", "git-hooks": "git-hooks", "nixpkgs": [ "nixpkgs" @@ -1041,6 +1270,27 @@ "type": "github" } }, + "treefmt-nix_2": { + "inputs": { + "nixpkgs": [ + "nixos-anywhere", + "nixpkgs" + ] + }, + "locked": { + "lastModified": 1780220602, + "narHash": "sha256-eynAfOmbmxJnkp7YewvCEbShNnnYJ9gLLqkzsYtBPeM=", + "owner": "numtide", + "repo": "treefmt-nix", + "rev": "db947814a175b7ca6ded66e21383d938df01c227", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "treefmt-nix", + "type": "github" + } + }, "utils": { "inputs": { "systems": "systems_2" diff --git a/flake.nix b/flake.nix index 587c34c9..90111197 100644 --- a/flake.nix +++ b/flake.nix @@ -1,5 +1,17 @@ { description = "A flake for all my stuff"; + # nixos-raspberrypi's own binary cache (pre-built vendor kernel/firmware + # for kiosk-gene-desk). Only inherited automatically when building *from + # their flake directly*, not when consumed as an input like here - hence + # declaring it ourselves. This is a build-time/client-side setting; it + # does not help kiosk-gene-desk's own future rebuilds once deployed - + # see its nix.settings.extra-substituters for that. + nixConfig = { + extra-substituters = [ "https://nixos-raspberrypi.cachix.org" ]; + extra-trusted-public-keys = [ + "nixos-raspberrypi.cachix.org-1:4iMO9LXa8BqhU+Rpg6LQKiGa2lsNh/j2oiYLNOQ5sPI=" + ]; + }; inputs = { # Where we get most of our software. Giant mono repo with recipes # called derivations that say how to build software. @@ -54,6 +66,16 @@ inputs.nixpkgs.follows = "nixpkgs"; }; + # Wipes / to tmpfs on every boot, bind-mounting an explicit allowlist + # of paths back in from a persistent partition + impermanence = { + url = "github:nix-community/impermanence"; + inputs = { + nixpkgs.follows = "nixpkgs"; + home-manager.follows = "home-manager"; + }; + }; + nix-auth = { url = "github:numtide/nix-auth"; inputs.nixpkgs.follows = "nixpkgs"; @@ -73,6 +95,11 @@ url = "github:zhaofengli-wip/nix-homebrew"; }; + nixos-anywhere = { + url = "github:nix-community/nixos-anywhere"; + inputs.nixpkgs.follows = "nixpkgs"; + }; + nixos-cosmic = { url = "github:lilyinstarlight/nixos-cosmic"; inputs.nixpkgs-stable.follows = "nixpkgs"; @@ -82,6 +109,14 @@ nixos-hardware.url = "github:NixOS/nixos-hardware/master"; + # Raspberry Pi hardware support: vendor kernel/firmware packages and a + # proper declarative bootloader (boot.loader.raspberry-pi) that keeps + # /boot/firmware in sync on every switch, instead of the one-shot + # image population nixpkgs' own sd-image module does. Deliberately + # not following our nixpkgs - its vendor kernel/firmware overlays are + # pinned to whatever nixpkgs revision it ships with. + nixos-raspberrypi.url = "github:nvmd/nixos-raspberrypi"; + plasma-manager = { url = "github:nix-community/plasma-manager"; inputs = { @@ -196,9 +231,20 @@ system = "aarch64-linux"; hostname = "kiosk-gene-desk"; additionalModules = [ - inputs.nixos-hardware.nixosModules.raspberry-pi-4 + inputs.impermanence.nixosModules.impermanence + inputs.nixos-raspberrypi.lib.inject-overlays + inputs.nixos-raspberrypi.nixosModules.nixpkgs-rpi + # Replaces inputs.nixos-hardware.nixosModules.raspberry-pi-4 - + # nixos-raspberrypi provides the same hardware support plus the + # declarative boot.loader.raspberry-pi module disko needs. + inputs.nixos-raspberrypi.nixosModules.raspberry-pi-4.base + inputs.nixos-raspberrypi.nixosModules.raspberry-pi-4.display-vc4 + inputs.nixos-raspberrypi.nixosModules.trusted-nix-caches inputs.private-flake.nixosModules.private.kiosk ]; + additionalSpecialArgs = { + inherit (inputs) nixos-raspberrypi; + }; }; nixnuc = localLib.mkNixosHost { hostname = "nixnuc"; @@ -241,7 +287,7 @@ let pkgs = nixpkgs.legacyPackages.${system}; in - import ./pkgs { inherit pkgs self; } + import ./pkgs { inherit inputs pkgs; } ); }; } diff --git a/modules/genebean/nixos/default.nix b/modules/genebean/nixos/default.nix index 62f73de0..cc7cae62 100644 --- a/modules/genebean/nixos/default.nix +++ b/modules/genebean/nixos/default.nix @@ -5,6 +5,7 @@ ./programs/askpass.nix ./programs/boinc.nix ./programs/firefox.nix + ./programs/kiosk-restic-full-restore.nix ./programs/onepassword.nix ./programs/thunderbird.nix ./programs/xfce4-terminal.nix diff --git a/modules/genebean/nixos/programs/kiosk-restic-full-restore.nix b/modules/genebean/nixos/programs/kiosk-restic-full-restore.nix new file mode 100644 index 00000000..a37f8f71 --- /dev/null +++ b/modules/genebean/nixos/programs/kiosk-restic-full-restore.nix @@ -0,0 +1,93 @@ +# Disaster-recovery tool for kiosks running cage+chromium(+tailscale): pulls +# hass-browser_mod's chromium state, atuin's login, and similar back from +# restic after a card death or fresh reinstall. Not for servers - the +# services.restic.backups. module already covers those on its own. +{ + config, + lib, + pkgs, + username, + ... +}: +let + cfg = config.genebean.programs.kiosk-restic-full-restore; +in +{ + options.genebean.programs.kiosk-restic-full-restore = { + enable = lib.mkEnableOption "kiosk-restic-full-restore recovery tool"; + + backupName = lib.mkOption { + type = lib.types.str; + default = "daily"; + description = "Name of the services.restic.backups. job to restore from."; + }; + + restorePaths = lib.mkOption { + type = lib.types.listOf lib.types.str; + description = "Paths to restore from the chosen snapshot (normally the same list passed to services.restic.backups..paths)."; + example = [ "/persist/home/gene/.config/chromium" ]; + }; + + stopServices = lib.mkOption { + type = lib.types.listOf lib.types.str; + default = [ ]; + description = "systemd services to stop before restoring and start again afterward, e.g. the kiosk browser session and tailscaled - anything that holds the restored files open."; + example = [ + "cage-tty1" + "tailscaled" + ]; + }; + }; + + config = lib.mkIf cfg.enable { + environment.systemPackages = [ + (pkgs.writeShellApplication { + name = "kiosk-restic-full-restore"; + text = '' + if [ $# -eq 0 ]; then + # No snapshot chosen yet - show what's available and stop. Doesn't + # default to "latest": confirmed on hardware that a boot-triggered + # catch-up backup capturing empty post-reinstall state can become + # "latest" before there's a chance to restore, silently making + # "latest" the wrong snapshot to restore from. + restic-${cfg.backupName} snapshots --host "$(hostname)" + echo + echo "Usage: kiosk-restic-full-restore " + exit 0 + fi + snapshot="$1" + + echo "Restoring snapshot '$snapshot' (host: $(hostname)) for:" + ${lib.concatMapStringsSep "\n" (p: "echo ' ${p}'") cfg.restorePaths} + echo + read -r -p "This overwrites the current state at these paths. Continue? [y/N] " confirm + [ "$confirm" = "y" ] || exit 1 + + ${lib.optionalString ( + cfg.stopServices != [ ] + ) "systemctl stop ${lib.concatStringsSep " " cfg.stopServices}"} + + # atuin isn't a service - each shell command spawns a short-lived + # `atuin history end` process to record it. One of these caught mid- + # write during a live restore held its old (now-deleted) sqlite file + # open, and new `atuin` invocations hung with "pool timed out + # waiting for an open connection" until it was killed. Harmless to + # kill - it's just finishing writing one command's exit status. + pkill -u ${username} -x atuin || true + + restic-${cfg.backupName} restore "$snapshot" --host "$(hostname)" --target / \ + ${lib.concatMapStringsSep " " (p: "--include ${lib.escapeShellArg p}") cfg.restorePaths} + + ${lib.optionalString ( + cfg.stopServices != [ ] + ) "systemctl start ${lib.concatStringsSep " " cfg.stopServices}"} + echo "Restored${ + lib.optionalString ( + cfg.stopServices != [ ] + ) " and restarted ${lib.concatStringsSep " + " cfg.stopServices}" + }." + ''; + }) + ]; + }; +} diff --git a/modules/hosts/darwin/mightymac/default.nix b/modules/hosts/darwin/mightymac/default.nix index 70b4a559..e36d1c17 100644 --- a/modules/hosts/darwin/mightymac/default.nix +++ b/modules/hosts/darwin/mightymac/default.nix @@ -1,16 +1,62 @@ -{ inputs, pkgs, ... }: +{ + inputs, + lib, + pkgs, + ... +}: { system.stateVersion = 4; + # Local aarch64-linux/x86_64-linux builder VM (Apple Virtualization + # framework) so aarch64-linux builds like kiosk-gene-desk's don't have + # to go over the network to hetznix02. Started/stopped manually via the + # `linux-builder` wrapper below rather than always-on, since it's + # only needed occasionally - KeepAlive/RunAtLoad would otherwise keep + # it (and its 4 cores/6GB) running in the background permanently. + # Its qcow2 disk (and anything already built on it) persists across + # start/stop either way; it's only lost if nix.linux-builder.enable + # is set back to false. + launchd.daemons.linux-builder.serviceConfig = { + KeepAlive = lib.mkForce false; + RunAtLoad = lib.mkForce false; + }; + + nix.linux-builder = { + enable = true; + config = { + virtualisation.cores = 4; + virtualisation.darwin-builder.memorySize = 6 * 1024; + }; + }; + environment = { systemPackages = with pkgs; [ chart-testing golangci-lint goreleaser inputs.flox.packages.${pkgs.stdenv.hostPlatform.system}.default + inputs.nixos-anywhere.packages.${pkgs.stdenv.hostPlatform.system}.default inputs.viscosity-cli.packages.${pkgs.stdenv.hostPlatform.system}.default kopia kubectx + (writeShellScriptBin "linux-builder" '' + set -euo pipefail + case "''${1:-}" in + start) + sudo launchctl kickstart -k system/org.nixos.linux-builder + ;; + stop) + sudo launchctl bootout system/org.nixos.linux-builder + ;; + status) + sudo launchctl print system/org.nixos.linux-builder 2>&1 | grep -i state || echo "not loaded" + ;; + *) + echo "usage: linux-builder start|stop|status" >&2 + exit 1 + ;; + esac + '') #reposurgeon # Nix is a major version behind brew rpiboot step-cli diff --git a/modules/hosts/nixos/default.nix b/modules/hosts/nixos/default.nix index 6a3c76c1..5c31d7ad 100644 --- a/modules/hosts/nixos/default.nix +++ b/modules/hosts/nixos/default.nix @@ -61,6 +61,7 @@ "https://cache.thalheim.io" "https://cosmic.cachix.org/" "https://nix-community.cachix.org" + "https://nixos-raspberrypi.cachix.org" ]; extra-trusted-public-keys = [ "cache.thalheim.io-1:R7msbosLEZKrxk/lKxf9BTjOOH7Ax3H0Qj0/6wiHOgc=" @@ -68,6 +69,7 @@ "flox-cache-public-1:7F4OyH7ZCnFhcze3fJdfyXYLQw/aV7GEed86nQ7IsOs=" "niks3.numtide.com-1:DTx8wZduET09hRmMtKdQDxNNthLQETkc/yaX7M4qK0g=" "nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=" + "nixos-raspberrypi.cachix.org-1:4iMO9LXa8BqhU+Rpg6LQKiGa2lsNh/j2oiYLNOQ5sPI=" ]; trusted-users = [ "${username}" ]; }; diff --git a/modules/hosts/nixos/kiosk-gene-desk/default.nix b/modules/hosts/nixos/kiosk-gene-desk/default.nix index 10890d81..ac031dd1 100644 --- a/modules/hosts/nixos/kiosk-gene-desk/default.nix +++ b/modules/hosts/nixos/kiosk-gene-desk/default.nix @@ -1,5 +1,4 @@ { - inputs, config, lib, pkgs, @@ -8,30 +7,40 @@ }: { imports = [ - # SD card image - "${inputs.nixpkgs}/nixos/modules/installer/sd-card/sd-image-aarch64.nix" - ./read-only-root.nix + ./disko.nix + ./persistence.nix + ../../../shared/nixos/restic.nix ]; system.stateVersion = "24.11"; - boot.supportedFilesystems = lib.mkForce [ - "vfat" - "ext4" - ]; + boot = { + # Independent of the GUI's own rotation (kiosk.sh's `wlr-randr + # --output HDMI-A-1 --transform 90`, which only affects the + # Wayland/cage compositor output) - this is the text console (fbcon), + # visible before/outside of cage. 0=normal, 1=90cw, 2=180, 3=270cw. + kernelParams = [ "fbcon=rotate:3" ]; + loader.raspberry-pi = { + enable = true; + variant = "4"; + # bootloader defaults to "uboot" for variant 4 + }; + supportedFilesystems = lib.mkForce [ + "ext4" + "f2fs" # /persist + "vfat" + ]; + }; environment.systemPackages = with pkgs; [ libraspberrypi raspberrypi-eeprom - raspberrypifw - ubootRaspberryPi4_64bit wlr-randr ]; hardware = { - enableRedistributableFirmware = true; + enableRedistributableFirmware = true; # wifi firmware, not boot firmware graphics.enable = true; - raspberry-pi."4".fkms-3d.enable = true; }; networking = { @@ -76,8 +85,6 @@ }) ]; - sdImage.compressImage = true; - services = { cage = let @@ -108,14 +115,63 @@ }; }; + # Public half of the sops-provided host key below - not secret, so it + # doesn't need to round-trip through /persist or sops. + environment.etc."ssh/ssh_host_ed25519_key.pub".text = + "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMSKkIHEABAkf/40QZTaJlrcoz1SmlG9nkQBxA/8HtGX root@kiosk-gene-desk\n"; + + services.openssh.hostKeys = [ + { + path = config.sops.secrets.ssh_host_ed25519_key.path; + type = "ed25519"; + } + ]; + sops = { - age.keyFile = "${config.users.users.${username}.home}/.config/sops/age/keys.txt"; + # Points directly at /persist rather than the impermanence-bind-mounted + # ~/.config/sops alias: sops-nix's setupSecrets activation script runs + # during initrd (confirmed on real hardware), which is earlier than + # impermanence can ever bind-mount an arbitrary path like this one - + # only a fixed set of core system paths qualify for that early stage + # (see nixos/lib/utils.nix's pathsNeededForBoot). /persist itself is + # already mounted by then (neededForBoot = true), so pointing here + # directly sidesteps the timing race entirely. + # + # Deliberately NOT under .config: scripts/prep-install-bootstrap.sh + # pre-seeds this file via nixos-anywhere's --extra-files *before* + # nixos-install's activation ever runs, and that copy step (running + # as root) creates .config itself as root:root as a side effect - + # confirmed on a real fresh install, where impermanence's directory + # creation happily created .config/chromium under it but never got + # the chance to fix .config's own ownership, since it already + # existed by then. Worse, this can't be patched with a tmpfiles `z` + # rule after the fact either - systemd-tmpfiles refuses with + # "Detected unsafe path transition ... (owned by gene) -> ... + # (owned by root)", a hardcoded symlink-attack safety check with no + # override. A flat path with no shared ancestor with anything + # impermanence creates sidesteps the whole problem. + age.keyFile = "/persist${config.users.users.${username}.home}/.sops-age-key"; defaultSopsFile = ./secrets.yaml; secrets = { + "ssh_private_key_gene_kiosk-gene-desk" = { + sopsFile = ../../../shared/secrets.yaml; + owner = "${username}"; + path = "${config.users.users.${username}.home}/.ssh/id_ed25519"; + mode = "0600"; + # pubkey is already declared in private-flake's ssh-keys lib and + # used for authorized_keys elsewhere; ssh derives it from the + # private key at runtime, so we don't need to place it separately. + }; local_private_env = { owner = "${username}"; path = "${config.users.users.${username}.home}/.private-env"; }; + ssh_host_ed25519_key = { + path = "/etc/ssh/ssh_host_ed25519_key"; + owner = "root"; + mode = "0600"; + restartUnits = [ "sshd.service" ]; + }; wifi_creds = { sopsFile = ../../../shared/secrets.yaml; owner = "wpa_supplicant"; diff --git a/modules/hosts/nixos/kiosk-gene-desk/disko.nix b/modules/hosts/nixos/kiosk-gene-desk/disko.nix new file mode 100644 index 00000000..ba45fa14 --- /dev/null +++ b/modules/hosts/nixos/kiosk-gene-desk/disko.nix @@ -0,0 +1,116 @@ +{ lib, ... }: +{ + disko.devices = { + disk.main = { + type = "disk"; + device = lib.mkDefault "/dev/mmcblk0"; + + content = { + type = "gpt"; + # `priority` pins physical partition order. This matters beyond + # readability: the Pi's GPU/EEPROM boot ROM always reads *physical + # partition 1* for its own firmware (start.elf, fixup.dat, + # config.txt), regardless of partition labels - it doesn't + # understand GPT names at that stage. Without explicit priorities + # here, disko orders partitions by attrset key (alphabetical), + # which put ESP before FIRMWARE and left the Pi unable to find its + # firmware at all. + partitions = { + # Loaded directly by the Pi's GPU/EEPROM boot ROM: RPi firmware + # (a.k.a "boot code") + the u-boot binary named in config.txt. + # Kept in sync by boot.loader.raspberry-pi on every switch. Must + # be physical partition 1 - see note above. + FIRMWARE = { + priority = 1; + label = "FIRMWARE"; + type = "0700"; # Microsoft basic data + attributes = [ 0 ]; # Required Partition + size = "512M"; + content = { + type = "filesystem"; + format = "vfat"; + mountpoint = "/boot/firmware"; + mountOptions = [ + "noatime" + "noauto" + "x-systemd.automount" + "x-systemd.idle-timeout=1min" + ]; + }; + }; + + # Read by u-boot (once it's running) for extlinux.conf, kernel + # and initrd. Needs to be its own real partition, not a + # directory under / - u-boot reads it before this boot's + # activation scripts (which repopulate it) ever run, and / is + # tmpfs so it wouldn't survive a reboot otherwise. + ESP = { + priority = 2; + label = "ESP"; + type = "EF00"; # EFI System Partition + attributes = [ 2 ]; # Legacy BIOS Bootable, so u-boot finds it + size = "256M"; + content = { + type = "filesystem"; + format = "vfat"; + mountpoint = "/boot"; + mountOptions = [ + "noatime" + "noauto" + "x-systemd.automount" + "x-systemd.idle-timeout=1min" + "umask=0077" + ]; + }; + }; + + # The nix store, rebuilt by every `nixos-rebuild switch`. Fixed + # size rather than 100% so /persist gets real space below; 20G + # is ~2x current store usage. + nix = { + priority = 3; + label = "NIX"; + type = "8305"; # Linux ARM64 root (/) + size = "20G"; + content = { + type = "filesystem"; + format = "ext4"; + mountpoint = "/nix"; + mountOptions = [ + "noatime" + "nodiratime" + ]; + }; + }; + + # Explicit persistence allowlist - see persistence.nix. f2fs + # for write-amplification reasons (this is where the write-heavy + # chromium cache lives). Takes whatever's left on the card. + persist = { + priority = 4; + label = "PERSIST"; + type = "8305"; + size = "100%"; + content = { + type = "filesystem"; + format = "f2fs"; + mountpoint = "/persist"; + mountOptions = [ + "noatime" + "nodiratime" + ]; + }; + }; + }; + }; + }; + + nodev."/" = { + fsType = "tmpfs"; + mountOptions = [ + "size=512M" + "mode=755" + ]; + }; + }; +} diff --git a/modules/hosts/nixos/kiosk-gene-desk/home-gene.nix b/modules/hosts/nixos/kiosk-gene-desk/home-gene.nix index d5ec9478..8410f739 100644 --- a/modules/hosts/nixos/kiosk-gene-desk/home-gene.nix +++ b/modules/hosts/nixos/kiosk-gene-desk/home-gene.nix @@ -1,6 +1,8 @@ { home.stateVersion = "24.11"; + genebean.services.tailscale.enable = true; + programs = { chromium = { enable = true; diff --git a/modules/hosts/nixos/kiosk-gene-desk/persistence.nix b/modules/hosts/nixos/kiosk-gene-desk/persistence.nix new file mode 100644 index 00000000..315cc7d2 --- /dev/null +++ b/modules/hosts/nixos/kiosk-gene-desk/persistence.nix @@ -0,0 +1,258 @@ +{ + config, + hostname, + lib, + pkgs, + username, + ... +}: +{ + boot.tmp = { + tmpfsSize = "20%"; + useTmpfs = true; + }; + + environment.persistence."/persist" = { + hideMounts = true; + + # System (non-home) paths: fine with impermanence's root:root default. + directories = [ + # uid/gid allocations - without this, gene's uid could get reassigned + # on reboot, breaking ownership of everything else persisted here + "/var/lib/nixos" + # No RTC on this Pi, so systemd-timesyncd's own persisted clock file + # here is what keeps the system clock from starting every single + # boot at whatever fallback the kernel/image build defaults to + # (confirmed on hardware: consistently the nixos-raspberrypi image's + # own build date) until NTP catches up moments later - not just + # cosmetic log confusion (explains every "up 122 days"/"since March + # 17" timestamp seen in journalctl all session), but a real window + # early in every boot where cert/timestamp validation could see the + # wrong time. + "/var/lib/systemd/timesync" + "/var/lib/tailscale" + ]; + files = [ "/etc/machine-id" ]; + + # Everything under gene's home goes through the dedicated per-user + # submodule (relative paths) instead of top-level directories/files + # with absolute /home/gene/... paths. This matters beyond style: the + # per-user submodule sets user/group to gene on every auto-created + # ancestor directory (/home/gene itself, .cache, .config, ...); + # top-level entries default those ancestors to root:root, which + # silently broke anything gene tried to create directly under $HOME + # (e.g. nix's own per-user profile dir - "Permission denied" on + # /home/gene/.cache/nix, which cascaded into home-manager failing + # with "Could not find suitable profile directory"). + users.${username} = { + directories = [ + ".cache/chromium" # actual browser cache + ".config/chromium" # profile, cookies, hass-browser_mod device registration + # genebean.programs.atuin-client (modules/shared/home/general) is + # enabled fleet-wide. Its local history db AND its sync-server + # login session/key both live here (not .config/atuin, which + # home-manager symlinks from the store and needs no persistence + # of its own) - without this, sync would need a manual `atuin + # login` again after every reboot. + ".local/share/atuin" + ]; + files = [ + # id_ed25519 itself is sops-managed (see default.nix) and + # intentionally NOT listed here - a whole-directory bind mount + # for .ssh would race with sops writing into the same directory. + # known_hosts has no such conflict. mode 0700 matches what ssh + # requires for the directory (still needs to be set explicitly - + # the per-user submodule's default is 0755, just with the right + # owner instead of root). + { + file = ".ssh/known_hosts"; + parentDirectory.mode = "0700"; + } + ]; + }; + }; + + # ------------------------------------------------------------------ # + # / is wiped to tmpfs on every boot (disko.nix: disko.devices.nodev."/"). + # /nix (the store, rebuilt by every `nixos-rebuild switch`) and + # /persist (the explicit allowlist above) are the only things that + # survive a reboot - both are real partitions, also declared in + # disko.nix. Both need to be mounted before most of boot happens. + # ------------------------------------------------------------------ # + fileSystems = { + "/nix".neededForBoot = true; + "/persist".neededForBoot = true; + }; + + # Recovery tool for the restic paths below - see modules/genebean/nixos/ + # programs/kiosk-restic-full-restore.nix. Shared across every kiosk + # running this same cage+chromium(+tailscale) setup, not just this host. + genebean.programs.kiosk-restic-full-restore = { + enable = true; + restorePaths = config.services.restic.backups.daily.paths; + stopServices = [ + "cage-tty1" + "tailscaled" + ]; + }; + + # What's persisted above survives a reboot (still on the same SD card), + # but not a card death - only what's backed up here does. Everything + # else in the persist allowlist above either comes from sops/git + # (ssh host key, gene's ssh key, wifi creds - already safe, backed up + # by the repo itself) or is cheap to lose (.ssh/known_hosts, uid/gid + # allocations). These three are the ones that would actually be a + # hassle to reconstruct by hand on a kiosk with no keyboard normally + # attached: + # - .config/chromium: hass-browser_mod's device registration + # - .local/share/atuin: sync-server login session/key + # - /var/lib/tailscale: node identity (else: re-approve in the + # Tailscale admin console) + # All backed up from their real /persist-side paths directly rather + # than the impermanence bind-mounted aliases under $HOME, matching + # sops.age.keyFile's precedent in ./default.nix of preferring the real + # path over the alias. + services.restic.backups = { + daily = { + paths = [ + "/persist${config.users.users.${username}.home}/.config/chromium" + "/persist${config.users.users.${username}.home}/.local/share/atuin" + "/persist/var/lib/tailscale" + ]; + + # The shared module's default (Persistent = true) fires a catch-up + # backup right after every boot if the last scheduled run was + # missed - which, on a host that gets reinstalled/rebooted a lot, + # means it nearly always fires before there's a chance to restore + # fresh state after a reinstall. Confirmed on hardware: that catch- + # up run captured the empty post-reinstall chromium/atuin/tailscale + # state as today's newest snapshot, and --keep-daily 7 then + # legitimately forgot the real (pre-reinstall) same-day snapshot as + # superseded before a restore could happen. A missed day here is a + # non-issue, so just wait for the next scheduled run instead. + timerConfig = { + OnCalendar = "daily"; + Persistent = false; + }; + }; + + # Companion prune-only job (no `paths` - see the shared restic + # module's own docs on that) so pre-reinstall-tagged snapshots don't + # just pile up forever: anything tagged pre-reinstall older than 45 + # days is fair game again. --tag and --host together scope this + # forget run to ONLY this host's pre-reinstall-tagged snapshots - + # everything else (including any other host that might someday reuse + # the same tag) is untouched by this job. --keep-within is based on + # each snapshot's own (immutable) creation time, not when it was + # tagged. + pre-reinstall-cleanup = { + environmentFile = config.sops.secrets.restic_env.path; + passwordFile = config.sops.secrets.restic_password.path; + pruneOpts = [ + "--tag pre-reinstall" + "--host ${config.networking.hostName}" + "--keep-within 45d" + ]; + repositoryFile = config.sops.secrets.restic_repo.path; + timerConfig = { + OnCalendar = "daily"; + Persistent = false; + }; + }; + }; + + systemd = { + services = { + # systemd-tmpfiles-setup.service is a one-shot that runs once, early + # in boot, and doesn't get a second chance if whatever creates a + # given directory fresh on tmpfs loses the race against it - nothing + # re-runs the rules afterward on a plain boot (as opposed to a + # `nixos-rebuild switch`, which does via systemd-tmpfiles- + # resetup.service). Confirmed on hardware three separate times now + # for three different directories ($HOME itself, breaking atuin's + # interactive search with a bare "Permission denied"; .config, + # breaking chromium's crash reporter; and + # /var/lib/systemd/timesync, breaking systemd-timesyncd's own + # ability to persist its clock file) - always self-resolves + # instantly with a manual re-run, so it's a boot-ordering race, not + # the "unsafe path transition" tmpfiles safety block that a couple + # of these also hit at various points (see sops.age.keyFile's + # comment in ./default.nix for that one - no ExecStartPre fixes + # that variant, only avoiding the unsafe ownership pattern in the + # first place does). The `+` prefix runs this specific command as + # root regardless of these services' own non-root User=. + cage-tty1.serviceConfig.ExecStartPre = [ + "+${pkgs.systemd}/bin/systemd-tmpfiles --create" + ]; + "home-manager-${username}".serviceConfig.ExecStartPre = [ + "+${pkgs.systemd}/bin/systemd-tmpfiles --create" + ]; + + # The shared restic module hardcodes RESTIC_CACHE_DIR to + # /var/cache/restic-backups-daily (no option exposed to change it), + # which lands on this host's 512M tmpfs root (disko.nix). That cache + # holds restic's local index for the *entire* shared repo - every + # host's snapshots, not just this one's - and filled the tmpfs to + # 100% from a single manual `restic ls` while testing this backup, + # confirmed via `du` afterwards. Redirecting it to /persist (real + # f2fs partition, effectively the whole card minus /nix's 20G) avoids + # that, and as a side benefit means the index doesn't have to be + # re-downloaded from scratch after every reboot. + restic-backups-daily.environment.RESTIC_CACHE_DIR = lib.mkForce "/persist/restic-cache"; + + systemd-timesyncd.serviceConfig.ExecStartPre = [ + "+${pkgs.systemd}/bin/systemd-tmpfiles --create" + ]; + }; + + tmpfiles.rules = [ + "d /var/log/journal 0755 root systemd-journal -" + # writable zsh history file since $HOME/.zsh_history isn't persisted + "f /tmp/zsh_history_gene 0600 ${username} users -" + # Something (NixOS's own createHome, ahead of impermanence's own + # directory-creation script) creates /home/gene as root:root before + # impermanence gets a chance to set it up with the right owner via + # its per-user submodule - and impermanence doesn't re-own a + # directory that already exists. `z` (unlike impermanence's + # create-if-missing) enforces ownership every boot regardless of who + # created it first. This is what broke gene creating anything + # directly under $HOME (nix's own per-user profile dir included), + # which cascaded into home-manager failing on every boot. + "z /home/${username} 0700 ${username} users -" + "z /home/${username}/.cache 0755 ${username} users -" + "z /home/${username}/.config 0755 ${username} users -" + # Same root:root staleness as $HOME itself, one level deeper: these two + # are impermanence-bind-mounted directories (see the users.${username} + # block above), so they're independently subject to the same + # create-if-missing-only limitation - chromium's crash reporter + # (chrome_crashpad_handler) hard-requires being able to mkdir + # ~/.config/chromium/Crash Reports on every launch, regardless of + # --user-data-dir, and fails the whole browser process (SIGTRAP) if it + # can't - confirmed via strace showing mkdirat(...) = -1 EACCES here. + "z /home/${username}/.cache/chromium 0755 ${username} users -" + "z /home/${username}/.config/chromium 0755 ${username} users -" + # Same reasoning, for the atuin state persisted above - .local and + # .local/share aren't otherwise forced, so cover the whole ancestor + # chain rather than assume they'll come out right. + "z /home/${username}/.local 0755 ${username} users -" + "z /home/${username}/.local/share 0755 ${username} users -" + "z /home/${username}/.local/share/atuin 0755 ${username} users -" + # scripts/prep-install-bootstrap.sh pre-seeds this on a fresh install + # via nixos-anywhere's --extra-files, which (same class of bug as + # .config's own root:root staleness earlier) creates it root:root - + # systemd-timesyncd runs as its own dedicated systemd-timesync user + # and needs write access to actually keep this file's mtime current + # going forward, not just read it once at boot. + "z /var/lib/systemd/timesync 0755 systemd-timesync systemd-timesync -" + "z /var/lib/systemd/timesync/clock 0644 systemd-timesync systemd-timesync -" + # id_ed25519.pub isn't sensitive and isn't persisted; reconstruct it + # every boot from the same value private-flake already declares for + # this host's authorized_keys entries elsewhere, instead of + # duplicating it via sops or a persisted file. + "d /home/${username}/.ssh 0700 ${username} users -" + "f /home/${username}/.ssh/id_ed25519.pub 0644 ${username} users - ${ + config.private-flake.sshKeys.${hostname}.key + }" + ]; + }; +} diff --git a/modules/hosts/nixos/kiosk-gene-desk/read-only-root.nix b/modules/hosts/nixos/kiosk-gene-desk/read-only-root.nix deleted file mode 100644 index b9cfb00d..00000000 --- a/modules/hosts/nixos/kiosk-gene-desk/read-only-root.nix +++ /dev/null @@ -1,126 +0,0 @@ -{ - lib, - pkgs, - username, - ... -}: -{ - # ------------------------------------------------------------------ # - # Read-only SD card mounts and tmpfs for writable paths - # ------------------------------------------------------------------ # - fileSystems = { - "/" = lib.mkForce { - device = "/dev/disk/by-label/NIXOS_SD"; - fsType = "ext4"; - options = [ - "ro" - "noatime" - "nodiratime" - ]; - }; - - "/boot/firmware" = lib.mkForce { - device = "/dev/disk/by-label/FIRMWARE"; - fsType = "vfat"; - options = [ - "ro" - "noatime" - "nofail" - "noauto" - ]; - }; - - "/var/log" = { - device = "tmpfs"; - fsType = "tmpfs"; - options = [ - "size=64m" - "mode=0755" - "nosuid" - "nodev" - ]; - neededForBoot = true; - }; - - "/var/lib" = { - device = "tmpfs"; - fsType = "tmpfs"; - options = [ - "size=256m" - "mode=0755" - "nosuid" - "nodev" - ]; - neededForBoot = true; - }; - - "/home/${username}/.cache" = { - device = "tmpfs"; - fsType = "tmpfs"; - options = [ - "size=256m" - "mode=0700" - "uid=1000" - "nosuid" - "nodev" - ]; - }; - - "/home/${username}/.local" = { - device = "tmpfs"; - fsType = "tmpfs"; - options = [ - "size=256m" - "mode=0700" - "uid=1000" - "nosuid" - "nodev" - ]; - }; - - "/home/${username}/.config/chromium" = { - device = "tmpfs"; - fsType = "tmpfs"; - options = [ - "size=128m" - "mode=0700" - "uid=1000" - "nosuid" - "nodev" - ]; - }; - }; - - # ------------------------------------------------------------------ # - # tmpfs for paths that need to be writable at runtime - # ------------------------------------------------------------------ # - - # /tmp - NixOS built-in option, cleaner than a manual fileSystems entry - boot.tmp.useTmpfs = true; - boot.tmp.tmpfsSize = "20%"; - - # ------------------------------------------------------------------ # - # systemd-journal needs its directory to exist after /var/log tmpfs - # is mounted - # ------------------------------------------------------------------ # - systemd.tmpfiles.rules = [ - "d /var/log/journal 0755 root systemd-journal -" - # create a writable zsh history file in /tmp for gene - "f /tmp/zsh_history_gene 0600 ${username} users -" - ]; - - # ------------------------------------------------------------------ # - # Helper scripts for doing a nixos-rebuild - # ------------------------------------------------------------------ # - environment.systemPackages = [ - (pkgs.writeShellScriptBin "remount-rw" '' - echo "Remounting / read-write..." - sudo mount -o remount,rw / - - echo "Starting nix-daemon..." - systemctl start nix-daemon.socket nix-daemon.service - - echo "Done. Run 'reboot' when finished." - '') - ]; -} diff --git a/modules/hosts/nixos/kiosk-gene-desk/secrets.yaml b/modules/hosts/nixos/kiosk-gene-desk/secrets.yaml index 67fffda0..642e890d 100644 --- a/modules/hosts/nixos/kiosk-gene-desk/secrets.yaml +++ b/modules/hosts/nixos/kiosk-gene-desk/secrets.yaml @@ -1,10 +1,8 @@ local_git_config: ENC[AES256_GCM,data:gMuCCJzRdZcDGZvdDCv3h5EZmAkK+bBewn2m6x4VT+23K5gCdajeV94=,iv:1pRM6QWSIw6xfTgcjpGkfHR8iHY/+xuWgeFQ+1pWSTM=,tag:lodSBhqgN8Yaagm6gK4xTQ==,type:str] local_private_env: ENC[AES256_GCM,data:i+uyo+/StMDUfIp6g1Pl8pOyqD+0f2X5AsFzOo+3VOd/n7CCOy7j/1F42QMpjgejvIYmQwLYSR2jEXyxig==,iv:QTUeDe/LnkKCEFB5g3y7pbySUwdWW7D8rVjVv4+ib9g=,tag:6hBzyfQus60i5c2EktD0Eg==,type:str] +ssh_host_ed25519_key: ENC[AES256_GCM,data:3zd8ldhlebTwbi1YPZRSODX30Jj/sb0owIFsCDCZhwrUphxljUmVa1Too78hdalSrQWp9B/OCVhEvV9r2RT1+gkvfz3jUsrQx9JlJsEAwDgIk5nrfpwUxfpH18VT69IrvfEOQaC303XGaDkq6q4P4NS54PqZlZ5EGNEd0b2pIZvi2hIGJDYbkodZKaiFu5jhiZdHZd5O1yox8+LwLLrlySSY6+37ygZjnzbQ9Y+9oBCz0wUEQ/xtn9olumihX9zcWbsKyv8yo4dCVyqWPSnKnu2DlR9kU8XRlB4sGnzBdbIc9uAEieJF7ZhLROoTfmQ5TfNZuQbuU//qEd/DdTs3YvdbddswxdBWuhbaJUkX2WaGZtUtuVB62acKE13zFQxAv1FSi7tA6l7nnzqhAm+wUI9nrPolc6bzqNHxkvdRjfZ5feLNl9C33b9vC1dXE8uggdUBKFjHG+pAF4OQBMmAtKx2ZPVMWX0pzlEQy7iHCCkLqzI5xhJ1aVVbxIzpQCtRJB+sj9MU4rWYeKmWUNQru7abqcvzsFcp2AF3,iv:EMEOtdhsUsc+Xnn72Y/FaNfGUMiuDXQPX2ZmJJe4wGQ=,tag:1FlKIhpoawoP3EIYG3MfrA==,type:str] +tailscale_key: ENC[AES256_GCM,data:zdgXW9K50aTUKU/k11iMwQLHxXrJgDiZAswDGnZEVnJ61A7+aQDBPYLK59A3ElwBd2VQZrqKU2lEqMuSfCg=,iv:1g15n4c40khkfhvTlzzvAYQFSjCyNrECT9pSGGW/Jsg=,tag:RKShOkGm/UeNPYaT2pANUg==,type:str] sops: - kms: [] - gcp_kms: [] - azure_kv: [] - hc_vault: [] age: - recipient: age1an6t5f0rr6h55rzsv5ejycxju72rp46jka840fwvupwfk65jegrq7hmkl9 enc: | @@ -15,8 +13,7 @@ sops: Z0JQanNPV2NTT2dxSThWOU9EUS90UlkKNpEald58B9SM98tqgyLV5Q/943nliZq2 vYd3ULeY9gF8NA7BlmbZrWKRbd+/eIZnYKSAht1lWTwhktwHEVZV0A== -----END AGE ENCRYPTED FILE----- - lastmodified: "2025-01-11T17:31:06Z" - mac: ENC[AES256_GCM,data:dGi6v1WdScNNMzaBlMQ3r+B6w4pKuVMo70x8J+d8suysW1Opsot3QehEkxPXQ8OmktuSJlSvPugKSX6CyM2N7pZtPL0ZeyVYQuHHHUmoIf0myc2tgIJ3OD3M+YYtnyEbZoRnDCE/geH/WfQ+ttNCDJbxtSYaQfhndJjouPQBRs4=,iv:zQXSbJLI4A85GcrU8VSOAaMWWprNTh/2PAVA47MALsk=,tag:MqBvBn9V5tXzZtUscuBpjA==,type:str] - pgp: [] + lastmodified: "2026-07-17T20:52:49Z" + mac: ENC[AES256_GCM,data:qV+m3h67B4Os0LfSiWaObJwMEBwbLOORWTyoI7akxpY5GO2q3W3RaGwVS9XHM29AN8dvDGjCHijyJEXRt2BkenUtsxbLlsuwxCVlOp7lZC2IYPByAGVxWj/7ZMzfCp0J0BkTLdQDFFPaVxUQKYmkDLbeW5b5UyHVKuQ2lYtoycg=,iv:pzWfW08aIxwZwFo9MnN3DlSCmyymjvNVyJcIXkch5SM=,tag:DufwgLdkET5Jgvn6KhW2qg==,type:str] unencrypted_suffix: _unencrypted - version: 3.9.2 + version: 3.12.1 diff --git a/modules/shared/nixos/restic.nix b/modules/shared/nixos/restic.nix index 4c0a04bd..19f12ca6 100644 --- a/modules/shared/nixos/restic.nix +++ b/modules/shared/nixos/restic.nix @@ -26,7 +26,43 @@ "--keep-daily 7" "--keep-weekly 5" "--keep-monthly 6" + # Every host shares this one repository and none of these forget + # invocations are scoped by --host, so any host's prune run + # processes retention for every host's snapshots, not just its + # own. This needs to live here (not on whichever host actually + # uses the tag) so it's honored no matter which host's prune job + # happens to run first - see kiosk-gene-desk/persistence.nix for + # where pre-reinstall gets applied. + "--keep-tag pre-reinstall" ]; }; + + # Fleet-wide backstop, deliberately not scoped by --host: whatever + # per-host cleanup job manages a normal expiry for its own + # pre-reinstall-tagged snapshots (see kiosk-gene-desk/persistence.nix) + # is the primary mechanism, but this catches anything that slips + # through - a host decommissioned before its own cleanup job ran, one + # disabled/misconfigured, etc. - so pre-reinstall tags can never pin a + # snapshot forever fleet-wide, only for up to a year. + pre-reinstall-backstop = { + environmentFile = config.sops.secrets.restic_env.path; + repositoryFile = config.sops.secrets.restic_repo.path; + passwordFile = config.sops.secrets.restic_password.path; + pruneOpts = [ + "--tag pre-reinstall" + "--keep-within 1y" + ]; + # Confirmed on hardware: the module default (Persistent = true) fired + # this immediately on the same reboot it was first created on, + # pegging CPU on a resource-constrained Pi actively running its + # kiosk display, right alongside two other newly-created jobs doing + # the same thing. A missed day here is a non-issue - see + # kiosk-gene-desk/persistence.nix's daily/pre-reinstall-cleanup + # timerConfig overrides for the same reasoning. + timerConfig = { + OnCalendar = "daily"; + Persistent = false; + }; + }; }; } diff --git a/modules/shared/secrets.yaml b/modules/shared/secrets.yaml index 05dfce3c..56987a8d 100644 --- a/modules/shared/secrets.yaml +++ b/modules/shared/secrets.yaml @@ -1,5 +1,6 @@ gandi_dns_pat: ENC[AES256_GCM,data:3L1RDSbCqkmLmguSgsJsf3gdnSi/zxS8xtl+B+kwBeaOnX3X5fmM7A==,iv:SWAfEAC/3klgreTppGZWV5SACrQEEL8tsXUvYFlJXyk=,tag:IiekQLSf1vnjRQr6ZRsVMQ==,type:str] gandi_api: ENC[AES256_GCM,data:YsdDMk75miIKO4LkCZjfwJw6gxfrmsTL,iv:BOPRxB661sPJnUH1AUKEALIJfBeyAHZpkWJEDbY+7i8=,tag:TvtW7qhPbOqi9kKDcIe28w==,type:str] +ssh_private_key_gene_kiosk-gene-desk: ENC[AES256_GCM,data:ZEEMh/Efo6Vl2FhafX2yGXJM8/mJUiU/hxnvsXvXpB1QorJvQ1ZJy8g9CYfnRIT+mbG/ygw1IBHG/AvBfbXf3IAVpYbNN7RG7WpiWpuP+UyLsCJhTayeHxSwCRcyzF6k7vLgLmZ7xf31pueIl4aVcpI1saFpK7vkLUEtT9tdqWpBxL99gO4NYef4LFeeLpVi0k8Bd0ctBb0YDTxB0fCxw641hRyqz/KI2aNs214T96/gVh4qm4Xp+bUjEz/Pjm3/MN1JXRj/4x8K3EdWESCmc+OlSm40vH01ek/7VwyWy6+jTkHe144AaMutw+mdHwvCmtY6TFvazLQwnTv4EJzubCKtSm/Yn/U24n9nmr4VEhyajJ2aGvFJ3jBf+JO7YQWKkFhruD8wNGvLcK603BApLv6ToDwwONERldXHi2cJbXHzDPqUHrvzFiMsAn4bCecLaLSEwcdwFQLq0k3B6M8JPv1QM5oVQnC7NRTyPUly4ix3w3impixdkQmSYX8YSxQfbZ9yipU9wErS61n95KvKP2RUbiLr7ZS3pGDV,iv:2DNMfbIy3424F9QIsUVwFygIek9IT6XGgZcTt3J9QIQ=,tag:lUED6mJEiGopE71GJ1fdyA==,type:str] hetzner_api_token: ENC[AES256_GCM,data:8+bYBnI6vSQ7QIDFv0zplU2A2lW2c7JA9WArCGeAgjg=,iv:Y92uRgjKfuGDY4HMr+j6uDweMmMCx0FBydP3alGgb3M=,tag:cbmeVnP1XcqE+T0qpzJfbw==,type:str] hetzner_lego_env: ENC[AES256_GCM,data:xRADnkMC/mTq8/oRpZ+NYTStB9qX2N6V0GNIpGsXNedgO3bTvowgMukyDW4nX19V627ykk5vPC/HTRhZ8ia2KxRJfqa+9n5+Eg83iAFtrQGOe2rvEGEHDUoCTSb/G8YA8XzB3t69Xc+o8g59Grf4rXvNLEEwewn92BP7YWoxvpPaeT3yl/g7/0m4SDXKR/D3LtiN4nikiUFYT6nBG+WipMK3oEw=,iv:dL4hw4/v1FgJKwmCzIpMKvryrm+mMb7SoohPi78paPY=,tag:Lq3vBkyVbv7w5/RIHcsiUg==,type:str] restic_env: ENC[AES256_GCM,data:FCYR8tkClRwfcjUotcr28D6uRz7sNihn50nw38CaYnqOD/U9+5kU0iAPSvqAbeuw+xUoKKKAPAfMHI12dPTYt17Wz1N7i4a+MRkiIR9pjyv5KZTK59G+,iv:jStc8GMbZUQUgooZiRdImSZskdckYN1cRm2gsKbUyYY=,tag:HpQQIj1j7fjCmxkSeY/k4g==,type:str] @@ -109,7 +110,7 @@ sops: ODFjcWxtRjkweGJvdzdWSEphMHRCdm8Kx0amHgaZZR26c+VRVTyBEnm+w5c5nA7R txHj1U349LbfEsovTqZAL1o2WuX+gmXSj1aeXPKW+S0bIagC6dDacA== -----END AGE ENCRYPTED FILE----- - lastmodified: "2026-02-01T03:15:26Z" - mac: ENC[AES256_GCM,data:gGk69AnZNAlA5fxViXZQMpGStn8v2L/IyIUAshyrlAKu8dcTDQfCDfC2kTlo0Q/gwhqchSKYFFKWWze7EUwUWmlYjJu5MEEK4aQ0HEObff27AKQloOln1X3jiIR7bnsKoakbAKJWYpGf6PWClbH64mjoJhBroG5amAwE25bevDE=,iv:bO2kFuGcq/1j2uo2y5/UwOHNQ5A8orStI190NMdAr1E=,tag:7xwClQ28I695qDICc0GS+A==,type:str] + lastmodified: "2026-07-17T20:52:15Z" + mac: ENC[AES256_GCM,data:YyG7Gz3zH8Y5RtHUrrCEm76yadjFngS2qicWCnDguaLlYnnKVtDG8XRshWvLqOCSqzUM78wUaRU4cphpJZOMo0PHQvmqRPrMQw9c+TuUAZJ5+A4Ja06uBGsiu9QCuAUopkIfiBNPrdRRzSSPxv4YR57HDqx5f61uKmt0LhHexdA=,iv:R7DW5y/IGm9medK7YwauS+31LW6By3SBCr7D5aJcdn8=,tag:FqIeChMQimP0wc4R3aOkRw==,type:str] unencrypted_suffix: _unencrypted - version: 3.11.0 + version: 3.12.1 diff --git a/pkgs/default.nix b/pkgs/default.nix index 58961214..f92177c5 100644 --- a/pkgs/default.nix +++ b/pkgs/default.nix @@ -1,11 +1,9 @@ -{ pkgs, self }: +{ inputs, pkgs }: let nixdiff = pkgs.callPackage ./nixdiff { }; + rpi4-installer = pkgs.callPackage ./rpi4-installer { inherit inputs; }; in { - inherit nixdiff; + inherit nixdiff rpi4-installer; default = nixdiff; } -// pkgs.lib.optionalAttrs (pkgs.stdenv.hostPlatform.system == "aarch64-linux") { - kiosk-gene-desk-sdImage = pkgs.callPackage ./kiosk-gene-desk-sdImage.nix { inherit self; }; -} diff --git a/pkgs/kiosk-gene-desk-sdImage.nix b/pkgs/kiosk-gene-desk-sdImage.nix deleted file mode 100644 index 18aa2560..00000000 --- a/pkgs/kiosk-gene-desk-sdImage.nix +++ /dev/null @@ -1,2 +0,0 @@ -{ self }: -self.nixosConfigurations.kiosk-gene-desk.config.system.build.sdImage diff --git a/pkgs/rpi4-installer/default.nix b/pkgs/rpi4-installer/default.nix new file mode 100644 index 00000000..49bd206c --- /dev/null +++ b/pkgs/rpi4-installer/default.nix @@ -0,0 +1,17 @@ +{ + inputs, + runCommand, + zstd, +}: +# Generic RPi4 installer image - bootstrap media for `nixos-anywhere`, not a +# build of any of our own host configs. Decompressed here so it's flashable +# straight from result/, without an extra manual zstd step (and without +# depending on the flashing tool's zstd support). +runCommand "rpi4-installer.img" + { + nativeBuildInputs = [ zstd ]; + } + '' + img=$(find ${inputs.nixos-raspberrypi.installerImages.rpi4}/sd-image -name '*.img.zst') + zstd -d "$img" -o $out + '' diff --git a/scripts/prep-install-bootstrap.sh b/scripts/prep-install-bootstrap.sh new file mode 100755 index 00000000..4633244a --- /dev/null +++ b/scripts/prep-install-bootstrap.sh @@ -0,0 +1,102 @@ +#!/usr/bin/env bash +# Prepares the nixos-anywhere --extra-files directory needed to bootstrap +# sops on a fresh install: disko wipes /persist from scratch every time, +# so the sops age key has to be re-seeded before activation runs, or +# sops-install-secrets fails on first boot with no way to recover it +# short of a manual SSH patch. +# +# Expects a per-host SSH private key named ssh_private_key_gene_ +# in modules/shared/secrets.yaml (decryptable by every host/user in the +# fleet, per .sops.yaml), matching the *_authorized_keys entry already +# declared in private-flake's ssh-keys lib for that host. The age key is +# deterministically derived from it via ssh-to-age, so it always +# reproduces the same identity already registered as a recipient in +# .sops.yaml for that host - nothing needs to be re-encrypted. +# +# Usage: scripts/prep-install-bootstrap.sh +# Then: nixos-anywhere --flake ~/repos/dots# \ +# --extra-files ./nixos-anywhere-extras- root@ +set -euo pipefail + +hostname="${1:?usage: $0 }" +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +secret_name="ssh_private_key_gene_${hostname}" +out_dir="${repo_root}/nixos-anywhere-extras-${hostname}" +# Flat path, not nested under .config: this file gets copied onto the +# target as root before nixos-install's activation ever runs, which +# would create .config itself as root:root as a side effect and (on +# kiosk-gene-desk at least) permanently break anything impermanence +# later needs to bind-mount under .config - systemd-tmpfiles refuses to +# fix it after the fact ("Detected unsafe path transition"). See the +# comment on sops.age.keyFile in kiosk-gene-desk/default.nix. +age_key_path="${out_dir}/persist/home/gene/.sops-age-key" +# No RTC on these Pis, so without a persisted clock file, systemd- +# timesyncd has no floor and every boot starts from whatever fallback the +# kernel/image build defaults to until NTP catches up moments later - a +# real window where cert/timestamp validation could see the wrong time, +# not just cosmetic log confusion. mightymac's own clock is already +# NTP-correct, so touching this file here (content doesn't matter, only +# its mtime) gives the very first boot a reasonable floor immediately, +# before that boot's own NTP sync has had a chance to run. Persistence.nix +# then keeps that floor accurate on every subsequent boot - see the +# comment there. Harmless if the target host doesn't persist this path. +timesync_clock_path="${out_dir}/persist/var/lib/systemd/timesync/clock" + +ssh_key_tmp="$(mktemp)" +restic_env_tmp="$(mktemp)" +trap 'rm -f "$ssh_key_tmp" "$restic_env_tmp"' EXIT + +# Wiped fresh, not just mkdir -p'd: out_dir persists between runs, so a +# stale file from a previous run's now-changed layout would otherwise +# silently tag along into --extra-files on a later run and get copied +# onto the target. +rm -rf "$out_dir" + +echo "Decrypting ${secret_name} from modules/shared/secrets.yaml..." +sops decrypt --extract "[\"${secret_name}\"]" "${repo_root}/modules/shared/secrets.yaml" > "$ssh_key_tmp" +chmod 600 "$ssh_key_tmp" + +mkdir -p "$(dirname "$age_key_path")" +nix run nixpkgs#ssh-to-age -- -private-key -i "$ssh_key_tmp" > "$age_key_path" +chmod 600 "$age_key_path" + +mkdir -p "$(dirname "$timesync_clock_path")" +touch "$timesync_clock_path" + +recipient="$(nix shell nixpkgs#age --command age-keygen -y "$age_key_path")" +echo "Derived age recipient: ${recipient}" +echo "(confirm this matches ${hostname}'s recipient in .sops.yaml before installing)" + +# Best-effort: tag this host's last 5 restic snapshots as pre-reinstall +# before the reinstall wipes it, so kiosk-restic-full-restore has known- +# good state to fall back on even if a boot-triggered catch-up backup on +# the fresh install captures empty state first and would otherwise get +# treated as "latest". Talks to the repo directly rather than over SSH to +# the device being reinstalled - mightymac is already a valid recipient +# for these same restic secrets in modules/shared/secrets.yaml, and this +# way it still works even if the device's own SSH/network is flaky right +# before a reinstall. See modules/shared/nixos/restic.nix and +# kiosk-gene-desk/persistence.nix for the retention side of this. +sops decrypt --extract '["restic_env"]' "${repo_root}/modules/shared/secrets.yaml" > "$restic_env_tmp" +set -a +# shellcheck disable=SC1090 +source "$restic_env_tmp" +set +a +export RESTIC_REPOSITORY +RESTIC_REPOSITORY="$(sops decrypt --extract '["restic_repo"]' "${repo_root}/modules/shared/secrets.yaml")" +export RESTIC_PASSWORD +RESTIC_PASSWORD="$(sops decrypt --extract '["restic_password"]' "${repo_root}/modules/shared/secrets.yaml")" + +echo "Tagging ${hostname}'s last 5 restic snapshots as pre-reinstall..." +snapshot_ids="$(nix run nixpkgs#restic -- snapshots --host "${hostname}" --latest 5 --json | nix run nixpkgs#jq -- -r '.[].short_id')" +if [ -n "$snapshot_ids" ]; then + # shellcheck disable=SC2086 + nix run nixpkgs#restic -- tag --add pre-reinstall $snapshot_ids + echo "Tagged: $(echo "$snapshot_ids" | tr '\n' ' ')" +else + echo "No existing snapshots found for ${hostname} - nothing to tag." +fi + +echo "" +echo "Ready. Run:" +echo " nixos-anywhere --flake ${repo_root}#${hostname} --extra-files ${out_dir} root@"