Repository navigation
Make config saves atomic and refuse to read a damaged config (#337) - #343
Merged
Merged
Conversation
…ntly Two tests fail on staging, pinning the load-side contract of the issue: - A config truncated at a resource boundary (simulating an interrupted in-place save) is served as a plausible smaller inventory with no diagnostic — rpk summary shows 2 of 3 servers, exit 0. - A config cut mid-token is swallowed entirely: rpk summary reports an EMPTY inventory with exit 0 and no error naming the file. This is quieter than the YamlDotNet stack trace the issue observed on 2.x — the boot-load catch now hides the failure, so the next write would persist the empty inventory over the damaged-but-recoverable file. The writer-side fix (atomic + durable saves in PhysicalTextFileStore: temp file, flush, rename) is not black-box observable; these tests cover the belt-and-braces load-side behaviour that must accompany it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Saving config.yaml was neither atomic nor durable, and a damaged file was then accepted without complaint on the next load. Together those could lose an inventory. Writes went through File.WriteAllTextAsync, which truncates the file to zero before writing a byte and never flushes — so an interrupted save could leave a truncated config, and a save that had returned successfully could still be lost to power failure. Migration backups shared that path, so the one existing safety net was written unsafely too. The store now writes a temp file, flushes it to disk, and renames over the destination; a failed write cleans up and leaves the original untouched. Reads were worse than the issue reported: on staging an unparseable config loaded as an EMPTY inventory with exit 0, because the boot-time catch for unreadable stores was swallowing parse failures too. A ConfigLoadException is now latched on the collection when the config exists but cannot be understood, and every read and write refuses while it is set — the CLI reports "Config error:" with exit 5, MCP forwards the message. Two cases are detected: the file fails to parse, and the file parses with no schema version (truncated before the version line, or not a RackPeek config). An empty file is still a legitimately empty inventory, and legacy versionless configs still migrate. Routes.razor gated the whole app on a successful load, so a damaged config left the Web UI stuck on "Loading…" — including the YAML editor that is how you fix it. Both hosts now render anyway, and the editor reports a still-broken save inline instead of tearing down the circuit. A save interrupted at a clean resource boundary leaves valid YAML that is indistinguishable from a smaller inventory; nothing at load time can detect that, which is why the atomic-write half is the primary remedy. Tests: 5 store tests (the concurrency one fails on the parent commit with a 12288-byte partial read), 6 CLI e2e tests, 2 Playwright tests covering repair through the YAML editor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #337.
Saving
config.yamlwas neither atomic nor durable, and a damaged file was then accepted without complaint on the next load. Together those could lose an inventory.What was wrong
Writes.
PhysicalTextFileStore.WriteAllTextAsyncwas a bareFile.WriteAllTextAsync, which opens withFileMode.Create— the file is truncated to zero before a byte goes back — and never flushes. So an interrupted write could leave a truncated config, and a save that had returned successfully could still be lost to power failure. The migration backup path (BackupOriginalAsync) went through the same unsafe write, so the one existing safety net was written unsafely too.Reads. A damaged file loaded without an error, and the next save made it permanent. Worse than the issue reported: on current
stagingan unparseable config loaded as an empty inventory with exit 0 — the boot-time catch added for unreadable stores was swallowing the parse failure too.rpk summarycheerfully printedHardware (0), and the next write would have persisted that emptiness over a recoverable file.What this changes
Atomic, durable saves. The store now writes to a temp file in the same directory, flushes through the OS cache to disk (
Flush(true)), then renames over the destination. A rename is atomic, so readers only ever see the old or the new file. Failed writes clean up their temp file and leave the destination untouched. This also makes migration backups safe, since they share the path.Damaged configs fail loudly instead of reading as empty. A new
ConfigLoadExceptionis latched on the collection when the config exists but cannot be understood, and every read path (ThrowIfLoadFailed()) and write path (the existingEnsureLoadedAsync()) refuses while it is set. The CLI reports it asConfig error:with exit code 5 rather than a stack trace; MCP surfaces the message instead of a generic failure.Two cases are detected: the file fails to parse, and the file parses but carries no schema version (so it is truncated before the version line, or is not a RackPeek config at all). An empty file — the one the CLI creates on first run — is still a legitimately empty inventory, and legacy configs with no
version:key still migrate normally.The Web UI stays usable for repair.
Routes.razorgated the whole app on a successful load, so a damaged config left the UI stuck on "Loading…" — including the/yamleditor that is how you fix it. It now renders anyway; pages that read the inventory surface the failure themselves, and the editor reports a still-broken save inline rather than tearing down the circuit.What is deliberately not fixed
A save interrupted at a clean resource boundary leaves valid YAML that is indistinguishable from a smaller inventory. Nothing at load time can detect that, which is exactly why the atomic-write half is the primary remedy rather than the detection half. I considered treating "no
connections:key" as a damage signal and rejected it: 11 of the 12 v4 test fixtures omit it, so it would break hand-written configs, which are explicitly supported.Tests
Tests/Yaml/PhysicalTextFileStoreTests.cs(5) — round-trip, full replacement, no temp debris, failed writes leave the original intact, and a concurrent reader sampling the file during 40 alternating writes never observes a partial one. That last test is the bug demonstration: against the old implementation it fails withObserved a partially written config (12288 bytes; expected 200011 or 50011).Tests/EndToEnd/CorruptConfigTests.cs(6) — an unparseable config and a non-RackPeek file both fail with a message naming the file and never printHardware (0); a damaged config is byte-for-byte unchanged after an attempted write; an empty config and a healthy config still work; a legacy versionless config still migrates.Tests.E2e/DamagedConfigTests.cs(2) — with a damaged config staged in the container, the app still renders, the/yamleditor loads it, and repairing it through the editor restores the inventory; and an inventory page never renders an empty list over the damaged file.The three load-side tests and the concurrency test all fail on the parent commit and pass here.
PlaywrightFixturegainedWriteConfigAsync/ReadConfigAsyncso a test can stage a damaged config, which cannot be produced through the UI (the editor validates before saving).Full suite green: 375 CLI + 260 discovery + 70 MCP + E2E,
dotnet formatclean.🤖 Generated with Claude Code