Skip to content

fix(storage): refuse a lost log its mark vouches for, cut only a zero tail past the committed length, remake an unfinished log, report an unwitnessed log - #99

Merged
magicaldave merged 5 commits into
fix/fair-claim-lapsefrom
fix/storage-witness-gaps
Nov 13, 2026
Merged

magicaldave merged 5 commits into
fix/fair-claim-lapsefrom
fix/storage-witness-gaps

Conversation

@magicaldave

Copy link
Copy Markdown
Member

Storage gaps around the committed-length witness, from the review of 2026-11-13: P2-14, P2-15 and two P3s. Dave decided to keep refusing a damaged mark; this PR keeps that and closes the gaps around it. Draft; stacked on #98; do not merge without review.

P2-14. A lost log that its mark vouches for is refused, not reopened empty

Before: if a sync store's log was gone while its mark survived, sql.open made an empty log and deleted the mark, which was the only evidence of the loss.

Now: a mark that says transactions were committed (sequence > 0), or that cannot be read, refuses the open:

the log is gone, and the committed-length mark beside it says N transactions were committed to it: it was lost or moved (remove <mark> only to accept that loss and start an empty log)

Removing the mark is the same explicit action as for a log found short. Managed and catalog databases are left as they were, through a new manifestNamesLog option set by catalog.jsl and hybrid.jsl: their manifest is the authority, and they already refuse a missing log it names.

P2-15. A zero tail past the committed length is cut, not refused

Before: a power cut can leave a log's end allocated and never written. A sound store then refused to open with invalid frame length or checksum mismatch.

Now: a broken frame is treated as a torn tail and cut only when all of these hold:

  • it lies wholly at or past the mark's committed length;
  • the log holds at least the mark's committed transactions;
  • everything from it to the end of the file is zeros: a zeroed header, or a header and then zeros.

Anything else is refused as before: non-zero garbage, or zeros where committed transactions should be. The mark is now read before the log, which also lets a damaged mark be refused before anything is read.

P3. A new log whose creation was cut off is made again

Before: a process killed after createDurable named a new log but before its header was written left a 0-byte or partial header. That was refused for good as invalid file header. The reviewer reproduced it with a SIGXFSZ kill.

Now: a sync log that is empty, or only a prefix of the log magic, can hold no transaction. It is removed and made again.

P3. Running without a mark is visible

store:witness() (optional on the Store type) answers:

  • witnessed;
  • unwitnessed: synchronized, but the mark could not be made, so a log found short will not be caught;
  • not-synchronized.

Tests (tests/wal-committed-length.jsl, 4 new; each fails on ba83c5a)

  • Lost log refused: a lost log with its mark is refused and the mark left in place; after the mark is removed, an empty log opens. On the old code: expected true to be false.
  • Zero tails: zeros and header-then-zeros past the committed length are cut, with 5 rows kept and the file truncated; garbage there is refused, and so are zeros where committed frames were. On the old code: expected "zeros: false" to be "zeros: true".
  • Unfinished new log: a log of "" or "JQL" is made again. On the old code: invalid file header.
  • Witness: witness() says witnessed, and unwitnessed when making the mark fails.
  • One existing test changed, deliberately. In "is not kept by a store that is not synchronized, and a stale one is discarded with a new log", the second half encoded the old contract (a stale mark beside a missing log is discarded). It now expects the refusal, then accepts the loss by removing the mark, and the new log opens and is witnessed as before.

Gates on this branch (24820e0)

jess main 3c7639f, l3i 2.10.0:

  • jess package test .: exit 0, 1,418 passed, 0 failed (style suite included).
  • jess package check .: errors: [].

A later PR (test/memfs-fidelity) adds a fault-lab torn mode. Its sweep fails on the storage before this PR with invalid frame length, and passes on this one.

Rollback

Revert this commit. A standalone store whose log is lost then opens empty again.

… tail past the committed length, remake an unfinished new log, and say when a log runs unwitnessed
…nowledged tail is cut, cut a last frame writeback tore, keep a damaged log and a standalone hybrid's lost log refused
@magicaldave

Copy link
Copy Markdown
Member Author

Follow-up f0fb2af on the re-review:

  • P2-c: hybrid takes manifestNamesLog from its options, default false; only managed passes true. A standalone hybrid over a lost log is now refused with its mark kept.
  • P2-d: the committed-length mark is now synchronized with every commit or batch (walmark.record syncs), so after a crash it covers every acknowledged commit. A zero cut is also bounded to one batch (256 × (8 + maxFrameBytes)). Your shape (mark 563, 267 acknowledged bytes zeroed) is refused with invalid frame length.
    • Cost (btrfs, bench/group-commit.jsl, 960 rows): solo went from 0.98 s to 1.51 s; batch of 32 went from 0.26 s to 0.35 s.
  • P2-e: a last frame that fails its checksum is cut when it lies past the mark and only zeros follow its claimed end. Your torn-page probe reopens with 5 rows.
    • The old garbage case in the zero-tail test was exactly this shape: a whole garbled frame at EOF past the mark. It now opens. It was replaced by two refused cases: garbage past a frame's end, and garbage as a header.
  • P3: an empty or partial-magic log whose mark claims transactions is refused unchanged; the log and mark are left as they were.

Tests: 5 new in tests/wal-committed-length.jsl; all fail on 24820e0.
Gates at f0fb2af: jess package test . 1,428 passed, 0 failed; jess package check . errors: [].

…ot be synchronized, and say the store is unwitnessed
…hronized at close and checkpoint only, and say a store whose mark cannot be written is unwitnessed
@magicaldave

Copy link
Copy Markdown
Member Author

Follow-up 10b5d34 (with 0c2de56 before it). Per Dave's decision, fsync is trusted.

  • The per-commit mark sync from f0fb2af is reverted. The mark is synchronized at close and before a checkpoint only, as before fix(storage): refuse a lost log its mark vouches for, cut only a zero tail past the committed length, remake an unfinished log, report an unwitnessed log #99.

    • An acknowledged commit was synchronized, so the zeros after the last whole frame can only be an append that was never acknowledged, and they are cut.
    • The cut stays bounded to one batch of frames.
    • src/walmark.jsl says what is trusted and why.
  • The reviewer's "zeroed acknowledged frames past a stale mark" test is replaced by a note in tests/wal-committed-length.jsl. It needs a disk that reports a synchronization it did not do, which is outside the threat model now. Zeros where the mark claims commits are still refused, and tested.

  • Kept: the torn-page cut, the lost-log refusal (a standalone hybrid included), refusing without remaking when the mark claims commits, and the zero cut's bound.

  • P2-1: a mark record that cannot be written now makes the store say unwitnessed. Commits go on, and the mark is behind but still true.

    • The writer is kept, so a later checkpoint still withdraws the mark before the shorter log takes its name. Dropping it outright broke that, and the fault-lab checkpoint sweep caught it.
    • Test: a write fault on the mark, 15 more acknowledged commits, witness() is unwitnessed, and all 20 rows survive a crash. It fails on f0fb2af.
  • bench/group-commit.jsl on btrfs, 960 rows, 3 runs each (the bench's os.clock, then wall):

    Mode Per-commit mark sync (f0fb2af) Now
    single commits 1,397–1,472 ms (1.63–1.82 s) 726–764 ms (0.95–1.01 s)
    batches of 32 61–67 ms (0.28–0.30 s) 42–44 ms (0.26–0.27 s)
  • Optional P3 not done: the inFlight bound is still one batch of maximal frames. Tightening it to the largest batch actually written would need that number persisted.

  • Gates at 10b5d34: jess package test . 1,428 passed, 0 failed; jess package check . errors: [].

@magicaldave
magicaldave marked this pull request as ready for review November 13, 2026 22:43
@magicaldave
magicaldave merged commit 593b378 into main Nov 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant