Repository navigation
Conversation
3 voters in parallel fork branches, quorum requires all 3. voterX and voterA vote before crash. After recovery, restoreContext processes branches in HashMap iteration order. Branch "a" (2 votes, newer snapshot) processes first, then branch "x" (1 vote, older snapshot) overwrites — voterA's vote is lost. voterZ then casts the deciding vote. The listener sees 2/3 votes instead of 3/3. The quorum never fires. Uses a single-threaded fork executor and branch names chosen to force deterministic HashMap iteration that triggers the overwrite. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mdproctor
force-pushed
the
test/fork-metadata-restore-order
branch
from
October 22, 2026 19:32
8c00391 to
2eb0361
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Timing-dependent setup undermines the reproduction, and phase-one failures bypass resource cleanup.
2 open issues
What changed in this PR
Adds a failing regression test demonstrating how fork crash recovery can overwrite newer instance metadata with an older branch snapshot. This supports the listener-state recovery discussion without changing production behavior.
Changes:
- Records completed task names in metadata across three fork branches.
- Uses MVStore persistence to simulate shutdown and recovery.
- Configures single-threaded recovery to expose stale metadata overwrites.
| File | Description |
|---|---|
impl/test/src/test/java/org/openworkflow/sdk/impl/test/ForkMetadataRestoreTest.java |
Adds the fork metadata-loss reproduction and recovery assertion. |
Review effort: Balanced
Comment on lines
+140
to
+141
| assertThat(twoTasksDone.await(5, TimeUnit.SECONDS)).isTrue(); | ||
| Thread.sleep(200); |
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Comment on lines
+144
to
+145
| assertThat(twoVotesCast.await(5, TimeUnit.SECONDS)).isTrue(); | ||
| Thread.sleep(200); |
Comment on lines
+204
to
+208
| restored.start(); | ||
|
|
||
| Awaitility.await() | ||
| .atMost(Duration.ofSeconds(10)) | ||
| .until(() -> restored.status() == WorkflowStatus.COMPLETED); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
removeMetadata+addMetadataIfAbsent. Two of three fork branches complete before crash. After recovery,restoreContextprocesses each branch's persisted metadata snapshot — but the last branch to process wins, and with fork branches the order is non-deterministic (HashMap iteration)What the test shows
taskXcompletes first (metadata ="taskX"),taskAcompletes second (metadata ="taskX,taskA")restoreContextfor branch "a" (newer:"taskX,taskA") runs first, then branch "x" (older:"taskX") overwritesonTaskCompleted— sees"taskX", missing"taskA"Discussion
Looking for guidance on the intended architecture for listener state recovery across crashes with forks. Options discussed:
onTaskRestoredevent so listeners can rebuild from per-task factsrestoreContext(PR fix: restoreContext overwrites newer metadata in parallel branches #1794, now closed)🤖 Generated with Claude Code