Sync → No randomized multi device simulation harness

[revett]

Every concurrency bug fixed so far (#83, #84, #86, #87, #88) was found by human reasoning, one scenario at a time. There is no harness that drives N simulated devices through random interleavings of edits, deletes, renames, syncs, crashes, and injected storage faults against the in-memory fakes, asserting the core invariants (no silent loss, convergence, manifest always matches bucket) and replaying failures from a seed. The pure module plus fakes architecture already pays for this; the class of bug most likely to survive into dogfooding is precisely the interleaving nobody hand-reasoned. Supersedes the vaguer #73.

[revett]

This is the same problem Dropbox solved when they rewrote their sync engine
(Nucleus) in Rust. Worth stealing their approach wholesale rather than inventing
one.

They built two harnesses, both fully deterministic and seed replayable:

• Trinity → end to end chaos harness. Mocks the filesystem, network, and clock, then uses a seeded RNG to inject failures and drop, duplicate, reorder, and interleave operations across simulated devices to shake out race conditions. Ran ~10x faster than native filesystem tests, tens of millions of randomized runs nightly. This is the direct analogue of what this issue asks for.
• CanopyCheck → narrower property based tester for the sync planner. Randomly generates initial trees, infers what the converged trees should be, checks the planner terminates and converges, then minimises any failing case to its simplest form.

The design principles that make it work, and that we should copy:

1. Everything mockable is mocked (fs, network, clock, storage) so a run is pure and deterministic. Our pure module plus fakes architecture already gives us this for free.
2. A single seed determines the entire run, so any failure replays exactly. No flaky tests, just seeds to reproduce.
3. Automatic minimisation → when a seed fails, shrink the operation sequence to the smallest reproduction before a human ever looks at it.

FoundationDB pioneered this (deterministic simulation testing) and it is arguably why both systems are trusted with data nobody can afford to lose. Same bar we hold ourselves to: would we point Geode at our own vault?

• Testing our new sync engine (Trinity, CanopyCheck):
  https://dropbox.tech/infrastructure/-testing-our-new-sync-engine
• Rewriting the heart of our sync engine (Nucleus background):
  https://dropbox.tech/infrastructure/rewriting-the-heart-of-our-sync-engine
• FoundationDB simulation and deterministic testing:
  https://apple.github.io/foundationdb/testing.html
• Talk: "Testing Distributed Systems w/ Deterministic Simulation" (Will Wilson):
  https://www.youtube.com/watch?v=4fFDFbi3toc