Skip to content

Fix flaky ipfs-cache warm tests (main is red) - #310

Merged
chiliec merged 1 commit into
mainfrom
fix/ipfs-cache-test-flake
Sep 9, 2026
Merged

Fix flaky ipfs-cache warm tests (main is red)#310
chiliec merged 1 commit into
mainfrom
fix/ipfs-cache-test-flake

Conversation

@chiliec

@chiliec chiliec commented Sep 9, 2026

Copy link
Copy Markdown
Member

main went red on the #309 merge run — but on a test unrelated to that PR: warm pulls images in the background and swallows failures (expected: 1, actual: 0). #309's own PR run passed the same file four minutes earlier. It's a long-standing flake that finally lost the dice.

Mechanism

settle() waited for warm()'s fire-and-forget chain by draining 20 setImmediate turns. That barrier expires in ~0.5 ms on an idle loop (measured), while the chain it guards contains real threadpool I/O — the cache-miss fs read. Turn count was never a proxy for wall clock.

On a 2-core CI runner, node --test forks one process per test file, so the threadpool and disk are heavily contended and that read takes milliseconds. The barrier had long since expired, h.errors was still empty, and the assertion read 0.

Fix

Poll for the effect each test actually asserts, with a 5s deadline so a genuine hang still fails the assertion rather than hanging the run.

The warm caches what a later get would have fetched case waits on the entry appearing under its final name — waiting on the fetch call isn't enough, since a get() racing the pending write would re-fetch and break the very cache hit the test checks for.

Verification

  • ipfs-cache.test.ts 20x locally: 14/14 pass every run
  • npm run test:backend 625/625, lint + typecheck clean
  • Measured the old barrier at 0.560 ms to confirm the mechanism

Honest caveat: I could not reproduce the original failure locally (12 parallel runs of the old code all passed on this machine) — it needs the contention of a small CI runner. The argument for the fix is structural rather than a red-to-green repro: the new barrier is condition-based, so it cannot expire early no matter the timing.

settle() drained 20 setImmediate turns to wait for warm()'s fire-and-forget
chain. That barrier expires in ~0.5ms on an idle loop, while the chain it
guards contains real threadpool I/O — so on a loaded CI runner (node --test
forks one process per file across 2 cores) the fs read had not landed and
the assertions read an empty result. Turn count was never a proxy for
wall clock.

Poll for the effect each test actually asserts instead, with a 5s deadline
so a genuine hang still fails the assertion rather than hanging the run.
The warm-caches case waits on the entry appearing under its final name: the
fetch call landing is not enough, since a get() racing the pending write
would re-fetch and break the very hit the test checks for.
@chiliec
chiliec merged commit c2b9e0c into main Sep 9, 2026
3 checks passed
@chiliec
chiliec deleted the fix/ipfs-cache-test-flake branch September 9, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant