test_checkpoint_summary_is_present_and_self_consistent is intermittently flaky; cause unidentified

[adityamukho]

What

tests/test_at_scale_ingestion_benchmark.py::TestRunIngestionBenchmark::test_checkpoint_summary_is_present_and_self_consistent fails intermittently at a low rate. Surfaced while verifying the DB lease protocol branch (PR #269).

Evidence, and its limits

observation	result
full-suite run on the lease branch	1 failure
second full-suite run, same tree	passed
12 isolated runs, lease branch	1 failure
12 isolated runs, master worktree	0 failures
20 further isolated runs, lease branch	0 failures
8 runs under 8 busy loops	0 failures

Roughly 1 failure in 48 runs on the branch, 0 in 12 on master.

Attribution is unproven in both directions. 1-vs-0 over those sample sizes is not significant, and I never captured the assertion text — so which of the four assertions failed is unknown. Do not record this as pre-existing, and do not record it as caused by the lease protocol. Both claims are currently unsupported.

What was ruled out

checkpoints == 0 is not the cause. Measured the distribution directly over 10 clean benchmark runs: 3-4 checkpoints, never 0. _CheckpointPolicy._budget_elapsed returns True when nothing has been measured yet ("checkpoint once to seed d"), so the first gated checkpoint always fires provided _db_checkpoint_gated is called at all.

Not load-sensitive in the way #261's matcher test was — 0/8 under 8 busy loops on an 8-core box.

Suggested first step

Capture the failure rather than sampling for it. Run with -l --tb=long in a loop until it reproduces, and record which assertion failed and the full checkpoint_summary dict at that moment. The remaining candidates are the realised_duty == approx(total_seconds / elapsed_seconds) self-consistency check and summary is not None.

Found during PR #269. Not a blocker for it.

[adityamukho]

Narrowed to one assertion, and made it self-diagnosing — PR #288

Two of the four assertions cannot fail, which removes one of the two candidates this issue named:

• summary["realised_duty"] == approx(total_seconds / elapsed_seconds) is a tautology. summary() computes realised_duty as self.total_seconds / elapsed from the same elapsed float it returns (mcp_server.py:3988-3995) — the assertion compares an expression against itself.
• summary["total_seconds"] >= 0.0 sums time.monotonic deltas.

checkpoints >= 1 is near-guaranteed by the seed rule and I measured its floor rather than assuming it: exactly 4 gated calls per run, of which 1-4 checkpoint; under 8x contention the floor is checkpoints=1, suppressed=3, never 0. (Note for anyone tempted to tighten that bound to the 3-4 recorded here: 1 is reachable under load.)

So the assertion that fired is summary is not None. The window that produces it is _run_ingestion's first ~130 lines, before _CheckpointPolicy is constructed at mcp_server.py:11165 — both finally blocks that publish checkpoint_summary are guarded on the policy being non-None. Proven by injecting a failure into _load_ingestion_preload_state (:11100, before the policy):

PROBE final_status= error
PROBE error= injected early failure (stand-in for a transient)
PROBE checkpoint_summary= None
E   assert None is not None

The _frontier_load failure that TestCheckpointDutySummaryPublication::test_summary_survives_a_pre_write_scope_failure covers runs at :11200, after the policy, and does publish an all-zeros summary — so that test does not cover this window.

Why the cause was unidentifiable: _run_ingestion catches its exception, stores it in _ingest_progress["error"], and returns without printing it. The text reached neither the metrics JSON (no such key), nor scan_ingestion_stderr's error_signals (nothing was written to fd 2), nor the test's failure message. PR #288 records it as metrics["ingest_error"], carries it through the cost probe, stops report.py from rendering an errored run as "pre-2026-08-08 harness; assume once-per-commit cadence", and makes the test assert final_status first so the error text lands in the message.

Reproduction attempts — 0 failures in ~800 runs on master

attempt	runs	failures
serial isolated single-test	300	0
isolated under 8 concurrent pytest processes	480	0
whole test_at_scale_ingestion_benchmark.py	20	0
full suite (1581 tests)	2	0
full suite with gc.disable()	1	0

The pre-policy code path has not changed since #269 merged, so today's master is the tree the flake was seen on.

A lease-leak theory was tested and rejected. _DbLeaseManager._detect_leaked_handle raises under pytest when the previously released handle is still alive, which is weakref/GC-timing dependent and would have been a clean explanation for a cross-test intermittent. Instrumenting it across full-suite runs with GC both enabled and disabled produced 0 stale-handle detections.

Leaving this open. The next occurrence will name its own cause instead of assert None is not None.