Exp 283: the second walk over the same rows

[danReynolds]

Hypothesis

A reactive stream re-runs its query whenever a write dirties something it
depends on, and has to decide whether the fresh result is worth emitting. Exp
075 made the common case cheap: resqlite_query_hash steps the statement to
completion in C, folds every cell into an FNV digest, and an unchanged stream
costs one SQLite pass and no Dart objects at all.

What nobody had looked at is what happens when the digest does move. The
statement has already been consumed, so it gets stepped a second time, from
the top, to build the result. That second walk has been there since April and
had never been measured — the two paths were always described as "hash only"
versus "hash plus decode", which quietly hides that the second one is two SQLite
passes rather than one.

resqlite already has a decoder that does both at once: exp 097's
decodeQueryWithInitialHash fills the result and folds the identical canonical
digest in a single pass, but was scoped to initial stream registration. Exp 228
later named this exact reopening — early rejection may be revisited if "the
changed-result decode can produce the canonical hash without another full pass."

The bet was that a stream's reruns do not change at random: a partition under
active writes changes on rerun after rerun, so the previous rerun's outcome is
enough to pick the right decoder.

Approach

StreamEntry gains one bool lastRerunChanged; ReaderPool.selectIfChanged
forwards it as SelectIfChangedRequest.decodeFirst; executeQueryIfChanged
grows a second arm that calls exp 097's decoder and discards the built result if
the digest turns out to match after all. The flag lives on the main isolate for
the reason exp 260 established for the row hint — a reader worker sees a sample
of a stream's reruns and is destroyed by the sacrifice path.

The runtime is reverted on this branch. It is preserved at
archive/exp-283,
and the temporary rerun census at
archive/exp-283-census.
What ships here is the writeup, two focused harnesses, the signal entry, and
retroactive claims 097.1 and 228.1. Full detail in
experiments/283-stream-rerun-one-pass.md.

Results

The premise is true. A temporary census counted reruns on scaled
reproductions of the release suite's three reactive lanes:

lane	reruns	of a possible	changed
high-cardinality fan-out	847	20,000	56.4%
keyed PK	1,071	10,000	0.28%
feed (latest-50)	100	100	0%

Per-stream coalescing is far stronger than the suites' own docstrings assume,
and the fan-out lane's reruns are not mostly unchanged — the familiar "99 of
100 streams are unchanged" describes the writes, not the reruns.

The mechanism is real. With no pool, no isolates and no message hop, the
removed pass is 35–45% of a changed rerun's SQLite-and-decode work at every
width from 1 row to 1,000 (10.72 → 6.50 µs at 100×2; 87.94 → 53.08 µs at
1000×2).

End to end it wins, but narrowly and not where expected. Four order-flipped
passes, two AOT bundles from separate worktrees, one lane per process, 41
samples after 8 warmup:

lane	median Δ	pooled Δ	drift verdict
fanout-wide — 20 streams × 1,000-row partitions	−11.1%	−11.3%	reproduced (−11.5% / −9.2%)
fanout — 100 streams × 100-row partitions	−3.0%	−1.7%	drift-suspected
keyed-pk (guard)	−1.5%	−4.2%	neutral
feed (guard)	−2.2%	−2.0%	neutral
writes (zero-ceiling control)	+3.7%	+5.6%	neutral

The win scales with the result the pass re-walks: at 1,000 rows the removed pass
is 34.86 µs and the lane moves −9% to −12% in every pass; at 100 rows it is
4.22 µs and the effect does not clear the harness floor (the control moved
+16.5% in its worst pass).

An earlier version of this PR reported −12.5% on fanout and −5.7% on
fanout-wide. Those figures are withdrawn. Review caught that the drain
harness's sentinel was a stream the burst could also change, with its
completer armed before the burst started, so each sample ended on whichever of
that stream's reruns fired first — a random prefix of the backlog whose length
depended on how fast each arm finished changed reruns. Reserving a partition
for the sentinel, excluding it from the burst, and arming the completer
afterwards produced the table above. The rule is recorded as claim 283.8. Four
order-flipped passes had agreed on the wrong number, which is the part worth
remembering: order-flipping catches drift, not a metric measuring the wrong
interval in both arms.

And then the release suite flagged it. --fail-on-regression fired on
High-Cardinality Stream Fan-out / 100 streams × 200 writes / resqlite at
+91.4% — the same shape the A/B says is 12–25% faster. Both readings are
correct, and unpicking that is the experiment's real result.

That lane's wall is quantized: its settle loop waits a 200 ms quiet window
and stops at the first one with no new emission, so a measured iteration that
emits nothing costs one window and one that emits anything costs two. The lane
is bimodal at ~246 ms and ~448 ms around roughly 46 ms of work, and +91.4% is
exactly one window. Filed as
#318.

What decides the mode is whether a convergence emission lands in a measured
iteration. Running the lane 35 times per arm:

arm	slow mode	rate
origin/main	2 / 35	5.7%
candidate	10 / 35	28.6%
candidate with the decode-first branch compiled out	1 / 14	7.1%

The third row is the attribution — wiring the flag through every layer while the
worker ignores it reproduces the baseline rate, so the shift is the decoder.
Harness-side content comparison found zero same-content emissions on either
arm: every extra emission carries genuinely different data, and the candidate
simply makes more of them during the burst (post-baseline emissions on the first
iteration: 91–101 on origin/main over six runs, 102–115 on the candidate over
eight — about +12%).

Why: the second walk was not waste

resqlite_query_hash resets the statement on exit, so the decodeQuery
that follows opens a fresh read transaction on a newer WAL snapshot. During a
burst, the rows a stream emits are fresher than the digest stored beside them,
and the stream converges in fewer emissions because each emission has skipped
ahead. The one-pass decoder takes exactly one snapshot — hash and rows always
agree, which is the more defensible contract and what exp 228's invariant asks
for in spirit — but it gives up that free refresh and needs one more rerun to
catch up.

So the second SQLite pass costs 35–45% of a changed rerun and it buys emission
freshness. Nobody knew it was buying anything, which is why this looked like
free money.

Outcome

Rejected, on two counts that compound. The win is narrower than first
measured — real and reproduced on wide results, invisible on narrow ones — and
what it does deliver, it cannot pay for: trading roughly 12% more emissions
during a write burst for less wall time is a semantic-shaped trade in the exact
dimension the library advertises — resqlite's reactive story is
that hash suppression keeps streams from re-emitting. This repo's record on
semantic-shape trades (exps 197, 212, 213) is that a reproduced win does not
settle them, so that call belongs to a maintainer rather than a scheduled run,
and the runtime is archived.

It reopens on any of three things: a design that keeps one snapshot per rerun
and still converges as fast (a cheap freshness re-check — row count, a
data-version probe — rather than a second full walk, gated on result size since
that is where the win lives); evidence that burst-time intermediate emissions
are immaterial because the UI layer coalesces them; or a maintainer's ruling
that the wall time is worth the emissions.

What stands regardless of the verdict: the rerun census (claim 283.1), the pass
price (283.2), the predictor's accuracy (283.3), the corrected end-to-end result
(283.4), the awaited-burst measurement trap (283.5), the freshness finding
(283.6), the lane quantization (283.7) and the sentinel rule (283.8).

Orthogonal to #155 (exp 160, incremental view maintenance): that prototype
reduces how often a stream reaches the fallback rerun; this was about making the
fallback rerun cheaper.

Host caveat. Load average ran 1.8–14.1; mediaanalysisd held a core for much
of the session, and the corrected collection was taken in the quietest window
available (1.8–4.1). The zero-ceiling writes control moved +16.5% in its worst
pass and +3.7% at the median, which is why only fanout-wide — consistent at
−9% to −12% in every pass — is read as an effect.

Test plan

• dart analyze --fatal-infos clean on the reverted tree
• dart test test/query_decoder_test.dart test/stream_test.dart test/reader_pool_test.dart test/experiment_outcomes_test.dart test/knowledge_impact_test.dart — all passing
• While the runtime was in place: 136 tests passing including a new one asserting the two arms agree on digest, row count and decoded values in both directions (preserved at archive/exp-283)
• Four order-flipped AOT A/B passes from separate worktrees on the corrected harness; benchmark/ab_drift_check.dart verdicts above
• Headline release suite, clean tree, --repeat=5 --fail-on-regression --fail-on-memory-regression --compare-to=…exp270-parent-baseline.md — the run that caught the trade-off; not committed, since no runtime ships
• benchmark/finalize_experiment.dart green

[github-actions]

Belief impact

Learned

• 097.1 · One-pass initial stream decode and hash
  decodeQueryWithInitialHash (driving the native resqlite_step_row_hash) produces a result and a result digest in one SQLite step pass, and that di…
• 228.1 · Restore canonical hashes after stream growth
  Any value stored in StreamEntry.lastResultHash must be the canonical digest of the complete corresponding result. Exp 077's row-count shortcut retu…
• 283.1 · The second walk over the same rows
  Per-stream rerun coalescing dominates the rerun count in reactive fan-out, and the reruns that survive it are not overwhelmingly unchanged. On a scal…
• 283.2 · The second walk over the same rows
  The second SQLite pass a changed stream rerun makes is 35-45% of its SQLite-and-decode work, at every width measured. Timed with no pool, no isolates…
• 283.3 · The second walk over the same rows
  A stream's reruns change in runs, so the previous rerun's outcome is a usable predictor of the next one and needs no more state than one bool per str…
• 283.4 · The second walk over the same rows
  Decoding during the hash pass when the previous rerun changed is worth a reproduced -11.3% pooled (-9% to -12% in every pass) on 20 streams over 1,00…
• 283.5 · The second walk over the same rows
  A reactive fan-out A/B cannot be read off an awaited write-by-write burst. With each write awaited, every rerun overlaps the next write's latency and…
• 283.6 · The second walk over the same rows
  The two SQLite passes a changed stream rerun makes do not read the same database state, and the second one is load-bearing. resqlite_query_hash res…
• 283.7 · The second walk over the same rows
  High-Cardinality Stream Fan-out (v1) / 100 streams x 200 writes / resqlite cannot resolve anything below 200 ms, and its headline value is bimodal …
• 283.8 · The second walk over the same rows
  A drain metric's sentinel must be unreachable by the workload it is draining. Exp 283's first harness armed its completer before the burst and let it…

What this changed

Initial stream registration decoded the result for subscribers and then replayed the same statement through resqlite_query_hash to establish the baseline. resqlite_step_row_hash folds the same masked-FNV accumulator while it fills the cell buffer, so decodeQueryWithInitialHash produces result and canonical digest in one SQLite pass. Stream setup improved 14-16% on fan-out and subscribe/cancel churn.

Stream re-queries were deliberately left on the hash-only path so an unchanged rerun could skip Dart decoding entirely; the one-pass decoder was scoped to initial registration only.

Exp 077's row-count shortcut correctly proved that a growing result differed, but resqlite_query_hash returned the prefix-only accumulator and executeQueryIfChanged cached it as the complete changed result's next baseline. Once the cached count advanced, the immediate identical rerun computed a full hash, mismatched the partial baseline, decoded all rows, and publicly re-emitted unchanged data once.

A stream result hash has two roles: change detection now and cached identity later. Any value stored in StreamEntry.lastResultHash must be canonical for the complete corresponding result; a fast-reject digest that stops early can only be safe as a visibly non-cacheable sentinel.

Deleting the shortcut and its private FFI row-count argument moves redundant immediate-no-op decodes from 9/9 to 0/9 in all three focused passes. The immediate unchanged p50 improves 56-64% and the complete growth-plus-no-op cycle improves 19-36%; the pure growth leg is noisy around parity, consistent with exp 077's original sub-noise saving.

We believed a stream rerun's changed path was one cheap C hash pass plus a Dart decode. It is two complete SQLite walks, and claim 283.2 prices the second at 35-45% of the rerun's SQLite-and-decode work at every width from one row to a thousand. That much was the expected finding. The unexpected one is claim 283.6: the two walks read two different snapshots, because resqlite_query_hash resets on exit and the decode that follows opens a fresh read transaction. The rows a stream emits are fresher than the digest stored beside them, and that accident is what makes it converge in as few emissions as it does. Anyone who reads the rerun path as 'hash, then decode if needed' should now read it as 'hash, then re-read', and should not treat the second walk as removable without replacing what it buys.

We believed reactive fan-out reruns were overwhelmingly unchanged - the release suites' own docstrings say so, and stream_rerun_latency.dart is built around one changed stream in a hundred. Claim 283.1 shows that describes the writes. Per-stream coalescing collapses 20,000 arithmetic reruns into 847 real ones on the high-cardinality shape, and 56% of the survivors change. Stop dividing stream count by write count: the quantity that matters is reruns after coalescing, and it is smaller and much more change-dense than the shape suggests. Keyed-PK (0.28%) and feed (0%) remain genuinely change-free, so the direction has two regimes rather than one.

We believed the release suite's largest resqlite lane measured stream fan-out performance. Claim 283.7 shows it mostly measures a 200 ms sleep, twice or once, decided by whether a convergence emission happens to land in a measured iteration. It cannot resolve anything smaller than a settle window, its two modes are ~246 ms and ~448 ms, and a +91% flag on it is one window. Do not size a stream candidate against it, and do not read its trend line as a performance signal; the drain metric in stream_rerun_one_pass_ab.dart is what to use instead.

A measurement rule this run had to learn twice. Claim 283.5: an awaited write-by-write fan-out burst cannot see reader-side rerun cost, because each rerun hides inside the next write's latency. Claim 283.8: the drain metric that replaces it must use a sentinel the burst cannot reach, or each sample ends on a race and the race reads as a code effect. The first cost a metric that reported +1.2% on a lane the mechanism moves 11%; the second cost a four-pass collection that agreed on a withdrawn number until a reviewer looked at the harness. Both are properties of measuring work that runs concurrently with something you await, and neither is specific to streams.

Claim 283.5 is a measurement rule for this direction, learned the expensive way. An awaited write-by-write fan-out burst cannot see reader-side rerun cost: each rerun hides inside the next write's latency, and a 25-sample pass of that metric put the candidate at +1.2% on a lane the mechanism moves 12%. Issue the burst concurrently and time a sentinel write through to the one stream it must change. The unchanged majority emits nothing and can never be waited on directly, which is exactly why the backlog needs something that can.

[Copilot]

🟡 Changes recommended

The new benchmark harnesses contain correctness/compilation issues (switch fall-through and sentinel-wait logic) and the experiment lacks a required benchmark/results/ artifact for chart linkage.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Records Experiment 283 (“the second walk over the same rows”) in the repo’s experiments system, including a new signal entry, retroactive claims for exps 097 and 228, and two focused benchmark harnesses intended to measure rerun-pass cost and an A/B drain metric.

Changes:

• Add experiment 283 writeup + index row + signal entry, and update the stream-rerun-dispatch direction synthesis.
• Mint retroactive typed claims for experiments 097 and 228 to support 283’s narrative/edges.
• Add two new benchmark harnesses: stream_rerun_pass_price.dart and stream_rerun_one_pass_ab.dart.

File summaries

File	Description
experiments/signals/entries/283.json	New signal entry capturing hypotheses, claims, and next signals for exp 283.
experiments/signals/entries/228.json	Adds retroactive claim 228.1 (canonical hash invariant) referenced by exp 283.
experiments/signals/entries/097.json	Adds retroactive claim 097.1 (one-pass decode+hash equivalence) referenced by exp 283.
experiments/signals/base.json	Updates stream-rerun-dispatch direction status/readout/notes to incorporate exp 283 findings.
experiments/JOURNAL.md	Adds three reusable experiment-methodology lessons from exp 283.
experiments/index/283.json	Adds index row fragment for experiment 283.
experiments/283-stream-rerun-one-pass.md	Adds the full experiment 283 writeup and disposition rationale.
benchmark/experiments/stream_rerun_pass_price.dart	New focused harness to price hash vs decode vs one-pass decode+hash.
benchmark/experiments/stream_rerun_one_pass_ab.dart	New A/B harness to time a concurrent burst + sentinel-drain metric.

Review details

Suppressed comments (1)

benchmark/experiments/stream_rerun_pass_price.dart:156

• This second switch (arm) has the same missing break problem, which would (if it compiled) add the same timing to multiple lists and corrupt the medians.

        switch (arm) {
          case 0:
            hash.add(us);
          case 1:
            decode.add(us);

• Files reviewed: 9/9 changed files
• Comments generated: 3
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.