Range sync bans peers when the execution client wrongly returns INVALID

[nflaig]

Describe the bug

EXECUTION_ENGINE_INVALID is treated as a peer attributable failure in range sync (isPeerAttributableFailure). When a batch fails MAX_BATCH_PROCESSING_ATTEMPTS times with it, shouldReportPeerOnBatchError reports every peer that served a failed attempt with LowToleranceError (-10), so a peer is banned (-50) after 5 rounds.

This is correct when a peer serves a block with a bad payload, but the same verdict is produced when our own execution client is wrong. On glamsterdam-devnet-9 (2026-09-04, genesis resync) besu 26.9-develop marked a canonical payload INVALID after a "World State Root does not match" on block 4334 and kept its descendants in the bad block cache, returning INVALID without re-execution (reported to besu). Lodestar retried the batch, every retry was served by other peers with the correct blocks (same roots, hash verified) and every round penalized them.

Observed on the besu-2 node over ~2.7 h: 219 peer bans and 1140 batch processing errors with the head stuck at slot 4832. The four other nodes resyncing at the same time with the same peer set had 7 to 21 bans.

Expected behavior

When independent peers serve identical data for a batch and the execution client still returns INVALID, the peers are not at fault and should not be penalized. Batch.failedProcessingAttempts already records the peers per attempt, and since #10025/#10028 the rejected block is dropped from the seen cache and re-downloaded, so the retries do compare independent peers.

Options

1. do not report peers for EXECUTION_ENGINE_INVALID once two or more distinct peers served the same block roots for the failing batch
2. or report EXECUTION_ENGINE_INVALID with MidToleranceError so it alone cannot ban a peer, and keep LowToleranceError for failures that prove bad content (INVALID_SIGNATURE, INVALID_STATE_ROOT, non-linear segments)

Additional context

• feat: surface range sync batches that fail after max attempts #10029 surfaces the stall itself (warn after max attempts, lodestar_sync_range_batch_process_error_total{code})
• fix: treat dial timeouts as HighToleranceError for all methods #9877 / fix: rate limit repeated peer score penalties #9878: the ban wave has the same self sustaining shape, fewer peers concentrate the penalties on the remaining ones
• besu side: in-memory bad block manager, cleared by a restart, re-poisoned within minutes under CL driven genesis sync

[twoeths]

this is controversial
on another day, when we have good EL, some peers serve us bad payload, we'll open another issue to penalize those peers for EXECUTION_ENGINE_INVALID
what if we don't penalize peers in this case? I think the node cannot function well with a bad EL, this will leads to other issues anyway

the question is more about do we trust the EL implementation or a peer?

[nflaig]

I agree, this issue is not well phrased, I think for EXECUTION_ENGINE_INVALID we should penalize peers during sync, generally, this seems more like a devnet issue, later on mainnet, we can trust the EL implementation and false invalids should not happen frequently. The problem is only if we downscore peers due to other engine errors, eg. connection related issues that cause http errors, timeouts etc., so differentiating based on the error code makes sense

[twoeths]

we have this function for that

lodestar/packages/beacon-node/src/sync/range/batch.ts

Line 837 in 4101ad2

function isPeerAttributableFailure(code: BlockErrorCode | PayloadErrorCode | null): boolean {

@lodekeeper please double check if follows what @nflaig asked for

[lodekeeper]

@twoeths @nflaig yes — isPeerAttributableFailure (batch.ts L837) already implements the code-based differentiation from @nflaig's comment. It's a whitelist: an attempt is peer-attributable only for content-proving codes — EXECUTION_ENGINE_INVALID, INVALID_SIGNATURE, INVALID_STATE_ROOT, PER_BLOCK_PROCESSING_ERROR, the non-linear-segment codes, envelope mismatches, and blacklisted blocks. Everything else hits default: return false.

The key point is that the two engine outcomes are already distinct codes (verifyBlocksExecutionPayloads.ts):

• ExecutionPayloadStatus.INVALID → EXECUTION_ENGINE_INVALID (L195-204) → peer-attributable (LowToleranceError). So a definitive INVALID verdict does downscore the serving peer — matching "we should penalize peers for EXECUTION_ENGINE_INVALID during sync."
• ELERROR / UNAVAILABLE / INVALID_BLOCK_HASH → EXECUTION_ENGINE_ERROR (L234-238) → not peer-attributable. So connection issues / HTTP errors / timeouts / EL malfunctions do not downscore peers — matching "the problem is only if we downscore peers due to other engine errors."

blockError.ts L68-74 documents exactly this split (EXECUTION_ENGINE_ERROR = local malfunction, not downscored; EXECUTION_ENGINE_INVALID = evidence about the data, MAY be downscored).

So comment [2]'s ask is already satisfied on unstable.

The one thing it does not do is the issue's option 1 — skip peer reporting for EXECUTION_ENGINE_INVALID once 2+ distinct peers served the same block roots for the failing batch. That's the guard specifically against a false INVALID from our own EL (the glamsterdam-devnet-9 besu case that opened this). It's not implemented today, and given the discussion leans toward keeping INVALID-penalization for mainnet (false invalids rare), it may be intentionally out of scope — flagging it only so it's an explicit call rather than an assumed gap.