Skip to content

[glamsterdam-devnet-8] Unknown sync gave up before gossip block came #10018

Description

@twoeths

Describe the bug

log by root

2026-09-06 10:27:25.909 | Sep-06 03:27:25.907[chain]           debug: Persisted payload envelope to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
-- | -- | --
  |   | 2026-09-06 10:27:25.909 | Sep-06 03:27:25.906[chain]           debug: Persisted dataColumnSidecars to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, dataColumnSidecars=128, custodyColumns=128, numBlobs=11 |  
  |   | 2026-09-06 10:27:25.891 | Sep-06 03:27:25.891[chain]         verbose: Execution payload imported slot=170237, builderIndex=10044, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, blockHash=0x3ccdee7a83d51bbeca735129f9ddb2cc2d7093b0ebb898e863ce0ef78cdb84b5, delaySec=1.8880000114440918 |  
  |   | 2026-09-06 10:27:25.842 | Sep-06 03:27:25.842[network]         debug: Received gossip payload envelope slot=170237, currentSlot=170237, peerId=16Uiu2HAmVHg5G6u7rvEuHZpBNUc7C3VxZuU38dtXC7NZrdt7oEsW, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=1.8369998931884766 |  
  |   | 2026-09-06 10:27:25.353 | Sep-06 03:27:25.353[chain]           debug: Cannot serve ExecutionPayloadEnvelopesByRoot: envelope not found slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peer=16...L3XHzb, client=Lodestar |  
  |   | 2026-09-06 10:27:24.428 | Sep-06 03:27:24.428[chain]         verbose: Completed getBlobsV2 for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.427 | Sep-06 03:27:24.427[chain]           debug: getBlobsV2 result for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, result=success_resolved |  
  |   | 2026-09-06 10:27:24.411 | Sep-06 03:27:24.411[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=78, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=9, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.411 | Sep-06 03:27:24.411[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=9, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1, delaySec=0.3970000743865967, secFromSlot=0.3989999294281006, gossipSubnet=78, columnIndex=78, recvToValLatency=0.0019998550415039062, recvToValidation=0.01399993896484375, validationTime=0.012000083923339844 |  
  |   | 2026-09-06 10:27:24.410 | Sep-06 03:27:24.410[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=36, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=8, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.410 | Sep-06 03:27:24.410[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=8, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm9rBtzvh7qpG5ipH6vZ6v11ymwKJPfprNRVpqLiL3XHzb, delaySec=0.39800000190734863, secFromSlot=0.40000009536743164, gossipSubnet=36, columnIndex=36, recvToValLatency=0.002000093460083008, recvToValidation=0.012000083923339844, validationTime=0.009999990463256836 |  
  |   | 2026-09-06 10:27:24.407 | Sep-06 03:27:24.407[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=0, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=7, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.407 | Sep-06 03:27:24.407[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=7, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1, delaySec=0.3970000743865967, secFromSlot=0.3989999294281006, gossipSubnet=0, columnIndex=0, recvToValLatency=0.0019998550415039062, recvToValidation=0.009999990463256836, validationTime=0.00800013542175293 |  
  |   | 2026-09-06 10:27:24.396 | Sep-06 03:27:24.396[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=13, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=6, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.396 | Sep-06 03:27:24.396[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=6, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2630000114440918, secFromSlot=0.375, gossipSubnet=13, columnIndex=13, recvToValLatency=0.1119999885559082, recvToValidation=0.13299989700317383, validationTime=0.020999908447265625 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=24, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=5, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=5, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2539999485015869, secFromSlot=0.375, gossipSubnet=24, columnIndex=24, recvToValLatency=0.12100005149841309, recvToValidation=0.1400001049041748, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=26, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=4, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=4, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.32399988174438477, secFromSlot=0.375, gossipSubnet=26, columnIndex=26, recvToValLatency=0.051000118255615234, recvToValidation=0.07000017166137695, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=68, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=3, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=3, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.31999993324279785, secFromSlot=0.375, gossipSubnet=68, columnIndex=68, recvToValLatency=0.05500006675720215, recvToValidation=0.07400012016296387, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=53, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=2, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=2, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.312000036239624, secFromSlot=0.375, gossipSubnet=53, columnIndex=53, recvToValLatency=0.06299996376037598, recvToValidation=0.0820000171661377, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[chain]           debug: Received gossip data column, payload envelope input not yet complete dataColumnIndex=28, slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=1, sampledColumnsCount=128 |  
  |   | 2026-09-06 10:27:24.394 | Sep-06 03:27:24.394[network]         debug: Received gossip dataColumn slot=170237, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, hasPayload=false, hasAllData=false, hasComputedAllData=false, isComplete=false, receivedColumns=1, sampledColumnsCount=128, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2560000419616699, secFromSlot=0.375, gossipSubnet=28, columnIndex=28, recvToValLatency=0.11899995803833008, recvToValidation=0.1380000114440918, validationTime=0.01900005340576172 |  
  |   | 2026-09-06 10:27:24.386 | Sep-06 03:27:24.380[chain]         verbose: Trigger getBlobsV2 for block slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.386 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.386 | Sep-06 03:27:24.379[sync]            debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.378 | Sep-06 03:27:24.376[sync]          verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Pruned block input slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Persisted blockInput to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Persist block to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, inputType=no-data, numBlobs=11 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]         verbose: Block processed slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.37400007247924805 |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain]         verbose: Strong block detected. Not override fcu call blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, slot=170237, reason=notProposerOfNextSlot |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain]         verbose: New chain head slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.372999906539917 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain]         verbose: Added block to forkchoice and state cache slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain]         verbose: SeenPayloadEnvelopeInput.add created new entry slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, daOutOfRange=false |  
  |   | 2026-09-06 10:27:24.278 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.276[sync]            debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.272 | Sep-06 03:27:24.267[sync]          verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 |  
  |   | 2026-09-06 10:27:24.265 | Sep-06 03:27:24.265[sync]          verbose: Added new payload rootHex to BlockInputSync.pendingPayloads slot=170237, delaySec=0.2650001049041748, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync]            debug: ignored Banning unknown block slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStrings=16...h1ckc1 |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync]            debug: Removing bad/unknown/incomplete BlockInputSyncCacheItem slot=unknown, blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.258 | Error: Error fetching UnknownBlockRoot slot=undefined root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.258 | Sep-06 03:27:24.258[sync]            debug: Ignoring unknown block root after many failed downloads slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingBlocks=1 - Error fetching UnknownBlockRoot slot=undefined root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
  |   | 2026-09-06 10:27:24.255 | Sep-06 03:27:24.255[sync]          verbose: BlockInputSync.downloadBlock() slot=unknown, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingBlocks=1 |  
  |   | 2026-09-06 10:27:24.255 | Sep-06 03:27:24.255[sync]          verbose: Added new rootHex to BlockInputSync.pendingBlocks root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAmLCFHKTFwgbPv4pHF7njWKqoDyrYDMt95axco35h1ckc1

log by slot

2026-09-06 10:27:24.386 | Sep-06 03:27:24.379[sync]            debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[0-127] |  
-- | -- | --
  |   | 2026-09-06 10:27:24.378 | Sep-06 03:27:24.376[sync]          verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.37599992752075195 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Pruned block input slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Persisted blockInput to db slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Persist block to hot DB slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, inputType=no-data, numBlobs=11 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]         verbose: Block processed slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.37400007247924805 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]           debug: Imported block slot=170237, recvToValLatency=0.11500000953674316, recvToValidation=0.11700010299682617, validationTime=0.002000093460083008 |  
  |   | 2026-09-06 10:27:24.374 | Sep-06 03:27:24.374[chain]         verbose: After importBlock caching postState without SSZ cache slot=170237 |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain]         verbose: Strong block detected. Not override fcu call blockRoot=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, slot=170237, reason=notProposerOfNextSlot |  
  |   | 2026-09-06 10:27:24.373 | Sep-06 03:27:24.373[chain]         verbose: New chain head slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, delaySec=0.372999906539917 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain]         verbose: Added block to forkchoice and state cache slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 |  
  |   | 2026-09-06 10:27:24.372 | Sep-06 03:27:24.372[chain]           debug: Verified block signatures slot=170237, recvToValLatency=0.10500001907348633, recvToValidation=0.11500000953674316, validationTime=0.009999990463256836 |  
  |   | 2026-09-06 10:27:24.362 | Sep-06 03:27:24.362[chain]           debug: Verified block state transition slot=170237, recvToValLatency=0.023000001907348633, recvToValidation=0.10500001907348633, validationTime=0.0820000171661377 |  
  |   | 2026-09-06 10:27:24.286 | Sep-06 03:27:24.279[network]         debug: Validated gossip block slot=170237, blockRoot=0xab60…92c0, timeCreatedSec=1788665244.257, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2569999694824219, recvToValLatency=0.019999980926513672, recvToValidation=0.02200007438659668, validationTime=0.002000093460083008, skippedSlots=0 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain]         verbose: SeenPayloadEnvelopeInput.add created new entry slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, daOutOfRange=false |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[chain]           debug: Attempt to cache block but is already cached on BlockInput slot=170237, blockRoot=0xab60…92c0, timeCreatedSec=1788665244.257 |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.277[network]         debug: Received gossip block slot=170237, currentSlot=170237, peerId=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ, delaySec=0.2569999694824219, recvToValLatency=0.019999980926513672 |  
  |   | 2026-09-06 10:27:24.278 | Error: Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.278 | Sep-06 03:27:24.276[sync]            debug: Ignoring unknown payload root after failed download slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 - Error fetching payload by root slot=170237 root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0 after 1: cannot find peer with needed columns=[] |  
  |   | 2026-09-06 10:27:24.272 | Sep-06 03:27:24.267[sync]          verbose: BlockInputSync.downloadPayload() slot=170237, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, pendingPayloads=100, delaySec=0.2669999599456787 |  
  |   | 2026-09-06 10:27:24.265 | Sep-06 03:27:24.265[sync]          verbose: Added new payload rootHex to BlockInputSync.pendingPayloads slot=170237, delaySec=0.2650001049041748, root=0xab60393e91386d1e0d555723b25442fa32b87fa9c24f1a96f57146525ef392c0, peerIdStr=16Uiu2HAm7Y79xd4RZZQsT5RV17EEdPa5UhN3tRjVM7ekDLt9JBWQ |  
  |   | 2026-09-06 10:27:24.001 | Sep-06 03:27:24.000[chain]         verbose: Clock slot slot=170237

Expected behavior

the pending block was removed 0.27s into the slot and recreated multiple times
we should keep the block (up until perhaps 1 slot = 12 seconds) and accumulate peer ids so that it's easier to download later
instead of giving up too early with only 1 peer tried

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

unstable

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

meta-bugIssues that identify a bug and require a fix.

Type

No type

Fields

Priority

None yet

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions