Skip to content

L1 PLEX-2601 LP Fetch Blocks in Batches#1504

Open
dhaidashenko wants to merge 7 commits intodevelopfrom
fix/LPEX-2601-lp-slot-fetches-in-batches-v2
Open

L1 PLEX-2601 LP Fetch Blocks in Batches#1504
dhaidashenko wants to merge 7 commits intodevelopfrom
fix/LPEX-2601-lp-slot-fetches-in-batches-v2

Conversation

@dhaidashenko
Copy link
Copy Markdown
Collaborator

@dhaidashenko dhaidashenko commented Apr 7, 2026

Summary

PLEX-2601 — Batch block fetching inside the log poller loader so slot discovery via getSignaturesForAddress runs once per backfill range, instead of restarting for every slot-range batch.

Problem

Previously, large backfills were split in the log poller into multiple independent slot ranges. Each range called BackfillForAddresses, which re-ran getSlotsToFetch from the newest signatures. Pagination (beforeSig) did not carry across adjacent batches, so every batch effectively rediscovered recent history and walked backward again. That added redundant RPC load, slowed deep catch-up on hot addresses, and could worsen throttling or recovery time in worst cases.

Solution

  • Revert batching at the poller layer (processBlocksRangeInBatches / per-batch BackfillForAddresses).
  • Move batching to EncodedLogCollector: after a single getSlotsToFetch pass for [fromSlot, toSlot], scheduleBlocksFetching processes the resulting slot list in chunks of LogPollerSlotsBatchSize (limits concurrent getBlock work / memory), while signature discovery pagination stays coherent for the whole range.
  • Lower the default LogPollerSlotsBatchSize to further limit memory usage, as getSlotsToFetch no longer executed multiple times.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Apr 7, 2026

⚠️ API Diff Results - github.com/smartcontractkit/chainlink-solana

⚠️ Breaking Changes (1)

pkg/solana/logpoller (1)
  • NewEncodedLogCollector — Type changed:
func(
  RPCClient, 
  github.com/smartcontractkit/chainlink-common/pkg/logger.Logger, 
  string, 
  *solLpMetrics, 
  *CPIEventExtractor, 
  + int64
)
*EncodedLogCollector

📄 View full apidiff report

archseer
archseer previously approved these changes Apr 8, 2026
@cl-sonarqube-production
Copy link
Copy Markdown

@dhaidashenko dhaidashenko requested a review from amit-momin April 8, 2026 20:31
Copy link
Copy Markdown
Contributor

@jadepark-dev jadepark-dev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@dhaidashenko dhaidashenko changed the title PLEX-2601 LP Fetch Blocks in Batches L1 PLEX-2601 LP Fetch Blocks in Batches Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants