lwwallet: speed up sync by bypassing the linear block scan - #1077
Conversation
474b2bc to
5fce2a4
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5fce2a49ad
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| t.seedHashHistory(height, hash) | ||
| if t.runCtx.Err() != nil { | ||
| return | ||
| } | ||
|
|
||
| t.pollLoop() |
There was a problem hiding this comment.
If any startup raw-header request fails transiently, seedHashHistory returns early but this goroutine still starts pollLoop with a hole in recentHashes. A later reorg that walks across that hole always exits through the new “history is incomplete” branch, and every subsequent poll retries the same walk without ever refilling the missing hash, so the cached tip and all subscriber block events remain stuck indefinitely. Propagate the seed failure and retry or repair the history before beginning normal polling.
AGENTS.md reference: lwwallet/AGENTS.md:L14-L18
Useful? React with 👍 / 👎.
| for script := range probes { | ||
| refs, err := s.scriptHistoryToHeight( | ||
| ctx, []byte(script), minHeight, | ||
| ) |
There was a problem hiding this comment.
During seed recovery, FilterBlocksRequest can contain a recovery window's worth of external and internal addresses (the production default is 100), but this loop serializes an independent HTTP history lookup for every script and potentially every pagination page. On the mobile latency motivating this change, even unused addresses therefore add tens of seconds per call before any result can be returned. Use bounded concurrency, as probeScripts already does with scriptProbeConcurrency, while merging candidates safely.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
This PR improves lwwallet sync/startup performance when using the Esplora backend by removing sequential block-body scans in favor of Esplora’s address index, and by moving TipPoller’s hash-history seeding off the critical startup path.
Changes:
- TipPoller: introduce a component-owned lifecycle context, publish the initial tip immediately, and seed the recent-hash ring asynchronously before starting the poll loop.
- Esplora chain backend: implement
FilterBlocksand (parts of)Rescanusing script history + raw-tx fetches, with correctness-preserving fallbacks to block scanning on index inconsistencies. - Add targeted tests to pin the non-block-scanning behavior, ordering requirements, and new TipPoller startup/shutdown behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| lwwallet/tip_poller.go | Adds poller-owned context + async history seeding to unblock wallet startup. |
| lwwallet/tip_poller_test.go | Updates stubs to support content-addressed header lookup across reorgs. |
| lwwallet/tip_poller_reorg_test.go | Adds tests for “no polling before seed complete” and Stop cancelling in-flight seed requests. |
| lwwallet/esplora_filterblocks_test.go | New test suite validating index-based FilterBlocks/Rescan behavior and fallbacks. |
| lwwallet/esplora_chain.go | Implements index-based FilterBlocks and index-assisted Rescan with dependency ordering. |
| lwwallet/chain_backend_reorg_test.go | Improves fake chain fixtures and adds timing test ensuring TipPoller Start doesn’t block on seeding. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| all = append(all, refs...) | ||
| last := refs[len(refs)-1] | ||
| if int32(last.Status.BlockHeight) <= minHeight { | ||
| return all, nil | ||
| } |
Move the sequential hash-history walk off the synchronous startup path. Keep the poll loop behind a fully seeded ring, so reorg handling never observes partial retained history. Retry transient seed failures with bounded exponential backoff until the service stops. Bind requests and retry timers to the component context, so shutdown cancels the entire operation promptly. Cover responsive startup, a reorg racing the seed, retry recovery, and cancellation. Co-authored-by: sputn1ck <kon@kon.ninja>
Use Esplora script histories to identify candidate transactions for FilterBlocks and Rescan instead of downloading every block body. Preserve canonical matching and ordering, and validate indexed block identities before advancing wallet state. Fall back to full block scans when history is truncated, inconsistent, or cannot represent a watched outpoint address. Page through every transaction at the minimum requested height before stopping. Cover spent addresses, pagination boundaries, ordering, reorg conflicts, fallbacks, cancellation, and the empty index contract. Co-authored-by: sputn1ck <kon@kon.ninja>
Document that an empty script history cannot be distinguished from a lagging index. Explain the recovery boundary and require a trusted, fully indexed Esplora endpoint for wallet rescans.
5fce2a4 to
7d514dc
Compare
|
Update after taking over the PR from @guggero. The branch is now organized as three atomic commits; Guggero remains the author of both implementation commits, with the follow-up fixes recorded as co-authored work. What this PR changes, and why1. Seed the tip poller's reorg history without blocking wallet startupPreviously, The history walk now runs under the poller's component-owned goroutine/context, so We also changed transient seed failures from "log and continue polling" to bounded exponential retries. This matters because entering the polling loop with a gap would make a later reorg look like internal corruption and disable reorg handling until the missing height aged out of the ring. Requests and retry timers use 2. Use Esplora's address index for
|
sputn1ck
added
the
backport-v0.1.x-branch
|
Successfully created backport PR for |
…ranch [v0.1.x-branch] Backport #1077: lwwallet: speed up sync by bypassing the linear block scan
Fixes #1073 and is complementary to #1076.
Implements three different speedups:
TipPoller.Start(), fetching the 99 recent block hashes synchronously blocks the whole wallet startup -> moved to background.EsploraChainService.FilterBlocks, since we're already talking to an address indexer, we can bypass the sequentials.esplora.GetRawBlock()calls by looking up the history of the addresses to scan.EsploraChainService.Rescan()we do the same by fetching the whole history of the addresses in question.