Skip to content

Commit 9134ff1

Browse files
authored
docs: update readme for sync pkg (#2776)
<!-- Please read and fill out this form before submitting your PR. Please make sure you have reviewed our contributors guide before submitting your first PR. NOTE: PR titles should follow semantic commits: https://www.conventionalcommits.org/en/v1.0.0/ --> ## Overview <!-- Please provide an explanation of the PR, including the appropriate context, background, goal, and rationale. If there is an issue with this information, please provide a tl;dr and link the issue. Ex: Closes #<issue number> -->
1 parent d64fa9b commit 9134ff1

1 file changed

Lines changed: 71 additions & 54 deletions

File tree

pkg/sync/README.md

Lines changed: 71 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -11,18 +11,27 @@ The sync mechanism consists of two main components:
1111
1. **Header Sync Service** - responsible for synchronizing block headers
1212
2. **Data Sync Service** - responsible for synchronizing block data (transactions and metadata)
1313

14+
Both services are instances of the same generic `SyncService` and differ only by the header type they specialize (signed headers vs. block data).
15+
1416
## Architecture
1517

1618
```mermaid
1719
graph TD
1820
subgraph "Node"
19-
BM[Block Manager]
21+
Exec["Block Executor (aggregator nodes)"]
22+
Sync["Block Syncer (follower nodes)"]
2023
HSS[Header Sync Service]
2124
DSS[Data Sync Service]
2225
P2P[P2P Client]
23-
24-
BM -->|Headers| HSS
25-
BM -->|Data| DSS
26+
Store[(Shared Store)]
27+
28+
Exec -->|WriteToStoreAndBroadcast| HSS
29+
Exec -->|WriteToStoreAndBroadcast| DSS
30+
HSS -->|Persist| Store
31+
DSS -->|Persist| Store
32+
Store -->|Load next block| Sync
33+
Sync -->|Republish DA data| HSS
34+
Sync -->|Republish DA data| DSS
2635
HSS <-->|P2P| P2P
2736
DSS <-->|P2P| P2P
2837
end
@@ -31,7 +40,8 @@ graph TD
3140
DAL[Data Availability Layer]
3241
end
3342
34-
BM <-->|Submit/Retrieve| DAL
43+
Exec -.->|Submit headers/data| DAL
44+
Sync -->|Retrieve blobs| DAL
3545
3646
subgraph "Other Nodes"
3747
ON[Other Nodes]
@@ -70,76 +80,83 @@ classDiagram
7080
HeaderSyncService --|> SyncService : H = *types.SignedHeader
7181
```
7282

73-
### 2. Block Manager (`block/manager.go`)
83+
#### Lifecycle and responsibilities
7484

75-
The Block Manager orchestrates the synchronization process through several key goroutines:
85+
- `node/full.go` wires the header and data services into both aggregator and follower nodes (see `initHeaderSyncService` and `initDataSyncService`).
86+
- Both services wrap a go-header `Store` instance that is prefixed per sync type, allowing them to share disk state while keeping namespaces separate.
87+
- `WriteToStoreAndBroadcast` (also used by the block executor) ensures the store is initialized with genesis data, starts the go-header syncer once via `SyncerStatus`, and gossips new items through libp2p.
88+
- When the node runs in follower mode, the go-header syncer fills the store from peers; when running as an aggregator, locally produced blocks flow through the same method.
7689

77-
#### a. SyncLoop
90+
### 2. Block Syncer (`block/internal/syncing/syncer.go`)
7891

79-
```mermaid
80-
flowchart TD
81-
SL[SyncLoop] --> |periodic| NBRCH[Send Signal to Retrieve Channel]
82-
SL --> |periodic| NBHCH[Send Signal to Header Store Channel]
83-
SL --> |periodic| NBDCH[Send Signal to Data Store Channel]
84-
SL --> |on header event| HC[Process Header]
85-
SL --> |on data event| DC[Process Data]
86-
87-
HC --> |cache header| TSYNC[Try Sync Next Block]
88-
DC --> |cache data| TSYNC
89-
90-
TSYNC --> |if header & data available| AB[Apply Block]
91-
AB --> |if successful| SB[Store Block]
92-
SB --> |if successful| UH[Update Height]
93-
```
92+
Follower nodes construct the block syncer to hydrate local state from the shared go-header stores and the DA layer. The syncer owns two long-lived goroutines that coordinate incoming events and outbound fetches.
93+
94+
#### a. `processLoop`
9495

95-
#### b. HeaderStoreRetrieveLoop
96+
- Listens on `heightInCh` for new `DAHeightEvent` values sourced from P2P or DA.
97+
- Uses the in-memory cache to de-duplicate and park out-of-order heights.
98+
- Calls `trySyncNextBlock` to execute the next block when header and data are available.
9699

97100
```mermaid
98101
flowchart TD
99-
HSRL[HeaderStoreRetrieveLoop] --> |on signal| CH[Check Height]
100-
CH --> |if new headers| GH[Get Headers]
101-
GH --> |for each header| VH[Validate Header]
102-
VH --> |if valid| SH[Send to headerInCh]
102+
Start[Start] --> Select{select}
103+
Select -->|"ctx.Done()"| Stop[Stop]
104+
Select -->|heightInCh| Handle[processHeightEvent]
105+
Handle --> Start
103106
```
104107

105-
#### c. DataStoreRetrieveLoop
108+
#### b. `syncLoop`
109+
110+
- Sleeps until genesis if necessary, then runs at the cadence of the configured block time.
111+
- Drains cached events, pulls new ranges from the go-header stores (`tryFetchFromP2P`), and queries the DA layer with backoff (`tryFetchFromDA`).
112+
- Pushes results into `heightInCh`, falling back to the cache when the queue is full.
106113

107114
```mermaid
108115
flowchart TD
109-
DSRL[DataStoreRetrieveLoop] --> |on signal| CD[Check Height]
110-
CD --> |if new data| GD[Get Data]
111-
GD --> |for each data| SD[Send to dataInCh]
116+
Begin[Start loop] --> Pending[processPendingEvents]
117+
Pending --> P2P[tryFetchFromP2P]
118+
P2P --> DA[tryFetchFromDA]
119+
DA --> Wait{"ctx.Done()?"}
120+
Wait -->|yes| End[Stop]
121+
Wait -->|no| Sleep[time.After/backoff]
122+
Sleep --> Begin
112123
```
113124

114-
#### d. RetrieveLoop
125+
#### c. Supporting helpers
115126

116-
```mermaid
117-
flowchart TD
118-
RL[RetrieveLoop] --> |on signal| PDA[Process Next DA Header]
119-
PDA --> |if successful| IH[Increment Height]
120-
IH --> RL
121-
```
127+
- `processPendingEvents` replays cached events once the next height becomes available.
128+
- `trySyncNextBlock` validates, executes, and persists the block via the execution client and shared store.
129+
- Persists the block through a `store.Store` batch, bumps height/state, and marks headers/data as seen to enforce sequential progress and metrics updates.
130+
- `tryFetchFromDA` manages DA backoff windows and advances `daHeight` on success.
131+
- `tryFetchFromP2P` reads the latest height from both header and data go-header stores, enqueueing any ranges the node has not yet processed.
122132

123-
## Communication Channels
133+
## Communication Paths
124134

125-
The Block Manager uses several channels for communication between its components:
135+
The block syncer relies on a handful of queues and shared stores to keep the node in sync:
126136

127-
1. `headerInCh` - Receives headers from both P2P and DA layer
128-
2. `dataInCh` - Receives data from both P2P and DA layer
129-
3. `headerStoreCh` - Signals to check for new headers in the store
130-
4. `dataStoreCh` - Signals to check for new data in the store
131-
5. `retrieveCh` - Signals to retrieve data from the DA layer
132-
6. `HeaderCh` - Sends headers to the HeaderSyncService for broadcasting
133-
7. `DataCh` - Sends data to the DataSyncService for broadcasting
137+
1. `heightInCh` – Buffered queue that carries `common.DAHeightEvent` values from both the P2P handler and DA retriever into `processLoop`.
138+
2. `cache.Manager` – In-memory structure that tracks pending events and deduplicates headers/data that arrive out of order.
139+
3. `headerStore` / `dataStore` – go-header stores exposed by the sync services. Aggregators append to them when producing blocks; followers poll them in `tryFetchFromP2P` to learn about new ranges.
140+
4. `errorCh` – Channel surfaced to the higher-level block components so critical execution failures inside the syncer can halt the node cleanly.
134141

135142
## Synchronization Process
136143

137-
1. Headers and data are received through P2P gossip or retrieved from the DA layer
138-
2. They are stored in the respective stores and cached in memory
139-
3. When both a header and its corresponding data are available, the block is applied
140-
4. The state is updated and the next block is processed
141-
5. New blocks created by the node are broadcast to peers via the P2P network
142-
6. Headers are submitted to the DA layer for finality
144+
1. Aggregator executors call `WriteToStoreAndBroadcast`, or remote peers gossip new headers and block data through the sync services.
145+
2. `SyncService` instances persist the payload in the prefixed go-header stores and broadcast it over libp2p.
146+
3. Follower syncers observe the updated store heights, fetch any missing data via P2P or the DA layer, and enqueue events on `heightInCh`.
147+
4. The syncer executes the block via the execution client, writes it to `store.Store` using a batch, updates in-memory state, and records metrics.
148+
5. For DA-sourced events, the syncer republishes the block by calling `WriteToStoreAndBroadcast` on the header and data services (`block/internal/syncing/syncer.go:389-392`) so gossip peers stay updated.
149+
6. Successfully applied blocks are now visible to both the local node and the sync services, keeping aggregator and follower paths in sync.
150+
7. Aggregator nodes additionally submit headers/data to the DA layer for finality.
151+
152+
## Integration with Block Components
153+
154+
The sync package is consumed by both the block executor (aggregator mode) and the block syncer (follower mode):
155+
156+
- **Aggregator nodes**`node/full.go:101` constructs `block.NewAggregatorComponents`, which in turn creates `block/internal/executing.Executor`. After the executor commits a block, it calls `WriteToStoreAndBroadcast` on the header and data services (`block/internal/executing/executor.go:415`). This persists the block in the shared store and gossips it to peers.
157+
- **Follower nodes**`node/full.go:116` builds `block.NewSyncComponents`, wiring the same sync services into `block/internal/syncing.Syncer`. The syncer consumes updates written by the services (`block/internal/syncing/syncer.go:77`) and merges them with DA retrieval to hydrate local state.
158+
- **Common broadcaster contract** – Both block paths depend only on the slim `block/internal/common.Broadcaster` interface, so alternate sync implementations can be plugged in as long as they expose `WriteToStoreAndBroadcast` and `Store`.
159+
- **Execution engine boundary** – Because the sync services operate on generic header types, swapping execution engines only requires satisfying the `core/execution.Executor` interface; the sync plumbing remains unchanged.
143160

144161
## Dependencies
145162

0 commit comments

Comments
 (0)