You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<!--
Please read and fill out this form before submitting your PR.
Please make sure you have reviewed our contributors guide before
submitting your
first PR.
NOTE: PR titles should follow semantic commits:
https://www.conventionalcommits.org/en/v1.0.0/
-->
## Overview
<!--
Please provide an explanation of the PR, including the appropriate
context,
background, goal, and rationale. If there is an issue with this
information,
please provide a tl;dr and link the issue.
Ex: Closes #<issue number>
-->
Copy file name to clipboardExpand all lines: pkg/sync/README.md
+71-54Lines changed: 71 additions & 54 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,18 +11,27 @@ The sync mechanism consists of two main components:
11
11
1.**Header Sync Service** - responsible for synchronizing block headers
12
12
2.**Data Sync Service** - responsible for synchronizing block data (transactions and metadata)
13
13
14
+
Both services are instances of the same generic `SyncService` and differ only by the header type they specialize (signed headers vs. block data).
15
+
14
16
## Architecture
15
17
16
18
```mermaid
17
19
graph TD
18
20
subgraph "Node"
19
-
BM[Block Manager]
21
+
Exec["Block Executor (aggregator nodes)"]
22
+
Sync["Block Syncer (follower nodes)"]
20
23
HSS[Header Sync Service]
21
24
DSS[Data Sync Service]
22
25
P2P[P2P Client]
23
-
24
-
BM -->|Headers| HSS
25
-
BM -->|Data| DSS
26
+
Store[(Shared Store)]
27
+
28
+
Exec -->|WriteToStoreAndBroadcast| HSS
29
+
Exec -->|WriteToStoreAndBroadcast| DSS
30
+
HSS -->|Persist| Store
31
+
DSS -->|Persist| Store
32
+
Store -->|Load next block| Sync
33
+
Sync -->|Republish DA data| HSS
34
+
Sync -->|Republish DA data| DSS
26
35
HSS <-->|P2P| P2P
27
36
DSS <-->|P2P| P2P
28
37
end
@@ -31,7 +40,8 @@ graph TD
31
40
DAL[Data Availability Layer]
32
41
end
33
42
34
-
BM <-->|Submit/Retrieve| DAL
43
+
Exec -.->|Submit headers/data| DAL
44
+
Sync -->|Retrieve blobs| DAL
35
45
36
46
subgraph "Other Nodes"
37
47
ON[Other Nodes]
@@ -70,76 +80,83 @@ classDiagram
70
80
HeaderSyncService --|> SyncService : H = *types.SignedHeader
71
81
```
72
82
73
-
###2. Block Manager (`block/manager.go`)
83
+
#### Lifecycle and responsibilities
74
84
75
-
The Block Manager orchestrates the synchronization process through several key goroutines:
85
+
-`node/full.go` wires the header and data services into both aggregator and follower nodes (see `initHeaderSyncService` and `initDataSyncService`).
86
+
- Both services wrap a go-header `Store` instance that is prefixed per sync type, allowing them to share disk state while keeping namespaces separate.
87
+
-`WriteToStoreAndBroadcast` (also used by the block executor) ensures the store is initialized with genesis data, starts the go-header syncer once via `SyncerStatus`, and gossips new items through libp2p.
88
+
- When the node runs in follower mode, the go-header syncer fills the store from peers; when running as an aggregator, locally produced blocks flow through the same method.
SL[SyncLoop] --> |periodic| NBRCH[Send Signal to Retrieve Channel]
82
-
SL --> |periodic| NBHCH[Send Signal to Header Store Channel]
83
-
SL --> |periodic| NBDCH[Send Signal to Data Store Channel]
84
-
SL --> |on header event| HC[Process Header]
85
-
SL --> |on data event| DC[Process Data]
86
-
87
-
HC --> |cache header| TSYNC[Try Sync Next Block]
88
-
DC --> |cache data| TSYNC
89
-
90
-
TSYNC --> |if header & data available| AB[Apply Block]
91
-
AB --> |if successful| SB[Store Block]
92
-
SB --> |if successful| UH[Update Height]
93
-
```
92
+
Follower nodes construct the block syncer to hydrate local state from the shared go-header stores and the DA layer. The syncer owns two long-lived goroutines that coordinate incoming events and outbound fetches.
93
+
94
+
#### a. `processLoop`
94
95
95
-
#### b. HeaderStoreRetrieveLoop
96
+
- Listens on `heightInCh` for new `DAHeightEvent` values sourced from P2P or DA.
97
+
- Uses the in-memory cache to de-duplicate and park out-of-order heights.
98
+
- Calls `trySyncNextBlock` to execute the next block when header and data are available.
RL[RetrieveLoop] --> |on signal| PDA[Process Next DA Header]
119
-
PDA --> |if successful| IH[Increment Height]
120
-
IH --> RL
121
-
```
127
+
-`processPendingEvents` replays cached events once the next height becomes available.
128
+
-`trySyncNextBlock` validates, executes, and persists the block via the execution client and shared store.
129
+
- Persists the block through a `store.Store` batch, bumps height/state, and marks headers/data as seen to enforce sequential progress and metrics updates.
130
+
-`tryFetchFromDA` manages DA backoff windows and advances `daHeight` on success.
131
+
-`tryFetchFromP2P` reads the latest height from both header and data go-header stores, enqueueing any ranges the node has not yet processed.
122
132
123
-
## Communication Channels
133
+
## Communication Paths
124
134
125
-
The Block Manager uses several channels for communication between its components:
135
+
The block syncer relies on a handful of queues and shared stores to keep the node in sync:
126
136
127
-
1.`headerInCh` - Receives headers from both P2P and DA layer
128
-
2.`dataInCh` - Receives data from both P2P and DA layer
129
-
3.`headerStoreCh` - Signals to check for new headers in the store
130
-
4.`dataStoreCh` - Signals to check for new data in the store
131
-
5.`retrieveCh` - Signals to retrieve data from the DA layer
132
-
6.`HeaderCh` - Sends headers to the HeaderSyncService for broadcasting
133
-
7.`DataCh` - Sends data to the DataSyncService for broadcasting
137
+
1.`heightInCh` – Buffered queue that carries `common.DAHeightEvent` values from both the P2P handler and DA retriever into `processLoop`.
138
+
2.`cache.Manager` – In-memory structure that tracks pending events and deduplicates headers/data that arrive out of order.
139
+
3.`headerStore` / `dataStore` – go-header stores exposed by the sync services. Aggregators append to them when producing blocks; followers poll them in `tryFetchFromP2P` to learn about new ranges.
140
+
4.`errorCh` – Channel surfaced to the higher-level block components so critical execution failures inside the syncer can halt the node cleanly.
134
141
135
142
## Synchronization Process
136
143
137
-
1. Headers and data are received through P2P gossip or retrieved from the DA layer
138
-
2. They are stored in the respective stores and cached in memory
139
-
3. When both a header and its corresponding data are available, the block is applied
140
-
4. The state is updated and the next block is processed
141
-
5. New blocks created by the node are broadcast to peers via the P2P network
142
-
6. Headers are submitted to the DA layer for finality
144
+
1. Aggregator executors call `WriteToStoreAndBroadcast`, or remote peers gossip new headers and block data through the sync services.
145
+
2.`SyncService` instances persist the payload in the prefixed go-header stores and broadcast it over libp2p.
146
+
3. Follower syncers observe the updated store heights, fetch any missing data via P2P or the DA layer, and enqueue events on `heightInCh`.
147
+
4. The syncer executes the block via the execution client, writes it to `store.Store` using a batch, updates in-memory state, and records metrics.
148
+
5. For DA-sourced events, the syncer republishes the block by calling `WriteToStoreAndBroadcast` on the header and data services (`block/internal/syncing/syncer.go:389-392`) so gossip peers stay updated.
149
+
6. Successfully applied blocks are now visible to both the local node and the sync services, keeping aggregator and follower paths in sync.
150
+
7. Aggregator nodes additionally submit headers/data to the DA layer for finality.
151
+
152
+
## Integration with Block Components
153
+
154
+
The sync package is consumed by both the block executor (aggregator mode) and the block syncer (follower mode):
155
+
156
+
-**Aggregator nodes** – `node/full.go:101` constructs `block.NewAggregatorComponents`, which in turn creates `block/internal/executing.Executor`. After the executor commits a block, it calls `WriteToStoreAndBroadcast` on the header and data services (`block/internal/executing/executor.go:415`). This persists the block in the shared store and gossips it to peers.
157
+
-**Follower nodes** – `node/full.go:116` builds `block.NewSyncComponents`, wiring the same sync services into `block/internal/syncing.Syncer`. The syncer consumes updates written by the services (`block/internal/syncing/syncer.go:77`) and merges them with DA retrieval to hydrate local state.
158
+
-**Common broadcaster contract** – Both block paths depend only on the slim `block/internal/common.Broadcaster` interface, so alternate sync implementations can be plugged in as long as they expose `WriteToStoreAndBroadcast` and `Store`.
159
+
-**Execution engine boundary** – Because the sync services operate on generic header types, swapping execution engines only requires satisfying the `core/execution.Executor` interface; the sync plumbing remains unchanged.
0 commit comments