diff --git a/.tide/lld-fork-export.md b/.tide/lld-fork-export.md new file mode 100644 index 0000000..b501d40 --- /dev/null +++ b/.tide/lld-fork-export.md @@ -0,0 +1,197 @@ +# LLD: Automated Fork Export via Temporary SeiNode + +## Overview + +Automates the fork export by creating a temporary "exporter" SeiNode that bootstraps from the source chain using the existing bootstrap pipeline (identical to replayer). The SeiNode controller brings it to the target height. The group plan then submits `export-state` to the exporter's sidecar and uploads the result. + +No new fields on FullNodeSpec. The exporter is a plain FullNode with BootstrapImage + TargetHeight — the same config replayers use. + +## Key Insight + +`seid export --height N` reads committed state at exactly height N from the database, regardless of whether seid has progressed beyond N. This means we don't need to halt the node precisely — we just need it to have reached height N. The export command is deterministic. + +## Flow + +``` +reconcileSeiNodes creates: + - N validator SeiNodes (start init plans, block at configure-genesis) + - 1 exporter SeiNode (FullNode, bootstraps from source chain snapshot) + +SeiNode controller drives exporter through standard bootstrap: + Pending → Initializing → Running + (snapshot-restore → config → sync to height → production StatefulSet) + +Group plan (once exporter reaches Running): + await-exporter-running ← polls exporter phase == Running + submit-export-state ← submits export-state task to exporter's sidecar + teardown-exporter ← deletes exporter SeiNode + assemble-genesis-fork ← existing: downloads export, strips validators, collect-gentxs + collect-and-set-peers ← existing + await-nodes-running ← existing + +Validators unblock: configure-genesis finds genesis.json → proceed → Running +``` + +## CRD Changes + +### ForkConfig (expanded) + +```go +type ForkConfig struct { + // SourceChainID is the chain ID of the network being forked. + SourceChainID string `json:"sourceChainId"` + + // SourceImage is the seid container image compatible with the source + // chain at ExportHeight. + SourceImage string `json:"sourceImage"` + + // ExportHeight is the block height at which to export state. + ExportHeight int64 `json:"exportHeight"` +} +``` + +No changes to FullNodeSpec, SnapshotSource, or any existing types. + +## Exporter SeiNode + +Created by `reconcileSeiNodes` as `{group}-exporter`. A plain FullNode: + +```go +Spec: SeiNodeSpec{ + ChainID: fork.SourceChainID, + Image: fork.SourceImage, + FullNode: &FullNodeSpec{ + Snapshot: &SnapshotSource{ + S3: &S3SnapshotSource{TargetHeight: fork.ExportHeight}, + BootstrapImage: fork.SourceImage, + }, + }, +} +``` + +This is the same bootstrap config pattern that replayers use. The SeiNode controller handles it through the standard `fullNodePlanner` → `buildBootstrapPlan` → bootstrap Job → StatefulSet. + +Labels: `sei.io/nodegroup: {group}`, `sei.io/role: exporter` +Excluded from `IncumbentNodes`. + +## Group Plan Tasks + +### `submit-export-state` (new controller-side task) + +Submits an `export-state` task to the exporter's sidecar via HTTP API: + +```go +type SubmitExportStateParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` + ExportHeight int64 `json:"exportHeight"` + SourceChainID string `json:"sourceChainId"` +} +``` + +The task: +1. Builds a sidecar client for the exporter node +2. Submits `export-state` with params: `{height: N, chainId: sourceChainId, s3Bucket: genesisBucket, s3Key: {sourceChainId}/exported-state.json, s3Region: genesisRegion}` +3. Polls the sidecar for task completion +4. Returns Complete when the sidecar task completes + +This is the same submit-and-poll pattern used by other sidecar tasks in the plan executor. + +### `await-exporter-running` (new controller-side task) + +Polls exporter SeiNode phase: + +```go +type AwaitExporterRunningParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` +} +``` + +Returns Complete when `Phase == Running`, Failed when `Phase == Failed`. + +### `teardown-exporter` (new controller-side task) + +Deletes the exporter SeiNode: + +```go +type TeardownExporterParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` +} +``` + +Polls until the SeiNode is gone (ownerReferences cascade the StatefulSet, PVC, etc.). + +## Full Group Plan Sequence + +When `ForkGenesisCeremonyNeeded`: + +``` +await-exporter-running (controller: poll exporter phase) +submit-export-state (controller: submit to exporter sidecar, poll completion) +teardown-exporter (controller: delete exporter SeiNode) +assemble-genesis-fork (sidecar: download export, strip validators, collect-gentxs) +collect-and-set-peers (existing) +await-nodes-running (existing) +``` + +When export already exists in S3 (checked by `needsForkExporter`): + +``` +assemble-genesis-fork (sidecar: download export, strip validators, collect-gentxs) +collect-and-set-peers (existing) +await-nodes-running (existing) +``` + +## Edge Case: Export Already Exists + +`needsForkExporter` checks S3 for `{sourceChainId}/exported-state.json`. If it exists, no exporter SeiNode is created and the group plan skips the first three tasks. This supports: +- Re-reconciling after a failed assembly (export already done) +- Multiple fork groups from the same source chain (export once, fork many) +- Pre-uploaded exports for faster iteration + +## Example YAML + +```yaml +apiVersion: sei.io/v1alpha1 +kind: SeiNodeGroup +metadata: + name: private-fork +spec: + replicas: 4 + genesis: + chainId: private-fork-1 + stakingAmount: "10000000usei" + fork: + sourceChainId: pacific-1 + sourceImage: ghcr.io/sei-protocol/seid:v5.9.0 + exportHeight: 200000000 + template: + spec: + image: ghcr.io/sei-protocol/seid:v6.0.0 + validator: {} +``` + +## Files Affected + +### Controller +| File | Change | +|------|--------| +| `api/v1alpha1/seinodegroup_types.go` | ForkConfig: add SourceImage, ExportHeight | +| `internal/controller/nodegroup/nodes.go` | ensureForkExporter, needsForkExporter, filter populateIncumbentNodes | +| `internal/planner/group.go` | Prepend export tasks to fork plan when exporter exists | +| `internal/task/fork_export.go` | New: SubmitExportStateParams, AwaitExporterRunningParams, TeardownExporterParams + executions | +| `internal/task/task.go` | Deserialize: 3 new cases | + +### Seictl +| File | Change | +|------|--------| +| (export-state task already exists from earlier work) | No changes needed if export-state handler is already registered | + +### What's Reused (no changes) +- `fullNodePlanner` — builds the exporter's bootstrap plan as-is +- `buildBootstrapPlan` — standard bootstrap task sequence +- `GenerateBootstrapJob` — standard bootstrap Job +- `SeiNode controller` — drives the exporter's full lifecycle +- `SnapshotSource` / `S3SnapshotSource` / `BootstrapImage` — existing types diff --git a/api/v1alpha1/seinodegroup_types.go b/api/v1alpha1/seinodegroup_types.go index a8c3f6e..48a1278 100644 --- a/api/v1alpha1/seinodegroup_types.go +++ b/api/v1alpha1/seinodegroup_types.go @@ -117,6 +117,35 @@ type GenesisCeremonyConfig struct { // assembly completion. Default: "15m". // +optional MaxCeremonyDuration *metav1.Duration `json:"maxCeremonyDuration,omitempty"` + + // Fork configures this genesis ceremony to fork from an existing + // chain's exported state rather than building genesis from scratch. + // When set, the assembler downloads the exported state, rewrites + // the chain identity, strips old validators, and runs collect-gentxs + // with the new validator set. + // +optional + Fork *ForkConfig `json:"fork,omitempty"` +} + +// ForkConfig configures forking from an existing chain. The controller +// creates a temporary exporter SeiNode that bootstraps from the source +// chain (using the same pipeline as replayers), then the group plan +// submits seid export to the exporter's sidecar and uploads the result. +type ForkConfig struct { + // SourceChainID is the chain ID of the network being forked. + // +kubebuilder:validation:MinLength=1 + SourceChainID string `json:"sourceChainId"` + + // SourceImage is the seid container image compatible with the source + // chain at ExportHeight. Used as both the bootstrap and main image + // for the temporary exporter node. + // +kubebuilder:validation:MinLength=1 + SourceImage string `json:"sourceImage"` + + // ExportHeight is the block height at which to export state. + // seid export --height N reads committed state at exactly this height. + // +kubebuilder:validation:Minimum=1 + ExportHeight int64 `json:"exportHeight"` } // GenesisAccount represents a non-validator genesis account to fund. @@ -270,13 +299,15 @@ type DeploymentStatus struct { // Status condition types for SeiNodeGroup. const ( - ConditionNodesReady = "NodesReady" - ConditionExternalServiceReady = "ExternalServiceReady" - ConditionRouteReady = "RouteReady" - ConditionIsolationReady = "IsolationReady" - ConditionServiceMonitorReady = "ServiceMonitorReady" - ConditionGenesisCeremonyComplete = "GenesisCeremonyComplete" - ConditionPlanInProgress = "PlanInProgress" + ConditionNodesReady = "NodesReady" + ConditionExternalServiceReady = "ExternalServiceReady" + ConditionRouteReady = "RouteReady" + ConditionIsolationReady = "IsolationReady" + ConditionServiceMonitorReady = "ServiceMonitorReady" + ConditionGenesisCeremonyComplete = "GenesisCeremonyComplete" + ConditionPlanInProgress = "PlanInProgress" + ConditionGenesisCeremonyNeeded = "GenesisCeremonyNeeded" + ConditionForkGenesisCeremonyNeeded = "ForkGenesisCeremonyNeeded" ) // +kubebuilder:object:root=true diff --git a/api/v1alpha1/zz_generated.deepcopy.go b/api/v1alpha1/zz_generated.deepcopy.go index b9b5906..8e3bb9d 100644 --- a/api/v1alpha1/zz_generated.deepcopy.go +++ b/api/v1alpha1/zz_generated.deepcopy.go @@ -197,6 +197,21 @@ func (in *ExternalServiceConfig) DeepCopy() *ExternalServiceConfig { return out } +// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. +func (in *ForkConfig) DeepCopyInto(out *ForkConfig) { + *out = *in +} + +// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new ForkConfig. +func (in *ForkConfig) DeepCopy() *ForkConfig { + if in == nil { + return nil + } + out := new(ForkConfig) + in.DeepCopyInto(out) + return out +} + // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil. func (in *FullNodeSpec) DeepCopyInto(out *FullNodeSpec) { *out = *in @@ -312,6 +327,11 @@ func (in *GenesisCeremonyConfig) DeepCopyInto(out *GenesisCeremonyConfig) { *out = new(metav1.Duration) **out = **in } + if in.Fork != nil { + in, out := &in.Fork, &out.Fork + *out = new(ForkConfig) + **out = **in + } } // DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new GenesisCeremonyConfig. diff --git a/config/crd/sei.io_seinodegroups.yaml b/config/crd/sei.io_seinodegroups.yaml index ca958f0..97a9ac2 100644 --- a/config/crd/sei.io_seinodegroups.yaml +++ b/config/crd/sei.io_seinodegroups.yaml @@ -111,6 +111,38 @@ spec: minLength: 1 pattern: ^[a-z0-9][a-z0-9-]*[a-z0-9]$ type: string + fork: + description: |- + Fork configures this genesis ceremony to fork from an existing + chain's exported state rather than building genesis from scratch. + When set, the assembler downloads the exported state, rewrites + the chain identity, strips old validators, and runs collect-gentxs + with the new validator set. + properties: + exportHeight: + description: |- + ExportHeight is the block height at which to export state. + seid export --height N reads committed state at exactly this height. + format: int64 + minimum: 1 + type: integer + sourceChainId: + description: SourceChainID is the chain ID of the network + being forked. + minLength: 1 + type: string + sourceImage: + description: |- + SourceImage is the seid container image compatible with the source + chain at ExportHeight. Used as both the bootstrap and main image + for the temporary exporter node. + minLength: 1 + type: string + required: + - exportHeight + - sourceChainId + - sourceImage + type: object maxCeremonyDuration: description: |- MaxCeremonyDuration is the maximum time from group creation to genesis diff --git a/go.mod b/go.mod index 47ccc2d..02a094d 100644 --- a/go.mod +++ b/go.mod @@ -10,7 +10,7 @@ require ( github.com/onsi/gomega v1.38.2 github.com/prometheus/client_golang v1.23.2 github.com/sei-protocol/sei-config v0.0.9 - github.com/sei-protocol/seictl v0.0.26 + github.com/sei-protocol/seictl v0.0.29 k8s.io/api v0.35.0 k8s.io/apiextensions-apiserver v0.35.0 k8s.io/apimachinery v0.35.0 diff --git a/go.sum b/go.sum index 4eb13ad..6fc4f01 100644 --- a/go.sum +++ b/go.sum @@ -175,8 +175,8 @@ github.com/rogpeppe/go-internal v1.14.1/go.mod h1:MaRKkUm5W0goXpeCfT7UZI6fk/L7L7 github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sei-protocol/sei-config v0.0.9 h1:ELCpE0XnsTvgjOfWe1fWU43vqqFGL2tnlWuF9U1A2l8= github.com/sei-protocol/sei-config v0.0.9/go.mod h1:IEAv5ynYw8Gu2F2qNfE4MQR0PPihAT6g7RWLpWdw5O0= -github.com/sei-protocol/seictl v0.0.26 h1:Rp0wcsOARR84d9phWfNvP8Ieup2UJJyGvCfo+Pw9bmA= -github.com/sei-protocol/seictl v0.0.26/go.mod h1:ffv0EkvWfTeJB5LbMdPEe83P5NYRcGW5TMVNo30LteY= +github.com/sei-protocol/seictl v0.0.29 h1:1cHn6hcPcLEVzylcl9/DVU6WVrYQYNm1ntuVduG0KWk= +github.com/sei-protocol/seictl v0.0.29/go.mod h1:ffv0EkvWfTeJB5LbMdPEe83P5NYRcGW5TMVNo30LteY= github.com/sei-protocol/seilog v0.0.3 h1:Zi7oWXdX5jv92dY8n482xH032LtNebC89Y+qYZlBn0Y= github.com/sei-protocol/seilog v0.0.3/go.mod h1:CKg58wraWnB3gRxWQ0v1rIVr0gmDHjkfP1bM2giKFFU= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= diff --git a/internal/controller/nodegroup/nodes.go b/internal/controller/nodegroup/nodes.go index 5556aee..9ba50fd 100644 --- a/internal/controller/nodegroup/nodes.go +++ b/internal/controller/nodegroup/nodes.go @@ -41,9 +41,42 @@ func (r *SeiNodeGroupReconciler) reconcileSeiNodes(ctx context.Context, group *s } r.detectDeploymentNeeded(group) + r.detectGenesisCeremonyNeeded(group) return nil } +// detectGenesisCeremonyNeeded sets either GenesisCeremonyNeeded or +// ForkGenesisCeremonyNeeded depending on whether the group has a fork +// config. The genesis planner reads this condition to select the +// appropriate assembler task. +func (r *SeiNodeGroupReconciler) detectGenesisCeremonyNeeded(group *seiv1alpha1.SeiNodeGroup) { + // Clear both conditions when not applicable. + if group.Spec.Genesis == nil { + removeCondition(group, seiv1alpha1.ConditionGenesisCeremonyNeeded) + removeCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) + return + } + if hasConditionTrue(group, seiv1alpha1.ConditionGenesisCeremonyComplete) { + removeCondition(group, seiv1alpha1.ConditionGenesisCeremonyNeeded) + removeCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) + return + } + if hasConditionTrue(group, seiv1alpha1.ConditionPlanInProgress) { + return + } + + if group.Spec.Genesis.Fork != nil { + removeCondition(group, seiv1alpha1.ConditionGenesisCeremonyNeeded) + setCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded, metav1.ConditionTrue, + "ForkConfigured", fmt.Sprintf("Fork genesis from %s", + group.Spec.Genesis.Fork.SourceChainID)) + } else { + removeCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) + setCondition(group, seiv1alpha1.ConditionGenesisCeremonyNeeded, metav1.ConditionTrue, + "GenesisConfigured", "Genesis ceremony configured") + } +} + // detectDeploymentNeeded checks if deployment-worthy fields have changed // by comparing the current template hash against the stored hash. Only // fields that require new nodes (image, entrypoint, chainId) are hashed; @@ -75,15 +108,18 @@ func (r *SeiNodeGroupReconciler) detectDeploymentNeeded(group *seiv1alpha1.SeiNo } // populateIncumbentNodes lists child SeiNodes and records their names -// on the group status so planners can read them from the group object. +// on the group status. Exporter nodes are excluded. func (r *SeiNodeGroupReconciler) populateIncumbentNodes(ctx context.Context, group *seiv1alpha1.SeiNodeGroup) error { nodes, err := r.listChildSeiNodes(ctx, group) if err != nil { return fmt.Errorf("listing child SeiNodes: %w", err) } - names := make([]string, len(nodes)) + names := make([]string, 0, len(nodes)) for i := range nodes { - names[i] = nodes[i].Name + if nodes[i].Labels["sei.io/role"] == "exporter" { + continue + } + names = append(names, nodes[i].Name) } group.Status.IncumbentNodes = names return nil diff --git a/internal/controller/nodegroup/plan.go b/internal/controller/nodegroup/plan.go index 447d079..aafb40a 100644 --- a/internal/controller/nodegroup/plan.go +++ b/internal/controller/nodegroup/plan.go @@ -73,8 +73,9 @@ func (r *SeiNodeGroupReconciler) startPlan(ctx context.Context, group *seiv1alph func (r *SeiNodeGroupReconciler) completePlan(ctx context.Context, group *seiv1alpha1.SeiNodeGroup, statusBase client.Patch) (ctrl.Result, error) { logger := log.FromContext(ctx) - // Deployment-specific finalization. - if group.Status.Deployment != nil { + isDeploymentPlan := group.Status.Deployment != nil + + if isDeploymentPlan { group.Status.ObservedGeneration = group.Generation if err := r.reconcileNetworking(ctx, group); err != nil { return ctrl.Result{}, fmt.Errorf("reconciling networking after deployment: %w", err) @@ -82,6 +83,11 @@ func (r *SeiNodeGroupReconciler) completePlan(ctx context.Context, group *seiv1a group.Status.Deployment = nil } + if group.Spec.Genesis != nil && !isDeploymentPlan { + setCondition(group, seiv1alpha1.ConditionGenesisCeremonyComplete, metav1.ConditionTrue, + "CeremonyComplete", "Genesis ceremony completed") + } + group.Status.Plan = nil clearPlanInProgress(group, "PlanComplete", "Plan completed successfully") diff --git a/internal/planner/bootstrap.go b/internal/planner/bootstrap.go index 6ee01ee..6afee56 100644 --- a/internal/planner/bootstrap.go +++ b/internal/planner/bootstrap.go @@ -207,12 +207,7 @@ func SnapshotUploadMonitorTask(node *seiv1alpha1.SeiNode) *sidecar.TaskRequest { if sg == nil || sg.Destination == nil || sg.Destination.S3 == nil { return nil } - dest := sg.Destination.S3 - req := sidecar.SnapshotUploadTask{ - Bucket: dest.Bucket, - Prefix: dest.Prefix, - Region: dest.Region, - }.ToTaskRequest() + req := sidecar.SnapshotUploadTask{}.ToTaskRequest() return &req } diff --git a/internal/planner/group.go b/internal/planner/group.go index 68b642b..878110a 100644 --- a/internal/planner/group.go +++ b/internal/planner/group.go @@ -3,11 +3,16 @@ package planner import ( "fmt" + sidecar "github.com/sei-protocol/seictl/sidecar/client" + seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1" "github.com/sei-protocol/sei-k8s-controller/internal/task" ) -const groupAssemblyMaxRetries = 60 +const ( + groupAssemblyMaxRetries = 60 + TaskAssembleGenesisFork = sidecar.TaskTypeAssembleGenesisFork +) type genesisGroupPlanner struct{} @@ -26,13 +31,28 @@ func (p *genesisGroupPlanner) BuildPlan( nodeParams[i] = task.GenesisNodeParam{Name: name} } - assembleParams := &task.AssembleAndUploadGenesisParams{ - AccountBalance: group.Spec.Genesis.AccountBalance, - Namespace: group.Namespace, - Nodes: nodeParams, + // Select assembler task based on whether this is a fork ceremony. + assembleTaskType := TaskAssembleGenesis + var assembleParams any + + if hasCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) && group.Spec.Genesis.Fork != nil { + assembleTaskType = TaskAssembleGenesisFork + assembleParams = &task.AssembleForkGenesisParams{ + SourceChainID: group.Spec.Genesis.Fork.SourceChainID, + ChainID: group.Spec.Genesis.ChainID, + AccountBalance: group.Spec.Genesis.AccountBalance, + Namespace: group.Namespace, + Nodes: nodeParams, + } + } else { + assembleParams = &task.AssembleAndUploadGenesisParams{ + AccountBalance: group.Spec.Genesis.AccountBalance, + Namespace: group.Namespace, + Nodes: nodeParams, + } } - assembleTask, err := buildGroupPlannedTask(group.Name, TaskAssembleGenesis, assembleParams) + assembleTask, err := buildGroupPlannedTask(group.Name, assembleTaskType, assembleParams) if err != nil { return nil, err } @@ -59,9 +79,63 @@ func (p *genesisGroupPlanner) BuildPlan( return nil, err } + var tasks []seiv1alpha1.PlannedTask + + // For fork ceremonies, prepend exporter lifecycle tasks. + if hasCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) && group.Spec.Genesis.Fork != nil { + fork := group.Spec.Genesis.Fork + exporterName := fmt.Sprintf("%s-exporter", group.Name) + + createExporter, err := buildGroupPlannedTask(group.Name, task.TaskTypeCreateExporter, + &task.CreateExporterParams{ + GroupName: group.Name, + ExporterName: exporterName, + Namespace: group.Namespace, + SourceChainID: fork.SourceChainID, + SourceImage: fork.SourceImage, + ExportHeight: fork.ExportHeight, + }) + if err != nil { + return nil, err + } + + awaitExporter, err := buildGroupPlannedTask(group.Name, task.TaskTypeAwaitExporterRunning, + &task.AwaitExporterRunningParams{ + ExporterName: exporterName, + Namespace: group.Namespace, + }) + if err != nil { + return nil, err + } + + submitExport, err := buildGroupPlannedTask(group.Name, task.TaskTypeSubmitExportState, + &task.SubmitExportStateParams{ + ExporterName: exporterName, + Namespace: group.Namespace, + ExportHeight: group.Spec.Genesis.Fork.ExportHeight, + SourceChainID: group.Spec.Genesis.Fork.SourceChainID, + }) + if err != nil { + return nil, err + } + + teardownExporter, err := buildGroupPlannedTask(group.Name, task.TaskTypeTeardownExporter, + &task.TeardownExporterParams{ + ExporterName: exporterName, + Namespace: group.Namespace, + }) + if err != nil { + return nil, err + } + + tasks = append(tasks, createExporter, awaitExporter, submitExport, teardownExporter) + } + + tasks = append(tasks, assembleTask, collectPeersTask, awaitTask) + return &seiv1alpha1.TaskPlan{ Phase: seiv1alpha1.TaskPlanActive, - Tasks: []seiv1alpha1.PlannedTask{assembleTask, collectPeersTask, awaitTask}, + Tasks: tasks, }, nil } diff --git a/internal/planner/group_test.go b/internal/planner/group_test.go index 1364222..d1a8eb7 100644 --- a/internal/planner/group_test.go +++ b/internal/planner/group_test.go @@ -21,6 +21,9 @@ func TestBuildGroupAssemblyPlan(t *testing.T) { }, Status: seiv1alpha1.SeiNodeGroupStatus{ IncumbentNodes: []string{"node-0", "node-1", "node-2"}, + Conditions: []metav1.Condition{ + {Type: seiv1alpha1.ConditionGenesisCeremonyNeeded, Status: metav1.ConditionTrue}, + }, }, } @@ -111,6 +114,9 @@ func TestBuildGroupAssemblyPlan_DefaultS3(t *testing.T) { }, Status: seiv1alpha1.SeiNodeGroupStatus{ IncumbentNodes: []string{"node-0"}, + Conditions: []metav1.Condition{ + {Type: seiv1alpha1.ConditionGenesisCeremonyNeeded, Status: metav1.ConditionTrue}, + }, }, } @@ -135,6 +141,155 @@ func TestBuildGroupAssemblyPlan_DefaultS3(t *testing.T) { } } +func TestBuildGroupForkPlan(t *testing.T) { + const sourceChain = "pacific-1" + + group := &seiv1alpha1.SeiNodeGroup{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group", Namespace: "default"}, + Spec: seiv1alpha1.SeiNodeGroupSpec{ + Replicas: 2, + Genesis: &seiv1alpha1.GenesisCeremonyConfig{ + ChainID: "fork-1", + AccountBalance: "1000usei", + Fork: &seiv1alpha1.ForkConfig{ + SourceChainID: sourceChain, + SourceImage: "sei:v5.0.0", + ExportHeight: 100000, + }, + }, + }, + Status: seiv1alpha1.SeiNodeGroupStatus{ + IncumbentNodes: []string{"fork-group-0", "fork-group-1"}, + Conditions: []metav1.Condition{ + {Type: seiv1alpha1.ConditionForkGenesisCeremonyNeeded, Status: metav1.ConditionTrue}, + }, + }, + } + + p, err := ForGroup(group) + if err != nil { + t.Fatalf("ForGroup: %v", err) + } + plan, err := p.BuildPlan(group) + if err != nil { + t.Fatalf("BuildPlan: %v", err) + } + + if plan.Phase != seiv1alpha1.TaskPlanActive { + t.Errorf("phase = %q, want Active", plan.Phase) + } + + // Fork plan: 4 exporter tasks + 3 shared tasks = 7 + if len(plan.Tasks) != 7 { + t.Fatalf("expected 7 tasks, got %d", len(plan.Tasks)) + } + + expectedTypes := []string{ + task.TaskTypeCreateExporter, + task.TaskTypeAwaitExporterRunning, + task.TaskTypeSubmitExportState, + task.TaskTypeTeardownExporter, + TaskAssembleGenesisFork, + task.TaskTypeCollectAndSetPeers, + TaskAwaitNodesRunning, + } + for i, wantType := range expectedTypes { + if plan.Tasks[i].Type != wantType { + t.Errorf("task[%d] type = %q, want %q", i, plan.Tasks[i].Type, wantType) + } + if plan.Tasks[i].Status != seiv1alpha1.TaskPending { + t.Errorf("task[%d] status = %q, want Pending", i, plan.Tasks[i].Status) + } + if plan.Tasks[i].ID == "" { + t.Errorf("task[%d] ID should not be empty", i) + } + } + + // Verify create-exporter params + var createParams task.CreateExporterParams + if err := json.Unmarshal(plan.Tasks[0].Params.Raw, &createParams); err != nil { + t.Fatalf("unmarshal create-exporter params: %v", err) + } + if createParams.GroupName != "fork-group" { + t.Errorf("GroupName = %q, want %q", createParams.GroupName, "fork-group") + } + if createParams.ExporterName != "fork-group-exporter" { + t.Errorf("ExporterName = %q, want %q", createParams.ExporterName, "fork-group-exporter") + } + if createParams.SourceChainID != sourceChain { + t.Errorf("SourceChainID = %q, want %q", createParams.SourceChainID, sourceChain) + } + if createParams.SourceImage != "sei:v5.0.0" { + t.Errorf("SourceImage = %q, want %q", createParams.SourceImage, "sei:v5.0.0") + } + if createParams.ExportHeight != 100000 { + t.Errorf("ExportHeight = %d, want %d", createParams.ExportHeight, 100000) + } + + // Verify submit-export-state params + var submitParams task.SubmitExportStateParams + if err := json.Unmarshal(plan.Tasks[2].Params.Raw, &submitParams); err != nil { + t.Fatalf("unmarshal submit-export-state params: %v", err) + } + if submitParams.SourceChainID != sourceChain { + t.Errorf("submit SourceChainID = %q, want %q", submitParams.SourceChainID, sourceChain) + } + + // Verify assemble-genesis-fork params + var assembleParams task.AssembleForkGenesisParams + if err := json.Unmarshal(plan.Tasks[4].Params.Raw, &assembleParams); err != nil { + t.Fatalf("unmarshal assemble-genesis-fork params: %v", err) + } + if assembleParams.SourceChainID != sourceChain { + t.Errorf("assemble SourceChainID = %q, want %q", assembleParams.SourceChainID, sourceChain) + } + if len(assembleParams.Nodes) != 2 { + t.Errorf("assemble Nodes = %d, want 2", len(assembleParams.Nodes)) + } + + // Verify assemble task has MaxRetries + if plan.Tasks[4].MaxRetries != groupAssemblyMaxRetries { + t.Errorf("assemble MaxRetries = %d, want %d", plan.Tasks[4].MaxRetries, groupAssemblyMaxRetries) + } +} + +func TestBuildGroupForkPlan_NilForkSpecNoExporterTasks(t *testing.T) { + // ForkGenesisCeremonyNeeded condition is set but Fork spec is nil. + // The planner should fall back to the standard genesis plan. + group := &seiv1alpha1.SeiNodeGroup{ + ObjectMeta: metav1.ObjectMeta{Name: "edge-group", Namespace: "default"}, + Spec: seiv1alpha1.SeiNodeGroupSpec{ + Replicas: 1, + Genesis: &seiv1alpha1.GenesisCeremonyConfig{ + ChainID: "test-1", + }, + }, + Status: seiv1alpha1.SeiNodeGroupStatus{ + IncumbentNodes: []string{"edge-group-0"}, + Conditions: []metav1.Condition{ + {Type: seiv1alpha1.ConditionForkGenesisCeremonyNeeded, Status: metav1.ConditionTrue}, + }, + }, + } + + p, err := ForGroup(group) + if err != nil { + t.Fatalf("ForGroup: %v", err) + } + plan, err := p.BuildPlan(group) + if err != nil { + t.Fatalf("BuildPlan: %v", err) + } + + // Should fall back to standard genesis (3 tasks, no exporter) + if len(plan.Tasks) != 3 { + t.Fatalf("expected 3 tasks (standard genesis fallback), got %d", len(plan.Tasks)) + } + if plan.Tasks[0].Type != TaskAssembleGenesis { + t.Errorf("task[0] type = %q, want %q", plan.Tasks[0].Type, TaskAssembleGenesis) + } +} + func TestBuildGroupAssemblyPlan_DeterministicIDs(t *testing.T) { group := &seiv1alpha1.SeiNodeGroup{ ObjectMeta: metav1.ObjectMeta{Name: "det-group", Namespace: "default"}, @@ -146,6 +301,9 @@ func TestBuildGroupAssemblyPlan_DeterministicIDs(t *testing.T) { }, Status: seiv1alpha1.SeiNodeGroupStatus{ IncumbentNodes: []string{"node-0", "node-1"}, + Conditions: []metav1.Condition{ + {Type: seiv1alpha1.ConditionGenesisCeremonyNeeded, Status: metav1.ConditionTrue}, + }, }, } diff --git a/internal/planner/planner.go b/internal/planner/planner.go index 6a5097d..8ad447b 100644 --- a/internal/planner/planner.go +++ b/internal/planner/planner.go @@ -58,8 +58,6 @@ type GroupPlanner interface { // ForGroup returns the appropriate GroupPlanner based on the group's // current state and spec. Returns (nil, nil) when no plan is needed. func ForGroup(group *seiv1alpha1.SeiNodeGroup) (GroupPlanner, error) { - // Genesis ceremony: needs a plan when genesis is configured, no plan - // exists yet, ceremony isn't complete, and all nodes are created. if needsGenesisPlan(group) { return &genesisGroupPlanner{}, nil } @@ -73,22 +71,34 @@ func ForGroup(group *seiv1alpha1.SeiNodeGroup) (GroupPlanner, error) { return nil, nil } +// needsGenesisPlan returns true when either GenesisCeremonyNeeded or +// ForkGenesisCeremonyNeeded condition is set with sufficient nodes. func needsGenesisPlan(group *seiv1alpha1.SeiNodeGroup) bool { if group.Spec.Genesis == nil { return false } - if group.Status.ObservedGeneration != 0 { - return false // genesis only runs on the first generation - } if group.Status.Plan != nil { return false } + genesisNeeded := hasCondition(group, seiv1alpha1.ConditionGenesisCeremonyNeeded) + forkNeeded := hasCondition(group, seiv1alpha1.ConditionForkGenesisCeremonyNeeded) + if !genesisNeeded && !forkNeeded { + return false + } + return allReplicasCreated(group) +} + +func allReplicasCreated(group *seiv1alpha1.SeiNodeGroup) bool { + return int32(len(group.Status.IncumbentNodes)) >= group.Spec.Replicas +} + +func hasCondition(group *seiv1alpha1.SeiNodeGroup, condType string) bool { for _, c := range group.Status.Conditions { - if c.Type == seiv1alpha1.ConditionGenesisCeremonyComplete && c.Status == metav1.ConditionTrue { - return false + if c.Type == condType && c.Status == metav1.ConditionTrue { + return true } } - return int32(len(group.Status.IncumbentNodes)) >= group.Spec.Replicas + return false } // ForNode returns the appropriate NodePlanner based on which mode sub-spec diff --git a/internal/task/fork.go b/internal/task/fork.go new file mode 100644 index 0000000..119ed36 --- /dev/null +++ b/internal/task/fork.go @@ -0,0 +1,34 @@ +package task + +import sidecar "github.com/sei-protocol/seictl/sidecar/client" + +// AssembleForkGenesisParams are the serialized fields for the +// assemble-genesis-fork sidecar task. The sidecar downloads exported +// state from the platform genesis bucket at {sourceChainId}/exported-state.json, +// strips old validators, rewrites chain identity, and runs collect-gentxs. +type AssembleForkGenesisParams struct { + SourceChainID string `json:"sourceChainId"` + ChainID string `json:"chainId"` + AccountBalance string `json:"accountBalance"` + Namespace string `json:"namespace"` + Nodes []GenesisNodeParam `json:"nodes"` +} + +func (p *AssembleForkGenesisParams) taskType() string { return sidecar.TaskTypeAssembleGenesisFork } + +func (p *AssembleForkGenesisParams) toRequestParams() *map[string]any { + nodes := make([]any, len(p.Nodes)) + for i, n := range p.Nodes { + nodes[i] = map[string]any{"name": n.Name} + } + m := map[string]any{ + "sourceChainId": p.SourceChainID, + "chainId": p.ChainID, + "accountBalance": p.AccountBalance, + "namespace": p.Namespace, + "nodes": nodes, + } + return &m +} + +var _ taskParamser = (*AssembleForkGenesisParams)(nil) diff --git a/internal/task/fork_export.go b/internal/task/fork_export.go new file mode 100644 index 0000000..d6d7fa1 --- /dev/null +++ b/internal/task/fork_export.go @@ -0,0 +1,362 @@ +package task + +import ( + "context" + "encoding/json" + "fmt" + "time" + + "github.com/google/uuid" + sidecar "github.com/sei-protocol/seictl/sidecar/client" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + ctrl "sigs.k8s.io/controller-runtime" + + seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1" +) + +// exportStateTimeout is the maximum time the export-state task can run +// before the controller fails the plan. Generous because seid export +// on large chains can take hours. +const exportStateTimeout = 6 * time.Hour + +const ( + TaskTypeCreateExporter = "create-exporter" + TaskTypeAwaitExporterRunning = "await-exporter-running" + TaskTypeSubmitExportState = "submit-export-state" + TaskTypeTeardownExporter = "teardown-exporter" +) + +// CreateExporterParams holds parameters for creating the temporary +// exporter SeiNode. +type CreateExporterParams struct { + GroupName string `json:"groupName"` + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` + SourceChainID string `json:"sourceChainId"` + SourceImage string `json:"sourceImage"` + ExportHeight int64 `json:"exportHeight"` +} + +type createExporterExecution struct { + taskBase + params CreateExporterParams + cfg ExecutionConfig +} + +func deserializeCreateExporter(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + var p CreateExporterParams + if len(params) > 0 { + if err := json.Unmarshal(params, &p); err != nil { + return nil, fmt.Errorf("deserializing create-exporter params: %w", err) + } + } + return &createExporterExecution{ + taskBase: taskBase{id: id, status: ExecutionRunning}, + params: p, + cfg: cfg, + }, nil +} + +func (e *createExporterExecution) Execute(ctx context.Context) error { + group, err := ResourceAs[*seiv1alpha1.SeiNodeGroup](e.cfg) + if err != nil { + return Terminal(err) + } + + existing := &seiv1alpha1.SeiNode{} + err = e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, existing) + if err == nil { + // If a previous attempt left a failed exporter, delete it so we + // can create a fresh one on the next reconcile. + if existing.Status.Phase == seiv1alpha1.PhaseFailed { + if delErr := e.cfg.KubeClient.Delete(ctx, existing); delErr != nil && !apierrors.IsNotFound(delErr) { + return fmt.Errorf("deleting failed exporter: %w", delErr) + } + return nil // next reconcile will re-enter Execute and create a new exporter + } + e.complete() + return nil + } + if !apierrors.IsNotFound(err) { + return fmt.Errorf("checking exporter existence: %w", err) + } + + // The exporter uses the source chain's binary (SourceImage) paired + // with the group's sidecar config. The sidecar is chain-agnostic — + // it only needs to talk to seid's RPC and manage files on disk. + node := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{ + Name: e.params.ExporterName, + Namespace: e.params.Namespace, + Labels: map[string]string{ + "sei.io/nodegroup": e.params.GroupName, + "sei.io/role": "exporter", + }, + }, + Spec: seiv1alpha1.SeiNodeSpec{ + ChainID: e.params.SourceChainID, + Image: e.params.SourceImage, + Sidecar: group.Spec.Template.Spec.Sidecar, + FullNode: &seiv1alpha1.FullNodeSpec{ + Snapshot: &seiv1alpha1.SnapshotSource{ + S3: &seiv1alpha1.S3SnapshotSource{ + TargetHeight: e.params.ExportHeight, + }, + BootstrapImage: e.params.SourceImage, + }, + }, + }, + } + + if err := ctrl.SetControllerReference(group, node, e.cfg.Scheme); err != nil { + return Terminal(fmt.Errorf("setting owner reference: %w", err)) + } + + if err := e.cfg.KubeClient.Create(ctx, node); err != nil { + if apierrors.IsAlreadyExists(err) { + e.complete() + return nil + } + return fmt.Errorf("creating exporter: %w", err) + } + + e.complete() + return nil +} + +func (e *createExporterExecution) Status(_ context.Context) ExecutionStatus { + return e.status +} + +// AwaitExporterRunningParams holds parameters for polling the exporter +// SeiNode until it reaches Running phase. +type AwaitExporterRunningParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` +} + +type awaitExporterRunningExecution struct { + taskBase + params AwaitExporterRunningParams + cfg ExecutionConfig +} + +func deserializeAwaitExporterRunning(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + var p AwaitExporterRunningParams + if len(params) > 0 { + if err := json.Unmarshal(params, &p); err != nil { + return nil, fmt.Errorf("deserializing await-exporter-running params: %w", err) + } + } + return &awaitExporterRunningExecution{ + taskBase: taskBase{id: id, status: ExecutionRunning}, + params: p, + cfg: cfg, + }, nil +} + +func (e *awaitExporterRunningExecution) Execute(_ context.Context) error { return nil } + +func (e *awaitExporterRunningExecution) Status(ctx context.Context) ExecutionStatus { + if s, done := e.isTerminal(); done { + return s + } + node := &seiv1alpha1.SeiNode{} + err := e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, node) + if apierrors.IsNotFound(err) { + e.setFailed(fmt.Errorf("exporter %s not found — create-exporter may have failed", e.params.ExporterName)) + return ExecutionFailed + } + if err != nil { + return ExecutionRunning + } + switch node.Status.Phase { + case seiv1alpha1.PhaseRunning: + e.complete() + return ExecutionComplete + case seiv1alpha1.PhaseFailed: + e.setFailed(fmt.Errorf("exporter %s failed", e.params.ExporterName)) + return ExecutionFailed + default: + return ExecutionRunning + } +} + +// SubmitExportStateParams holds parameters for submitting the export-state +// task to the exporter's sidecar. +type SubmitExportStateParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` + ExportHeight int64 `json:"exportHeight"` + SourceChainID string `json:"sourceChainId"` +} + +type submitExportStateExecution struct { + taskBase + params SubmitExportStateParams + cfg ExecutionConfig +} + +func deserializeSubmitExportState(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + var p SubmitExportStateParams + if len(params) > 0 { + if err := json.Unmarshal(params, &p); err != nil { + return nil, fmt.Errorf("deserializing submit-export-state params: %w", err) + } + } + return &submitExportStateExecution{ + taskBase: taskBase{id: id, status: ExecutionRunning}, + params: p, + cfg: cfg, + }, nil +} + +// sidecarTaskID returns the deterministic UUID used for the export-state +// submission to the exporter's sidecar. Recomputed on every call so it +// survives re-deserialization across reconcile boundaries. +func (e *submitExportStateExecution) sidecarTaskID() uuid.UUID { + return uuid.MustParse(DeterministicTaskID(e.params.ExporterName, "export-state", 0)) +} + +func (e *submitExportStateExecution) Execute(ctx context.Context) error { + node := &seiv1alpha1.SeiNode{} + if err := e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, node); err != nil { + return fmt.Errorf("getting exporter node: %w", err) + } + + sc, err := sidecarClientForNode(node) + if err != nil { + return fmt.Errorf("building sidecar client for exporter: %w", err) + } + + taskID := e.sidecarTaskID() + exportParams := map[string]any{ + "height": e.params.ExportHeight, + "chainId": e.params.SourceChainID, + "s3Bucket": e.cfg.Platform.GenesisBucket, + "s3Key": fmt.Sprintf("%s/exported-state.json", e.params.SourceChainID), + "s3Region": e.cfg.Platform.GenesisRegion, + } + + req := sidecar.TaskRequest{ + Id: &taskID, + Type: sidecar.TaskTypeExportState, + Params: &exportParams, + } + if _, err := sc.SubmitTask(ctx, req); err != nil { + return fmt.Errorf("submitting export-state to exporter: %w", err) + } + + return nil +} + +func (e *submitExportStateExecution) Status(ctx context.Context) ExecutionStatus { + if s, done := e.isTerminal(); done { + return s + } + + node := &seiv1alpha1.SeiNode{} + if err := e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, node); err != nil { + return ExecutionRunning + } + + // Timeout: fail if the exporter has been alive longer than the limit. + if !node.CreationTimestamp.IsZero() && time.Since(node.CreationTimestamp.Time) > exportStateTimeout { + e.setFailed(fmt.Errorf("export-state timed out after %s", exportStateTimeout)) + return ExecutionFailed + } + + sc, err := sidecarClientForNode(node) + if err != nil { + return ExecutionRunning + } + + taskID := e.sidecarTaskID() + result, err := sc.GetTask(ctx, taskID) + if err != nil { + return ExecutionRunning + } + + switch result.Status { + case sidecar.Completed: + e.complete() + return ExecutionComplete + case sidecar.Failed: + errMsg := "unknown error" + if result.Error != nil { + errMsg = *result.Error + } + e.setFailed(fmt.Errorf("export-state failed: %s", errMsg)) + return ExecutionFailed + default: + return ExecutionRunning + } +} + +// TeardownExporterParams holds parameters for deleting the exporter SeiNode. +type TeardownExporterParams struct { + ExporterName string `json:"exporterName"` + Namespace string `json:"namespace"` +} + +type teardownExporterExecution struct { + taskBase + params TeardownExporterParams + cfg ExecutionConfig +} + +func deserializeTeardownExporter(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + var p TeardownExporterParams + if len(params) > 0 { + if err := json.Unmarshal(params, &p); err != nil { + return nil, fmt.Errorf("deserializing teardown-exporter params: %w", err) + } + } + return &teardownExporterExecution{ + taskBase: taskBase{id: id, status: ExecutionRunning}, + params: p, + cfg: cfg, + }, nil +} + +func (e *teardownExporterExecution) Execute(ctx context.Context) error { + node := &seiv1alpha1.SeiNode{} + err := e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, node) + if apierrors.IsNotFound(err) { + e.complete() + return nil + } + if err != nil { + return fmt.Errorf("getting exporter for deletion: %w", err) + } + if err := e.cfg.KubeClient.Delete(ctx, node); err != nil && !apierrors.IsNotFound(err) { + return fmt.Errorf("deleting exporter: %w", err) + } + return nil +} + +func (e *teardownExporterExecution) Status(ctx context.Context) ExecutionStatus { + if s, done := e.isTerminal(); done { + return s + } + err := e.cfg.KubeClient.Get(ctx, types.NamespacedName{ + Name: e.params.ExporterName, Namespace: e.params.Namespace, + }, &seiv1alpha1.SeiNode{}) + if apierrors.IsNotFound(err) { + e.complete() + return ExecutionComplete + } + return ExecutionRunning +} diff --git a/internal/task/fork_export_test.go b/internal/task/fork_export_test.go new file mode 100644 index 0000000..f31ebed --- /dev/null +++ b/internal/task/fork_export_test.go @@ -0,0 +1,427 @@ +package task + +import ( + "context" + "encoding/json" + "testing" + "time" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/types" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + + seiv1alpha1 "github.com/sei-protocol/sei-k8s-controller/api/v1alpha1" + "github.com/sei-protocol/sei-k8s-controller/internal/platform/platformtest" +) + +func testGroup() *seiv1alpha1.SeiNodeGroup { + return &seiv1alpha1.SeiNodeGroup{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group", Namespace: "default", UID: "uid-group"}, + Spec: seiv1alpha1.SeiNodeGroupSpec{ + Replicas: 2, + Template: seiv1alpha1.SeiNodeTemplate{ + Spec: seiv1alpha1.SeiNodeSpec{ + Image: "sei:v6.0.0", + Sidecar: &seiv1alpha1.SidecarConfig{ + Image: "seictl:latest", + Port: 7777, + }, + }, + }, + Genesis: &seiv1alpha1.GenesisCeremonyConfig{ + ChainID: "fork-1", + Fork: &seiv1alpha1.ForkConfig{ + SourceChainID: "pacific-1", + SourceImage: "sei:v5.0.0", + ExportHeight: 100000, + }, + }, + }, + } +} + +func testGroupCfg(t *testing.T, group *seiv1alpha1.SeiNodeGroup, objs ...metav1.Object) ExecutionConfig { + t.Helper() + s := testScheme(t) + clientObjs := make([]interface{ GetName() string }, 0, len(objs)+1) + _ = clientObjs // satisfy unused + + builder := fake.NewClientBuilder().WithScheme(s).WithObjects(group) + for _, obj := range objs { + if sn, ok := obj.(*seiv1alpha1.SeiNode); ok { + builder = builder.WithObjects(sn) + } + } + c := builder.Build() + return ExecutionConfig{ + KubeClient: c, + Scheme: s, + Resource: group, + Platform: platformtest.Config(), + } +} + +// --- create-exporter --- + +func TestCreateExporter_Execute_CreatesSeiNode(t *testing.T) { + group := testGroup() + cfg := testGroupCfg(t, group) + + params := CreateExporterParams{ + ExporterName: "fork-group-exporter", + Namespace: "default", + SourceChainID: "pacific-1", + SourceImage: "sei:v5.0.0", + ExportHeight: 100000, + } + raw, _ := json.Marshal(params) + exec, err := deserializeCreateExporter("id-1", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + ctx := context.Background() + if err := exec.Execute(ctx); err != nil { + t.Fatalf("Execute: %v", err) + } + if exec.Status(ctx) != ExecutionComplete { + t.Fatalf("expected Complete, got %s", exec.Status(ctx)) + } + + node := &seiv1alpha1.SeiNode{} + if err := cfg.KubeClient.Get(ctx, types.NamespacedName{Name: "fork-group-exporter", Namespace: "default"}, node); err != nil { + t.Fatalf("exporter node not found: %v", err) + } + if node.Spec.ChainID != "pacific-1" { + t.Errorf("ChainID = %q, want %q", node.Spec.ChainID, "pacific-1") + } + if node.Spec.Image != "sei:v5.0.0" { + t.Errorf("Image = %q, want %q", node.Spec.Image, "sei:v5.0.0") + } + if node.Labels["sei.io/role"] != "exporter" { + t.Errorf("role label = %q, want %q", node.Labels["sei.io/role"], "exporter") + } + if node.Spec.FullNode == nil || node.Spec.FullNode.Snapshot == nil { + t.Fatal("expected FullNode with Snapshot config") + } + if node.Spec.FullNode.Snapshot.S3 == nil || node.Spec.FullNode.Snapshot.S3.TargetHeight != 100000 { + t.Errorf("TargetHeight = %d, want %d", node.Spec.FullNode.Snapshot.S3.TargetHeight, 100000) + } +} + +func TestCreateExporter_Execute_Idempotent(t *testing.T) { + group := testGroup() + existing := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{ + Name: "fork-group-exporter", + Namespace: "default", + }, + Spec: seiv1alpha1.SeiNodeSpec{ + ChainID: "pacific-1", + Image: "sei:v5.0.0", + }, + Status: seiv1alpha1.SeiNodeStatus{ + Phase: seiv1alpha1.PhaseInitializing, + }, + } + cfg := testGroupCfg(t, group, existing) + + params := CreateExporterParams{ + ExporterName: "fork-group-exporter", + Namespace: "default", + SourceChainID: "pacific-1", + SourceImage: "sei:v5.0.0", + ExportHeight: 100000, + } + raw, _ := json.Marshal(params) + exec, err := deserializeCreateExporter("id-1", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + ctx := context.Background() + if err := exec.Execute(ctx); err != nil { + t.Fatalf("Execute on existing: %v", err) + } + if exec.Status(ctx) != ExecutionComplete { + t.Fatalf("expected Complete for existing exporter, got %s", exec.Status(ctx)) + } +} + +func TestCreateExporter_Execute_DeletesFailedExporter(t *testing.T) { + group := testGroup() + failed := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{ + Name: "fork-group-exporter", + Namespace: "default", + }, + Spec: seiv1alpha1.SeiNodeSpec{ + ChainID: "pacific-1", + Image: "sei:v5.0.0", + }, + Status: seiv1alpha1.SeiNodeStatus{ + Phase: seiv1alpha1.PhaseFailed, + }, + } + cfg := testGroupCfg(t, group, failed) + + params := CreateExporterParams{ + ExporterName: "fork-group-exporter", + Namespace: "default", + SourceChainID: "pacific-1", + SourceImage: "sei:v5.0.0", + ExportHeight: 100000, + } + raw, _ := json.Marshal(params) + exec, err := deserializeCreateExporter("id-1", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + ctx := context.Background() + if err := exec.Execute(ctx); err != nil { + t.Fatalf("Execute: %v", err) + } + // Should NOT be complete — the failed exporter was deleted, + // next reconcile will re-enter and create a fresh one. + if exec.Status(ctx) != ExecutionRunning { + t.Fatalf("expected Running (pending recreate), got %s", exec.Status(ctx)) + } + + // Verify the failed exporter was deleted + node := &seiv1alpha1.SeiNode{} + err = cfg.KubeClient.Get(ctx, types.NamespacedName{Name: "fork-group-exporter", Namespace: "default"}, node) + if !apierrors.IsNotFound(err) { + t.Fatalf("expected NotFound after deleting failed exporter, got %v", err) + } +} + +// --- await-exporter-running --- + +func TestAwaitExporterRunning_Running(t *testing.T) { + group := testGroup() + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group-exporter", Namespace: "default"}, + Status: seiv1alpha1.SeiNodeStatus{Phase: seiv1alpha1.PhaseInitializing}, + } + cfg := testGroupCfg(t, group, exporter) + + params := AwaitExporterRunningParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeAwaitExporterRunning("id-2", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionRunning { + t.Fatal("expected Running for Initializing exporter") + } +} + +func TestAwaitExporterRunning_Complete(t *testing.T) { + group := testGroup() + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group-exporter", Namespace: "default"}, + Status: seiv1alpha1.SeiNodeStatus{Phase: seiv1alpha1.PhaseRunning}, + } + cfg := testGroupCfg(t, group, exporter) + + params := AwaitExporterRunningParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeAwaitExporterRunning("id-2", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionComplete { + t.Fatal("expected Complete for Running exporter") + } +} + +func TestAwaitExporterRunning_Failed(t *testing.T) { + group := testGroup() + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group-exporter", Namespace: "default"}, + Status: seiv1alpha1.SeiNodeStatus{Phase: seiv1alpha1.PhaseFailed}, + } + cfg := testGroupCfg(t, group, exporter) + + params := AwaitExporterRunningParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeAwaitExporterRunning("id-2", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionFailed { + t.Fatal("expected Failed for Failed exporter") + } + if exec.Err() == nil { + t.Fatal("expected non-nil error") + } +} + +func TestAwaitExporterRunning_NotFound_Fails(t *testing.T) { + group := testGroup() + cfg := testGroupCfg(t, group) + + params := AwaitExporterRunningParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeAwaitExporterRunning("id-2", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionFailed { + t.Fatal("expected Failed when exporter not found") + } + if exec.Err() == nil { + t.Fatal("expected non-nil error") + } +} + +// --- teardown-exporter --- + +func TestTeardownExporter_Execute_DeletesNode(t *testing.T) { + group := testGroup() + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group-exporter", Namespace: "default"}, + } + cfg := testGroupCfg(t, group, exporter) + + params := TeardownExporterParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeTeardownExporter("id-4", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + ctx := context.Background() + if err := exec.Execute(ctx); err != nil { + t.Fatalf("Execute: %v", err) + } + + node := &seiv1alpha1.SeiNode{} + err = cfg.KubeClient.Get(ctx, types.NamespacedName{Name: "fork-group-exporter", Namespace: "default"}, node) + if !apierrors.IsNotFound(err) { + t.Fatalf("expected exporter to be deleted, got %v", err) + } +} + +func TestTeardownExporter_Execute_Idempotent(t *testing.T) { + group := testGroup() + cfg := testGroupCfg(t, group) + + params := TeardownExporterParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeTeardownExporter("id-4", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if err := exec.Execute(context.Background()); err != nil { + t.Fatalf("expected no error for nonexistent exporter, got %v", err) + } +} + +func TestTeardownExporter_Status_CompleteWhenGone(t *testing.T) { + group := testGroup() + cfg := testGroupCfg(t, group) + + params := TeardownExporterParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeTeardownExporter("id-4", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionComplete { + t.Fatal("expected Complete when exporter already gone") + } +} + +func TestTeardownExporter_Status_RunningWhilePresent(t *testing.T) { + group := testGroup() + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{Name: "fork-group-exporter", Namespace: "default"}, + } + cfg := testGroupCfg(t, group, exporter) + + params := TeardownExporterParams{ExporterName: "fork-group-exporter", Namespace: "default"} + raw, _ := json.Marshal(params) + exec, err := deserializeTeardownExporter("id-4", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + if exec.Status(context.Background()) != ExecutionRunning { + t.Fatal("expected Running while exporter still exists") + } +} + +// --- submit-export-state --- + +func TestSubmitExportState_SidecarTaskIDDeterministic(t *testing.T) { + group := testGroup() + cfg := testGroupCfg(t, group) + + params := SubmitExportStateParams{ + ExporterName: "fork-group-exporter", + Namespace: "default", + ExportHeight: 100000, + SourceChainID: "pacific-1", + } + raw, _ := json.Marshal(params) + + exec1, err := deserializeSubmitExportState("id-3", raw, cfg) + if err != nil { + t.Fatalf("deserialize 1: %v", err) + } + exec2, err := deserializeSubmitExportState("id-3", raw, cfg) + if err != nil { + t.Fatalf("deserialize 2: %v", err) + } + + se1 := exec1.(*submitExportStateExecution) + se2 := exec2.(*submitExportStateExecution) + + if se1.sidecarTaskID() != se2.sidecarTaskID() { + t.Errorf("sidecar task IDs not deterministic: %s vs %s", + se1.sidecarTaskID(), se2.sidecarTaskID()) + } +} + +func TestSubmitExportState_Timeout(t *testing.T) { + group := testGroup() + // Create an exporter that was created long ago + exporter := &seiv1alpha1.SeiNode{ + ObjectMeta: metav1.ObjectMeta{ + Name: "fork-group-exporter", + Namespace: "default", + CreationTimestamp: metav1.NewTime(time.Now().Add(-7 * time.Hour)), + }, + Status: seiv1alpha1.SeiNodeStatus{Phase: seiv1alpha1.PhaseRunning}, + } + cfg := testGroupCfg(t, group, exporter) + + params := SubmitExportStateParams{ + ExporterName: "fork-group-exporter", + Namespace: "default", + ExportHeight: 100000, + SourceChainID: "pacific-1", + } + raw, _ := json.Marshal(params) + exec, err := deserializeSubmitExportState("id-3", raw, cfg) + if err != nil { + t.Fatalf("deserialize: %v", err) + } + + status := exec.Status(context.Background()) + if status != ExecutionFailed { + t.Fatalf("expected Failed after timeout, got %s", status) + } + if exec.Err() == nil { + t.Fatal("expected non-nil timeout error") + } +} diff --git a/internal/task/genesis_peers.go b/internal/task/genesis_peers.go index 63b86cf..cec14a8 100644 --- a/internal/task/genesis_peers.go +++ b/internal/task/genesis_peers.go @@ -74,7 +74,7 @@ func (e *collectAndSetPeersExecution) collectPeers(ctx context.Context) ([]strin return nil, fmt.Errorf("getting node ID for %s: %w", name, err) } - dns := fmt.Sprintf("%s.%s.svc.cluster.local", name, e.params.Namespace) + dns := fmt.Sprintf("%s-0.%s.%s.svc.cluster.local", name, name, e.params.Namespace) peers = append(peers, fmt.Sprintf("%s@%s:%d", nodeID, dns, defaultP2PPort)) } return peers, nil diff --git a/internal/task/task.go b/internal/task/task.go index 5e96791..71a62c0 100644 --- a/internal/task/task.go +++ b/internal/task/task.go @@ -152,77 +152,68 @@ func ResourceAs[T client.Object](cfg ExecutionConfig) (T, error) { return r, nil } -// Deserialize reconstructs a TaskExecution from its serialized CRD -// representation. Dependencies are injected via the ExecutionConfig bundle. -// Returns UnknownTaskTypeError for unrecognized types. -func Deserialize(taskType, id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { - buildSC := cfg.BuildSidecarClient - switch taskType { - // Bootstrap tasks - case sidecar.TaskTypeSnapshotRestore: - return deserializeSidecar[SnapshotRestoreParams](id, params, buildSC, false) - case sidecar.TaskTypeConfigureStateSync: - return deserializeSidecar[ConfigureStateSyncParams](id, params, buildSC, false) - case sidecar.TaskTypeAwaitCondition: - return deserializeSidecar[AwaitConditionParams](id, params, buildSC, false) - - // Config tasks - case sidecar.TaskTypeConfigApply: - return deserializeSidecar[ConfigApplyParams](id, params, buildSC, false) - case sidecar.TaskTypeConfigValidate: - return deserializeSidecar[ConfigValidateParams](id, params, buildSC, true) - case sidecar.TaskTypeConfigureGenesis: - return deserializeSidecar[ConfigureGenesisParams](id, params, buildSC, false) - case sidecar.TaskTypeDiscoverPeers: - return deserializeSidecar[DiscoverPeersParams](id, params, buildSC, false) - case sidecar.TaskTypeMarkReady: - return deserializeSidecar[MarkReadyParams](id, params, buildSC, true) - - // Genesis ceremony tasks - case sidecar.TaskTypeGenerateIdentity: - return deserializeSidecar[GenerateIdentityParams](id, params, buildSC, false) - case sidecar.TaskTypeGenerateGentx: - return deserializeSidecar[GenerateGentxParams](id, params, buildSC, false) - case sidecar.TaskTypeUploadGenesisArtifacts: - return deserializeSidecar[UploadGenesisArtifactsParams](id, params, buildSC, false) - case sidecar.TaskTypeAssembleGenesis: - return deserializeSidecar[AssembleAndUploadGenesisParams](id, params, buildSC, false) - case sidecar.TaskTypeSetGenesisPeers: - return deserializeSidecar[SetGenesisPeersParams](id, params, buildSC, false) +// taskDeserializer reconstructs a TaskExecution from serialized params. +type taskDeserializer func(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) + +// sidecarTask creates a deserializer for a sidecar-backed task type. +func sidecarTask[T any](fireAndForget bool) taskDeserializer { + return func(id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + return deserializeSidecar[T](id, params, cfg.BuildSidecarClient, fireAndForget) + } +} + +// registry maps task type strings to their deserializers. +var registry = map[string]taskDeserializer{ + // Sidecar tasks + sidecar.TaskTypeSnapshotRestore: sidecarTask[SnapshotRestoreParams](false), + sidecar.TaskTypeConfigureStateSync: sidecarTask[ConfigureStateSyncParams](false), + sidecar.TaskTypeAwaitCondition: sidecarTask[AwaitConditionParams](false), + sidecar.TaskTypeConfigApply: sidecarTask[ConfigApplyParams](false), + sidecar.TaskTypeConfigValidate: sidecarTask[ConfigValidateParams](true), + sidecar.TaskTypeConfigureGenesis: sidecarTask[ConfigureGenesisParams](false), + sidecar.TaskTypeDiscoverPeers: sidecarTask[DiscoverPeersParams](false), + sidecar.TaskTypeMarkReady: sidecarTask[MarkReadyParams](true), + sidecar.TaskTypeGenerateIdentity: sidecarTask[GenerateIdentityParams](false), + sidecar.TaskTypeGenerateGentx: sidecarTask[GenerateGentxParams](false), + sidecar.TaskTypeUploadGenesisArtifacts: sidecarTask[UploadGenesisArtifactsParams](false), + sidecar.TaskTypeAssembleGenesis: sidecarTask[AssembleAndUploadGenesisParams](false), + sidecar.TaskTypeSetGenesisPeers: sidecarTask[SetGenesisPeersParams](false), + sidecar.TaskTypeAssembleGenesisFork: sidecarTask[AssembleForkGenesisParams](false), // Controller-side group tasks - case TaskTypeAwaitNodesRunning: - return deserializeAwaitNodesRunning(id, params, cfg) - case TaskTypeCollectAndSetPeers: - return deserializeCollectAndSetPeers(id, params, cfg) + TaskTypeAwaitNodesRunning: deserializeAwaitNodesRunning, + TaskTypeCollectAndSetPeers: deserializeCollectAndSetPeers, // Controller-side bootstrap tasks - case TaskTypeDeployBootstrapSvc: - return deserializeBootstrapService(id, params, cfg) - case TaskTypeDeployBootstrapJob: - return deserializeBootstrapJob(id, params, cfg) - case TaskTypeAwaitBootstrapComplete: - return deserializeBootstrapAwait(id, params, cfg) - case TaskTypeTeardownBootstrap: - return deserializeBootstrapTeardown(id, params, cfg) + TaskTypeDeployBootstrapSvc: deserializeBootstrapService, + TaskTypeDeployBootstrapJob: deserializeBootstrapJob, + TaskTypeAwaitBootstrapComplete: deserializeBootstrapAwait, + TaskTypeTeardownBootstrap: deserializeBootstrapTeardown, // Controller-side deployment tasks - case TaskTypeCreateEntrantNodes: - return deserializeCreateEntrantNodes(id, params, cfg) - case TaskTypeSubmitHaltSignal: - return deserializeSubmitHaltSignal(id, params, cfg) - case TaskTypeAwaitNodesAtHeight: - return deserializeAwaitNodesAtHeight(id, params, cfg) - case TaskTypeAwaitNodesCaughtUp: - return deserializeAwaitNodesCaughtUp(id, params, cfg) - case TaskTypeSwitchTraffic: - return deserializeSwitchTraffic(id, params, cfg) - case TaskTypeTeardownNodes: - return deserializeTeardownNodes(id, params, cfg) - - default: + TaskTypeCreateEntrantNodes: deserializeCreateEntrantNodes, + TaskTypeSubmitHaltSignal: deserializeSubmitHaltSignal, + TaskTypeAwaitNodesAtHeight: deserializeAwaitNodesAtHeight, + TaskTypeAwaitNodesCaughtUp: deserializeAwaitNodesCaughtUp, + TaskTypeSwitchTraffic: deserializeSwitchTraffic, + TaskTypeTeardownNodes: deserializeTeardownNodes, + + // Controller-side fork tasks + TaskTypeCreateExporter: deserializeCreateExporter, + TaskTypeAwaitExporterRunning: deserializeAwaitExporterRunning, + TaskTypeSubmitExportState: deserializeSubmitExportState, + TaskTypeTeardownExporter: deserializeTeardownExporter, +} + +// Deserialize reconstructs a TaskExecution from its serialized CRD +// representation. Dependencies are injected via the ExecutionConfig bundle. +// Returns UnknownTaskTypeError for unrecognized types. +func Deserialize(taskType, id string, params json.RawMessage, cfg ExecutionConfig) (TaskExecution, error) { + fn, ok := registry[taskType] + if !ok { return nil, &UnknownTaskTypeError{Type: taskType} } + return fn(id, params, cfg) } // deserializeSidecar is a generic helper that unmarshals params into a typed diff --git a/manifests/sei.io_seinodegroups.yaml b/manifests/sei.io_seinodegroups.yaml index ca958f0..97a9ac2 100644 --- a/manifests/sei.io_seinodegroups.yaml +++ b/manifests/sei.io_seinodegroups.yaml @@ -111,6 +111,38 @@ spec: minLength: 1 pattern: ^[a-z0-9][a-z0-9-]*[a-z0-9]$ type: string + fork: + description: |- + Fork configures this genesis ceremony to fork from an existing + chain's exported state rather than building genesis from scratch. + When set, the assembler downloads the exported state, rewrites + the chain identity, strips old validators, and runs collect-gentxs + with the new validator set. + properties: + exportHeight: + description: |- + ExportHeight is the block height at which to export state. + seid export --height N reads committed state at exactly this height. + format: int64 + minimum: 1 + type: integer + sourceChainId: + description: SourceChainID is the chain ID of the network + being forked. + minLength: 1 + type: string + sourceImage: + description: |- + SourceImage is the seid container image compatible with the source + chain at ExportHeight. Used as both the bootstrap and main image + for the temporary exporter node. + minLength: 1 + type: string + required: + - exportHeight + - sourceChainId + - sourceImage + type: object maxCeremonyDuration: description: |- MaxCeremonyDuration is the maximum time from group creation to genesis