Skip to content
Konstellation-NetworkPublic

About

No description, website, or topics provided.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

infra

OpenTofu (Terraform HCL) + Ansible for the Konstellation validator fleet and supporting infrastructure. Private, permanently (ENGINEERING.md §6.4) — this is a map of where the validators live.

Structure follows ENGINEERING.md §6.4:

terraform/
├── modules/
│   ├── hetzner/{validator,sentry,rpc,archive,bastion,monitoring,cosigner}/   # hcloud
│   └── gcp/{validator,sentry,rpc,archive,bastion,monitoring,cosigner}/        # google
└── envs/
    └── testnet-1/                                  # instantiates both providers for one network
        ├── topology.tf                             # the cross-cloud decisions, as locals
        ├── inventory.tpl.yml, ssh_config.tpl       # -> ansible/inventories/testnet-1/{hosts.yml,ssh_config}
        └── backend.tf                              # partial gcs config — bucket NOT chosen (open, P4)
                                                    # devnet-1 has NO env here: Contabo, ordered by hand
                                                    # (2026-09-30); the Hetzner env was removed (git history)
ansible/
├── roles/
│   ├── {node,cosmovisor,horcrux,monitoring,firewall}/   # chain nodes (monitoring = node_exporter, every host)
│   ├── bastion/, wireguard/                        # jump hosts + the site-to-site tunnel
│   └── monitoring_server/                          # prometheus, alertmanager, grafana, tenderduty
├── inventories/testnet-1/                          # hosts.yml + ssh_config generated by OpenTofu
├── inventories/devnet-1/                           # hosts.yml + ssh_config HAND-WRITTEN from the .example files
├── tests/                                          # render_test.yml (testnet-1 fixture) + devnet_render_test.yml
│                                                   # (fixture = the devnet-1 example): template regressions, no hosts
├── bootstrap.yml                                   # devnet-1 only: `deploy` user on a fresh hand-ordered VPS
├── site.yml                                        # full fleet config, idempotent; bastions first
├── upgrade.yml                                     # stages a checksummed binary via cosmovisor (chain_nodes)
└── requirements.yml                                # ansible-galaxy collections (community.general, ansible.posix)
monitoring/
├── prometheus/alerts.yml                           # deployed verbatim by roles/monitoring_server
└── grafana/                                        # empty — see grafana/README.md
runbooks/                                           # on-call, emergency-halt, coordinated-upgrade,
                                                    # validator-key-rotation, validator-admission,
                                                    # incident-comms — none rehearsed yet
CODEOWNERS

runbooks/ is written (2026-09-20) but every file says Rehearsed: never at the top. The first halt drill (ENGINEERING.md §13, §15 Phase 5) is where they get corrected — edit them in the same PR as the drill write-up.

Status

Scaffolding only. No tofu apply or ansible-playbook run has happened against real infrastructure — there's no Hetzner project, no GCP project, no Contabo server, no release binary to install, and no key ceremony yet. Everything below is what exists in code, not what's running.

Verified so far (no cloud credentials needed for any of this): tofu fmt -check, tofu validate on every module and on envs/testnet-1 (OpenTofu — a drop-in, terraform-compatible CLI; terraform itself works identically), ansible-playbook --syntax-check on site.yml/upgrade.yml/bootstrap.yml, yamllint, ansible-lint clean except the intentional var-naming trade-off documented in ansible/.ansible-lint, a credential-free tofu plan of envs/testnet-1 against a scratch local backend for every topology combination (bastion_ssh_entry × monitoring_cloud × horcrux_mode; 2026-09-21 counts, for the then 10-validator fleet: 58–60 resources colocated, 63–65 dedicated — the range is the optional operator-SSH firewall rule and the monitoring host's cloud; not re-run after the 2026-09-29 reduction to 4 validators, which has tofu validate only), the inventory/ssh_config templates rendered via tofu console, and ansible/tests/render_test.yml — rendered-template regressions (ansible-playbook -i tests/inventory/hosts.yml tests/render_test.yml, controller-only, touches no host). It imports the roles' own task files (import_role … tasks_from) so the shipped tasks run with real variable precedence, and it is kept honest by the mutation set in its header — deliberate breakages that each must fail it (seven as of 2026-09-21, all verified). Covers: the cosmovisor current symlink guard, the data-rebuild guard (roles/node/tasks/data_guard.yml, §2.7), the mount guard on the node unit, WireGuard source routes, bastion scrape targets, Prometheus's own port not colliding with CometBFT's prometheus_port (a real bug the test found), tenderduty's isolation, digest pin and targets, peer wiring (validators ← same-cloud sentries only, sentries cross-peered), horcrux's per-validator instances and metrics endpoints. ansible/tests/devnet_render_test.yml (-i tests/devnet_inventory/hosts.yml, whose hosts.yml and group_vars are symlinks to the shipped devnet-1 example; rewritten 2026-09-30 for Contabo, ten mutations in its header) covers the three-server shape: the firewall plan per host (roles/firewall/tasks/plan.yml — the validator admits SSH and p2p from server 2's /32 only and metrics from server 3's /32 only, and has no RPC rule at all; server 2 is sentry + public RPC + operator-only SSH; server 3 opens nothing publicly; a validator sharing a host is refused), server 2's single node carrying both sentry peering and public RPC config, nothing binding RPC publicly on servers 1 and 3, the port guard (Prometheus vs the archive's SDK gRPC on 9090), scrape and tenderduty targets covering all three hosts once each, the single-disk state guard, no mount guard where there is no volume, no IPv4 forwarding on server 2, horcrux_mode: none, and the WireGuard role gated off. The firewall refactor (rules computed in plan.yml, applied in main.yml) was checked against testnet-1's fixture: the planned rules for all 17 hosts are the rules the old tasks applied, one for one. Not verified: an actual apply (needs a real Hetzner token and GCP project), anything ansible actually touching a live host — including the WireGuard tunnel, the Hetzner NAT path, the per-validator horcrux instances and bootstrap.yml against a real Contabo image — all of which are written against the providers' documented behaviour, not observed.

Current scope: devnet-1 and testnet-1

Three networks (D18, decided 2026-09-29, Solana-style): devnet-1 for dapp developers (one validator, three hand-ordered Contabo servers — section "devnet-1" below), testnet-1 for validator and upgrade rehearsal (this section), and konstellation-1 (mainnet, 4 validators — no environment here yet).

testnet-1: 4 validators, split across Hetzner and GCP

4 foundation-run validators (ENGINEERING.md D7, re-decided 2026-09-29; was 10 = 5 Hetzner + 5 GCP), further admissions permissioned via D16 — runbooks/validator-admission.md. Placement rule: the 4 validators belong in 4 separate failure domains, so losing one provider or region never halts the chain — at equal power, 1 of 4 down leaves 75 % (live), 2 of 4 down leaves 50 % (halt: CometBFT needs more than ⅔). Which four domains is still open (Contabo, more clouds, more regions — STATUS §5a P16), so the split is two variables, hetzner_validator_count / gcp_validator_count, defaulting to 2 + 2: v1 fsn1, v2 hel1 on Hetzner; v3, v4 in two GCP zones of one region. That default puts 2 of 4 — 50 % of voting power — behind each provider, so losing either provider (or GCP's region) halts the chain. Accepted for testnet-1 only; every plan prints validator_placement_warning saying so, in the style of the P21 cosigner warning. Each cloud has its own private network; they are joined by one routed site-to-site WireGuard tunnel between the two bastions (see "Cross-cloud connectivity"):

Hetzner (10.0.1.0/24) GCP (10.0.2.0/24)
Validators 2 (v1 fsn1, v2 hel1), no public IP, .11-.12 (slots to .15) 2 (v3, v4, zones a/b), no public IP, .16-.17 (slots to .20)
Sentries 2, 1:1 with their validator, public, .21-.22 2, 1:1 with their validator, public, .26-.27
Archive 1, internal only, .41 1, internal only, .42
RPC — 1, public, .31
Bastion / gateway 1, public, .5 — SSH entry, NAT egress for the private hosts, tunnel endpoint 1, public, .5 — SSH entry, tunnel endpoint (egress is Cloud NAT)
Monitoring — 1, internal only, .50 (monitoring_cloud = "gcp", default)
Horcrux cosigners c1 fsn1, c3 hel1 (.61, .63) c2 (.62) — only with horcrux_mode = "dedicated"; default is colocated, so none

Each validator's sentry lives in the same cloud — the validator↔sentry p2p link stays inside one private network and never crosses the tunnel. Sentries from both clouds peer with each other over the public internet like any other node. Only management traffic (SSH from a single-entry bastion, metrics scrape, horcrux signing in dedicated mode) crosses the tunnel.

Host count with the defaults (horcrux_mode = colocated): 4 validators + 4 sentries + 2 archives + 1 RPC + 2 bastions + 1 monitoring = 14 hosts (was 26); dedicated adds 3 cosigners (17), each running one horcrux instance per validator (4). Mainnet also has 4 validators, so terraform/envs/konstellation-1/ (not yet created — deliberately not in this change) should start as a copy of this environment with persistent disks, horcrux_mode = "dedicated" (§18) and a placement that actually meets the 4-failure-domain rule — the open decision above.

Cross-cloud connectivity

Simplest safe default, chosen 2026-09-20 and flagged for a human to confirm: one kernel-WireGuard tunnel between the two bastions, with a cloud route on each side (hcloud_network_route / google_compute_route) sending the other cloud's /24 through its local bastion. Traffic is routed, not NAT'd, so every firewall rule sees the real source IP and "internal only" means fleet_cidrs (both /24s) rather than one subnet. Why this and not the alternatives:

  • Public scrape with mTLS needs a TLS proxy and a certificate on every node, and doesn't help horcrux at all. Rejected.
  • Managed interconnect / VPN gateways — Hetzner has none. Rejected.
  • WireGuard on every node is the same trust model with 15× the key material. Rejected.

Costs of the choice, stated plainly: the bastions are now routers (kernel forwarding; sshd is still their only userland listener); the tunnel is a single point of failure for cross-cloud management traffic (validator↔sentry p2p never crosses it); and in dedicated-horcrux mode it is a single point of failure for consensus too (adversarial review, 2026-09-21; STATUS §5a P22): the cosigners reach each other through the tunnel, so a tunnel outage partitions the far cloud's cosigner from the other two — and since each cluster's quorum path for the validators in that cloud runs across the tunnel, that cloud's validators (2 of 4 by default) stop signing and the chain halts within seconds. CloudUnreachable says so. An earlier version of this paragraph claimed "still quorate"; it was wrong. Design note (P22, not built): the cosigners need their own peer-pinned mesh — WireGuard between the three cosigner hosts over their own public IPs (or redundant tunnels), pinned to each other's keys, so that horcrux raft/grpc never depends on the bastions. Until that exists, dedicated mode is a liveness downgrade, not only a key-safety upgrade; testnet-1 runs colocated. WireGuard keys are generated on each bastion by ansible/roles/wireguard and never leave it; only public keys travel, as Ansible facts within one play, and each side refuses a fact that is not a 32-byte base64 key before templating it into root's wg0.conf.

Hetzner-specific: hcloud firewalls filter the public interface only, so hosts with no public IP are not covered by them at all — ansible/roles/ firewall (ufw) is the primary control on Hetzner, not a redundant one. And a Hetzner server with no public IP has no internet egress unless something NATs for it; the bastion does (nat_gateway = true, an hcloud_network_route 0.0.0.0/0, and a cloud-init unit on every private-only Hetzner host pointing its default route at the network gateway). Without this the validators could never apt-get or fetch their checksummed binary — a gap in the original scaffold, closed here.

Cosigner admin domains (P21) — interim

One deploy SSH key with NOPASSWD sudo on every host was the original scaffold, cosigners included: one leaked key = three shards = the consensus key of every validator. Interim (2026-09-21): var.cosigner_ssh_public_key is a separate key wired only into modules/{hetzner,gcp}/cosigner, required in dedicated mode and validated to differ from ssh_public_key. What it does not do — and what STATUS §5a P21 has to decide before mainnet: the cosigner hosts still get the same deploy user with unrestricted sudo (cloud-init), the three cosigners still share one key, and the default cosigner_placement still puts two shards in one Hetzner account (fsn1 + hel1), so one hcloud API token can rescue-boot a threshold — the plan prints cosigner_placement_warning for it. Per-cosigner keys/operators, hardware-backed keys, restricted sudo and a third provider are P21/P16, not this branch.

Sentry topology (P23) — interim

One sentry per validator was the original scaffold. Interim (2026-09-21): ansible/roles/node/tasks/peers.yml wires every validator to all its cloud's sentries (2 with the default counts) and cross-peers every sentry with every other on public IPs, with private_peer_ids = the same-cloud validators; node ids are read after init so nothing is hand-copied. The cloud firewalls and ufw admit p2p on validators from those sentries' /32s only — the /24 used to include the internet-facing RPC node. Still P23: the sentry count (≥ 2 per validator is the target; today the redundancy comes from sharing the cloud's two) and a DoS on both of a cloud's sentry IPs still isolates that cloud's validators.

Decisions parametrised here (defaults chosen, not decided)

All in terraform/envs/testnet-1/variables.tf, all documented inline:

Variable Default The decision a human still owns
hetzner_validator_count / gcp_validator_count 2 / 2 how the 4 validators (D7, 2026-09-29) are split, i.e. which failure domains they occupy. The founder's rule is 4 separate domains so one provider/region loss never halts; which domains (Contabo, more clouds, more regions) is open (P16). 2 + 2 halts on either provider's loss — validator_placement_warning. Adding a third provider means new modules, not just a count.
bastion_ssh_entry per-cloud one bastion reachable from both clouds vs one per cloud. Both bastion hosts exist either way (tunnel endpoints); this only picks which run an internet-facing sshd. per-cloud means a tunnel outage never locks you out of either side, which matters most in an emergency halt. "hetzner"/"gcp" gives a single audited entry point at the cost of that dependency — and is applied in two steps (first per-cloud, so ansible can reach both bastions to build the tunnel, then switch).
monitoring_cloud gcp where the one Prometheus/Grafana/Alertmanager/tenderduty box lives. GCP because Cloud NAT keeps its bastion out of the egress path.
horcrux_mode + cosigner_placement colocated (no cosigner hosts) testnet-1 keeps the original co-located mode; dedicated provisions 3 cosigner hosts (fsn1, us-central1-b, hel1 — no location or cloud holds two shards) and ansible runs one horcrux instance per validator on each. Mandatory for mainnet (§18). Switching a live validator is a key ceremony (runbooks/validator-key-rotation.md).
backend.tf bucket none the terraform state bucket. Still a partial gcs config; terraform init -backend-config=backend.hcl with a bucket a human creates. Not parametrised further because there is nothing to default to.

GCP validators use Local SSD, and that choice needs to be read, not just trusted: terraform/modules/gcp/validator's local_ssd_count variable has the full explanation, but the short version — GCP Local SSD doesn't survive a stop or a host error. Scheduled maintenance is fine: per GCP's current live-migration docs (checked 2026-09-21) VMs with Local SSD live-migrate with the disk, so the module sets on_host_maintenance = MIGRATE (the original scaffold had TERMINATE, which would have discarded the disk on every maintenance event — flagged for a human to re-confirm against the docs of the day before the first apply). For the host-error path, where the VM auto-restarts with empty Local SSDs, ansible/roles/cosmovisor's unit carries RequiresMountsFor + ConditionPathIsMountPoint on the data directory: the node stays down (a NodeDown page) rather than starting with its key and a priv_validator_state.json at height 0 — the double-sign scenario ENGINEERING.md §2.7 calls unrecoverable (5% slash, no undo). Local SSD was chosen for testnet-1 to match Hetzner's local-NVMe performance profile on the other half of the fleet. ansible/roles/node stripes the Local SSDs into one RAID0 array (a single 375 GiB disk doesn't meet the multi-TB spec either way). Re-decide this before any mainnet validator runs on GCP, probably in favor of pd-ssd despite its worse commit latency; with dedicated Horcrux cosigners the key is no longer on that disk, which is most of the risk.

devnet-1: one validator, three Contabo servers (D18; Contabo decided 2026-09-30)

The dapp-developer network: chain-id devnet-1, EIP-155 56672, one foundation-run validator, and it runs the mainnet binary version (ansible/inventories/devnet-1/group_vars/all.yml pins the same release as konstellation-1, not testnet-1's). Contabo, devnet-1 only (testnet-1 and mainnet placement stays open under STATUS P16). The three VPSes are ordered by hand in Contabo's panel; there is no OpenTofu for devnet-1 (the Hetzner env that used to be terraform/envs/devnet-1 was removed on 2026-09-30; git history keeps it). Ansible does everything from the OS up, from a hand-written inventory: ansible/inventories/devnet-1/hosts.yml.example and ssh_config.example.

Server Inventory host Groups Listens publicly Why it exists
1 devnet-1-contabo-validator-v1 validators nothing: SSH and p2p from server 2's /32, metrics from server 3's /32; CometBFT RPC on 127.0.0.1, REST and JSON-RPC off the only block producer; plain priv_validator_key.json, horcrux_mode: none
2 devnet-1-contabo-sentry-rpc-1 sentries, rpc, bastions 26656, 26657, 1317, 8545-8546; SSH from devnet_operator_ssh_cidrs only the sentry the validator peers with (§9.2), the public RPC dapps, wallets and the faucet use, and the one SSH entry point (hardened sshd, fail2ban) — no NAT, no forwarding
3 devnet-1-contabo-archive-mon-1 archive, monitoring nothing yet: SSH from server 2 only pruning = nothing + tracing (the explorer's backend), Prometheus/Alertmanager/Grafana/tenderduty; the explorer and faucet containers will land here later

Each VPS has one public IPv4 and one disk. There is no private network (a paid add-on we are not using), no cloud firewall and no separate volume: ansible/roles/firewall (ufw) is the only network control. The inventory sets private_network: false, which makes the firewall role insist that every internal source (fleet_cidrs, metrics_scrape_cidrs, bastion_private_ip, monitoring_host_private_ip — names kept from the private-network fleets, holding public addresses here) is an exact /32, and refuse NAT/tunnel settings on the bastion. Nothing the archive serves binds to its public IP (loopback only), so a firewall mistake cannot expose the debug JSON-RPC.

Design choices, and why:

  • SSH: ProxyJump through server 2, not direct SSH to all three. Servers 1 and 3 admit SSH from server 2's /32 only, so the internet sees exactly one sshd, and an operator IP change is one edit on server 2, never on the validator. The cost is that server 2 down = no SSH to 1 and 3: the break-glass path is Contabo's VNC console with the root password (which bootstrap.yml leaves working for the console only).
  • Several roles on one host. A host is listed in every group whose role it holds; roles/firewall/tasks/plan.yml applies the union of those roles' rules (on testnet-1, one group per host, it is the same rule set as before, checked host by host). node_role is pinned per host in hosts.yml, because with several groups the group_vars value is just whichever file sorts last — and without that, server 2 would have lost the operator SSH rule in the bastions play. Server 2 runs one konstellationd: sentry peering (private_peer_ids = the validator) from sentries.yml, public listeners and the mempool floor from rpc.yml. A validator sharing its host with any other role is refused.
  • Server 3 port collision. The archive's konstellationd binds the SDK gRPC on localhost:9090, Prometheus's default web port, so devnet-1 moves prometheus_web_port to 9095; roles/monitoring_server/tasks/port_guard.yml refuses any overlap between monitoring listeners and a co-located node's ports. Alertmanager's gossip port binds loopback there.
  • Scrape and tenderduty targets list each konstellationd once (server 2 is in two chain groups) on its public IP; the validator's and server 2's metrics admit server 3's /32 only.
  • The archive peers with server 2 (peer_non_validators_with_sentries): there is no seed node, and on testnet-1 the archive relies on seeds.

Key handling — why no Horcrux, and what §2.7 still demands. Horcrux's two benefits are (a) no single host holds the whole key and (b) a validator host can be replaced without moving the key. For a single validator on a valueless dev network neither buys much, while 3 extra hosts (or 3 colocated processes on the one host, which gives neither benefit) would add a signing-cluster outage mode to the network dapp developers most need up. So the validator signs with a plain priv_validator_key.json (horcrux_enabled: false; roles/node refuses true when horcrux_mode: none); the founder holds its offline backup (STATUS P29). ENGINEERING.md §2.7 applies unchanged: the key and its priv_validator_state.json live on exactly one running host, ever — never a "warm standby", never a second node started "to get devnet back up", never a Contabo snapshot restored over a running chain (it rolls the state file back).

The disk guards on one disk (single_disk: true). The mount guard (roles/cosmovisor RequiresMountsFor) and the data-rebuild guard (roles/node/tasks/data_guard.yml) exist for a data volume lost while the key survives on another disk. On a single disk key, state and chain data go together — a lost disk is downtime, never a node signing from height 0 — so neither applies, and the unit says why instead of carrying a guard with nothing to mount. What can still happen on one disk is the same failure by hand: the data directory wiped or replaced under the key. So roles/node/tasks/state_guard.yml stops site.yml on the validator whenever the key is present and its state file is not (override -e allow_data_rebuild=true only after runbooks/validator-key-rotation.md §B). single_disk and a data volume are mutually exclusive (asserted).

Liveness is not a property of devnet-1. One validator holds 100 % of voting power: server 1 or Contabo going down halts devnet-1. Server 2 down does not stop blocks (the validator needs no peers to commit), but cuts the public RPC, the archive and SSH to the other two until it is back. runbooks/validator-key-rotation.md §B's "state = chain height + a margin" failover deadlocks a one-validator chain; its devnet-1 paragraph says what to do instead.

Contabo plans (recommendation, confirm names/specs in the panel of the day; all three in one region, Debian 12):

Server Plan Why
1 validator Cloud VPS 30 (8 vCPU, 24 GB, NVMe) pruned state stays small on a quiet devnet; NVMe for commit latency (§9.2); headroom for the EVM. Cloud VPS 20 works while traffic is low
2 sentry + RPC Cloud VPS 30 (8 vCPU, 24 GB, NVMe) the public JSON-RPC (eth_getLogs, eth_call) is the CPU/RAM hot spot; pruned
3 archive + monitoring (+ explorer, faucet) Cloud VPS 40 (12 vCPU, 48 GB) with the largest disk offered pruning = nothing grows forever; Blockscout's Postgres and indexer want ~16 GB; Prometheus keeps 90 days. Disk is the limit — watch DiskHeadroomLow

devnet-1: deploying

Prerequisites: the release pins in group_vars/all.yml (konstellationd_version/_sha256), genesis_url/genesis_sha256 from networks/devnet-1, and every other empty pin the roles fail closed on (cosmovisor_sha256, node_exporter_sha256, prometheus_sha256, alertmanager_sha256, grafana_apt_key_fingerprint, tenderduty_image_digest) — grep -rn '""' ansible/inventories/devnet-1/group_vars. Commands from the infra repo root unless noted:

# 1. One key for `deploy` (and, simplest, for root at order time too)
ssh-keygen -t ed25519 -f ~/.ssh/konstellation_devnet_deploy -C devnet-1-deploy

# 2. Contabo panel: order 3 VPSes (plans above), image Debian 12, same region,
#    and add ~/.ssh/konstellation_devnet_deploy.pub as the root SSH key.
#    (Password-only root works too: add --ask-pass in step 5, needs sshpass.)
#    Keep the root passwords: after step 5 they only work on the VNC console.

# 3. Inventory: copy, then fill in the 4 FILL IN values in hosts.yml and the
#    3 IPs in ssh_config (the same addresses)
cd ansible
cp inventories/devnet-1/hosts.yml.example inventories/devnet-1/hosts.yml
cp inventories/devnet-1/ssh_config.example inventories/devnet-1/ssh_config
$EDITOR inventories/devnet-1/hosts.yml inventories/devnet-1/ssh_config

# 4. Collections
ansible-galaxy collection install -r requirements.yml

# 5. First contact, as root, directly to each IP: creates `deploy` with the
#    key and NOPASSWD sudo, proves deploy can become root, THEN turns off
#    root and password SSH logins
ansible-playbook -i inventories/devnet-1/hosts.yml bootstrap.yml \
  --private-key ~/.ssh/konstellation_devnet_deploy \
  -e deploy_ssh_public_key_file=~/.ssh/konstellation_devnet_deploy.pub

# 6. The jump path works (server 2 direct, 1 and 3 through it)
ssh -F inventories/devnet-1/ssh_config devnet-1-contabo-validator-v1 true
ssh -F inventories/devnet-1/ssh_config devnet-1-contabo-archive-mon-1 true

# 7. Server 2 first (SSH entry, firewall), then everything. Every run
#    needs -i: ansible.cfg's default inventory is testnet-1's.
ansible-playbook -i inventories/devnet-1/hosts.yml site.yml --limit bastions
ansible-playbook -i inventories/devnet-1/hosts.yml site.yml
# (a second full run once every node is initialised is harmless and makes
#  sure all node ids were seen when the peer lists were computed)

# 8. Install the GENESIS validator key over the one `init` generated.
#    Stop first; keep init's priv_validator_state.json (height 0, pre-genesis).
S=inventories/devnet-1/ssh_config; V=devnet-1-contabo-validator-v1
ssh -F $S $V sudo systemctl stop cosmovisor
scp -F $S /path/to/devnet-1/priv_validator_key.json $V:/tmp/pvk.json
ssh -F $S $V 'sudo install -o konstellation -g konstellation -m 0600 /tmp/pvk.json \
  /home/konstellation/.konstellationd/config/priv_validator_key.json && shred -u /tmp/pvk.json \
  && sudo systemctl start cosmovisor'

# 9. Check from OUTSIDE: server 2 answers, the validator does not
curl -s http://<server2>:26657/status | jq .result.sync_info.latest_block_height
curl -s -X POST -H 'content-type: application/json' \
  --data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' http://<server2>:8545   # "0xdd60" = 56672
nc -zv -w 3 <server1> 26657; nc -zv -w 3 <server1> 22     # both must time out

Dashboards: ssh -F inventories/devnet-1/ssh_config -L 3000:localhost:3000 -L 9093:localhost:9093 -L 8888:localhost:8888 devnet-1-contabo-archive-mon-1, then Grafana :3000, Alertmanager :9093, tenderduty :8888. Fill the validator's valoper in group_vars/monitoring.yml once the gentx exists. hosts.yml and ssh_config are not gitignored (like testnet-1's generated ones); committing them to this private repo lets every operator run the same inventory.

Known gaps — closed 2026-09-20 (branch close-known-gaps)

  • Bastion host. terraform/modules/{hetzner,gcp}/bastion, wired in envs/testnet-1; the old var.bastion_ipv4 input is gone — every host's SSH source is now its own cloud's bastion private IP, from the module output, in both the cloud firewalls and ufw. ansible/roles/bastion hardens sshd (key-only, AllowUsers deploy, no agent forwarding, fail2ban with the operator CIDRs in ignoreip). Operators SSH in from var.operator_ssh_cidrs only (validated non-empty and never 0.0.0.0/0); ansible/inventories/testnet-1/ssh_config (generated) ProxyJumps every other host.
  • Monitoring host. terraform/modules/{hetzner,gcp}/monitoring + ansible/roles/monitoring_server: Prometheus (scrape config templated from the inventory, monitoring/prometheus/alerts.yml deployed verbatim and promtool-checked), Alertmanager with a placeholder on-call route (nothing paged until alert_* vars are set from a vault — see runbooks/on-call.md), Grafana (loopback only, provisioned datasource), and tenderduty for validator paging (§9.2, §17). Everything is reached with an SSH port-forward through the bastion; nothing is public.
  • Dedicated horcrux cosigners. terraform/modules/{hetzner,gcp}/cosigner behind horcrux_mode = "dedicated"; ansible/roles/horcrux builds one instance per validator per cosigner host from the inventory, ports 2222+i, cosigner list and shard_ids templated, units installed disabled. ansible/roles/node sets priv_validator_laddr only when a validator's horcrux_enabled is flipped after its ceremony. Co-located mode is unchanged and remains the testnet-1 default. No key material anywhere in this repo, by construction.
  • Runbooks. All four from ENGINEERING.md §6.4 plus on-call.md, grounded in §2.7, §13/D14 (x/circuit CLI and its protected types), cosmovisor, networks/templates/upgrade.md. None rehearsed.
  • CODEOWNERS from the org template, with terraform/envs/konstellation-1/ and runbooks/ as separate lines.

Known gaps — still open

  • Public RPC node exposes CometBFT RPC 26657 and REST 1317 to the internet (group_vars/rpc.yml, rpc_public firewall) — unauthenticated, unmetered, a DoS surface. Acceptable for testnet-1 behind the minimum-gas-prices floor; put a rate-limiting reverse proxy in front (or move RPC to the app tier, ENGINEERING.md §9.1) before mainnet.

  • Terraform state backend (terraform/envs/testnet-1/backend.tf) is a partial gcs config with no bucket chosen. Human decision.

  • bastion_ssh_entry / monitoring_cloud / horcrux_mode have defaults, not decisions — table above. Human decision, before the first apply for the first two, before real validator keys for the third.

  • Version pins are empty. konstellationd_version, konstellationd_sha256, cosmovisor_sha256, horcrux_sha256, node_exporter_sha256, and now prometheus_sha256, alertmanager_sha256 are blank on purpose — konstellation hasn't cut a release build yet. Every download task asserts its checksum var is non-empty and fails closed.

  • tenderduty is unmaintained. github.com/blockpane/tenderduty was archived on 2025-01-02; it ships as a container image only (no binaries), so roles/monitoring_server runs it under Debian's docker.io with a latest tag until a digest is chosen (tenderduty_image_digest, tenderduty_require_digest). Works today, will not get fixes. Evaluate a maintained fork before mainnet.

  • On-call rota (§17): mechanics in runbooks/on-call.md, schedule TBD until validators exist. The alert routes stay placeholders until then.

  • GCP validator disk durability — see the callout above. Not a gap so much as a decision deliberately deferred past testnet-1; with dedicated cosigners the key is no longer on that disk, which is most of the risk.

  • Metric names in alerts.yml are CometBFT's documented convention, not yet confirmed against a live /metrics endpoint (see the caveat comment in that file). promtool check rules validates syntax, not names.

  • Grafana dashboards — none (monitoring/grafana/README.md).

  • Not verified against real providers: Hetzner's DHCP/route behaviour for private-only hosts (the cloud-init default-route unit and the WireGuard ip route replace lines are written to be harmless if DHCP already pushed the route — verify on first apply), Hetzner's private network handing out no resolvers (cloud-init writes 185.12.64.1/2), GCP can_ip_forward + custom-route return paths through the tunnel, GCP live migration preserving Local SSD data on N2, and the exact Grafana apt package version.

  • Public RPC node minimum-gas-prices (group_vars/rpc.yml): a node-local 1 gwei-equivalent floor so the one internet-facing mempool is not a free spam target. Not a protocol parameter (TOKENOMICS.md §3 keeps that at 0), but it does set what users of the public RPC pay at minimum — flagged for the TOKENOMICS owner to confirm or lower.

  • Explorer reachability (found by the explorer's live run, 2026-09-20): archive nodes now bind JSON-RPC/WS to their private address instead of init's 127.0.0.1 (group_vars/archive.yml jsonrpc_bind_address), and var.explorer_cidrs / archive_rpc_allowed_cidrs gate who may reach them — both empty/fleet-wide until the explorer host exists and its route into a private network (a bastion WireGuard peer, or a host inside one cloud) is decided. Open: that route.

  • devnet-1 (2026-09-30, Contabo): the explorer and faucet containers on server 3 are not built here yet — they need the archive's loopback RPC (host networking, or bind to the docker bridge address) and a reverse proxy for their public ports, and ports docker publishes bypass ufw, the only firewall there. The lost-state-file recovery for the one validator is open (STATUS P29). Server 2 is a single point of failure for SSH to servers 1 and 3 (break-glass: Contabo's VNC console). The validator↔sentry p2p link and the metrics scrape cross the public internet (unencrypted metrics; p2p is CometBFT's authenticated encryption).

  • IPv6 is only opened on the Hetzner sentry firewall, not GCP's (would need a dual-stack VPC there) — a minor asymmetry, not a security gap.

Apply flow (once the open gaps above are closed)

cd terraform/envs/testnet-1
cp terraform.tfvars.example terraform.tfvars   # fill in, do not commit
gcloud auth application-default login           # or set GOOGLE_APPLICATION_CREDENTIALS
terraform init -backend-config=backend.hcl      # backend.hcl gitignored, not yet written
terraform plan
terraform apply
# writes ansible/inventories/testnet-1/hosts.yml and ssh_config from outputs

cd ../../../ansible
ansible-galaxy collection install -r requirements.yml
ansible-playbook site.yml --limit bastions            # first: everything else is reached through them,
                                                      # and Hetzner's private hosts have no egress until NAT is up
ansible-playbook site.yml --limit validators --check  # dry run one group first
ansible-playbook site.yml

devnet-1 has no OpenTofu step at all — see "devnet-1: deploying" above.

Humans: ssh -F ansible/inventories/testnet-1/ssh_config testnet-1-gcp-validator-v3. Dashboards: ssh -F … -L 3000:localhost:3000 -L 9093:localhost:9093 -L 8888:localhost:8888 testnet-1-gcp-monitoring-1 then Grafana on :3000, Alertmanager :9093, tenderduty :8888.

Fleet upgrades: ansible-playbook -i inventories/testnet-1/hosts.yml upgrade.yml -e upgrade_name=... -e binary_sha256=... -e binary_url=... (ENGINEERING.md §16, runbooks/coordinated-upgrade.md). One invocation covers both clouds — upgrade.yml targets chain_nodes.

About

No description, website, or topics provided.

Resources

Security policy

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages