OpenTofu (Terraform HCL) + Ansible for the Konstellation validator fleet and
supporting infrastructure. Private, permanently (ENGINEERING.md §6.4) — this is a map
of where the validators live.
Structure follows ENGINEERING.md §6.4:
terraform/
├── modules/
│ ├── hetzner/{validator,sentry,rpc,archive,bastion,monitoring,cosigner}/ # hcloud
│ └── gcp/{validator,sentry,rpc,archive,bastion,monitoring,cosigner}/ # google
└── envs/
└── testnet-1/ # instantiates both providers for one network
├── topology.tf # the cross-cloud decisions, as locals
├── inventory.tpl.yml, ssh_config.tpl # -> ansible/inventories/testnet-1/{hosts.yml,ssh_config}
└── backend.tf # partial gcs config — bucket NOT chosen (open, P4)
# devnet-1 has NO env here: Contabo, ordered by hand
# (2026-09-30); the Hetzner env was removed (git history)
ansible/
├── roles/
│ ├── {node,cosmovisor,horcrux,monitoring,firewall}/ # chain nodes (monitoring = node_exporter, every host)
│ ├── bastion/, wireguard/ # jump hosts + the site-to-site tunnel
│ └── monitoring_server/ # prometheus, alertmanager, grafana, tenderduty
├── inventories/testnet-1/ # hosts.yml + ssh_config generated by OpenTofu
├── inventories/devnet-1/ # hosts.yml + ssh_config HAND-WRITTEN from the .example files
├── tests/ # render_test.yml (testnet-1 fixture) + devnet_render_test.yml
│ # (fixture = the devnet-1 example): template regressions, no hosts
├── bootstrap.yml # devnet-1 only: `deploy` user on a fresh hand-ordered VPS
├── site.yml # full fleet config, idempotent; bastions first
├── upgrade.yml # stages a checksummed binary via cosmovisor (chain_nodes)
└── requirements.yml # ansible-galaxy collections (community.general, ansible.posix)
monitoring/
├── prometheus/alerts.yml # deployed verbatim by roles/monitoring_server
└── grafana/ # empty — see grafana/README.md
runbooks/ # on-call, emergency-halt, coordinated-upgrade,
# validator-key-rotation, validator-admission,
# incident-comms — none rehearsed yet
CODEOWNERS
runbooks/ is written (2026-09-20) but every file says Rehearsed: never
at the top. The first halt drill (ENGINEERING.md §13, §15 Phase 5) is
where they get corrected — edit them in the same PR as the drill write-up.
Scaffolding only. No tofu apply or ansible-playbook run has
happened against real infrastructure — there's no Hetzner project, no GCP
project, no Contabo server, no release binary to install, and no key
ceremony yet. Everything below is what exists in code, not what's running.
Verified so far (no cloud credentials needed for any of this): tofu fmt -check, tofu validate on every module and on envs/testnet-1 (OpenTofu —
a drop-in, terraform-compatible CLI; terraform itself works identically),
ansible-playbook --syntax-check on site.yml/upgrade.yml/bootstrap.yml,
yamllint, ansible-lint clean except the intentional var-naming
trade-off documented in ansible/.ansible-lint, a credential-free tofu plan of envs/testnet-1 against a scratch local backend for every topology
combination (bastion_ssh_entry × monitoring_cloud × horcrux_mode;
2026-09-21 counts, for the then 10-validator fleet: 58–60 resources
colocated, 63–65 dedicated — the range is the optional operator-SSH firewall
rule and the monitoring host's cloud; not re-run after the 2026-09-29
reduction to 4 validators, which has tofu validate only), the
inventory/ssh_config templates rendered via tofu console, and
ansible/tests/render_test.yml — rendered-template regressions
(ansible-playbook -i tests/inventory/hosts.yml tests/render_test.yml,
controller-only, touches no host). It imports the roles' own task files
(import_role … tasks_from) so the shipped tasks run with real variable
precedence, and it is kept honest by the mutation set in its header —
deliberate breakages that each must fail it (seven as of 2026-09-21, all
verified). Covers:
the cosmovisor current symlink guard, the data-rebuild guard
(roles/node/tasks/data_guard.yml, §2.7), the mount guard on the node
unit, WireGuard source routes, bastion scrape targets, Prometheus's own
port not colliding with CometBFT's prometheus_port (a real bug the test
found), tenderduty's isolation, digest pin and targets, peer wiring (validators ←
same-cloud sentries only, sentries cross-peered), horcrux's per-validator
instances and metrics endpoints. ansible/tests/devnet_render_test.yml
(-i tests/devnet_inventory/hosts.yml, whose hosts.yml and group_vars are
symlinks to the shipped devnet-1 example; rewritten 2026-09-30 for Contabo,
ten mutations in its header) covers the three-server shape: the
firewall plan per host (roles/firewall/tasks/plan.yml — the validator
admits SSH and p2p from server 2's /32 only and metrics from server 3's /32
only, and has no RPC rule at all; server 2 is sentry + public RPC +
operator-only SSH; server 3 opens nothing publicly; a validator sharing a
host is refused), server 2's single node carrying both sentry peering and
public RPC config, nothing binding RPC publicly on servers 1 and 3, the
port guard (Prometheus vs the archive's SDK gRPC on 9090), scrape and
tenderduty targets covering all three hosts once each, the single-disk
state guard, no mount guard where there is no volume, no IPv4 forwarding
on server 2, horcrux_mode: none, and the WireGuard role gated off. The
firewall refactor (rules computed in plan.yml, applied in main.yml) was
checked against testnet-1's fixture: the planned rules for all 17 hosts are
the rules the old tasks applied, one for one. Not verified: an actual
apply (needs a real Hetzner token and GCP project), anything ansible
actually touching a live host — including the WireGuard tunnel, the
Hetzner NAT path, the per-validator horcrux instances and bootstrap.yml
against a real Contabo image — all of which are written against the
providers' documented behaviour, not observed.
Three networks (D18, decided 2026-09-29, Solana-style): devnet-1 for dapp developers (one validator, three hand-ordered Contabo servers — section "devnet-1" below), testnet-1 for validator and upgrade rehearsal (this section), and konstellation-1 (mainnet, 4 validators — no environment here yet).
4 foundation-run validators (ENGINEERING.md D7, re-decided 2026-09-29;
was 10 = 5 Hetzner + 5 GCP), further admissions permissioned via D16 —
runbooks/validator-admission.md. Placement rule: the 4 validators
belong in 4 separate failure domains, so losing one provider or region never
halts the chain — at equal power, 1 of 4 down leaves 75 % (live), 2 of 4
down leaves 50 % (halt: CometBFT needs more than ⅔). Which four domains
is still open (Contabo, more clouds, more regions — STATUS §5a P16), so the
split is two variables, hetzner_validator_count / gcp_validator_count,
defaulting to 2 + 2: v1 fsn1, v2 hel1 on Hetzner; v3, v4 in two GCP zones
of one region. That default puts 2 of 4 — 50 % of voting power — behind
each provider, so losing either provider (or GCP's region) halts the
chain. Accepted for testnet-1 only; every plan prints
validator_placement_warning saying so, in the style of the P21 cosigner
warning. Each cloud has its own private network; they are joined by one
routed site-to-site WireGuard tunnel between the two bastions (see
"Cross-cloud connectivity"):
Hetzner (10.0.1.0/24) |
GCP (10.0.2.0/24) |
|
|---|---|---|
| Validators | 2 (v1 fsn1, v2 hel1), no public IP, .11-.12 (slots to .15) |
2 (v3, v4, zones a/b), no public IP, .16-.17 (slots to .20) |
| Sentries | 2, 1:1 with their validator, public, .21-.22 |
2, 1:1 with their validator, public, .26-.27 |
| Archive | 1, internal only, .41 |
1, internal only, .42 |
| RPC | — | 1, public, .31 |
| Bastion / gateway | 1, public, .5 — SSH entry, NAT egress for the private hosts, tunnel endpoint |
1, public, .5 — SSH entry, tunnel endpoint (egress is Cloud NAT) |
| Monitoring | — | 1, internal only, .50 (monitoring_cloud = "gcp", default) |
| Horcrux cosigners | c1 fsn1, c3 hel1 (.61, .63) |
c2 (.62) — only with horcrux_mode = "dedicated"; default is colocated, so none |
Each validator's sentry lives in the same cloud — the validator↔sentry p2p link stays inside one private network and never crosses the tunnel. Sentries from both clouds peer with each other over the public internet like any other node. Only management traffic (SSH from a single-entry bastion, metrics scrape, horcrux signing in dedicated mode) crosses the tunnel.
Host count with the defaults (horcrux_mode = colocated): 4 validators +
4 sentries + 2 archives + 1 RPC + 2 bastions + 1 monitoring = 14 hosts
(was 26); dedicated adds 3 cosigners (17), each running one horcrux
instance per validator (4). Mainnet also has 4 validators, so
terraform/envs/konstellation-1/ (not yet created — deliberately not in
this change) should start as a copy of this environment with persistent
disks, horcrux_mode = "dedicated" (§18) and a placement that actually
meets the 4-failure-domain rule — the open decision above.
Simplest safe default, chosen 2026-09-20 and flagged for a human to
confirm: one kernel-WireGuard tunnel between the two bastions, with a
cloud route on each side (hcloud_network_route / google_compute_route)
sending the other cloud's /24 through its local bastion. Traffic is
routed, not NAT'd, so every firewall rule sees the real source IP and
"internal only" means fleet_cidrs (both /24s) rather than one subnet.
Why this and not the alternatives:
- Public scrape with mTLS needs a TLS proxy and a certificate on every node, and doesn't help horcrux at all. Rejected.
- Managed interconnect / VPN gateways — Hetzner has none. Rejected.
- WireGuard on every node is the same trust model with 15× the key material. Rejected.
Costs of the choice, stated plainly: the bastions are now routers (kernel
forwarding; sshd is still their only userland listener); the tunnel is a
single point of failure for cross-cloud management traffic
(validator↔sentry p2p never crosses it); and in dedicated-horcrux mode
it is a single point of failure for consensus too (adversarial review,
2026-09-21; STATUS §5a P22): the cosigners reach each other through the
tunnel, so a tunnel outage partitions the far cloud's cosigner from the
other two — and since each cluster's quorum path for the validators in
that cloud runs across the tunnel, that cloud's validators (2 of 4 by
default) stop signing and the chain halts within seconds. CloudUnreachable says so. An
earlier version of this paragraph claimed "still quorate"; it was wrong.
Design note (P22, not built): the cosigners need their own
peer-pinned mesh — WireGuard between the three cosigner hosts over their
own public IPs (or redundant tunnels), pinned to each other's keys, so
that horcrux raft/grpc never depends on the bastions. Until that exists,
dedicated mode is a liveness downgrade, not only a key-safety upgrade;
testnet-1 runs colocated. WireGuard keys are generated on each bastion
by ansible/roles/wireguard and never leave it; only public keys travel,
as Ansible facts within one play, and each side refuses a fact that is not
a 32-byte base64 key before templating it into root's wg0.conf.
Hetzner-specific: hcloud firewalls filter the public interface only,
so hosts with no public IP are not covered by them at all — ansible/roles/ firewall (ufw) is the primary control on Hetzner, not a redundant one.
And a Hetzner server with no public IP has no internet egress unless
something NATs for it; the bastion does (nat_gateway = true, an
hcloud_network_route 0.0.0.0/0, and a cloud-init unit on every private-only
Hetzner host pointing its default route at the network gateway). Without
this the validators could never apt-get or fetch their checksummed
binary — a gap in the original scaffold, closed here.
One deploy SSH key with NOPASSWD sudo on every host was the original
scaffold, cosigners included: one leaked key = three shards = the consensus
key of every validator. Interim (2026-09-21): var.cosigner_ssh_public_key
is a separate key wired only into modules/{hetzner,gcp}/cosigner,
required in dedicated mode and validated to differ from ssh_public_key.
What it does not do — and what STATUS §5a P21 has to decide before
mainnet: the cosigner hosts still get the same deploy user with
unrestricted sudo (cloud-init), the three cosigners still share one
key, and the default cosigner_placement still puts two shards in one
Hetzner account (fsn1 + hel1), so one hcloud API token can rescue-boot
a threshold — the plan prints cosigner_placement_warning for it.
Per-cosigner keys/operators, hardware-backed keys, restricted sudo and a
third provider are P21/P16, not this branch.
One sentry per validator was the original scaffold. Interim
(2026-09-21): ansible/roles/node/tasks/peers.yml wires every validator
to all its cloud's sentries (2 with the default counts) and cross-peers every sentry with
every other on public IPs, with private_peer_ids = the same-cloud
validators; node ids are read after init so nothing is hand-copied. The
cloud firewalls and ufw admit p2p on validators from those sentries' /32s
only — the /24 used to include the internet-facing RPC node. Still P23: the
sentry count (≥ 2 per validator is the target; today the redundancy
comes from sharing the cloud's two) and a DoS on both of a cloud's
sentry IPs still isolates that cloud's validators.
All in terraform/envs/testnet-1/variables.tf, all documented inline:
| Variable | Default | The decision a human still owns |
|---|---|---|
hetzner_validator_count / gcp_validator_count |
2 / 2 |
how the 4 validators (D7, 2026-09-29) are split, i.e. which failure domains they occupy. The founder's rule is 4 separate domains so one provider/region loss never halts; which domains (Contabo, more clouds, more regions) is open (P16). 2 + 2 halts on either provider's loss — validator_placement_warning. Adding a third provider means new modules, not just a count. |
bastion_ssh_entry |
per-cloud |
one bastion reachable from both clouds vs one per cloud. Both bastion hosts exist either way (tunnel endpoints); this only picks which run an internet-facing sshd. per-cloud means a tunnel outage never locks you out of either side, which matters most in an emergency halt. "hetzner"/"gcp" gives a single audited entry point at the cost of that dependency — and is applied in two steps (first per-cloud, so ansible can reach both bastions to build the tunnel, then switch). |
monitoring_cloud |
gcp |
where the one Prometheus/Grafana/Alertmanager/tenderduty box lives. GCP because Cloud NAT keeps its bastion out of the egress path. |
horcrux_mode + cosigner_placement |
colocated (no cosigner hosts) |
testnet-1 keeps the original co-located mode; dedicated provisions 3 cosigner hosts (fsn1, us-central1-b, hel1 — no location or cloud holds two shards) and ansible runs one horcrux instance per validator on each. Mandatory for mainnet (§18). Switching a live validator is a key ceremony (runbooks/validator-key-rotation.md). |
backend.tf bucket |
none | the terraform state bucket. Still a partial gcs config; terraform init -backend-config=backend.hcl with a bucket a human creates. Not parametrised further because there is nothing to default to. |
GCP validators use Local SSD, and that choice needs to be read, not just
trusted: terraform/modules/gcp/validator's local_ssd_count variable has
the full explanation, but the short version — GCP Local SSD doesn't survive
a stop or a host error. Scheduled maintenance is fine: per GCP's current
live-migration docs (checked 2026-09-21) VMs with Local SSD live-migrate
with the disk, so the module sets on_host_maintenance = MIGRATE (the
original scaffold had TERMINATE, which would have discarded the disk on
every maintenance event — flagged for a human to re-confirm against the
docs of the day before the first apply). For the host-error path, where the
VM auto-restarts with empty Local SSDs, ansible/roles/cosmovisor's unit
carries RequiresMountsFor + ConditionPathIsMountPoint on the data
directory: the node stays down (a NodeDown page) rather than starting
with its key and a priv_validator_state.json at height 0 — the
double-sign scenario ENGINEERING.md §2.7 calls unrecoverable (5% slash,
no undo). Local SSD was chosen for testnet-1 to match Hetzner's local-NVMe
performance profile on the other half of the fleet. ansible/roles/node
stripes the Local SSDs into one RAID0 array (a single 375 GiB disk doesn't
meet the multi-TB spec either way). Re-decide this before any mainnet
validator runs on GCP, probably in favor of pd-ssd despite its worse
commit latency; with dedicated Horcrux cosigners the key is no longer on
that disk, which is most of the risk.
The dapp-developer network: chain-id devnet-1, EIP-155 56672, one
foundation-run validator, and it runs the mainnet binary version
(ansible/inventories/devnet-1/group_vars/all.yml pins the same release as
konstellation-1, not testnet-1's). Contabo, devnet-1 only (testnet-1 and
mainnet placement stays open under STATUS P16). The three VPSes are ordered
by hand in Contabo's panel; there is no OpenTofu for devnet-1 (the
Hetzner env that used to be terraform/envs/devnet-1 was removed on
2026-09-30; git history keeps it). Ansible does everything from the OS up,
from a hand-written inventory: ansible/inventories/devnet-1/hosts.yml.example
and ssh_config.example.
| Server | Inventory host | Groups | Listens publicly | Why it exists |
|---|---|---|---|---|
| 1 | devnet-1-contabo-validator-v1 |
validators | nothing: SSH and p2p from server 2's /32, metrics from server 3's /32; CometBFT RPC on 127.0.0.1, REST and JSON-RPC off | the only block producer; plain priv_validator_key.json, horcrux_mode: none |
| 2 | devnet-1-contabo-sentry-rpc-1 |
sentries, rpc, bastions | 26656, 26657, 1317, 8545-8546; SSH from devnet_operator_ssh_cidrs only |
the sentry the validator peers with (§9.2), the public RPC dapps, wallets and the faucet use, and the one SSH entry point (hardened sshd, fail2ban) — no NAT, no forwarding |
| 3 | devnet-1-contabo-archive-mon-1 |
archive, monitoring | nothing yet: SSH from server 2 only | pruning = nothing + tracing (the explorer's backend), Prometheus/Alertmanager/Grafana/tenderduty; the explorer and faucet containers will land here later |
Each VPS has one public IPv4 and one disk. There is no private network (a
paid add-on we are not using), no cloud firewall and no separate volume:
ansible/roles/firewall (ufw) is the only network control. The
inventory sets private_network: false, which makes the firewall role
insist that every internal source (fleet_cidrs, metrics_scrape_cidrs,
bastion_private_ip, monitoring_host_private_ip — names kept from the
private-network fleets, holding public addresses here) is an exact /32, and
refuse NAT/tunnel settings on the bastion. Nothing the archive serves binds
to its public IP (loopback only), so a firewall mistake cannot expose the
debug JSON-RPC.
Design choices, and why:
- SSH: ProxyJump through server 2, not direct SSH to all three. Servers
1 and 3 admit SSH from server 2's /32 only, so the internet sees exactly
one sshd, and an operator IP change is one edit on server 2, never on the
validator. The cost is that server 2 down = no SSH to 1 and 3: the
break-glass path is Contabo's VNC console with the root password (which
bootstrap.ymlleaves working for the console only). - Several roles on one host. A host is listed in every group whose role
it holds;
roles/firewall/tasks/plan.ymlapplies the union of those roles' rules (on testnet-1, one group per host, it is the same rule set as before, checked host by host).node_roleis pinned per host inhosts.yml, because with several groups the group_vars value is just whichever file sorts last — and without that, server 2 would have lost the operator SSH rule in the bastions play. Server 2 runs one konstellationd: sentry peering (private_peer_ids= the validator) fromsentries.yml, public listeners and the mempool floor fromrpc.yml. A validator sharing its host with any other role is refused. - Server 3 port collision. The archive's konstellationd binds the SDK
gRPC on
localhost:9090, Prometheus's default web port, so devnet-1 movesprometheus_web_portto 9095;roles/monitoring_server/tasks/port_guard.ymlrefuses any overlap between monitoring listeners and a co-located node's ports. Alertmanager's gossip port binds loopback there. - Scrape and tenderduty targets list each konstellationd once (server 2 is in two chain groups) on its public IP; the validator's and server 2's metrics admit server 3's /32 only.
- The archive peers with server 2 (
peer_non_validators_with_sentries): there is no seed node, and on testnet-1 the archive relies onseeds.
Key handling — why no Horcrux, and what §2.7 still demands. Horcrux's
two benefits are (a) no single host holds the whole key and (b) a validator
host can be replaced without moving the key. For a single validator on a
valueless dev network neither buys much, while 3 extra hosts (or 3
colocated processes on the one host, which gives neither benefit) would
add a signing-cluster outage mode to the network dapp developers most need
up. So the validator signs with a plain priv_validator_key.json
(horcrux_enabled: false; roles/node refuses true when
horcrux_mode: none); the founder holds its offline backup (STATUS P29).
ENGINEERING.md §2.7 applies unchanged: the key and its
priv_validator_state.json live on exactly one running host, ever — never
a "warm standby", never a second node started "to get devnet back up", never
a Contabo snapshot restored over a running chain (it rolls the state file
back).
The disk guards on one disk (single_disk: true). The mount guard
(roles/cosmovisor RequiresMountsFor) and the data-rebuild guard
(roles/node/tasks/data_guard.yml) exist for a data volume lost while the
key survives on another disk. On a single disk key, state and chain data go
together — a lost disk is downtime, never a node signing from height 0 — so
neither applies, and the unit says why instead of carrying a guard with
nothing to mount. What can still happen on one disk is the same failure by
hand: the data directory wiped or replaced under the key. So
roles/node/tasks/state_guard.yml stops site.yml on the validator whenever
the key is present and its state file is not (override -e allow_data_rebuild=true only after runbooks/validator-key-rotation.md
§B). single_disk and a data volume are mutually exclusive (asserted).
Liveness is not a property of devnet-1. One validator holds 100 % of
voting power: server 1 or Contabo going down halts devnet-1. Server 2 down
does not stop blocks (the validator needs no peers to commit), but cuts
the public RPC, the archive and SSH to the other two until it is back.
runbooks/validator-key-rotation.md §B's "state = chain height + a margin"
failover deadlocks a one-validator chain; its devnet-1 paragraph says what
to do instead.
Contabo plans (recommendation, confirm names/specs in the panel of the day; all three in one region, Debian 12):
| Server | Plan | Why |
|---|---|---|
| 1 validator | Cloud VPS 30 (8 vCPU, 24 GB, NVMe) | pruned state stays small on a quiet devnet; NVMe for commit latency (§9.2); headroom for the EVM. Cloud VPS 20 works while traffic is low |
| 2 sentry + RPC | Cloud VPS 30 (8 vCPU, 24 GB, NVMe) | the public JSON-RPC (eth_getLogs, eth_call) is the CPU/RAM hot spot; pruned |
| 3 archive + monitoring (+ explorer, faucet) | Cloud VPS 40 (12 vCPU, 48 GB) with the largest disk offered | pruning = nothing grows forever; Blockscout's Postgres and indexer want ~16 GB; Prometheus keeps 90 days. Disk is the limit — watch DiskHeadroomLow |
Prerequisites: the release pins in group_vars/all.yml
(konstellationd_version/_sha256), genesis_url/genesis_sha256 from
networks/devnet-1, and every other empty pin the roles fail closed on
(cosmovisor_sha256, node_exporter_sha256, prometheus_sha256,
alertmanager_sha256, grafana_apt_key_fingerprint,
tenderduty_image_digest) — grep -rn '""' ansible/inventories/devnet-1/group_vars.
Commands from the infra repo root unless noted:
# 1. One key for `deploy` (and, simplest, for root at order time too)
ssh-keygen -t ed25519 -f ~/.ssh/konstellation_devnet_deploy -C devnet-1-deploy
# 2. Contabo panel: order 3 VPSes (plans above), image Debian 12, same region,
# and add ~/.ssh/konstellation_devnet_deploy.pub as the root SSH key.
# (Password-only root works too: add --ask-pass in step 5, needs sshpass.)
# Keep the root passwords: after step 5 they only work on the VNC console.
# 3. Inventory: copy, then fill in the 4 FILL IN values in hosts.yml and the
# 3 IPs in ssh_config (the same addresses)
cd ansible
cp inventories/devnet-1/hosts.yml.example inventories/devnet-1/hosts.yml
cp inventories/devnet-1/ssh_config.example inventories/devnet-1/ssh_config
$EDITOR inventories/devnet-1/hosts.yml inventories/devnet-1/ssh_config
# 4. Collections
ansible-galaxy collection install -r requirements.yml
# 5. First contact, as root, directly to each IP: creates `deploy` with the
# key and NOPASSWD sudo, proves deploy can become root, THEN turns off
# root and password SSH logins
ansible-playbook -i inventories/devnet-1/hosts.yml bootstrap.yml \
--private-key ~/.ssh/konstellation_devnet_deploy \
-e deploy_ssh_public_key_file=~/.ssh/konstellation_devnet_deploy.pub
# 6. The jump path works (server 2 direct, 1 and 3 through it)
ssh -F inventories/devnet-1/ssh_config devnet-1-contabo-validator-v1 true
ssh -F inventories/devnet-1/ssh_config devnet-1-contabo-archive-mon-1 true
# 7. Server 2 first (SSH entry, firewall), then everything. Every run
# needs -i: ansible.cfg's default inventory is testnet-1's.
ansible-playbook -i inventories/devnet-1/hosts.yml site.yml --limit bastions
ansible-playbook -i inventories/devnet-1/hosts.yml site.yml
# (a second full run once every node is initialised is harmless and makes
# sure all node ids were seen when the peer lists were computed)
# 8. Install the GENESIS validator key over the one `init` generated.
# Stop first; keep init's priv_validator_state.json (height 0, pre-genesis).
S=inventories/devnet-1/ssh_config; V=devnet-1-contabo-validator-v1
ssh -F $S $V sudo systemctl stop cosmovisor
scp -F $S /path/to/devnet-1/priv_validator_key.json $V:/tmp/pvk.json
ssh -F $S $V 'sudo install -o konstellation -g konstellation -m 0600 /tmp/pvk.json \
/home/konstellation/.konstellationd/config/priv_validator_key.json && shred -u /tmp/pvk.json \
&& sudo systemctl start cosmovisor'
# 9. Check from OUTSIDE: server 2 answers, the validator does not
curl -s http://<server2>:26657/status | jq .result.sync_info.latest_block_height
curl -s -X POST -H 'content-type: application/json' \
--data '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}' http://<server2>:8545 # "0xdd60" = 56672
nc -zv -w 3 <server1> 26657; nc -zv -w 3 <server1> 22 # both must time outDashboards: ssh -F inventories/devnet-1/ssh_config -L 3000:localhost:3000 -L 9093:localhost:9093 -L 8888:localhost:8888 devnet-1-contabo-archive-mon-1,
then Grafana :3000, Alertmanager :9093, tenderduty :8888. Fill the
validator's valoper in group_vars/monitoring.yml once the gentx exists.
hosts.yml and ssh_config are not gitignored (like testnet-1's generated
ones); committing them to this private repo lets every operator run the
same inventory.
Bastion host.terraform/modules/{hetzner,gcp}/bastion, wired inenvs/testnet-1; the oldvar.bastion_ipv4input is gone — every host's SSH source is now its own cloud's bastion private IP, from the module output, in both the cloud firewalls and ufw.ansible/roles/bastionhardens sshd (key-only,AllowUsers deploy, no agent forwarding, fail2ban with the operator CIDRs inignoreip). Operators SSH in fromvar.operator_ssh_cidrsonly (validated non-empty and never0.0.0.0/0);ansible/inventories/testnet-1/ssh_config(generated) ProxyJumps every other host.Monitoring host.terraform/modules/{hetzner,gcp}/monitoring+ansible/roles/monitoring_server: Prometheus (scrape config templated from the inventory,monitoring/prometheus/alerts.ymldeployed verbatim andpromtool-checked), Alertmanager with a placeholderon-callroute (nothing paged untilalert_*vars are set from a vault — seerunbooks/on-call.md), Grafana (loopback only, provisioned datasource), and tenderduty for validator paging (§9.2,§17). Everything is reached with an SSH port-forward through the bastion; nothing is public.Dedicated horcrux cosigners.terraform/modules/{hetzner,gcp}/cosignerbehindhorcrux_mode = "dedicated";ansible/roles/horcruxbuilds one instance per validator per cosigner host from the inventory, ports2222+i, cosigner list andshard_ids templated, units installed disabled.ansible/roles/nodesetspriv_validator_laddronly when a validator'shorcrux_enabledis flipped after its ceremony. Co-located mode is unchanged and remains the testnet-1 default. No key material anywhere in this repo, by construction.Runbooks.All four fromENGINEERING.md §6.4pluson-call.md, grounded in §2.7, §13/D14 (x/circuitCLI and its protected types), cosmovisor,networks/templates/upgrade.md. None rehearsed.from the org template, withCODEOWNERSterraform/envs/konstellation-1/andrunbooks/as separate lines.
-
Public RPC node exposes CometBFT RPC 26657 and REST 1317 to the internet (
group_vars/rpc.yml,rpc_publicfirewall) — unauthenticated, unmetered, a DoS surface. Acceptable for testnet-1 behind theminimum-gas-pricesfloor; put a rate-limiting reverse proxy in front (or move RPC to the app tier,ENGINEERING.md §9.1) before mainnet. -
Terraform state backend (
terraform/envs/testnet-1/backend.tf) is a partialgcsconfig with no bucket chosen. Human decision. -
bastion_ssh_entry/monitoring_cloud/horcrux_modehave defaults, not decisions — table above. Human decision, before the firstapplyfor the first two, before real validator keys for the third. -
Version pins are empty.
konstellationd_version,konstellationd_sha256,cosmovisor_sha256,horcrux_sha256,node_exporter_sha256, and nowprometheus_sha256,alertmanager_sha256are blank on purpose —konstellationhasn't cut a release build yet. Every download task asserts its checksum var is non-empty and fails closed. -
tenderduty is unmaintained.
github.com/blockpane/tenderdutywas archived on 2025-01-02; it ships as a container image only (no binaries), soroles/monitoring_serverruns it under Debian'sdocker.iowith alatesttag until a digest is chosen (tenderduty_image_digest,tenderduty_require_digest). Works today, will not get fixes. Evaluate a maintained fork before mainnet. -
On-call rota (
§17): mechanics inrunbooks/on-call.md, schedule TBD until validators exist. The alert routes stay placeholders until then. -
GCP validator disk durability — see the callout above. Not a gap so much as a decision deliberately deferred past testnet-1; with dedicated cosigners the key is no longer on that disk, which is most of the risk.
-
Metric names in
alerts.ymlare CometBFT's documented convention, not yet confirmed against a live/metricsendpoint (see the caveat comment in that file).promtool check rulesvalidates syntax, not names. -
Grafana dashboards — none (
monitoring/grafana/README.md). -
Not verified against real providers: Hetzner's DHCP/route behaviour for private-only hosts (the cloud-init default-route unit and the WireGuard
ip route replacelines are written to be harmless if DHCP already pushed the route — verify on first apply), Hetzner's private network handing out no resolvers (cloud-init writes185.12.64.1/2), GCPcan_ip_forward+ custom-route return paths through the tunnel, GCP live migration preserving Local SSD data on N2, and the exact Grafana apt package version. -
Public RPC node
minimum-gas-prices(group_vars/rpc.yml): a node-local 1 gwei-equivalent floor so the one internet-facing mempool is not a free spam target. Not a protocol parameter (TOKENOMICS.md §3 keeps that at 0), but it does set what users of the public RPC pay at minimum — flagged for the TOKENOMICS owner to confirm or lower. -
Explorer reachability (found by the explorer's live run, 2026-09-20): archive nodes now bind JSON-RPC/WS to their private address instead of init's
127.0.0.1(group_vars/archive.ymljsonrpc_bind_address), andvar.explorer_cidrs/archive_rpc_allowed_cidrsgate who may reach them — both empty/fleet-wide until the explorer host exists and its route into a private network (a bastion WireGuard peer, or a host inside one cloud) is decided. Open: that route. -
devnet-1 (2026-09-30, Contabo): the explorer and faucet containers on server 3 are not built here yet — they need the archive's loopback RPC (host networking, or bind to the docker bridge address) and a reverse proxy for their public ports, and ports docker publishes bypass ufw, the only firewall there. The lost-state-file recovery for the one validator is open (STATUS P29). Server 2 is a single point of failure for SSH to servers 1 and 3 (break-glass: Contabo's VNC console). The validator↔sentry p2p link and the metrics scrape cross the public internet (unencrypted metrics; p2p is CometBFT's authenticated encryption).
-
IPv6 is only opened on the Hetzner sentry firewall, not GCP's (would need a dual-stack VPC there) — a minor asymmetry, not a security gap.
cd terraform/envs/testnet-1
cp terraform.tfvars.example terraform.tfvars # fill in, do not commit
gcloud auth application-default login # or set GOOGLE_APPLICATION_CREDENTIALS
terraform init -backend-config=backend.hcl # backend.hcl gitignored, not yet written
terraform plan
terraform apply
# writes ansible/inventories/testnet-1/hosts.yml and ssh_config from outputs
cd ../../../ansible
ansible-galaxy collection install -r requirements.yml
ansible-playbook site.yml --limit bastions # first: everything else is reached through them,
# and Hetzner's private hosts have no egress until NAT is up
ansible-playbook site.yml --limit validators --check # dry run one group first
ansible-playbook site.ymldevnet-1 has no OpenTofu step at all — see "devnet-1: deploying" above.
Humans: ssh -F ansible/inventories/testnet-1/ssh_config testnet-1-gcp-validator-v3.
Dashboards: ssh -F … -L 3000:localhost:3000 -L 9093:localhost:9093 -L 8888:localhost:8888 testnet-1-gcp-monitoring-1
then Grafana on :3000, Alertmanager :9093, tenderduty :8888.
Fleet upgrades: ansible-playbook -i inventories/testnet-1/hosts.yml upgrade.yml -e upgrade_name=... -e binary_sha256=... -e binary_url=... (ENGINEERING.md §16, runbooks/coordinated-upgrade.md). One invocation covers both clouds — upgrade.yml targets chain_nodes.