# [Discussion] Survivability of etcd in 2-AZ deployments — seeking maintainers' guidance

[icecoolr1]

Hi etcd maintainers and community,

Before writing any design doc, we would like to ask for your opinion on whether a specific operational pain point is in scope for etcd, and if so, what shape of solution you would be open to.

The problem

etcd's recommended HA topology assumes 3 independent failure domains (AZs), so that a single-AZ outage still leaves a Raft quorum.

In practice, a non-trivial number of production environments only have 2 AZs available (edge sites, regional private clouds, cost- or compliance-constrained deployments). In a pure 2-AZ topology, no distribution of voting members can survive every single-AZ failure — this is a property of majority quorum on two failure domains, not a misconfiguration.

When etcd becomes unavailable, the data plane on top of it (typically Kubernetes) loses its control plane at exactly the moment operators need it most: the mechanisms used to evacuate or failover workloads away from the failed AZ are themselves blocked.

Why existing options don't fully solve it for us

• Third-region witness / arbiter member — requires a third failure domain, which is precisely what these environments don't have. Adds a cross-region network dependency, latency tail, and operational cost.
• --force-new-cluster + snapshot restore — works, but is manual, destructive, poorly observable, and risky under incident pressure.
• Learners — don't vote, so they don't help with quorum.
• Membership change via etcdctl — itself requires quorum, so unavailable exactly when needed.

We fully acknowledge that Raft is etcd's cornerstone and we are not proposing to weaken its safety model in steady state.

What we would like to explore

An opt-in, operator-gated "emergency survivability" path for 2-AZ clusters, roughly:

1. Off by default; no behavior change for anyone who doesn't enable it.
2. When one AZ is confirmed lost, an authenticated operator request can reconfigure the cluster to continue on the surviving AZ (essentially a first-class, auditable version of what --force-new-cluster already does today).
3. A monotonic "emergency epoch" + fencing of the lost members prevents split-brain if the other AZ returns — recovered members rejoin as learners and must be explicitly promoted.
4. Full observability: metrics / events for entering, operating in, and exiting the degraded mode, including an explicit "entries potentially lost" counter.

In short: make the existing "rebuild from survivors" escape hatch explicit, safer, reversible, and automatable, without changing Raft itself.

Questions for the maintainers

We would genuinely like your guidance before investing in a full KEP-style design:

1. Do you consider the 2-AZ survivability problem in scope for etcd, or should it remain entirely in downstream tooling (etcd-operator, cloud operators, etc.)?
2. If in scope, is an operator-gated, epoch-fenced, reversible degradation path (shape sketched above) acceptable in principle, or is there a hard line we would be crossing?
3. Would you prefer an incremental first step — e.g. just standardizing AZ labels on members and exposing the fencing / epoch primitive — before any state-machine changes?
4. Are there prior discussions / issues / design notes we should read first so we don't re-litigate settled ground?

Any answer, including "no, and here's why," is very valuable to us and to the operators running etcd in 2-AZ environments today. Happy to turn this into a proper design doc once the direction is clear.

Thanks for your time.

[icecoolr1]

@ahrtr please cc

[ahrtr]

2. When one AZ is confirmed lost, an authenticated operator request can reconfigure the cluster to continue on the surviving AZ (essentially a first-class, auditable version of what --force-new-cluster already does today).

This isn't planned in etcd v3.7. Also it's what etcd-operator v0.3.0 has planned to do.

[icecoolr1]

Thank you for the information. Also, has the timeline for this requirement been finalized? Can I go ahead and take ownership of implementing it?

[ahrtr]

Also, has the timeline for this requirement been finalized? Can I go ahead and take ownership of implementing it?

Do you mean the etcd-operator v0.3.0? There is no planned timeline. Contributions are welcome.

FYI. we have bi-weekly WG etcd-operator meeting https://github.com/kubernetes/community/tree/main/wg-etcd-operator