Long-running learner use cases

[silentred]

TL;DR

Core argument: An etcd learner should not be treated only as a short-lived
pre-promotion state. It is also a long-running, non-voting, read-capable
replica. etcd already lets learners serve Status, serializable Range reads,
and Snapshot streams — but the Defragment RPC is still rejected, so a
long-lived learner's backend can grow unbounded with no supported way to
reclaim space.

Why learner defrag matters

• Long-running learner use cases already enabled by current behavior: remote
  read-only service, hot-standby capacity, backup target, operational
  isolation, and topology flexibility.
• All of these depend on the same local bbolt maintenance primitives voters
  have. Compaction changes logical revisions; defragment is what reclaims
  physical backend space.

Industry precedent (non-voting replicas as durable infrastructure, not
transient state):

• TiDB TiFlash — Raft Learner role as the steady-state mechanism for
  columnar OLAP replicas.
• OceanBase read-only replicas — non-Paxos-voting replicas as a
  read-scaling primitive.
• YugabyteDB read replicas — observer nodes, async replication, independent
  replication factors.
• CockroachDB follower reads — local-replica reads for locality and
  availability.

Proposed implementation (minimal, low-risk):

• Keep learners rejecting writes, linearizable reads, watch, lease, auth
  mutation, and membership changes.
• Allow only local maintenance that does not change the replicated keyspace and
  does not require quorum, starting with Defragment.
• Change scope: add DefragmentRequest to isRPCSupportedForLearner in
  server/etcdserver/api/v3rpc/util.go, plus a small test update. No new
  server logic or replication coordination.
• CLI should remain explicit enough that operators can deliberately include or
  directly target learner endpoints for maintenance.

Bottom line: Opening Defragment for learners is low-hanging fruit. It
preserves the non-voting nature of learners while making the long-running
learner use cases etcd already supports operationally viable.

Background

This note summarizes why an etcd learner should not be treated only as a
short-lived pre-promotion state. It was written in the context of
etcd issue #21740, where a
learner member can accumulate backend fragmentation but currently rejects the
online Defragment RPC.

Current learner behavior

The current RPC allowlist already gives a learner useful operational roles:

• A learner can serve endpoint status. StatusRequest is explicitly accepted
  for learners, and the status response reports fields such as raft index,
  applied index, backend size, backend size in use, and IsLearner.
• A learner can serve serializable range reads. RangeRequest is accepted when
  Serializable is true, while linearizable reads and mutating KV operations
  are rejected.
• A learner can serve the streaming Snapshot RPC. The stream interceptor
  rejects learner stream RPCs except the snapshot method, and etcdctl snapshot save --endpoints=<learner> can therefore use a learner as the snapshot
  source.
• A learner currently cannot serve the unary Defragment RPC, because
  DefragmentRequest is not in the learner unary allowlist. The maintenance
  server implementation itself is local backend maintenance, but the learner
  interceptor rejects it before it reaches the handler.

Code references:

• server/etcdserver/api/v3rpc/util.go: isRPCSupportedForLearner allows
  StatusRequest and serializable RangeRequest.
• server/etcdserver/api/v3rpc/interceptor.go: unary learner requests outside
  that allowlist return ErrGRPCNotSupportedForLearner; stream learner
  requests are rejected except for Snapshot.
• server/etcdserver/api/v3rpc/maintenance.go: Defragment performs local
  backend defragmentation, and Snapshot streams a backend snapshot.
• tests/integration/clientv3/kv_test.go: TestKVForLearner locks the
  current contract that a learner accepts serializable reads and rejects
  linearizable reads, put, delete, and txn.

Why long-running learners are valuable

It is reasonable to view learner membership primarily as a safe path toward
promotion, especially because that was the original operational motivation for
adding learners. At the same time, the behavior etcd already exposes suggests
there may be room for a few carefully scoped, longer-running learner use cases.
A learner is also a replicated, non-voting, read-capable member. That shape is
useful when an operator wants an up-to-date copy of the keyspace without adding
quorum cost or write-path risk.

Long-running learners can support several practical deployment patterns:

• Remote read-only service. A learner in another region can provide
  serializable reads close to local consumers. This is not a replacement for
  linearizable reads, but it can serve read-heavy workflows that can tolerate
  serializable semantics.
• Hot standby capacity. A learner kept close to the raft log can be
  promoted when the cluster needs to restore voter count. This is materially
  faster and less risky than adding a cold member after capacity is already
  degraded.
• Backup execution. A learner can be the target for snapshot save, moving
  snapshot read I/O and network egress away from voting members.
• Operational isolation. Read-only scans, monitoring, inventory jobs, and
  backup jobs can be routed to a non-voting member so voters remain focused on
  consensus and write service.
• Topology flexibility. A learner can be placed where a voter would be too
  expensive for quorum latency, such as a distant region, a constrained
  disaster-recovery site, or a network segment dedicated to backup tooling.

These patterns are only sustainable if the learner has the same local storage
maintenance primitives as other bbolt-backed members. Compaction changes the
logical revision history, but defragmentation reclaims physical backend space.
If learners can be long-lived read and snapshot sources, they also need a
supported defrag path. Otherwise a learner that is intentionally retained for
read, standby, or backup use can grow unbounded even though the equivalent voter
maintenance is available.

Similar designs in other storage systems

Other storage systems describe non-voting or learner-like replicas as durable
read-serving infrastructure, not only as temporary membership states.

TiDB TiFlash

TiFlash is described as a columnar storage extension that makes TiDB an HTAP
database. TiFlash replicas are asynchronously replicated by the Raft Learner
consensus algorithm. The docs describe this as a way to store columnar replicas
with snapshot isolation, workload isolation, and computing acceleration for
analytical queries.

Important details from the TiFlash documentation:

• TiFlash connects to the TiDB cluster as a Learner role, so data is written to
  TiKV and replicated to TiFlash rather than written directly to TiFlash.
• TiFlash uses asynchronous learner replication so TiKV applications can
  continue normally when TiFlash is down or has high network latency.
• TiFlash reads validate replication progress and MVCC state to provide the
  same snapshot isolation level as TiKV.
• The stated use case is not promotion to a voter. The learner role is the
  steady-state mechanism for an OLAP-serving replica.

Source: https://docs.pingcap.com/tidb/stable/tiflash-overview/

OceanBase read-only replicas

OceanBase and Alibaba Cloud OceanBase describe read-only replicas as replicas
that do not participate in Paxos voting. Instead, they observe and replay logs
from Paxos members, then provide read-only service when the application can
accept the consistency model.

Important details from the OceanBase documentation:

• A read-only replica does not act as a Paxos voting member; it follows logs
  and replays them locally.
• Read-only replicas are presented as a read scaling mechanism that offloads
  read requests and reduces pressure on the primary cluster.
• Alibaba Cloud OceanBase describes scenarios such as large read volume,
  read/write isolation, complex reporting, and OLAP analysis without
  interfering with transaction processing on the primary cluster.
• OceanBase also documents conversion between full-featured replicas and
  read-only replicas, which treats read-only status as an explicit operational
  replica type.

Sources:

• https://en.oceanbase.com/docs/common-oceanbase-database-10000000003683602

YugabyteDB read replicas

YugabyteDB describes read replicas as observer nodes that extend Raft. They do
not participate in writes, receive an asynchronously replicated,
timeline-consistent copy of data, and can be placed in remote data centers for
read-only workloads.

Important details from the YugabyteDB documentation:

• Read replicas are also called observer nodes.
• They do not participate in Raft consensus for writes, so they do not add
  synchronous write latency.
• They are explicitly intended for cases where remote distributed-consensus
  write latency is not tolerable.
• Read replica clusters can have independent replication factors, including
  even replica counts, because they are outside Raft write consensus.

Sources:

• https://docs.yugabyte.com/stable/architecture/docdb-replication/read-replicas/
• https://docs.yugabyte.com/stable/deploy/multi-dc/read-replica-clusters/

CockroachDB follower reads

CockroachDB follower reads are not identical to etcd learners, because they
read from ordinary replicas rather than a dedicated learner member type.
However, the design shows the same broader product pattern: a replicated copy
can intentionally serve stale or bounded-staleness reads to improve locality,
availability, and read scalability while avoiding write-path coordination for
those reads.

Important details from the CockroachDB documentation:

• Follower reads are read-only operations served from a consistent historical
  view.
• Topology guidance describes reading from the closest replica instead of
  routing to a remote leaseholder.
• Bounded staleness reads are framed as improving availability because local
  replicas can continue serving reads even when the SQL gateway cannot reach the
  leaseholder.

Sources:

• https://www.cockroachlabs.com/docs/stable/follower-reads
• https://www.cockroachlabs.com/docs/stable/topology-follower-reads

Implication for learner defragmentation

Learner defrag is not only about helping a soon-to-be-promoted member. It is a
maintenance primitive for a bbolt-backed replica that etcd already permits to
serve status, serializable reads, and snapshots. If etcd supports these
long-running read and backup roles, then the learner needs a safe way to reclaim
backend space after compaction.

A narrow implementation principle follows from the current contract:

• Keep learner writes, linearizable reads, watch, lease, auth mutation, and
  membership-changing operations rejected.
• Allow local maintenance that does not change the replicated keyspace and does
  not require quorum participation, starting with Defragment. Opening up
  Defragment for learners is a low-hanging fruit: the maintenance handler
  already performs purely local backend defragmentation, so the change is
  effectively limited to adding DefragmentRequest to the learner unary
  allowlist in isRPCSupportedForLearner plus a small test update — no new
  server logic or replication coordination is required.
• Keep CLI behavior explicit enough that operators can intentionally include or
  directly target learner endpoints for maintenance.

This preserves the non-voting nature of learners while making the already
supported long-running learner use cases operationally viable.

[serathius]

tldr?

[silentred]

etcd already lets learners serve Status, serializable Range reads, and Snapshot streams. A long-running learner can be useful for remote
read-only service, hot-standby capacity, backup target, operational isolation, and topology flexibility.

Industry precedent: TiDB TiFlash, OceanBase read-only replicas, YugabyteDB read replicas, CockroachDB follower reads.

Opening Defragment for learners is low-hanging fruit. It preserves the non-voting nature of learners while making the long-running learner use cases etcd already supports operationally viable.

[serathius]

For read only service we just use build a cache server utilizing library #19371. This is how K8s scaled it's reads. Reason: Main bottleneck for reads is on deserialization, horizontal scaling will not help with that. Focus on problem, not on particular solution. Industry precedent is bad argument in this case.

[silentred]

Reason: Main bottleneck for reads is on deserialization, horizontal scaling will not help with that

You're right. I don't have a concrete use case for a read-only service in K8s. However, I've outlined two use cases below regarding recovery and workload isolation.

[ahrtr]

Long-running learners can support several practical deployment patterns:

• Remote read-only service. A learner in another region can provide
  serializable reads close to local consumers. This is not a replacement for
  linearizable reads, but it can serve read-heavy workflows that can tolerate
  serializable semantics.
• Hot standby capacity. A learner kept close to the raft log can be
  promoted when the cluster needs to restore voter count. This is materially
  faster and less risky than adding a cold member after capacity is already
  degraded.
• Backup execution. A learner can be the target for snapshot save, moving
  snapshot read I/O and network egress away from voting members.
• Operational isolation. Read-only scans, monitoring, inventory jobs, and
  backup jobs can be routed to a non-voting member so voters remain focused on
  consensus and write service.
• Topology flexibility. A learner can be placed where a voter would be too
  expensive for quorum latency, such as a distant region, a constrained
  disaster-recovery site, or a network segment dedicated to backup tooling.

It's interesting. Some of them seem reasonable, such as Remote read-only service, but are you aware of any real-world usage of these use cases?

Also, learner doesn't support snapshot requests either, so the Backup execution use case may not be valid?

[ahrtr]

Also, learner doesn't support snapshot requests either, so the Backup execution use case may not be valid?

It does support snapshot.

[silentred]

It truely does.

# version [etcd-v3.6.9-linux-amd64]

$./etcdctl --endpoints=http://127.0.0.1:13379 member list -wtable
+------------------+---------+--------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |  NAME  |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+--------+-----------------------------+-----------------------------+------------+
| 6c208b54421817e8 | started | infra2 | http://127.0.0.1:23380 | http://127.0.0.1:23379 |       true |
| af9284128b6f2c0c | started | infra1 | http://127.0.0.1:13380 | http://127.0.0.1:13379 |      false |
+------------------+---------+--------+-----------------------------+-----------------------------+------------+

$./etcdctl --endpoints=http://127.0.0.1:23379 snapshot save learnerdb
{"level":"info","ts":"2026-05-15T17:23:25.718841+0800","caller":"snapshot/v3_snapshot.go:83","msg":"created temporary db file","path":"learnerdb.part"}
{"level":"info","ts":"2026-05-15T17:23:25.719796+0800","logger":"client","caller":"v3@v3.6.9/maintenance.go:236","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2026-05-15T17:23:25.722267+0800","caller":"snapshot/v3_snapshot.go:96","msg":"fetching snapshot","endpoint":"http://127.0.0.1:23379"}
{"level":"info","ts":"2026-05-15T17:23:25.722486+0800","logger":"client","caller":"v3@v3.6.9/maintenance.go:302","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2026-05-15T17:23:25.724077+0800","caller":"snapshot/v3_snapshot.go:111","msg":"fetched snapshot","endpoint":"http://127.0.0.1:23379","size":"20 kB","took":"5.116952ms","etcd-version":"3.6.0"}
{"level":"info","ts":"2026-05-15T17:23:25.724131+0800","caller":"snapshot/v3_snapshot.go:121","msg":"saved","path":"learnerdb"}
Snapshot saved at learnerdb
Server version 3.6.0

[silentred]

Main bottleneck for reads is on deserialization, horizontal scaling will not help with that

It is true for K8S use case.

you aware of any real-world usage of these use cases?

Sharing a concrete production scenario from our side that maps to two of the use cases in the OP.

Environment

• Multiple etcd clusters serving as the backing store for core Kubernetes clusters.
• Per-cluster db size: ~60–100 GB.
• Per-cluster write TPS: ~1k.
• Hard requirements on data durability and availability.

Use case 1 — Hot standby for fast voter recovery

When a voter goes down, replacing it with a fresh member requires the full snapshot transfer + restore path. At our db size this consistently takes ~3–4 minutes before the
new member catches up and the cluster is back to its target voter count. During that window the cluster is operating with reduced fault tolerance, which is exactly when
we least want a long recovery.

If we keep a learner as a warm standby that is already caught up on the raft log, MemberPromote returns the cluster to full voter count in seconds rather than minutes. The
MTTR(Mean Time to Repair) difference is the entire point — the learner doesn't need to be there forever, but it needs to be there long enough to be useful when something fails, which in
practice means "indefinitely, until it's needed." That's the long-running shape.

Use case 2 — Backup isolation via snapshot save against a learner

We take periodic snapshots for disaster recovery. Running snapshot save against a voter at this db size produces a non-trivial amount of read I/O and network egress on a member that is
also serving consensus and client traffic. Routing snapshot save to a learner gives us physical isolation of the backup
workload from the voting quorum, so backup jobs don't compete with the write path or with linearizable reads on voters.

This is not a read-scaling argument. It's about moving heavy local I/O off the voter set, which is something a non-voting replica is uniquely well-suited for.

Happy to share more numbers if useful.

[serathius]

Multiple etcd clusters serving as the backing store for core Kubernetes clusters.

What do you mean by that? Please note, I focus on problem, not a particular solution. You are giving me solutions for a problem, I still not fully understand.

[silentred]

"Multiple etcd clusters serving as the backing store for core Kubernetes clusters." means multiple K8S clusters in deifferent regions, each K8S cluster has multiple etcd clusters partitioned by resources.

Problem 1

In a 3-node etcd cluster, when a voting member goes down, adding a fresh new member takes ~3-4 minutes due to the large db size. The majority of this time is spent transferring and restoring the db. If another node fails during this window, the cluster will lose quorum.

Problem 2

We observed that performing a snapshot save on a voting member causes the db-in-use or db-size to grow. Additionally, it negatively impacts performance, specifically increasing the P90 request latency for Txn and Range operations.

The yellow line is the node saving snapshot.

@fuweid pointed out a long-running learner may hit the db-size limit. That might not be the only issue with long-running learners. Therefore, I don't think it's the right time for the community to commit to a fully functional long-running learner just yet. That said, I will share more details about how we use learners in our environment.

cc @ahrtr

[serathius]

During outage it's expected that for some time you will only have 2 members available in 3 member cluster. Time here 3 minutes seems reasonable for me. Especially with your 70GB of data.

If you want higher availability you can have 5 member clusters. 3 node cluster has failure tolerance of 1, 5 node cluster has failure tolerance of 2.

If sending snapshots is the bottleneck, we can discuss how to solve that.

As for 70GB snapshots, people think etcd doesn't support >8GB of data. This sounds like a success story. Thanks for sharing.

[silentred]

sending snapshots is the bottleneck

I haven't measured it yet, but I suspect the bottleneck might be restoring db.

[serathius]

I haven't measured it yet, but I suspect the bottleneck might be restoring db.

That sounds like a good start. Let's get some benchmarks!