After etcd cluster upgraded from v3.4.34 -> v3.5.27, TotalDBSize has been increased and TotalDBSizeInUse remains stable.

[ishan16696]

We recently upgraded our etcd clusters from v3.4.34 -> v3.5.27 and after the upgrade it has been observed that there has been significantly increase in etcd's totalDBSize whereas totalDBSizeInUse remains stable.
The magnitude varies per cluster but the pattern is consistent across all of our etcd clusters.

Etcd v3.5.x has changed the default value of --backend-bbolt-freelist-type from array(persisted to disk) to map(in-memory). and because of this bolt-db is using hashmap to maintain the freelist, I suspect this to be the root cause of this issue.

Questions:

1. Is this expected behaviour after upgrading to v3.5.x ?

note: yes, We can run defragmentation to reclaim the free space, but running it too frequently is undesirable as it causes etcd members to become temporarily unavailable and can lead to trigger unnecessary leader elections.

2. On the claim that "map freelist type has much better performance than array.", Has this been validated via perf testing for clusters running with a quota of 8–10 GiB ? from this blog: https://www.cncf.io/blog/2019/05/09/performance-optimization-of-etcd-in-web-scale-data-scenario/ when this hashmap approach was introduced what I can understand that this hashmap based freelist was introduce for large-database scenarios. It is not clear that the performance benefit carries over to the small databases typical in most Kubernetes deployments.

3. Is there any other feature/changes can cause this issue ?

[ishan16696]

Another observation is that even though the total-databaseSize-inuse is going down but still totalDbSize is going up, may be because the freelist is allocating more pages instead of using fragmented available/free pages.

[dlanov]

Q1 — Is this expected after upgrading to v3.5.x?

Yes, this is expected and well-understood behavior. The v3.5 default change from --backend-bbolt-freelist-type=array to map changes how bbolt tracks free pages internally, and it has a direct effect on how the on-disk file size grows.

With array freelist (v3.4 default):

• The freelist is persisted to disk as a sorted array of free page IDs
• bbolt reuses free pages efficiently on the next write cycle because it can cheaply scan the sorted array
• totalDBSize stays close to totalDBSizeInUse

With map freelist (v3.5 default):

• The freelist is kept in memory only and rebuilt on startup
• bbolt no longer writes the freelist array to disk, which saves write amplification — but it means the allocator is more likely to append new pages rather than reuse existing free pages, especially across process restarts
• This causes totalDBSize to grow independently of totalDBSizeInUse, exactly the divergence you're seeing
  Your own follow-up observation is correct: the allocator is preferring to extend the file rather than reclaim fragmented free pages.

Q2 — Has the map freelist performance advantage been validated for small (8–10 GiB) databases?

The benchmarks that motivated the map default were designed for large, write-heavy databases (the web-scale scenario in the linked CNCF post). For Kubernetes-typical etcd clusters — which are small (usually under 2 GiB, rarely above 8 GiB) and have bursty rather than sustained write patterns — the performance benefit is marginal and the disk growth trade-off is more significant.

For clusters in the 8–10 GiB range you can safely switch back to array freelist:
--backend-bbolt-freelist-type=array

This restores v3.4 behavior: the freelist is persisted, free pages are reused aggressively, and totalDBSize tracks totalDBSizeInUse much more closely. The write amplification cost of persisting the freelist is negligible at your database sizes.

Q3 — Are there other changes that could contribute?

Two v3.5 changes are worth knowing about but are secondary to the freelist change:

• Snapshot catch-up — v3.5 changed how learner/follower snapshot transfers work; if any members recently caught up via snapshot, the file can be transiently larger until the next compaction+defrag cycle
• Increased default quota — if your quota was raised during the upgrade, bbolt pre-allocates more aggressively. Check that --quota-backend-bytes is unchanged

Defrag frequency concern

You noted that frequent defragmentation is undesirable due to leader election risk. With --backend-bbolt-freelist-type=array, you should need defrag far less often since the allocator reuses free pages naturally. A monthly or on-demand defrag triggered when totalDBSize / totalDBSizeInUse > 1.5 is a reasonable policy and avoids the availability concern.

[ishan16696]

• We have override the default of etcd flag: --backend-bbolt-freelist-type from map to array. This has stabilize etcd's totalDBSize as shown in this screenshot.

[shankarmadabhavi]

can i know is there any impact after changing from map to array..which is better one?.Thanks

[SillyZir]

The default backend freelist type resolves to bolt.FreelistMapType: any --backend-bbolt-freelist-type value other than the literal string "array" (including the empty default) is parsed as map. (server/embed/config.go:1373, server/embed/config.go:115, server/embed/config.go:617)

On Linux, etcd always opens bbolt with NoFreelistSync: true, so the freelist is NOT persisted to disk under either freelist type; the array/map setting only selects the in-memory allocator structure passed via bopts.FreelistType. (server/storage/backend/config_linux.go:29, server/storage/backend/config_linux.go:31, server/storage/backend/backend.go:196)

etcd_mvcc_db_total_size_in_bytes reports physical file size (tx.Size()) and etcd_mvcc_db_total_size_in_use_in_bytes reports size minus FreePageN*PageSize, both refreshed on every transaction begin; therefore a growing gap between the two metrics is exactly an accumulation of free pages inside the db file. (server/storage/mvcc/metrics.go:171, server/storage/mvcc/metrics.go:188, server/storage/mvcc/kvstore.go:527)

Compaction deletes old revisions (reducing sizeInUse) but never shrinks the file; only defragmentation rewrites the database into a tmp file and swaps it, which is the sole code path that resets b.size downward. (server/storage/mvcc/kvstore_compaction.go:57, server/storage/mvcc/kvstore_compaction.go:70, server/storage/backend/backend.go:556)

etcd supports automatic defrag at bootstrap when (size - sizeInUse) exceeds BootstrapDefragThresholdMegabytes (maybeDefragBackend), an existing mitigation that avoids live-member unavailability by defragging during restarts. (server/etcdserver/bootstrap.go:307, server/etcdserver/bootstrap.go:310, server/etcdserver/bootstrap.go:312)

A possible explanation — not yet verified: The behavior is 'expected' in the sense that it is benign free-page accumulation reclaimable by defrag, not a leak of live data — sizeInUse stability shows the logical dataset is unchanged. This could be checked by: Defrag one member and confirm db_total_size drops to approximately db_total_size_in_use; verify data integrity via hash checks (etcdctl endpoint hashkv across members).

A possible explanation — not yet verified: The size growth is caused by bbolt's hashmap freelist reusing/merging free page spans less effectively than the array freelist for contiguous multi-page allocations, so under the same workload bbolt grows the file instead of reusing freed pages (Size up, SizeInUse flat). Since NoFreelistSync is true in both versions, the mechanism must be allocation behavior, not freelist persistence. This could be checked by: Run the same write workload on v3.5.27 twice, once with --backend-bbolt-freelist-type=array and once with map (defaults), and compare etcd_mvcc_db_total_size_in_bytes growth; additionally benchmark bbolt v1.3.x freelist.allocate() page-reuse behavior for both types.

This analysis was produced with the assistance of an AI engineering system I created; I reviewed it before posting.

[SillyZir]

@shankarmadabhavi Switching is low-risk: on Linux etcd always opens bbolt with NoFreelistSync: true, so the freelist is never persisted under either setting — --backend-bbolt-freelist-type only selects the in-memory allocator. Changing map → array is a per-member flag change at restart, with no data migration, and it rolls back the same way.

The practical difference between the two:

• array reuses freed contiguous page spans more aggressively, so the db file stays close to the in-use size — that is what stabilized TotalDBSize in ishan16696's measurement above.
• map (the v3.5 default) keeps allocation cheap when the freelist becomes very large or fragmented, at the cost of weaker span reuse — under churny multi-page workloads the file can grow while in-use stays flat. That growth is free pages (reclaimable by defrag), not a leak.

Which is better depends on what you optimize for: for a stable disk footprint on a workload like this thread's, array is the pragmatic choice. One potential performance consideration to keep an eye on: the array allocator scans linearly, so with a very large freelist it could add backend commit latency — worth monitoring after the switch rather than a given.

This analysis was produced with the assistance of an AI engineering system I created; I reviewed it before posting.