Describe the enhancement
checkin_limit.max (the concurrent long-poll connection cap that, when exceeded, returns HTTP 429 MaxLimit) is currently coupled to cache sizing through the shared tier system in internal/pkg/config/. Both are selected by the same loadLimits() call:
// internal/pkg/config/config.go
agentLimits := loadLimits(log, fleetInput.Server.Limits.MaxAgents)
fleetInput.Cache.LoadLimits(agentLimits) // cache: num_counters, max_cost
fleetInput.Server.Limits.LoadLimits(agentLimits) // server: checkin_limit.max, etc.
When max_agents is unset (the default), loadLimits falls through to memEnvLimits, which selects the tier by available container RAM. PRs #7568 and #7573 correctly changed memMB() to read container RAM (via GOMEMLIMIT / cgroup) rather than host RAM — this was the right fix for OOMKills caused by oversized ristretto caches. However, because the same tier drives both cache sizing and checkin_limit.max, Fleet Server instances running in small containers on large hosts now select a lower checkin limit tier than they did before those PRs.
Concrete example: a 4 GB container previously read the host's 32+ GB RAM and selected the max tier (checkin_limit.max = 80,000). It now correctly reads 4 GB and selects lte10000 (checkin_limit.max = 10,000). The cache sizing change is correct. The checkin limit change is an unintended regression for deployments that spread agents across multiple small-container nodes.
The coupling is not necessary
Cache sizing and checkin capacity have different natural inputs:
- Cache (
max_cost, num_counters): should be bounded by available container memory — reading container RAM is correct.
checkin_limit.max: is a concurrent-connection semaphore (not a pre-allocation). Its natural input is expected agent count, not RAM. No memory is reserved for unused semaphore slots.
Memory analysis confirms bumping checkin_limit.max is safe:
semaphore.NewWeighted(N) allocates a fixed-size struct regardless of N; slots are never pre-allocated- Actual per-agent RAM (goroutine stacks ~2 KB,
checkin.Bulk.pending entries ~250 B, policy/action subscription nodes ~450 B) scales with actual concurrent connections, not the tier ceiling
- The ristretto
max_cost tier values for lte0500 through lte10000 are identical (all 50 MiB), meaning cache sizing already doesn't benefit from the coupled tier selection in the lower half of the range
Proposed solution
Decouple the RAM-based path from the checkin limit selection. Two options worth evaluating:
-
Split the tier lookup: use containerMemoryMB() only for cache-related fields and a separate agent-count signal for checkin_limit.max. When max_agents is unset, default checkin_limit.max to a high/unlimited value (as defaultCheckinMax = 0 already does) and let max_agents remain the explicit knob.
-
Keep a single tier but source it from agent count, not RAM: always use max_agents (requiring operators to set it explicitly) and remove the RAM-based auto-selection of the server limits. Cache sizing would use a separate, RAM-only lookup.
Either way, max_agents should be clearly documented as the primary knob for checkin_limit.max, with the RAM-based path as a conservative fallback for cache sizing only.
Related
Describe the enhancement
checkin_limit.max(the concurrent long-poll connection cap that, when exceeded, returns HTTP 429MaxLimit) is currently coupled to cache sizing through the shared tier system ininternal/pkg/config/. Both are selected by the sameloadLimits()call:When
max_agentsis unset (the default),loadLimitsfalls through tomemEnvLimits, which selects the tier by available container RAM. PRs #7568 and #7573 correctly changedmemMB()to read container RAM (viaGOMEMLIMIT/ cgroup) rather than host RAM — this was the right fix for OOMKills caused by oversized ristretto caches. However, because the same tier drives both cache sizing andcheckin_limit.max, Fleet Server instances running in small containers on large hosts now select a lower checkin limit tier than they did before those PRs.Concrete example: a 4 GB container previously read the host's 32+ GB RAM and selected the
maxtier (checkin_limit.max = 80,000). It now correctly reads 4 GB and selectslte10000(checkin_limit.max = 10,000). The cache sizing change is correct. The checkin limit change is an unintended regression for deployments that spread agents across multiple small-container nodes.The coupling is not necessary
Cache sizing and checkin capacity have different natural inputs:
max_cost,num_counters): should be bounded by available container memory — reading container RAM is correct.checkin_limit.max: is a concurrent-connection semaphore (not a pre-allocation). Its natural input is expected agent count, not RAM. No memory is reserved for unused semaphore slots.Memory analysis confirms bumping
checkin_limit.maxis safe:semaphore.NewWeighted(N)allocates a fixed-size struct regardless ofN; slots are never pre-allocatedcheckin.Bulk.pendingentries ~250 B, policy/action subscription nodes ~450 B) scales with actual concurrent connections, not the tier ceilingmax_costtier values forlte0500throughlte10000are identical (all 50 MiB), meaning cache sizing already doesn't benefit from the coupled tier selection in the lower half of the rangeProposed solution
Decouple the RAM-based path from the checkin limit selection. Two options worth evaluating:
Split the tier lookup: use
containerMemoryMB()only for cache-related fields and a separate agent-count signal forcheckin_limit.max. Whenmax_agentsis unset, defaultcheckin_limit.maxto a high/unlimited value (asdefaultCheckinMax = 0already does) and letmax_agentsremain the explicit knob.Keep a single tier but source it from agent count, not RAM: always use
max_agents(requiring operators to set it explicitly) and remove the RAM-based auto-selection of the server limits. Cache sizing would use a separate, RAM-only lookup.Either way,
max_agentsshould be clearly documented as the primary knob forcheckin_limit.max, with the RAM-based path as a conservative fallback for cache sizing only.Related
containerMemoryMB()(cache OOMKill fix)containerMemoryMB()