Postgres default of 4Gi is not viable at 4,500 hosts — 531 OOM kills per run, and it makes unrelated components look broken

[shayan1995]

Summary

The default Postgres sizing in helm-prereqs/values.yaml:197-203 is 4Gi memory / 4 CPU, and at 4,500 hosts that is not survivable. The database runs at 99.96% of the limit and the kernel kills it repeatedly, which degrades or breaks the entire ingestion pipeline in ways that look like unrelated defects.

## Resource limits for each postgres instance.
## Production uses 32 CPU / 16Gi — tune down for dev clusters.
resources:
  limits:
    cpu: "4"
    memory: "4Gi"
  requests:
    cpu: "500m"
    memory: "1Gi"

Consumed by helm-prereqs/templates/postgresql.yaml:37-43 into the postgresql.acid.zalan.do CR.

Measured

A 4,500-host (13,500 machine) run on dev6 with the 4Gi default:

	Value
Memory	4,293 MB of a 4,295 MB limit — 99.96%
OOM kills	531
Full crash-recoveries	112, roughly one every 2-3 minutes
Effect of each	every connection dropped, all in-flight transactions rolled back

The same run with the limit raised to 24Gi: peak usage 2.1 GiB, zero OOM kills, and the full fleet ingested — 13,500 machines created and 13,500 reaching ready, reproduced three times at 54-56 machines/min.

The node had 240 GB free. This was purely a configured limit, not a hardware constraint.

Why it matters beyond the number

Three issues were filed as separate product defects during the period this was in effect, and none reproduced once the memory was adequate:

• bug: readiness plateaus at 79% at 4,500 hosts — 2,796 machines stuck in dpuinit/hostinit with healthy components #4750 — readiness plateau at 79%, 2,796 machines stuck (closed as not reproducible)
• bug: nico-api panics on cleanup assertion after a Postgres deadlock, crashing the whole API server on a recoverable condition #4753 — cleanup-path panic following a Postgres deadlock (re-scoped)
• the deadlock itself

When the database is being killed every couple of minutes, aborted transactions and rolled-back state make almost any downstream component look broken. Days were spent tracing individual state machines before the memory pressure was noticed.

Suggested change

1. Raise the default so a default install survives a realistic fleet, or at minimum make the failure obvious rather than silent.
2. Document the relationship to fleet size. The existing comment says "Production uses 32 CPU / 16Gi — tune down for dev clusters", which reads as though 4Gi is a safe dev value. It is fine for small fleets and actively harmful at 4,500 hosts. A short table (hosts to recommended memory) would prevent the same trap.
3. Consider a preflight check in setup.sh that warns when configured Postgres memory looks low for the requested host count.

Operational note that cost a full run

Teardown deletes the postgres namespace, so a resize applied before teardown is silently reverted by the subsequent setup. It must be applied after setup.sh. Anything scripted around this should account for it.

Diagnostic worth adding to the runbook

If ingestion stalls, check this first:

kubectl exec -n postgres <pg-pod> -c postgres -- sh -c \
  'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max; grep oom_kill /sys/fs/cgroup/memory.events'

A non-zero and rising oom_kill count means everything downstream is a symptom.

Related: #3738 (scaling epic), #4750, #4753.

[shayan1995]

Additional finding from run forensics: even at the 8-CPU limit PR #4844 proposes, the Postgres primary is CPU-throttled in 14% of scheduler periods (39,691 of 282,487) under sustained 4,500-host ingestion + steady-state heartbeat load. Memory is comfortable (~8 GiB of 24). Suggest the sizing guidance cover CPU alongside memory — throttling adds latency to every query on a path where 60–70% of exclusive-lock time is already client round trips.