Skip to content

fix: fail Deployment tracking fast on ReplicaSet pod-create errors - #404

Merged
ilya-lesikov merged 5 commits into
mainfrom
fix/398-replicaset-pod-create-failures
Oct 29, 2026
Merged

ilya-lesikov merged 5 commits into
mainfrom
fix/398-replicaset-pod-create-failures

Conversation

@ilya-lesikov

@ilya-lesikov ilya-lesikov commented Oct 27, 2026 •

Copy link
Copy Markdown
Member

Summary

The Deployment tracker now reports pod-create failures of the Deployment's new ReplicaSet (the ReplicaFailure/FailedCreate condition, e.g. exceeded quota) instead of waiting for the tracking timeout. Failures the rollout cannot recover from on its own fail the deploy immediately; the rest are charged against the allowed-failures budget.

This is a port of #399 by @Sharvash. When branch 1 was merged into main (#389) and deleted, GitHub closed #399 without merging it. His two commits are cherry-picked unchanged, with his authorship and sign-off. Our commits on top fix the quota classification and a lint error.

Closes #398

What

  • Commits by @Sharvash (from fix: fail Deployment tracking fast on ReplicaSet pod-create errors #399):
    • fix: fail Deployment tracking on ReplicaSet pod-create errors: reads the durable ReplicaFailure condition of the new ReplicaSet rather than transient events, so failures that happened before tracking started are also seen. Only ReplicaSets controlled by the Deployment are considered.
    • fix: handle ReplicaSet failure transitions consistently: waits for the initial ReplicaSet list before classifying, keeps informer events in order (unbuffered, cancellation-aware sends), handles ReplicaSet replacement, selector leave and re-entry, and deletion. Reports a failure once per ReplicaSet incarnation and re-arms it when the condition clears. Ignores a lagging condition once the Deployment is ready.
  • Our commits:
    • fix: keep ReplicaSet quota failures counted while the rollout can release quota: see Quota classification below.
    • chore: fix errorlint in deployment tracker test harness and document FailureMode field.
    • fix: bound quota quantities parsed from a ReplicaSet failure message: the FailedCreate message can be written by anyone who can update the ReplicaSet status, and a quantity with a huge exponent kept the tracker busy for minutes. Only the bounded forms the quota admission plugin prints are parsed now; anything else keeps the failure counted.

Quota classification

The original heuristic reported a quota failure as fatal when the rollout could not scale an old ReplicaSet down at that moment. Several rollouts that do recover were therefore misclassified. One example: replicas: 3, maxSurge: 2, maxUnavailable: 0, a quota of 4 pods. The new ReplicaSet gets 1 of its 2 pods. Once that pod becomes available, an old pod is scaled down, quota is freed and the rollout continues. A quota failure is now fatal only if all of the following hold:

  1. Rollout projection. Assuming every existing pod of the new ReplicaSet becomes available, the Deployment controller still would not scale any old ReplicaSet down: sum(old spec.replicas) + existing new pods <= replicas - maxUnavailable. The same formula as reconcileOldReplicaSets.
  2. Fresh pod list. A quorum list pods confirms the projection: no pod of the rollout is terminating, no old ReplicaSet has surplus pods, and the new ReplicaSet is still missing pods. This check is needed because the status that brings the condition counts only the pods listed before the failed sync created any, and status.terminatingReplicas is feature-gated.
  3. Quota status. get resourcequotas/<name from the message> shows that the quota still has no room for the request. The ReplicaSet controller keeps FailedCreate across its retry backoff, which can last minutes, so the condition can outlive the shortage.
  4. Persistence. All of the above still hold after 10 seconds, re-evaluated from scratch. The checks read different objects at different moments, and quota usage drops only once the quota controller catches up with deleted pods.

If a check cannot be made (the list or get fails, for example because of RBAC, or the message does not parse), the failure stays counted, as on main.

Behaviour changes

  • A Deployment whose new ReplicaSet cannot create pods is reported as failed. Before, tracking waited for the timeout.
    • Fatal, failing the deploy regardless of failures-allowed: an invalid pod spec (... is invalid:), or a quota failure confirmed as described above. A quota deadlock is reported about 10 seconds after it is observed.
    • Counted against the failures budget: everything else, such as webhook or RBAC rejections, recoverable quota failures, and quota failures that cannot be confirmed.
    • A quota failure that clears within the confirmation period is not reported at all.
  • New public API in pkg/tracker/deployment: type FailureMode, FailureModeCounted (the zero value, so existing code that builds a DeploymentStatus keeps the old meaning) and FailureModeFatal, plus the DeploymentStatus.FailureMode field. It is meaningful only when IsFailed is set.
  • dyntracker: a fatal Deployment failure marks the resource Failed, which fails the readiness task immediately.
  • dyntracker fail modes (statestore/readiness_task_state.go): with IgnoreAndContinueDeployProcess, a resource marked Failed does not fail the task. Only FailWholeDeployProcessImmediately and LegacyHopeUntilEndOfDeployProcess fail on it. Before this PR nothing set ResourceStatusFailed, so the change affects only the new fatal failures: they respect werf.io/fail-mode: IgnoreAndContinueDeployProcess.
  • Sticky failures are reported once. After an event-sourced failure, the Deployment tracker no longer re-stamps IsFailed/FailedReason into every later status. Each failure is charged to the budget once, where before it was charged on every subsequent update. The StatefulSet and DaemonSet trackers still re-stamp.
  • utils.FindNewReplicaSet only considers ReplicaSets controlled by the Deployment.
  • RBAC: fast-failing on a quota deadlock needs list pods (already required for tracking) and get resourcequotas in the namespace. The built-in view, edit and admin roles include both. Without them, quota failures stay counted.

Why

#398: a deploy that hits exceeded quota (or any other pod-create error) hangs until the tracking timeout, and the reason only shows up in ReplicaSet events, which the tracker did not watch.

Testing

  • go test ./... and go test -race on the touched packages pass. task lint reports 0 issues.
  • Unit tests for every classification case. Each counted case fails with the corresponding check disabled.
  • kind (Kubernetes v1.36.1) with a ResourceQuota on pods, using the Deployment from the example above, tracked through dyntracker the way nelm does it:
    • quota 3 (deadlock): main waits for the timeout. This branch fails with the FailedCreate ... exceeded quota diagnostic about 10 seconds after the failure appears.
    • quota 4 (recoverable): fix: fail Deployment tracking fast on ReplicaSet pod-create errors #399 as it was failed after 2s. This branch keeps waiting, and the rollout completes once the ReplicaSet controller retries. That took several minutes because of its retry backoff.
    • Tracking started in the middle of that backoff, after quota had been freed but with the condition still set: counted, and the rollout completes.

Merging

Please merge with a merge commit or a rebase, not a squash, so that @Sharvash stays the author of his commits.

Sharvash and others added 4 commits October 27, 2026 14:22
Read FailedCreate from the durable ReplicaSetReplicaFailure condition instead of transient Warning events, so failures that predate informer startup terminate tracking immediately.

Only consider ReplicaSets controlled by the tracked Deployment to avoid selector collisions.

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
(cherry picked from commit a2e5c1f)
Force Failed payloads to set IsFailed and clear IsReady, while ignoring lagging FailedCreate conditions after the Deployment becomes ready.

Wait for the complete initial ReplicaSet snapshot before classifying failures. Preserve informer ordering with unbuffered, cancellation-aware sends, and handle same-name UID replacement and selector leave/reentry without losing a failure transition.

Classify invalid specs and quota failures that cannot recover through the rollout as fatal. Keep rollout-recoverable quota failures and event-sourced failures counted against the allowed-failures budget. Latch the first classification until the condition clears so a stale FailedCreate condition cannot escalate after old pods terminate, and account for TerminatingReplicas when checking whether the rollout can release quota.

Deduplicate failures per ReplicaSet incarnation, rearm reporting after condition recovery or Deployment readiness, ignore stale events after deletion, and re-evaluate a template-equivalent successor when deletion promotes it to the new ReplicaSet.

Fail the readiness task on a failed resource state only for fail modes that are supposed to fail the deploy process, keeping IgnoreAndContinueDeployProcess unaffected.

Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com>
(cherry picked from commit 78ec367)
…ease quota

The classifier subtracted every unavailable new pod, so a rollout whose pods
were created but not ready yet, or were terminating, was failed as stuck. It
also trusted the FailedCreate condition, which outlives the shortage while the
ReplicaSet controller backs off. Assume existing new pods become available,
confirm a deadlock with a fresh pod list and the named ResourceQuota, and
report it only once it has held for ten seconds.

Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
…FailureMode field

Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
Anyone allowed to write the ReplicaSet status controls the FailedCreate
message, and a quantity with a huge exponent kept the tracker busy for
minutes in arbitrary-precision arithmetic. Accept only the bounded forms
the quota admission plugin prints and count the failure otherwise.

Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
@ilya-lesikov
ilya-lesikov merged commit 97f4f13 into main Oct 29, 2026
2 checks passed
@ilya-lesikov
ilya-lesikov deleted the fix/398-replicaset-pod-create-failures branch October 29, 2026 01:10
ilya-lesikov added a commit to werf/nelm that referenced this pull request Oct 29, 2026
## Summary

Bumps `github.com/werf/kubedog` to the merge commit of werf/kubedog#404
on kubedog `main` (`97f4f13`), which makes Deployment tracking fail fast
on ReplicaSet pod-create errors such as `exceeded quota`
(werf/kubedog#398).

werf/kubedog#404 is merged; this PR pins its merge commit.

## What

- `go.mod`/`go.sum`: `github.com/werf/kubedog`
`v0.13.1-0.20260915153156-438ef109f804` →
`v0.13.1-0.20261029011008-97f4f131413e`.
- The bump also brings in the kubedog `main` commits since the previous
pin (`438ef10..b201189`): golangci-lint cleanups and reformatting. They
do not change behaviour.
- It also brings in hardening from werf/kubedog#404: quota quantities
parsed from a `FailedCreate` message are bounded, so a crafted
ReplicaSet status cannot keep the tracker busy.

## Why

Without this bump, `nelm` (and `werf`) keep waiting for the tracking
timeout when a Deployment's new ReplicaSet cannot create pods. The
failure is only visible in ReplicaSet events.

## Behaviour changes for nelm users

These come from `pkg/dyntracker`, which `pkg/plan/plan_execute.go` uses:

- An invalid pod spec, or a quota deadlock the rollout cannot resolve on
its own, now fails the resource immediately, regardless of
`werf.io/failures-allowed-per-replica`. A quota deadlock is reported
about 10s after it is observed, with the `FailedCreate ... exceeded
quota` diagnostic.
- Other pod-create failures, including quota failures that the rollout
can recover from, are added once as errors against the failures budget.
Before, they were not seen at all.
- With `werf.io/fail-mode: IgnoreAndContinueDeployProcess`, these fatal
failures do not fail the deploy.
- Only ReplicaSets controlled by the Deployment are treated as its new
ReplicaSet. An orphan ReplicaSet that matches the template is ignored
until the Deployment adopts it.
- A Deployment failure from an event is charged to the budget once, not
again on every later status update.
- Fast-failing on a quota deadlock needs `list pods` (already needed for
tracking) and `get resourcequotas` in the namespace. The built-in
`view`, `edit` and `admin` roles have both. Without them, quota failures
are counted rather than fatal.

## Testing

- `task build`, `task test:unit` (67 suites) and `task lint` pass.
- The kubedog PR has unit tests for each case and was verified on kind
with a `ResourceQuota`, tracking through `dyntracker` as nelm does.

---------

Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Event "exceeded quota" in replicaset should be tracked and treated as a failure

2 participants