Repository navigation
fix: fail Deployment tracking fast on ReplicaSet pod-create errors - #404
Merged
Merged
Conversation
Read FailedCreate from the durable ReplicaSetReplicaFailure condition instead of transient Warning events, so failures that predate informer startup terminate tracking immediately. Only consider ReplicaSets controlled by the tracked Deployment to avoid selector collisions. Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com> (cherry picked from commit a2e5c1f)
Force Failed payloads to set IsFailed and clear IsReady, while ignoring lagging FailedCreate conditions after the Deployment becomes ready. Wait for the complete initial ReplicaSet snapshot before classifying failures. Preserve informer ordering with unbuffered, cancellation-aware sends, and handle same-name UID replacement and selector leave/reentry without losing a failure transition. Classify invalid specs and quota failures that cannot recover through the rollout as fatal. Keep rollout-recoverable quota failures and event-sourced failures counted against the allowed-failures budget. Latch the first classification until the condition clears so a stale FailedCreate condition cannot escalate after old pods terminate, and account for TerminatingReplicas when checking whether the rollout can release quota. Deduplicate failures per ReplicaSet incarnation, rearm reporting after condition recovery or Deployment readiness, ignore stale events after deletion, and re-evaluate a template-equivalent successor when deletion promotes it to the new ReplicaSet. Fail the readiness task on a failed resource state only for fail modes that are supposed to fail the deploy process, keeping IgnoreAndContinueDeployProcess unaffected. Signed-off-by: Alexey Gorovenko <sharvashinho@gmail.com> (cherry picked from commit 78ec367)
…ease quota The classifier subtracted every unavailable new pod, so a rollout whose pods were created but not ready yet, or were terminating, was failed as stuck. It also trusted the FailedCreate condition, which outlives the shortage while the ReplicaSet controller backs off. Assume existing new pods become available, confirm a deadlock with a fresh pod list and the named ResourceQuota, and report it only once it has held for ten seconds. Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
…FailureMode field Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
This was referenced Oct 27, 2026
Anyone allowed to write the ReplicaSet status controls the FailedCreate message, and a quantity with a huge exponent kept the tracker busy for minutes in arbitrary-precision arithmetic. Accept only the bounded forms the quota admission plugin prints and count the failure otherwise. Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
ilya-lesikov
added a commit
to werf/nelm
that referenced
this pull request
Oct 29, 2026
## Summary Bumps `github.com/werf/kubedog` to the merge commit of werf/kubedog#404 on kubedog `main` (`97f4f13`), which makes Deployment tracking fail fast on ReplicaSet pod-create errors such as `exceeded quota` (werf/kubedog#398). werf/kubedog#404 is merged; this PR pins its merge commit. ## What - `go.mod`/`go.sum`: `github.com/werf/kubedog` `v0.13.1-0.20260915153156-438ef109f804` → `v0.13.1-0.20261029011008-97f4f131413e`. - The bump also brings in the kubedog `main` commits since the previous pin (`438ef10..b201189`): golangci-lint cleanups and reformatting. They do not change behaviour. - It also brings in hardening from werf/kubedog#404: quota quantities parsed from a `FailedCreate` message are bounded, so a crafted ReplicaSet status cannot keep the tracker busy. ## Why Without this bump, `nelm` (and `werf`) keep waiting for the tracking timeout when a Deployment's new ReplicaSet cannot create pods. The failure is only visible in ReplicaSet events. ## Behaviour changes for nelm users These come from `pkg/dyntracker`, which `pkg/plan/plan_execute.go` uses: - An invalid pod spec, or a quota deadlock the rollout cannot resolve on its own, now fails the resource immediately, regardless of `werf.io/failures-allowed-per-replica`. A quota deadlock is reported about 10s after it is observed, with the `FailedCreate ... exceeded quota` diagnostic. - Other pod-create failures, including quota failures that the rollout can recover from, are added once as errors against the failures budget. Before, they were not seen at all. - With `werf.io/fail-mode: IgnoreAndContinueDeployProcess`, these fatal failures do not fail the deploy. - Only ReplicaSets controlled by the Deployment are treated as its new ReplicaSet. An orphan ReplicaSet that matches the template is ignored until the Deployment adopts it. - A Deployment failure from an event is charged to the budget once, not again on every later status update. - Fast-failing on a quota deadlock needs `list pods` (already needed for tracking) and `get resourcequotas` in the namespace. The built-in `view`, `edit` and `admin` roles have both. Without them, quota failures are counted rather than fatal. ## Testing - `task build`, `task test:unit` (67 suites) and `task lint` pass. - The kubedog PR has unit tests for each case and was verified on kind with a `ResourceQuota`, tracking through `dyntracker` as nelm does. --------- Signed-off-by: Ilya Lesikov <ilya@lesikov.com>
This was referenced Nov 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The Deployment tracker now reports pod-create failures of the Deployment's new ReplicaSet (the
ReplicaFailure/FailedCreatecondition, e.g.exceeded quota) instead of waiting for the tracking timeout. Failures the rollout cannot recover from on its own fail the deploy immediately; the rest are charged against the allowed-failures budget.This is a port of #399 by @Sharvash. When branch
1was merged intomain(#389) and deleted, GitHub closed #399 without merging it. His two commits are cherry-picked unchanged, with his authorship and sign-off. Our commits on top fix the quota classification and a lint error.Closes #398
What
fix: fail Deployment tracking on ReplicaSet pod-create errors: reads the durableReplicaFailurecondition of the new ReplicaSet rather than transient events, so failures that happened before tracking started are also seen. Only ReplicaSets controlled by the Deployment are considered.fix: handle ReplicaSet failure transitions consistently: waits for the initial ReplicaSet list before classifying, keeps informer events in order (unbuffered, cancellation-aware sends), handles ReplicaSet replacement, selector leave and re-entry, and deletion. Reports a failure once per ReplicaSet incarnation and re-arms it when the condition clears. Ignores a lagging condition once the Deployment is ready.fix: keep ReplicaSet quota failures counted while the rollout can release quota: see Quota classification below.chore: fix errorlint in deployment tracker test harness and document FailureMode field.fix: bound quota quantities parsed from a ReplicaSet failure message: theFailedCreatemessage can be written by anyone who can update the ReplicaSet status, and a quantity with a huge exponent kept the tracker busy for minutes. Only the bounded forms the quota admission plugin prints are parsed now; anything else keeps the failure counted.Quota classification
The original heuristic reported a quota failure as fatal when the rollout could not scale an old ReplicaSet down at that moment. Several rollouts that do recover were therefore misclassified. One example:
replicas: 3,maxSurge: 2,maxUnavailable: 0, a quota of 4 pods. The new ReplicaSet gets 1 of its 2 pods. Once that pod becomes available, an old pod is scaled down, quota is freed and the rollout continues. A quota failure is now fatal only if all of the following hold:sum(old spec.replicas) + existing new pods <= replicas - maxUnavailable. The same formula asreconcileOldReplicaSets.list podsconfirms the projection: no pod of the rollout is terminating, no old ReplicaSet has surplus pods, and the new ReplicaSet is still missing pods. This check is needed because the status that brings the condition counts only the pods listed before the failed sync created any, andstatus.terminatingReplicasis feature-gated.get resourcequotas/<name from the message>shows that the quota still has no room for the request. The ReplicaSet controller keepsFailedCreateacross its retry backoff, which can last minutes, so the condition can outlive the shortage.If a check cannot be made (the list or get fails, for example because of RBAC, or the message does not parse), the failure stays counted, as on
main.Behaviour changes
failures-allowed: an invalid pod spec (... is invalid:), or a quota failure confirmed as described above. A quota deadlock is reported about 10 seconds after it is observed.pkg/tracker/deployment:type FailureMode,FailureModeCounted(the zero value, so existing code that builds aDeploymentStatuskeeps the old meaning) andFailureModeFatal, plus theDeploymentStatus.FailureModefield. It is meaningful only whenIsFailedis set.Failed, which fails the readiness task immediately.statestore/readiness_task_state.go): withIgnoreAndContinueDeployProcess, a resource markedFaileddoes not fail the task. OnlyFailWholeDeployProcessImmediatelyandLegacyHopeUntilEndOfDeployProcessfail on it. Before this PR nothing setResourceStatusFailed, so the change affects only the new fatal failures: they respectwerf.io/fail-mode: IgnoreAndContinueDeployProcess.IsFailed/FailedReasoninto every later status. Each failure is charged to the budget once, where before it was charged on every subsequent update. The StatefulSet and DaemonSet trackers still re-stamp.utils.FindNewReplicaSetonly considers ReplicaSets controlled by the Deployment.list pods(already required for tracking) andget resourcequotasin the namespace. The built-inview,editandadminroles include both. Without them, quota failures stay counted.Why
#398: a deploy that hits
exceeded quota(or any other pod-create error) hangs until the tracking timeout, and the reason only shows up in ReplicaSet events, which the tracker did not watch.Testing
go test ./...andgo test -raceon the touched packages pass.task lintreports 0 issues.ResourceQuotaon pods, using the Deployment from the example above, tracked throughdyntrackerthe way nelm does it:mainwaits for the timeout. This branch fails with theFailedCreate ... exceeded quotadiagnostic about 10 seconds after the failure appears.Merging
Please merge with a merge commit or a rebase, not a squash, so that @Sharvash stays the author of his commits.