Repository navigation
fix: stop leaking informers and their caches after tracking ends - #407
Merged
Merged
Conversation
Backport of #406 (1ed920d) to 0. Handlers sent to tracker channels without watching the context, so once the tracker stopped reading, a handler blocked the client-go listener and the shared informer never stopped, keeping its cache in memory. Send through utils.SendOrDone and cancel the sends on cleanup, including the ReplicaSet informer, whose channel capacities stay unchanged. Also release what is already started when tracking a deployment, job, daemonset, statefulset or pod fails halfway, and cover every informer handler with regression tests through internal/informertest. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Contributor
Author
Verification
Follow-up
|
dmmordvi
marked this pull request as ready for review
November 3, 2026 16:13
dmmordvi
added a commit
to werf/nelm
that referenced
this pull request
Nov 3, 2026
## Summary A long-lived process that runs a nelm release install with tracking on every reconcile (for example a controller embedding nelm v1) keeps shared informers alive after each tracking ends, each holding the cache of a resource type for a whole namespace, so memory grows until the process restarts. This bumps `github.com/werf/kubedog` from `ad1e8a1` to `f1eefb5` on kubedog's `0` branch, which releases those informers (werf/kubedog#407, backport of werf/kubedog#406). The range also brings werf/kubedog#405, which fixes the resource elimination wait of the legacy `release uninstall` path. ## What - `go.mod`/`go.sum`: `github.com/werf/kubedog` `v0.13.1-0.20260912145640-ad1e8a134607` → `v0.13.1-0.20261103161329-f1eefb59d19a`; no other module changes. - Kubedog range `ad1e8a1..f1eefb5` is two commits: werf/kubedog#405 and werf/kubedog#407. ### Tracking during `release install` / `release rollback` / native `release uninstall` (werf/kubedog#407) - After resource tracking ends, the shared informers started for it stop and release their caches, including when the tracking side stops reading events mid-stream. Before, a handler blocked on a send kept the informer and its cache alive for the life of the process; most often the ReplicaSet informer of a Deployment. - While tracking runs, every event is delivered as before: channel capacities and send order are unchanged, nothing is dropped. - A fatal watch error raised after tracking stopped no longer blocks the informer from stopping. - When starting a Deployment, StatefulSet, DaemonSet, Job or Pod tracker fails halfway, the informers it already started are released. - Not changed: Deployment failure handling. The fail-fast on ReplicaSet pod-create errors from kubedog `main` (38643e9) is not part of kubedog's `0` branch and does not come with this bump. - UNVERIFIED: the leak fix is not reproduced against a real cluster here; kubedog covers every affected handler with regression tests on a fake dynamic client. ### Resource elimination wait in the legacy uninstall path (werf/kubedog#405) - Applies to `nelm release uninstall` without `NELM_FEAT_NATIVE_RELEASE_UNINSTALL`/`NELM_FEAT_PREVIEW_V2`, and to `action.LegacyReleaseUninstall` callers, at the `Waiting for resources elimination` step. - The wait no longer misses a deletion that happens between its initial LIST and WATCH, which made it run until the timeout. - A consumer that builds nelm v1 with client-go v0.35+ (where the `WatchListClient` feature gate is on by default) no longer hangs at `Waiting for resources elimination` until the timeout when the resources are already gone. nelm itself pins client-go v0.29.3, where the gate is off. ## Why Kubedog's informer event handlers on `0` sent to tracker channels without watching for cancellation. Once the tracker stopped reading, a handler blocked forever, so the stopped informer waited on it and never returned, keeping its namespace-wide cache referenced; in a process that deploys repeatedly this accumulated into gigabytes. The elimination tracker replaced the reflector's list/watch options with a bare field selector, dropping the resourceVersion and WatchList parameters, which caused both the missed deletion and the WatchList hang. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #406 to
0.Summary
A long-lived process that tracks resources over and over (for example a controller that runs a Helm-compatible deploy per reconcile) keeps shared informers alive after tracking ends, each holding the cache of a resource type for a whole namespace, until the process restarts. A goroutine dump shows informers that got the stop signal but wait in
wg.Waitforever, and client-go listeners stuck inchansendinsideReplicaSetInformer.Run. On0it happens on almost every Deployment tracking: the ReplicaSet informer replays an Add for every ReplicaSet of the Deployment into a channel with capacity 1 while the tracker is busy reporting readiness, and stops reading right after.What
utils.SendOrDone(ctx, ch, v): sendsvor gives up oncectxis done.Ready,Failed,Status,PodError,AddedReplicaSet, …) still block when the consumer is gone. They can leave a tracker goroutine behind, but not an informer and its cache.Why
Informer handlers run in the client-go listener goroutine. A handler blocked on a channel send keeps
sharedProcessor.runinwg.Wait()after the stop, sosharedIndexInformer.Runnever returns and the indexer stays referenced. On0no handler, including the ReplicaSet one, was cancellation-aware. Porting 38643e9 instead was rejected: it also changes Deployment failure semantics, which a maintenance branch should not pick up for a leak fix.