Skip to content

fix: stop leaking informers and their caches after tracking ends - #406

Merged
dmmordvi merged 2 commits into
mainfrom
fix/informer-handler-ctx-sends
Nov 3, 2026
Merged

dmmordvi merged 2 commits into
mainfrom
fix/informer-handler-ctx-sends

Conversation

@dmmordvi

@dmmordvi dmmordvi commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Summary

A long-lived process that tracks resources over and over (for example a controller that runs a Helm-compatible deploy per reconcile) keeps shared informers alive after tracking ends, each holding the cache of a resource type for a whole namespace, until the process restarts. A goroutine dump shows informers that got the stop signal but wait in wg.Wait forever, and client-go listeners stuck in chansend inside a kubedog event handler. It happens when a handler has more events to deliver than the tracker channel can buffer at the moment the tracker stops reading, e.g. the initial Add events for several Pods of a Deployment that is already ready.

What

  • After a tracker stops (its context is canceled or its informer cleanup is called), informer event handlers stop waiting on tracker channels, so the shared informer stops on the factory stop channel and releases its cache. Applies to the Pod, Deployment, StatefulSet, DaemonSet, Job, Canary and ReplicaSet trackers, the generic resource state and events watchers, and the event informer.
  • While the tracker keeps reading, every event is delivered as before: channel capacities and send order are unchanged, nothing is dropped.
  • A fatal watch error is no longer sent to the factory's watch error channel once the factory is stopped. Before, a full channel nobody read blocked the reflector and kept the informer alive the same way.
  • When starting a Job, DaemonSet, StatefulSet or Pod tracker fails halfway, the informers it already started are released.
  • New exported helper utils.SendOrDone(ctx, ch, v): sends v or gives up once ctx is done.
  • Not changed: sends from tracker loops to consumer-facing channels (Ready, Failed, Status, PodError, AddedReplicaSet, …) still block when the consumer is gone. They can leave a tracker goroutine behind, but not an informer and its cache.
  • UNVERIFIED: not reproduced against a real cluster. Every handler is covered by a regression test on a fake dynamic client that shows the informer goroutines stay alive without the fix and exit with it.

Why

Informer handlers run in the client-go listener goroutine. A handler blocked on a channel send keeps sharedProcessor.run in wg.Wait() after the stop, so sharedIndexInformer.Run never returns and the indexer stays referenced. 38643e9 made the ReplicaSet informer sends cancellation-aware; every other handler, and the watch error handler, kept the plain send. Bigger channel buffers were not an option: they only move the threshold, since the number of objects a handler replays on start is unbounded.

Handlers sent to tracker channels without watching the context, so once the
tracker stopped reading, a handler blocked the client-go listener and the
shared informer never stopped, keeping its cache in memory. Send through
utils.SendOrDone and cancel the sends on cleanup, as the ReplicaSet informer
already did.

Also release what is already started when tracking a job, daemonset,
statefulset or pod fails halfway, and cover every informer handler with
regression tests through internal/informertest.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The reflector calls the handler synchronously, so a send to a full watch error
channel nobody reads kept the informer and its cache alive after the stop.
Give up on the send once the informer is stopped.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
@dmmordvi

dmmordvi commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Verification

  • Mutation: a plain send instead of utils.SendOrDone in any one of the 11 handlers → only that handler's *_whenConsumerStopsReading test fails (4 subtests, ~10 s each, with a message), neighbours in the same package pass; same with -shuffle=on.
  • Mutation: RemoveEventHandler before stopSending() in cleanup → the cleanup after factory stopped subtest fails in every lifecycle test.
  • Mutation: plain send in the watch error handler → TestSetWatchErrorHandler_whenNobodyReadsWatchErrors fails.
  • Mutation: SendOrDone that never sends / never gives up → pkg/utils/chan_test.go fails.

Review focus

  • pkg/tracker/internal/informertest detects a stuck informer by client-go goroutine frame names (processorListener).run, sharedIndexInformer).Run, sharedProcessor).run + wait.(*Group).Wait). A client-go rename makes the tests fail loudly, not pass.

Follow-up

  • Backport to the 0 branch (werf/kubedog) for consumers pinned to the 0.x line.
  • Bump kubedog in werf/nelm and release.
  • Tests: cover the three sends to Failures in the event informer (the seeded Event blocks on Messages first).
  • Tests: run the cleanup registered in t.Cleanup with a time limit, so a broken fix combined with renamed client-go frames fails instead of hanging until -timeout.

@dmmordvi
dmmordvi marked this pull request as ready for review November 3, 2026 16:07
@dmmordvi
dmmordvi merged commit 1ed920d into main Nov 3, 2026
2 checks passed
@dmmordvi
dmmordvi deleted the fix/informer-handler-ctx-sends branch November 3, 2026 16:07
dmmordvi added a commit that referenced this pull request Nov 3, 2026
Backport of #406 to `0`.

## Summary

A long-lived process that tracks resources over and over (for example a
controller that runs a Helm-compatible deploy per reconcile) keeps
shared informers alive after tracking ends, each holding the cache of a
resource type for a whole namespace, until the process restarts. A
goroutine dump shows informers that got the stop signal but wait in
`wg.Wait` forever, and client-go listeners stuck in `chansend` inside
`ReplicaSetInformer.Run`. On `0` it happens on almost every Deployment
tracking: the ReplicaSet informer replays an Add for every ReplicaSet of
the Deployment into a channel with capacity 1 while the tracker is busy
reporting readiness, and stops reading right after.

## What

- After a tracker stops (its context is canceled or its informer cleanup
is called), informer event handlers stop waiting on tracker channels, so
the shared informer stops on the factory stop channel and releases its
cache. Applies to the Pod, Deployment, StatefulSet, DaemonSet, Job,
Canary and ReplicaSet trackers, the generic resource state and events
watchers, and the event informer.
- While the tracker keeps reading, every event is delivered as before:
channel capacities (including the ReplicaSet channels, capacity 1) and
send order are unchanged, nothing is dropped.
- A fatal watch error is no longer sent to the factory's watch error
channel once the factory is stopped. Before, a full channel nobody read
blocked the reflector and kept the informer alive the same way.
- When starting a Deployment, Job, DaemonSet, StatefulSet or Pod tracker
fails halfway, the informers it already started are released.
- New exported helper `utils.SendOrDone(ctx, ch, v)`: sends `v` or gives
up once `ctx` is done.
- Not changed: the ReplicaSet failure handling from 38643e9 (fail fast
on pod-create errors) is not backported; only the cancellation-aware
sends are.
- Not changed: sends from tracker loops to consumer-facing channels
(`Ready`, `Failed`, `Status`, `PodError`, `AddedReplicaSet`, …) still
block when the consumer is gone. They can leave a tracker goroutine
behind, but not an informer and its cache.
- UNVERIFIED: not reproduced against a real cluster. Every handler is
covered by a regression test on a fake dynamic client that shows the
informer goroutines stay alive without the fix and exit with it.

## Why

Informer handlers run in the client-go listener goroutine. A handler
blocked on a channel send keeps `sharedProcessor.run` in `wg.Wait()`
after the stop, so `sharedIndexInformer.Run` never returns and the
indexer stays referenced. On `0` no handler, including the ReplicaSet
one, was cancellation-aware. Porting 38643e9 instead was rejected: it
also changes Deployment failure semantics, which a maintenance branch
should not pick up for a leak fix.

---------

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
dmmordvi added a commit to werf/nelm that referenced this pull request Nov 3, 2026
## Summary

A long-lived process that runs a nelm release install with tracking on
every reconcile (for example a controller embedding nelm v1) keeps
shared informers alive after each tracking ends, each holding the cache
of a resource type for a whole namespace, so memory grows until the
process restarts. This bumps `github.com/werf/kubedog` from `ad1e8a1` to
`f1eefb5` on kubedog's `0` branch, which releases those informers
(werf/kubedog#407, backport of werf/kubedog#406). The range also brings
werf/kubedog#405, which fixes the resource elimination wait of the
legacy `release uninstall` path.

## What

- `go.mod`/`go.sum`: `github.com/werf/kubedog`
`v0.13.1-0.20260912145640-ad1e8a134607` →
`v0.13.1-0.20261103161329-f1eefb59d19a`; no other module changes.
- Kubedog range `ad1e8a1..f1eefb5` is two commits: werf/kubedog#405 and
werf/kubedog#407.

### Tracking during `release install` / `release rollback` / native
`release uninstall` (werf/kubedog#407)

- After resource tracking ends, the shared informers started for it stop
and release their caches, including when the tracking side stops reading
events mid-stream. Before, a handler blocked on a send kept the informer
and its cache alive for the life of the process; most often the
ReplicaSet informer of a Deployment.
- While tracking runs, every event is delivered as before: channel
capacities and send order are unchanged, nothing is dropped.
- A fatal watch error raised after tracking stopped no longer blocks the
informer from stopping.
- When starting a Deployment, StatefulSet, DaemonSet, Job or Pod tracker
fails halfway, the informers it already started are released.
- Not changed: Deployment failure handling. The fail-fast on ReplicaSet
pod-create errors from kubedog `main` (38643e9) is not part of kubedog's
`0` branch and does not come with this bump.
- UNVERIFIED: the leak fix is not reproduced against a real cluster
here; kubedog covers every affected handler with regression tests on a
fake dynamic client.

### Resource elimination wait in the legacy uninstall path
(werf/kubedog#405)

- Applies to `nelm release uninstall` without
`NELM_FEAT_NATIVE_RELEASE_UNINSTALL`/`NELM_FEAT_PREVIEW_V2`, and to
`action.LegacyReleaseUninstall` callers, at the `Waiting for resources
elimination` step.
- The wait no longer misses a deletion that happens between its initial
LIST and WATCH, which made it run until the timeout.
- A consumer that builds nelm v1 with client-go v0.35+ (where the
`WatchListClient` feature gate is on by default) no longer hangs at
`Waiting for resources elimination` until the timeout when the resources
are already gone. nelm itself pins client-go v0.29.3, where the gate is
off.

## Why

Kubedog's informer event handlers on `0` sent to tracker channels
without watching for cancellation. Once the tracker stopped reading, a
handler blocked forever, so the stopped informer waited on it and never
returned, keeping its namespace-wide cache referenced; in a process that
deploys repeatedly this accumulated into gigabytes. The elimination
tracker replaced the reflector's list/watch options with a bare field
selector, dropping the resourceVersion and WatchList parameters, which
caused both the missed deletion and the WatchList hang.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
dmmordvi added a commit to werf/nelm that referenced this pull request Nov 3, 2026
## Summary

Bumps `github.com/werf/kubedog` to the merge commit of werf/kubedog#406
on kubedog `main` (`1ed920d`). A long-lived process that runs `nelm`
release actions over and over with tracking (for example, a controller
that deploys a release on every reconcile) no longer accumulates
informers that hold the cache of a resource type for a whole namespace
after each action ends. Before, the memory such a process held grew with
every deploy until it restarted.

## What

- `go.mod`/`go.sum`: `github.com/werf/kubedog`
`v0.13.1-0.20261029011008-97f4f131413e` →
`v0.13.1-0.20261103160714-1ed920dee18b`. The range `97f4f13..1ed920d` is
the single commit of werf/kubedog#406.
- After a release install, upgrade, uninstall or rollback with tracking
finishes and its context is canceled, the shared informers it started
stop and release their caches, even when the trackers stopped reading
before an informer delivered all its events. Applies to the Pod,
Deployment, StatefulSet, DaemonSet, Job, Canary and ReplicaSet trackers,
the generic resource state and events watchers and the event informer
that `pkg/dyntracker` uses.
- A fatal watch error raised after the action finished no longer blocks
the informer: it is dropped once the informer factory is stopped instead
of waiting forever on the watch error channel.
- When a Job, DaemonSet, StatefulSet or Pod tracker fails halfway
through starting, the informers it already started are released.
- While tracking runs, every event is delivered as before: channel
capacities and send order are unchanged, nothing is dropped. Tracking
results, progress output and errors do not change.
- Library API: kubedog adds the exported helper `utils.SendOrDone(ctx,
ch, v)`. Nothing in the `nelm` API changes.
- `go.mod`: `golang.org/x/sync v0.20.0` moves from the indirect to the
direct block, same version. `pkg/resource/spec/patches_debug_test.go`
(#785) already imports it directly; `go mod tidy` on `main` without this
bump makes the same change.
- UNVERIFIED: not reproduced against a real cluster. kubedog covers
every changed handler with a regression test on a fake dynamic client
that shows the informer goroutines stay alive without the fix and exit
with it.

## Why

Informer event handlers run in the client-go listener goroutine and sent
to tracker channels without watching for cancellation. When a handler
had more events to deliver than the channel could buffer at the moment
the tracker stopped reading (e.g. the initial Add events for several
Pods of a Deployment that is already ready), the send blocked forever,
the informer stayed in `wg.Wait()` after the stop signal, and its
indexer stayed referenced. 38643e9 in kubedog (brought in by #780) fixed
this for the ReplicaSet informer only. Bigger channel buffers only move
the threshold, since the number of objects an informer replays on start
is unbounded.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant