Skip to content

chore: bump kubedog to stop leaking informers after tracking - #813

Merged
dmmordvi merged 1 commit into
mainfrom
chore/bump-kubedog-informer-leak
Nov 3, 2026
Merged

dmmordvi merged 1 commit into
mainfrom
chore/bump-kubedog-informer-leak

Conversation

@dmmordvi

@dmmordvi dmmordvi commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Bumps github.com/werf/kubedog to the merge commit of werf/kubedog#406 on kubedog main (1ed920d). A long-lived process that runs nelm release actions over and over with tracking (for example, a controller that deploys a release on every reconcile) no longer accumulates informers that hold the cache of a resource type for a whole namespace after each action ends. Before, the memory such a process held grew with every deploy until it restarted.

What

  • go.mod/go.sum: github.com/werf/kubedog v0.13.1-0.20261029011008-97f4f131413e → v0.13.1-0.20261103160714-1ed920dee18b. The range 97f4f13..1ed920d is the single commit of fix: stop leaking informers and their caches after tracking ends kubedog#406.
  • After a release install, upgrade, uninstall or rollback with tracking finishes and its context is canceled, the shared informers it started stop and release their caches, even when the trackers stopped reading before an informer delivered all its events. Applies to the Pod, Deployment, StatefulSet, DaemonSet, Job, Canary and ReplicaSet trackers, the generic resource state and events watchers and the event informer that pkg/dyntracker uses.
  • A fatal watch error raised after the action finished no longer blocks the informer: it is dropped once the informer factory is stopped instead of waiting forever on the watch error channel.
  • When a Job, DaemonSet, StatefulSet or Pod tracker fails halfway through starting, the informers it already started are released.
  • While tracking runs, every event is delivered as before: channel capacities and send order are unchanged, nothing is dropped. Tracking results, progress output and errors do not change.
  • Library API: kubedog adds the exported helper utils.SendOrDone(ctx, ch, v). Nothing in the nelm API changes.
  • go.mod: golang.org/x/sync v0.20.0 moves from the indirect to the direct block, same version. pkg/resource/spec/patches_debug_test.go (feat: support debug in jq patches #785) already imports it directly; go mod tidy on main without this bump makes the same change.
  • UNVERIFIED: not reproduced against a real cluster. kubedog covers every changed handler with a regression test on a fake dynamic client that shows the informer goroutines stay alive without the fix and exit with it.

Why

Informer event handlers run in the client-go listener goroutine and sent to tracker channels without watching for cancellation. When a handler had more events to deliver than the channel could buffer at the moment the tracker stopped reading (e.g. the initial Add events for several Pods of a Deployment that is already ready), the send blocked forever, the informer stayed in wg.Wait() after the stop signal, and its indexer stayed referenced. 38643e9 in kubedog (brought in by #780) fixed this for the ReplicaSet informer only. Bigger channel buffers only move the threshold, since the number of objects an informer replays on start is unbounded.

View guided diff

BEGIN_COMMIT_OVERRIDE
fix: stop leaking informers and their caches after tracking
END_COMMIT_OVERRIDE

Bumps github.com/werf/kubedog to 1ed920d (werf/kubedog#406). Informer
event handlers no longer block on tracker channels after tracking ends,
so long-lived processes stop accumulating informers and their caches.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
@dmmordvi

dmmordvi commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Verification

  • TestPullCmd and TestPullOCIWithTagAndDigest (pkg/helm/pkg/cmd) fail intermittently on macOS with error logging into registry with good credentials: The specified item already exists in the keychain. (-25299) when ~/.docker/config.json uses "credsStore": "osxkeychain". Reproduced on main without this bump (task test:unit paths=./pkg/helm/pkg/cmd -- --focus-file pull_test.go); the package's only kubedog dependency, pkg/dyntracker/util, is unchanged in the range.
  • Mutation: not run here; the regression tests live in fix: stop leaking informers and their caches after tracking ends kubedog#406, which states that each one fails without its fix.
  • Not run: a long-lived process doing repeated deploys against a real cluster with a heap/goroutine profile before and after.

Review focus

Follow-up

  • pkg/action/release_rollback.go: with Timeout == 0, ReleaseRollback never cancels its context, so the informer factory and the watch error goroutine stop only when the caller cancels the context it passed in. Install and uninstall cancel theirs on return.

@dmmordvi
dmmordvi marked this pull request as ready for review November 3, 2026 17:29
@dmmordvi
dmmordvi merged commit dc619a2 into main Nov 3, 2026
8 checks passed
@dmmordvi
dmmordvi deleted the chore/bump-kubedog-informer-leak branch November 3, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant