Skip to content

chore: bump kubedog to stop leaking informers after tracking - #812

Merged
dmmordvi merged 1 commit into
1from
chore/bump-kubedog-informer-leak-1
Nov 3, 2026
Merged

dmmordvi merged 1 commit into
1from
chore/bump-kubedog-informer-leak-1

Conversation

@dmmordvi

@dmmordvi dmmordvi commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

A long-lived process that runs a nelm release install with tracking on every reconcile (for example a controller embedding nelm v1) keeps shared informers alive after each tracking ends, each holding the cache of a resource type for a whole namespace, so memory grows until the process restarts. This bumps github.com/werf/kubedog from ad1e8a1 to f1eefb5 on kubedog's 0 branch, which releases those informers (werf/kubedog#407, backport of werf/kubedog#406). The range also brings werf/kubedog#405, which fixes the resource elimination wait of the legacy release uninstall path.

What

Tracking during release install / release rollback / native release uninstall (werf/kubedog#407)

  • After resource tracking ends, the shared informers started for it stop and release their caches, including when the tracking side stops reading events mid-stream. Before, a handler blocked on a send kept the informer and its cache alive for the life of the process; most often the ReplicaSet informer of a Deployment.
  • While tracking runs, every event is delivered as before: channel capacities and send order are unchanged, nothing is dropped.
  • A fatal watch error raised after tracking stopped no longer blocks the informer from stopping.
  • When starting a Deployment, StatefulSet, DaemonSet, Job or Pod tracker fails halfway, the informers it already started are released.
  • Not changed: Deployment failure handling. The fail-fast on ReplicaSet pod-create errors from kubedog main (38643e9) is not part of kubedog's 0 branch and does not come with this bump.
  • UNVERIFIED: the leak fix is not reproduced against a real cluster here; kubedog covers every affected handler with regression tests on a fake dynamic client.

Resource elimination wait in the legacy uninstall path (werf/kubedog#405)

  • Applies to nelm release uninstall without NELM_FEAT_NATIVE_RELEASE_UNINSTALL/NELM_FEAT_PREVIEW_V2, and to action.LegacyReleaseUninstall callers, at the Waiting for resources elimination step.
  • The wait no longer misses a deletion that happens between its initial LIST and WATCH, which made it run until the timeout.
  • A consumer that builds nelm v1 with client-go v0.35+ (where the WatchListClient feature gate is on by default) no longer hangs at Waiting for resources elimination until the timeout when the resources are already gone. nelm itself pins client-go v0.29.3, where the gate is off.

Why

Kubedog's informer event handlers on 0 sent to tracker channels without watching for cancellation. Once the tracker stopped reading, a handler blocked forever, so the stopped informer waited on it and never returned, keeping its namespace-wide cache referenced; in a process that deploys repeatedly this accumulated into gigabytes. The elimination tracker replaced the reflector's list/watch options with a bare field selector, dropping the resourceVersion and WatchList parameters, which caused both the missed deletion and the WatchList hang.

View guided diff

BEGIN_COMMIT_OVERRIDE
fix: stop leaking informers and their caches after tracking
fix: stop resource elimination waiting from missing deletions and hanging when WatchListClient is enabled
END_COMMIT_OVERRIDE

Pins werf/kubedog f1eefb5 on its `0` branch (werf/kubedog#407, backport of #406): informers are released after tracking ends. The range also brings werf/kubedog#405, which keeps the legacy uninstall wait from hanging with WatchListClient and from missing a deletion between LIST and WATCH.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
@dmmordvi

dmmordvi commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Verification

  • task lint:golangci-lint reports the same 24 issues (gci, gofumpt, gosec, staticcheck, wrapcheck, wsl_v5) on this branch and on origin/1 without the bump; none come from this change, since no Go file in nelm is touched.
  • Not run: a tracking or uninstall run against a cluster; kubedog's own regression tests are the only evidence for the leak fix.

Review focus

Follow-up

  • After merge, release a nelm 1 patch version through the release-please PR on werf/nelm branch 1, then bump consumers pinned to nelm v1.

@dmmordvi
dmmordvi marked this pull request as ready for review November 3, 2026 17:29
@dmmordvi
dmmordvi merged commit 1db96ad into 1 Nov 3, 2026
8 checks passed
@dmmordvi
dmmordvi deleted the chore/bump-kubedog-informer-leak-1 branch November 3, 2026 17:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant