Skip to content

fix(informer, tracker): warn instead of failing on inaccessible events - #401

Merged
dmmordvi merged 8 commits into
mainfrom
fix/skip-cluster-scope-events-watcher
Sep 12, 2026
Merged

dmmordvi merged 8 commits into
mainfrom
fix/skip-cluster-scope-events-watcher

Conversation

@dmmordvi

@dmmordvi dmmordvi commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Tracking a cluster-scoped resource aborted the whole release when the user had no access to events. Events of a cluster-scoped resource are stored in the default namespace, so ResourceEventsWatcher lists and watches v1/events there; a user allowed to read the tracked resource is not necessarily allowed to read events in default. The reflector's 403 Forbidden was reported through watchErrCh as an unrecoverable watch error, which cancels the tracking context.

Repro: grant a role with get/list/watch on some cluster-scoped kind but nothing on events in default, then deploy a release containing that kind. Expected: the resource is tracked, with a warning that its events are unavailable. Actual: unrecoverable watch error for /v1, Resource=events: ... forbidden, release fails.

Key changes

  • Losing the events feed no longer stops the tracking:
    • InformerOptions{ForbiddenIsNotFatal} makes the watch error handler treat apierrors.IsForbidden the way it already treats EOF and ResourceExpired — the reflector keeps retrying, nothing reaches watchErrCh.
    • Both events call sites opt in: the generic ResourceEventsWatcher and the legacy event.EventInformer. The leniency does not depend on the scope of the tracked resource, because the scope says nothing about the right to read events — a role granting a namespaced resource but not the events of its own namespace failed a release exactly the same way.
  • The lost feed is now actually reported: ConcurrentInformerFactoryOptions.OnNonFatalWatchError receives the resource, the namespace and the error, defaulting to a warning on the display output stream. Reported once per informer, since the reflector retries indefinitely.
  • Consumers disagreeing on the options get separate informers, keyed by {namespace, options}: a SharedIndexInformer has a single watch error handler which can only be set once, so sharing would make the policy depend on which consumer created the informer first.
  • ForNamespace now rejects more than one InformerOptions argument instead of silently dropping the extras.

Why

A missing auxiliary feed should degrade the tracking, not abort the release. The alternative — a SelfSubjectAccessReview precheck before creating the informer — was rejected: it is a time-of-check/time-of-use snapshot (access can be revoked mid-release, which still must not kill the release), it needs both list and watch checked, and create selfsubjectaccessreviews can itself be denied in hardened clusters. Runtime handling is required regardless, so the precheck would only add an earlier message.

Verification

  • Unit tests cover the watch error classification (forbidden reported through the callback and not the channel, only once, detected through the reflector's error wrapping, other errors still fatal) and the factory behavior (separate informers for differing options, shared informer for equal ones, default warning reaching the display stream).
  • Tests driving a real reflector over a client denying list and watch of events assert which channel the error reaches: for ForNamespace with and without the option, for ResourceEventsWatcher with a cluster-scoped and a namespaced resource, and for event.EventInformer. Without them the fix was untested — dropping the option passthrough in ForNamespace, or the opt-in of either events call site, kept the whole suite green.
  • All the tests carry the ai_tests build tag, which CI does not run — so they need -tags ai_tests to execute.
  • Not run: a real cluster with a role denying events. The failing path was traced through client-go@v0.29.3 reflector.go rather than reproduced end to end.

Review focus / risks

  • Event-based failure detection is lost when the events feed is denied. event.EventInformer turns an event whose Reason contains "Failed" into a resource failure and pod probe events into ProbeTriggeredRestart/ReadinessProbeFailure; the generic tracker fails on EventStatus.IsFailure(), which never fires today, so for it the loss is cosmetic. ImagePullBackOff/ErrImagePull/CrashLoopBackOff come from the pod status rather than from events and still fail fast; probe failures, FailedScheduling and FailedMount now surface via noActivityTimeout instead. That detection is unreachable in this scenario anyway, since today the release dies outright. Readiness itself is derived from the resource and unaffected.
  • Tracking a v1/Event itself is the one case where the same resource and namespace is requested with both options, producing two reflectors on events in that namespace. dynamicinformer creates informers lazily per resource, so the second factory is otherwise free.
  • ForbiddenIsNotFatal is exposed on ForNamespace for any resource, including a primary readiness watch. Narrowing it to events would need either a hardcoded GVR in the factory or a separate event-feed abstraction; for now only the two events call sites pass it.

…o events

Events of cluster-scoped resources are stored in the "default" namespace, so
ResourceEventsWatcher lists and watches v1/events there. A user allowed to read
the tracked resource is not necessarily allowed to read events in "default",
and the resulting "403 Forbidden" from the reflector was reported through
watchErrCh as an unrecoverable watch error, which aborts the whole tracking.
Losing the events feed shouldn't stop tracking the resource itself.

The events informer of a cluster-scoped resource is now created with
InformerOptions{ForbiddenIsNotFatal: true}, and its watch error handler treats
apierrors.IsForbidden the same way it already treats EOF and ResourceExpired:
logs it once under KUBEDOG_DEBUG and lets the reflector keep retrying. Only
!namespaced resources get the leniency; for namespaced ones a 403 on events
still means the tracking scope is wrong, and stays fatal.

A SharedIndexInformer has a single watch error handler which can only be set
once, so consumers disagreeing on the policy must not share an informer.
The policy is therefore part of the factory-map key, which keeps the two
policies on separate informers and avoids depending on which consumer happened
to create the informer first. A namespace name can't contain a slash, so the
"<namespace>/forbidden-is-not-fatal" key never collides with a plain namespace.
The cost is a second events informer on "default" for a release deployed into
"default" that has both cluster-scoped and namespaced resources.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
A "403 Forbidden" tolerated via ForbiddenIsNotFatal was only printed under
KUBEDOG_TRACKER_DEBUG=1, so by default the user lost the feed with no
explanation at all. The factory now takes OnNonFatalWatchError, which defaults
to printing a warning naming the resource and the namespace, and the sync.Once
wraps the reporting itself rather than the debug check.

Consumers agreeing on the policy share one informer, and one informer has one
watch error handler, so the options are now stored per resource and namespace
and a disagreeing consumer gets an error instead of a silently split factory,
a duplicate reflector, and a policy decided by creation order.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Leniency was limited to cluster-scoped resources because that is how the
missing access was reported, but the scope of the tracked resource says nothing
about the right to read events. A role granting the resource but not the events
in its own namespace killed the release the same way, and a release in "default"
got opposite verdicts for the same denied right depending on the scope.

Both events call sites are now lenient. Generic events are cosmetic, since
EventStatus never reports a failure; legacy trackers lose the event-based
failure detection, which is unreachable anyway when the feed 403s.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
A registry rejecting disagreeing consumers broke tracking of a v1/Event itself:
its state watcher requests the events of the namespace with the default options
and its events watcher requests the very same resource leniently, so a healthy
release got "conflicting options". Keying the namespaced factory by the options
lets both coexist, and informers are created lazily per resource, so the second
factory costs nothing until the same resource is requested with both options.

The exported signature is variadic again to stay source-compatible for the
library users, but more than one options argument is now an error instead of
being silently dropped. The default warning goes to the display error stream,
which the consumers can redirect, instead of straight to stdout.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The invariant is that consumers disagreeing on the options get separate
informers, which the factory count only implies. The tests now compare the
shared informers themselves, and the display stream is restored to whatever it
was instead of assuming the default.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The tests asserted the watch error classification by calling the handler directly
and the informer separation by the factory key, so the fix itself was uncovered:
dropping the option passthrough in ForNamespace, or the opt-in of either events
call site, kept the whole suite green.

The new tests drive a real reflector over a client denying list and watch of
events and assert which channel the error reaches, for the factory and for both
consumers that opt in.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The sync.Once is one per informer, not one per kind of tolerated error, so a
second non-fatal kind would be swallowed by the first one having fired.

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
@dmmordvi
dmmordvi marked this pull request as ready for review September 12, 2026 14:43
@dmmordvi
dmmordvi merged commit ad1e8a1 into main Sep 12, 2026
1 of 2 checks passed
@dmmordvi
dmmordvi deleted the fix/skip-cluster-scope-events-watcher branch September 12, 2026 14:56
kubeop pushed a commit to kubeop/kubedog that referenced this pull request Oct 8, 2026
werf#401)

Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant