Repository navigation
fix(informer, tracker): warn instead of failing on inaccessible events - #401
Merged
Merged
Conversation
…o events
Events of cluster-scoped resources are stored in the "default" namespace, so
ResourceEventsWatcher lists and watches v1/events there. A user allowed to read
the tracked resource is not necessarily allowed to read events in "default",
and the resulting "403 Forbidden" from the reflector was reported through
watchErrCh as an unrecoverable watch error, which aborts the whole tracking.
Losing the events feed shouldn't stop tracking the resource itself.
The events informer of a cluster-scoped resource is now created with
InformerOptions{ForbiddenIsNotFatal: true}, and its watch error handler treats
apierrors.IsForbidden the same way it already treats EOF and ResourceExpired:
logs it once under KUBEDOG_DEBUG and lets the reflector keep retrying. Only
!namespaced resources get the leniency; for namespaced ones a 403 on events
still means the tracking scope is wrong, and stays fatal.
A SharedIndexInformer has a single watch error handler which can only be set
once, so consumers disagreeing on the policy must not share an informer.
The policy is therefore part of the factory-map key, which keeps the two
policies on separate informers and avoids depending on which consumer happened
to create the informer first. A namespace name can't contain a slash, so the
"<namespace>/forbidden-is-not-fatal" key never collides with a plain namespace.
The cost is a second events informer on "default" for a release deployed into
"default" that has both cluster-scoped and namespaced resources.
Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
A "403 Forbidden" tolerated via ForbiddenIsNotFatal was only printed under KUBEDOG_TRACKER_DEBUG=1, so by default the user lost the feed with no explanation at all. The factory now takes OnNonFatalWatchError, which defaults to printing a warning naming the resource and the namespace, and the sync.Once wraps the reporting itself rather than the debug check. Consumers agreeing on the policy share one informer, and one informer has one watch error handler, so the options are now stored per resource and namespace and a disagreeing consumer gets an error instead of a silently split factory, a duplicate reflector, and a policy decided by creation order. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Leniency was limited to cluster-scoped resources because that is how the missing access was reported, but the scope of the tracked resource says nothing about the right to read events. A role granting the resource but not the events in its own namespace killed the release the same way, and a release in "default" got opposite verdicts for the same denied right depending on the scope. Both events call sites are now lenient. Generic events are cosmetic, since EventStatus never reports a failure; legacy trackers lose the event-based failure detection, which is unreachable anyway when the feed 403s. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
A registry rejecting disagreeing consumers broke tracking of a v1/Event itself: its state watcher requests the events of the namespace with the default options and its events watcher requests the very same resource leniently, so a healthy release got "conflicting options". Keying the namespaced factory by the options lets both coexist, and informers are created lazily per resource, so the second factory costs nothing until the same resource is requested with both options. The exported signature is variadic again to stay source-compatible for the library users, but more than one options argument is now an error instead of being silently dropped. The default warning goes to the display error stream, which the consumers can redirect, instead of straight to stdout. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The invariant is that consumers disagreeing on the options get separate informers, which the factory count only implies. The tests now compare the shared informers themselves, and the display stream is restored to whatever it was instead of assuming the default. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The tests asserted the watch error classification by calling the handler directly and the informer separation by the factory key, so the fix itself was uncovered: dropping the option passthrough in ForNamespace, or the opt-in of either events call site, kept the whole suite green. The new tests drive a real reflector over a client denying list and watch of events and assert which channel the error reaches, for the factory and for both consumers that opt in. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
The sync.Once is one per informer, not one per kind of tolerated error, so a second non-fatal kind would be swallowed by the first one having fired. Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
dmmordvi
marked this pull request as ready for review
September 12, 2026 14:43
kubeop
pushed a commit
to kubeop/kubedog
that referenced
this pull request
Oct 8, 2026
werf#401) Signed-off-by: Dmitry Mordvinov <dmitry.mordvinov@flant.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Tracking a cluster-scoped resource aborted the whole release when the user had no access to events. Events of a cluster-scoped resource are stored in the
defaultnamespace, soResourceEventsWatcherlists and watchesv1/eventsthere; a user allowed to read the tracked resource is not necessarily allowed to read events indefault. The reflector's403 Forbiddenwas reported throughwatchErrChas an unrecoverable watch error, which cancels the tracking context.Repro: grant a role with
get/list/watchon some cluster-scoped kind but nothing oneventsindefault, then deploy a release containing that kind. Expected: the resource is tracked, with a warning that its events are unavailable. Actual:unrecoverable watch error for /v1, Resource=events: ... forbidden, release fails.Key changes
InformerOptions{ForbiddenIsNotFatal}makes the watch error handler treatapierrors.IsForbiddenthe way it already treats EOF andResourceExpired— the reflector keeps retrying, nothing reacheswatchErrCh.ResourceEventsWatcherand the legacyevent.EventInformer. The leniency does not depend on the scope of the tracked resource, because the scope says nothing about the right to read events — a role granting a namespaced resource but not the events of its own namespace failed a release exactly the same way.ConcurrentInformerFactoryOptions.OnNonFatalWatchErrorreceives the resource, the namespace and the error, defaulting to a warning on the display output stream. Reported once per informer, since the reflector retries indefinitely.{namespace, options}: aSharedIndexInformerhas a single watch error handler which can only be set once, so sharing would make the policy depend on which consumer created the informer first.ForNamespacenow rejects more than oneInformerOptionsargument instead of silently dropping the extras.Why
A missing auxiliary feed should degrade the tracking, not abort the release. The alternative — a
SelfSubjectAccessReviewprecheck before creating the informer — was rejected: it is a time-of-check/time-of-use snapshot (access can be revoked mid-release, which still must not kill the release), it needs bothlistandwatchchecked, andcreate selfsubjectaccessreviewscan itself be denied in hardened clusters. Runtime handling is required regardless, so the precheck would only add an earlier message.Verification
listandwatchof events assert which channel the error reaches: forForNamespacewith and without the option, forResourceEventsWatcherwith a cluster-scoped and a namespaced resource, and forevent.EventInformer. Without them the fix was untested — dropping the option passthrough inForNamespace, or the opt-in of either events call site, kept the whole suite green.ai_testsbuild tag, which CI does not run — so they need-tags ai_teststo execute.events. The failing path was traced throughclient-go@v0.29.3reflector.gorather than reproduced end to end.Review focus / risks
event.EventInformerturns an event whoseReasoncontains "Failed" into a resource failure and pod probe events intoProbeTriggeredRestart/ReadinessProbeFailure; the generic tracker fails onEventStatus.IsFailure(), which never fires today, so for it the loss is cosmetic.ImagePullBackOff/ErrImagePull/CrashLoopBackOffcome from the pod status rather than from events and still fail fast; probe failures,FailedSchedulingandFailedMountnow surface vianoActivityTimeoutinstead. That detection is unreachable in this scenario anyway, since today the release dies outright. Readiness itself is derived from the resource and unaffected.v1/Eventitself is the one case where the same resource and namespace is requested with both options, producing two reflectors oneventsin that namespace.dynamicinformercreates informers lazily per resource, so the second factory is otherwise free.ForbiddenIsNotFatalis exposed onForNamespacefor any resource, including a primary readiness watch. Narrowing it to events would need either a hardcoded GVR in the factory or a separate event-feed abstraction; for now only the two events call sites pass it.