A footgun related to:
The content-addressed kubenix/hash label that gets stamped onto every object conflicted with Valkey CRs: valkey-operator doesn't reject such CRs, but accepts them and propagates the CR's labels onto the resources it reconciles, including the Service's selector and onto managed pods.
Because the hash is derived from the rendered content, any edit to the Valkey CR changes kubenix/hash; the operator rewrites the Service selector to the new hash but doesn't re-label the already-running pods, so the selector matches zero pods and the Service falls to 0 endpoints. So I self-inflicted an outage.
The deeper issue is that kubenix labels every resource as if it owns it end-to-end, but that isn't true for an operator's CR: kubenix owns the CR spec, while the operator owns everything the CR reconciles into (Service, StatefulSet, pods). Operators that treat labels on their CRs as semantically meaningful may copy them downstream and quietly turn kubenix's bookkeeping label into part of a live selector. The only way I can manipulate this is addHashLabel, and it's global. There's no per-resource or per-GVK opt-out. So the choice is the hash label on everything or nothing.
I fix this by post-processing the generated List and strip kubenix/* from a small list of group/kind pairs (just hyperspike.io/Valkey so far), but I'd like something slightly less hacky, perhaps related to CRs in general, and perhaps with either a property test or a linter warning.
A footgun related to:
The content-addressed
kubenix/hashlabel that gets stamped onto every object conflicted with Valkey CRs: valkey-operator doesn't reject such CRs, but accepts them and propagates the CR's labels onto the resources it reconciles, including the Service's selector and onto managed pods.Because the hash is derived from the rendered content, any edit to the Valkey CR changes kubenix/hash; the operator rewrites the Service selector to the new hash but doesn't re-label the already-running pods, so the selector matches zero pods and the Service falls to 0 endpoints. So I self-inflicted an outage.
The deeper issue is that kubenix labels every resource as if it owns it end-to-end, but that isn't true for an operator's CR: kubenix owns the CR spec, while the operator owns everything the CR reconciles into (Service, StatefulSet, pods). Operators that treat labels on their CRs as semantically meaningful may copy them downstream and quietly turn kubenix's bookkeeping label into part of a live selector. The only way I can manipulate this is
addHashLabel, and it's global. There's no per-resource or per-GVK opt-out. So the choice is the hash label on everything or nothing.I fix this by post-processing the generated List and strip kubenix/* from a small list of group/kind pairs (just hyperspike.io/Valkey so far), but I'd like something slightly less hacky, perhaps related to CRs in general, and perhaps with either a property test or a linter warning.