fix(kube-ovn): give ovn-central hostPID so its pidfile means something - #263
Merged
Conversation
The last of the four alerts, OVNComponentUnhealthy, survived #258 and #261 and needed a different fix from the other three. #261 gave kube-ovn-monitor hostPID, and that part worked — the error message changed from "being deleted by pid 0" to "by pid 234187", i.e. the monitor now sees real host PIDs. It still cannot verify northd, because the PID it is checking does not exist on the host: /var/run/ovn/ovn-northd.pid: stale pidfile for pid 544 544 is northd's PID inside ovn-central's own namespace. ovn-central runs without hostPID, so that is what northd writes into a pidfile living in a hostPath shared with every other kube-ovn component. Nobody outside that one container can make sense of it. This is why kube-ovn-cni and ovs-ovn have always worked: they run with hostPID: true, so the pidfiles they write carry host PIDs, and the pinger's checks against them passed the moment #258 landed. ovn-central is now the only writer left in the shared directory whose PIDs are namespace-local. Same postRenderer, third target. NOTE FOR ROLLOUT, unlike the previous two: this restarts the OVN control plane. ovn-central runs a single replica here, so northd, the northbound and the southbound databases all go down briefly while the pod is replaced. Existing datapath flows keep forwarding — ovn-controller and OVS are untouched — but no new logical configuration is programmed until it is back. Worth doing at a quiet moment rather than alongside other changes. Verified the same two ways as the previous patches: the JSON6902 patch run through kustomize against the live Deployment flips hostPID from unset to true without disturbing the pod spec, and the manifest builds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The last of the four, and it needed a different fix
OVNComponentUnhealthysurvived both #258 and #261.#261 gave
kube-ovn-monitorhostPID, and that part worked — the error changed frombeing deleted by pid 0toby pid 234187, i.e. the monitor now sees real host PIDs. It still cannot verify northd, because the PID it checks does not exist on the host:544 is northd's PID inside ovn-central's own namespace.
ovn-centralruns withouthostPID, so that is what northd writes into a pidfile living in a hostPath shared with every other kube-ovn component. Nobody outside that one container can make sense of it.Why the other two worked immediately
kube-ovn-cniandovs-ovnhave always run withhostPID: true, so the pidfiles they write carry host PIDs — which is why the pinger's checks against ovs-vswitchd, ovsdb-server and ovn-controller all passed the moment #258 landed.ovn-centralis now the only writer left in that shared directory whose PIDs are namespace-local.Rollout note — this one is not free
Unlike #258 and #261, this restarts the OVN control plane.
ovn-centralruns a single replica here, so northd and the northbound and southbound databases all go down briefly while the pod is replaced.Existing datapath flows keep forwarding —
ovn-controllerand OVS are untouched — but no new logical configuration is programmed until it is back. Worth doing at a quiet moment rather than alongside other changes.Verification
Same two ways as the previous patches: the JSON6902 patch run through kustomize against the live Deployment flips
hostPIDfrom unset totruewithout disturbing the pod spec, andkubectl kustomize flux/apps/kube-system/kube-ovn/appbuilds.