fix(kube-ovn): restore the datapath alerts — the metrics were never gone - #260
Merged
Merged
Conversation
#251 deleted the kube-ovn-datapath group on the grounds that the dp_* family had been dropped in 1.16 with no replacement. That was wrong, and the reason it looked right is now understood. The metrics were absent from every exporter endpoint because the pinger could not collect them: `ovs-dpctl dump-dps` failed with the same stale- pidfile error that made it report ovs and ovn-controller down. #258 gave the pinger hostPID, and the family came back immediately — 78 series, including exactly the names the deleted rules used, under the kube_ovn_ prefix: kube_ovn_dp_flows_lookup_hit 4884614762 kube_ovn_dp_flows_lookup_missed 128981846 kube_ovn_dp_flows_lookup_lost 100 kube_ovn_dp_masks_hit_ratio 6.24 So the three rules are restored with the prefixed names. The label for the node is `hostname`, not `instance`, which the annotations now use. Worth recording how this was got wrong. An adversarial review raised the deletion twice, and it was rejected twice with growing confidence, on the strength of reading all three /metrics endpoints in full and finding no dp_* anywhere. The measurement was sound; the inference from it was not. "The exporter does not expose this" and "the exporter cannot currently collect this" produce identical evidence at the endpoint, and only the second one is reversible. The reviewer was right about the conclusion while being wrong about the reason, which made it easy to dismiss. Verified: all three expressions evaluate against the live stage cluster and return empty, i.e. the thresholds are not currently tripped, with mask hit ratio at 6.24 against a threshold of < 1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The metrics were never gone
#251 deleted the
kube-ovn-datapathgroup on the grounds that thedp_*family had been dropped in 1.16 with no replacement. That was wrong.They were absent from every exporter endpoint because the pinger could not collect them:
ovs-dpctl dump-dpsfailed with the same stale-pidfile error that made it report OVS and ovn-controller down. #258 gave the pingerhostPID, and the family came back immediately — 78 series, under thekube_ovn_prefix, with exactly the names the deleted rules used:kube_ovn_dp_flows_lookup_hitkube_ovn_dp_flows_lookup_missedkube_ovn_dp_flows_lookup_lostkube_ovn_dp_masks_hit_ratioThe three rules are restored with prefixed names. The node label is
hostname, notinstance, which the annotations now use.How this was got wrong
Worth recording, because the failure was in reasoning rather than in measurement.
An adversarial review raised this deletion twice, and it was rejected twice with growing confidence — on the strength of reading all three
/metricsendpoints in full and finding nodp_*anywhere. The measurement was sound. The inference from it was not."The exporter does not expose this" and "the exporter cannot currently collect this" produce identical evidence at the endpoint, and only the second is reversible. The reviewer was right about the conclusion while being wrong about the stated reason, which made it easy to dismiss and easy to feel justified in dismissing.
Verification
All three expressions evaluate against the live stage cluster and return empty — the thresholds are not currently tripped, with mask hit ratio at 6.24 against a threshold of
< 1.kubectl kustomize flux/apps/kube-system/kube-ovn/observabilitybuilds.