OCPBUGS-81500: fix: udpate healthcheck previous state only after successful operator status update

[fracappa]

Previously, getPacemakerStatus updated c.previous before updateOperatorStatus was called. If the status update failed, subsequent syncs would skip processing because c.previous already reflected the new CR timestamp, leaving PacemakerHealthCheckDegraded stuck a True indefinitely.

Summary by CodeRabbit

Bug Fixes

• Improved Pacemaker health check status tracking for more accurate grace-period timing and state transitions.
• Prevented unknown status values from overwriting meaningful health information.
• Improved persistence of health check status between checks.
• Ensured status timestamps remain accurate when health conditions are unchanged.
• Improved consistency between reported health status and recorded events during repeated checks.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci-robot]

@fracappa: This pull request references Jira Issue OCPBUGS-81500, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Previously, getPacemakerStatus updated c.previous before updateOperatorStatus was called. If the status update failed, subsequent syncs would skip processing because c.previous already reflected the new CR timestamp, leaving PacemakerHealthCheckDegraded stuck a True indefinitely.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: dd3be512-6089-4db7-b4a7-79127e48f638

📥 Commits

Reviewing files that changed from the base of the PR and between 31ad611 and fd9198a.

📒 Files selected for processing (1)

• pkg/tnf/pkg/pacemaker/healthcheck.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/tnf/pkg/pacemaker/healthcheck.go

---

Walkthrough

Pacemaker health checks now update the stored non-Unknown status after processing and event recording. Status construction preserves CRLastUpdated. The unchanged-timestamp test initializes the stored previous status.

Changes

Pacemaker health status tracking

Layer / File(s)	Summary
Status synchronization and validation pkg/tnf/pkg/pacemaker/healthcheck.go, pkg/tnf/pkg/pacemaker/healthcheck_test.go	sync stores non-Unknown status after processing and event recording. Status construction uses the lowercase health-status builder and preserves CRLastUpdated. The test stores the first returned status before the second call.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jaypoulz

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error, 1 warning)

Check name	Status	Explanation	Resolution
Ote Binary Stdout Contract	❌ Error	The OTE binary calls klog.Fatal in main() and Cobra setup, but no klog.SetOutput(os.Stderr) or klog.LogToStderr(true) exists in the process-level code.	Configure klog to write to os.Stderr at the start of main(), before newOperatorTestCommand() or any logging call.
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (13 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the fix to update healthcheck previous state only after a successful operator status update.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	The changed test uses a stable Go test name, TestHealthCheck_getPacemakerStatus_UnchangedTimestamp; no Ginkgo title declarations or dynamic test-title values were added.
Test Structure And Quality	✅ Passed	The changed test is a focused standard Go unit test, not Ginkgo; it uses fake clients, performs no cluster waits or resource creation, and all assertions have diagnostic messages.
Microshift Test Compatibility	✅ Passed	The only test change is in a standard Go TestHealthCheck_getPacemakerStatus_UnchangedTimestamp unit test; no new Ginkgo e2e tests or MicroShift-sensitive API usage was added.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	The commit changes only healthcheck.go and a standard Go testing unit test; it adds no Ginkgo e2e test or SNO-sensitive multi-node assumption.
Topology-Aware Scheduling Compatibility	✅ Passed	The patch changes only Pacemaker health-check state handling and its test; it adds no manifests, replicas, affinity, topology spread, selectors, tolerations, or disruption budgets.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	The commit changes only pacemaker controller code and a standard Go unit test; it adds no Ginkgo e2e test and no new network or external-connectivity dependency.
No-Weak-Crypto	✅ Passed	The parent-to-HEAD diff only changes Pacemaker health-status state handling and a test; it adds no weak crypto, custom crypto, or secret/token comparisons.
Container-Privileges	✅ Passed	The patch changes only two Go files. No manifest files or privilege indicators were added, and existing privileged manifests are unchanged.
No-Sensitive-Data-In-Logs	✅ Passed	The PR adds no logging statements or log arguments. It only moves state assignment, changes status construction, and updates a test; existing status logs are unchanged.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.12.2)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[openshift-ci-robot]

@fracappa: This pull request references Jira Issue OCPBUGS-81500, which is valid.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (5.0.0) matches configured target version for branch (5.0.0)
• bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

Details

In response to this:

Previously, getPacemakerStatus updated c.previous before updateOperatorStatus was called. If the status update failed, subsequent syncs would skip processing because c.previous already reflected the new CR timestamp, leaving PacemakerHealthCheckDegraded stuck a True indefinitely.

Summary by CodeRabbit

Bug Fixes

• Corrected Pacemaker health check status tracking to ensure accurate grace-period timing and state transitions
• Fixed handling of unknown status values during operator condition updates
• Improved reliability of health check status persistence logic

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[fonta-rh]

LGTM — the fix correctly moves c.previous assignment from getPacemakerStatus() to after updateOperatorStatus() succeeds in sync(), so a failed status write no longer poisons the skip-optimization cache.

One minor nit: the doc comment on getPacemakerStatus() (line 254) still says "For Unknown status, previous is not updated" — this referred to c.previous being written inside the function, which no longer happens for any status. Consider removing that line or replacing with something like:

// Note: c.previous is updated by sync() after successful status write, not here.

Non-blocking, cosmetic only.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: fonta-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• pkg/tnf/OWNERS [fonta-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[fonta-rh]

/lgtm

[openshift-merge-bot]

Scheduling required tests:
/test e2e-agnostic-ovn
/test e2e-agnostic-ovn-upgrade
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-aws-ovn-single-node
/test e2e-gcp-operator
/test e2e-gcp-operator-disruptive
/test e2e-metal-ipi-ovn-ipv6
/test e2e-operator

[fracappa]

/retest-required

[fracappa]

/retest-required

[coderabbitai]

Note

GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/tnf/pkg/pacemaker/healthcheck.go`:
- Around line 268-274: Remove the pre-update assignment to c.previous in
getPacemakerStatus, leaving only the post-success assignment guarded by
currentStatus.OverallStatus != statusUnknown. Ensure failed updateOperatorStatus
calls leave the prior state intact so the next sync does not skip the status
change, and add a regression test covering that failure path.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 1b78b6f2-89dc-48cc-9862-837997f3c2d0

📥 Commits

Reviewing files that changed from the base of the PR and between fb904c8 and d12813a.

📒 Files selected for processing (2)

• pkg/tnf/pkg/pacemaker/healthcheck.go
• pkg/tnf/pkg/pacemaker/healthcheck_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/tnf/pkg/pacemaker/healthcheck_test.go

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/tnf/pkg/pacemaker/healthcheck.go`:
- Around line 268-274: Update the guard in the pacemaker health-check processing
flow to compare currentStatus.OverallStatus against the declared StatusUnknown
constant, replacing the undefined statusUnknown reference while preserving the
existing previous-status update behavior.
- Around line 314-316: Update the health-status construction around
buildHealthStatusFromCR so currentStatus.CRLastUpdated receives the
PacemakerCluster status.lastUpdated value from pacemakerCR, replacing the
undeclared crLastUpdated reference. Preserve the existing
buildHealthStatusFromCR flow.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3ef598e5-96e9-4b50-a4eb-35163df83e7c

📥 Commits

Reviewing files that changed from the base of the PR and between d12813a and 31ad611.

📒 Files selected for processing (1)

• pkg/tnf/pkg/pacemaker/healthcheck.go

[fonta-rh]

/lgtm

[fonta-rh]

/retest

[fracappa]

/retest

[fracappa]

/retest

[openshift-ci]

New changes are detected. LGTM label has been removed.

[fracappa]

/retest-required

[fracappa]

/retest-required

[openshift-ci]

@fracappa: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-agnostic-ovn	e897f46	link	true	/test e2e-agnostic-ovn
ci/prow/e2e-aws-ovn-single-node	e897f46	link	true	/test e2e-aws-ovn-single-node
ci/prow/e2e-aws-ovn-serial-2of2	e897f46	link	true	/test e2e-aws-ovn-serial-2of2
ci/prow/configmap-scale	fd9198a	link	false	/test configmap-scale
ci/prow/unit	fd9198a	link	true	/test unit
ci/prow/periodics-images	fd9198a	link	true	/test periodics-images
ci/prow/okd-scos-images	fd9198a	link	true	/test okd-scos-images
ci/prow/images	fd9198a	link	true	/test images
ci/prow/verify	fd9198a	link	true	/test verify

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.