Skip to content

fix(operator): retry deadline JobSet cleanup - #3871

Open
DevaanshPathak wants to merge 1 commit into
kubeflow:masterfrom
DevaanshPathak:fix/issue-3833-deadline-cleanup
Open

fix(operator): retry deadline JobSet cleanup#3871
DevaanshPathak wants to merge 1 commit into
kubeflow:masterfrom
DevaanshPathak:fix/issue-3833-deadline-cleanup

Conversation

@DevaanshPathak

Copy link
Copy Markdown

What this PR does / why we need it:

  • propagates transient JobSet deletion errors after a TrainJob exceeds its active deadline
  • keeps deadline-exceeded TrainJobs eligible for cleanup-only retries while runtime object reconciliation remains skipped for terminal jobs
  • adds a regression test that fails the first JobSet DELETE and verifies the retry removes it

Which issue(s) this PR fixes:
Fixes #3833

Testing:

  • go test ./pkg/controller -run TestReconcileDeadline_RetryJobSetDeletion -count=1
  • go test ./pkg/controller/... -count=1
  • make test
  • make vet
  • golangci-lint on the changed controller files: 0 issues

Checklist:

  • Docs are not required; this fixes internal controller cleanup behavior without changing the API

AI assistance disclosure: Codex assisted with implementation and technical drafting. I reviewed the control flow and verified the changes with the tests and checks listed above.

Propagate transient JobSet deletion failures and keep deadline-exceeded TrainJobs eligible for cleanup-only retries without reconciling runtime objects again.

Assisted-by: OpenAI Codex
Signed-off-by: DevaanshPathak <devaanshpathak08@gmail.com>
Copilot AI balanced review requested due to automatic review settings August 7, 2026 14:49

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@DevaanshPathak DevaanshPathak changed the title fix(controller): retry deadline JobSet cleanup fix(operator): retry deadline JobSet cleanup Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TrainJob does not retry JobSet deletion after active deadline

2 participants