Skip to content

fix(initializer): propagate cache initializer Kubernetes API failures - #3867

Open
sahilleth wants to merge 2 commits into
kubeflow:masterfrom
sahilleth:fix/3835-cache-initializer-k8s-errors
Open

fix(initializer): propagate cache initializer Kubernetes API failures#3867
sahilleth wants to merge 2 commits into
kubeflow:masterfrom
sahilleth:fix/3835-cache-initializer-k8s-errors

Conversation

@sahilleth

@sahilleth sahilleth commented Aug 7, 2026

Copy link
Copy Markdown

Summary

  • Re-raise Kubernetes API errors from cache dataset initializer instead of exiting successfully
  • Treat AlreadyExists (409) as idempotent for ServiceAccount, LeaderWorkerSet, and Service
  • Track resources created in the current invocation and only clean those up on failure (reverse order: Service → LWS → SA)
  • Re-raise the original error after best-effort cleanup; cleanup failures do not mask it

Fixes #3835

Test plan

  • pytest pkg/initializers/dataset/cache_test.py -v (24 tests)
  • pytest pkg/initializers/dataset/main_test.py -v (cache ApiException propagates through main())
  • TrainJob GET failure propagates ApiException
  • ServiceAccount / LeaderWorkerSet / Service create failures propagate and clean up only owned resources
  • All three resource types handle 409 idempotently without deleting pre-existing objects
  • Readiness poll GET failure cleans up all newly created resources
  • Cleanup delete errors are logged without replacing the original failure

Re-raise non-recoverable Kubernetes API errors instead of returning
success, handle LeaderWorkerSet AlreadyExists idempotently, and only
clean up resources created during the current invocation.

Fixes kubeflow#3835

Signed-off-by: Sahil Bhardwaj <sahillbhardwaj23@gmail.com>
@google-oss-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign terrytangyuan for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@google-oss-prow
google-oss-prow Bot requested a review from kuizhiqing August 7, 2026 06:43
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

🎉 Welcome to the Kubeflow Trainer! 🎉

Thanks for opening your first PR! We're happy to have you as part of our community 🚀

Here's what happens next:

  • If you haven't already, please check out our Contributing Guide for repo-specific guidelines and the Kubeflow Contributor Guide for general community standards.
  • Our team will review your PR soon! cc @kubeflow/kubeflow-trainer-team

Join the community:

Feel free to ask questions in the comments if you need any help or clarification!
Thanks again for contributing to Kubeflow! 🙏

Add regression tests for every Kubernetes API failure mode described in
kubeflow#3835, including readiness poll errors, scoped cleanup at each stage,
cleanup error masking, and main() exception propagation.

Signed-off-by: Sahil Bhardwaj <sahillbhardwaj23@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cache initializer silently succeeds on Kubernetes API failures

1 participant