feat: KEP-3744: Store trial history natively in OptimizationJob status

[Sridhar1030]

What this PR does / why we need it:

This PR introduces KEP-3744, proposing native trial history in OptimizationJob.status.trials.

In the Phase 1 design of KEP-3562, the controller reconstructs trial history from child TrainJob annotations, so deleting any trial TrainJob (manually, or via TTL/GC cleanup) silently corrupts a running optimization. This KEP makes the OptimizationJob self-contained: each trial's record is written at creation and finalized once at terminal state, the suggestion snapshot is assembled entirely from status, and no child finalizers are needed.

Raised during the KEP-3562 API review: #3565 (comment)

Which issue(s) this PR fixes:
Part of #3744

Does this PR introduce a user-facing change?
No, design proposal only.

Checklist:

• Docs included if any changes are user facing
• KEP doc included

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Krishna-kg732]

Left a couple of status suggestions on #3552 that overlap with this — result becoming a list, and recording the winning trial's objective value. Both are cheaper while that PR is still open, so maybe worth landing them there and building this on top? Happy either way, mostly just don't want us to end up with two status designs to reconcile later.

One thing I'd find useful in the proposal: where does the boundary sit between what the controller reconstructs and what status owns? If OptimizationJobStatus becomes the source of truth for history, then buildSuggestionRequest() would read from status rather than the opt-param-* annotations, and those drop to being a debugging aid. That seems like a good simplification to me, but it does change KEP-3562 §8.2 and §8.4 — might be worth calling out explicitly so it reads as a deliberate change rather than an implied one.

Also, have you thought about how to bound the growth? 100 trials × up to 100 parameters in a single status object gets fairly large, and I think that's part of why Katib kept trial state in separate CRs.

[Sridhar1030]

Thanks @Krishna-kg732! On growth: this is covered in Risks under "Status object growth" (capped list, and the controller checks the bound before appending, so the failure mode is an explicit condition rather than a rejected status write); curious whether you think the cap needs to be lower. On #3552: would you prefer result become a list there first and this KEP build on top, or keep result as a projection of trials and land the shape change later? And on the KEP-3562 boundary: would an explicit note in Compatibility, calling out that this deliberately amends §8.2 and §8.4, address your concern?