OLS-3566 Add terminal-run TTL cleanup and oc agentic run cleanup CLI

.ai/spec/how/cli.md

@@ -49,6 +49,7 @@ Audience: AI agents. Command behavior and user-facing rules belong in **what/**
 | `delete.go` | `DeleteOptions` | `NewDeleteCmd`, `Complete`, `Run` |
 | `watch.go` | `WatchOptions`; package var `agenticRunGVR` | `NewWatchCmd`, `Complete`, `Run`, `doWatch`, `extractConditions` |
 | `logs.go` | `LogsOptions` | `NewLogsCmd`, `Complete`, `Validate`, `Run`, `resolveSandbox` |
+| `cleanup.go` | `CleanupOptions` | `NewCleanupCmd`, `Complete`, `Validate`, `Run`, `printRunsTable`, `isTerminalPhaseIncludingNoAction`, `parseDuration` |
 
 `*_test.go` files under `cli/` exercise commands (not fully enumerated here).
 
@@ -66,7 +67,8 @@ oc-agentic
 │   ├── deny NAME
 │   ├── watch NAME
 │   ├── logs NAME
-│   └── delete NAME
+│   ├── delete NAME
+│   └── cleanup
 ├── status
 ├── suspend
 ├── resume
@@ -103,6 +105,7 @@ There is **no** unstructured client for run CRUD in the main commands; only watc
 - **`approve`:** Loads `AgenticRun`; `getOrCreateApproval` (get or create `AgenticRunApproval` with owner ref — create path omits controller flags present in controller’s `ensureAgenticRunApproval`; operator reconciler may enrich). Builds `[]ApprovalStage` entries, `client.Patch(MergeFrom)` on approval. `--all` uses `pendingStages` derived from spec non-zero steps vs existing stage types. `--wait` delegates to `doWatch`.
 - **`deny`:** Requires existing `AgenticRunApproval`; appends denied stage with `ApprovalDecisionDenied`. `--stage` defaults via `nextPendingStage` walk order analysis → execution → verification.
 - **`delete`:** `client.Delete` minimal `AgenticRun` object keyed by name/namespace.
+- **`cleanup`:** `client.List` `AgenticRunList` (namespace or `-A` cluster-wide), client-side filters to terminal phases (including `NoActionRequired`, which `IsTerminalPhase` does not cover — see `isTerminalPhaseIncludingNoAction`) matching optional `--state` and `--older-than` (against `status.terminalTime`; runs missing it are skipped with a warning when `--older-than` is set). `--dry-run` prints the match table and exits without prompting or deleting. Otherwise prints the match table and prompts for confirmation (`[y/N]`), skippable with `--yes`/`-y` (same pattern as `cli/system/suspend.go`), before calling `client.Delete` per run; per-run delete failures are reported but do not halt the batch.
 - **`watch`:** Dynamic watch; `extractConditions` pulls `status.conditions` into `[]metav1.Condition`; phase from `DerivePhase`; prints only on phase change; stops when `IsTerminalPhase` (Completed, Failed, Escalated, Denied — note helpers.go set).
 - **`logs`:** Loads run via controller-runtime client; `resolveSandbox` picks explicit `--step` (normalized via `NormalizeStep`) or prefers verification, then execution, then analysis sandbox info. Uses **`SandboxInfo.ClaimName` as pod name** and `SandboxInfo.Namespace` (fallback run namespace). Streams with optional `-f`.
 

.ai/spec/what/crd-api.md

@@ -9,7 +9,9 @@ Kubernetes API surface for the agentic operator. **Lifecycle and gates** are in
 3. **Scope — cluster**: `Agent`, `LLMProvider`, `ApprovalPolicy`, and `AgenticOLSConfig` MUST be cluster-scoped; `metadata.name` is the global identifier.
 4. **AgenticRun identity**: A `AgenticRun` MUST include required immutable fields per CEL: at minimum `spec.request` and `spec.analysis`. Omitting `spec.execution` or `spec.verification` means those steps do not exist for that run (see `run-lifecycle.md`).
 5. **AgenticRun — `spec.request`**: Human/agent input text; immutable after creation; max length enforced by validation.
-6. **AgenticRun — `spec.revisionFeedback`**: Only mutable spec field; when set/non-empty and `metadata.generation` advances beyond the analyzed condition’s `observedGeneration`, operators MUST trigger re-analysis per `run-lifecycle.md`.
+6. **AgenticRun — `spec.revisionFeedback`**: User-mutable spec field for iterative feedback; when set/non-empty and `metadata.generation` advances beyond the analyzed condition’s `observedGeneration`, operators MUST trigger re-analysis per `run-lifecycle.md`. `spec.ttlAfterTerminal` (rule 6a) is the only other mutable spec field; the controller MUST advance `Analyzed.observedGeneration` in the same write whenever it patches `ttlAfterTerminal`, so that its own generation bump is never misread as an outstanding revision request.
+6a. **AgenticRun — `spec.ttlAfterTerminal`**: Optional time-to-live in seconds after the run reaches a terminal state (see `run-lifecycle.md` rule 4 for the terminal set), after which the operator deletes the `AgenticRun` CR (Kubernetes GC cascades to owned resources via owner references). Mutable — adapters or admins MAY pre-set it before the run reaches terminal state; once terminal, if unset, the operator stamps it from `AgenticOLSConfig.spec.lifecycle.terminalTTL` (cluster default) without overwriting a pre-set value. `0` explicitly disables auto-deletion for that run. When omitted and no cluster default exists, no automatic deletion occurs (backwards-compatible). See `run-lifecycle.md` rule 23.
+6b. **AgenticRun — `status.terminalTime`**: Timestamp the operator stamps once, the first time a run reaches a terminal state; not updated again while the run remains terminal. Cleared by the revision handler when a terminal run re-enters analysis (rule 6/23), so a subsequent terminal phase gets a fresh timestamp instead of reusing the prior terminal event's. Used with `spec.ttlAfterTerminal` to compute the deletion deadline.
 7. **AgenticRun — `spec.targetNamespaces`**: Optional list of namespaces for context and RBAC targeting; immutable once set; when empty, RBAC targeting MAY fall back to namespaces declared in analysis RBAC output at execution time (see `sandbox-execution.md`).
 8. **AgenticRun — `spec.analysisOutput`**: Immutable after set. `mode` defaults to full analysis schema when empty/default. `mode=Minimal` REQUIRES `schema` to be set, forbids `spec.execution` and `spec.verification`, and restricts option shape accordingly.
 9. **AgenticRun — `spec.tools`**: Default `ToolsSpec` for all steps; immutable once set. Per-step `tools` on `spec.analysis` / `spec.execution` / `spec.verification` replaces the default for that step only when non-zero.
@@ -52,13 +54,14 @@ Kubernetes API surface for the agentic operator. **Lifecycle and gates** are in
 45. **AgenticOLSConfig — absence**: When no `AgenticOLSConfig` CR exists, the system MUST behave as if `spec.suspended` is `false`.
 46. **AgenticOLSConfig — status subresource**: `AgenticOLSConfig` MUST have a `/status` subresource with `conditions` array (`metav1.Condition`). Condition type `Suspended` tracks whether the operator has acknowledged and acted on `spec.suspended`. See `system-config.md` rules 5a–5e for full semantics.
 47. **AgenticOLSConfig — status RBAC**: The operator’s service account MUST have `get`, `update`, `patch` on `agenticolsconfigs/status` in addition to existing permissions on the main resource.
+48. **AgenticOLSConfig — `spec.lifecycle.terminalTTL`**: Optional cluster-wide default time-to-live in seconds for terminal `AgenticRun` resources, applied only when a given run's `spec.ttlAfterTerminal` is unset (per-run override always wins — see rule 6a). `spec.lifecycle` requires at least one property when present (`minProperties: 1`) and is omitted entirely from the object when zero-valued. When omitted or no `AgenticOLSConfig` exists, no cluster-wide default gets stamped, so runs relying on the default are never auto-deleted — but this does NOT affect runs that already carry a pre-set `spec.ttlAfterTerminal` independently of the cluster config; those are still deleted on schedule regardless of whether `AgenticOLSConfig` exists.
 
 ## Configuration Surface (by path)
 
 ### AgenticRun
 - `metadata.*`
-- `spec.request`, `spec.targetNamespaces`, `spec.revisionFeedback`, `spec.analysisOutput`, `spec.tools`, `spec.analysis`, `spec.execution`, `spec.verification`
-- `status.conditions`, `status.steps.analysis|execution|verification|escalation.*`
+- `spec.request`, `spec.targetNamespaces`, `spec.revisionFeedback`, `spec.analysisOutput`, `spec.tools`, `spec.analysis`, `spec.execution`, `spec.verification`, `spec.ttlAfterTerminal`
+- `status.conditions`, `status.steps.analysis|execution|verification|escalation.*`, `status.terminalTime`
 
 ### Agent
 - `metadata.name`, `spec.llmProvider.name`, `spec.model`, `spec.reasoningConfig`, `spec.timeouts.*`, `spec.maxTurns`, `status.conditions`
@@ -70,8 +73,9 @@ Kubernetes API surface for the agentic operator. **Lifecycle and gates** are in
 - `metadata.name` (must be `cluster`), `spec.stages[]`, `spec.maxAttempts`, `spec.maxConcurrentRuns`
 
 ### AgenticOLSConfig
-- `metadata.name` (must be `cluster`), `spec.suspended`, `spec.templog`
+- `metadata.name` (must be `cluster`), `spec.suspended`, `spec.templog`, `spec.lifecycle.terminalTTL`
 - `spec.templog` (bool, default `true`): When `true` or absent, the lightspeed-operator deploys a custom OTel Collector for temporary audit log storage in PostgreSQL. See `templog.md`.
+- `spec.lifecycle.terminalTTL` (optional int, seconds): Cluster-wide default TTL for terminal `AgenticRun` deletion; see rule 48.
 - `status.conditions` — condition types: `Suspended`
 - See `system-config.md` for full behavioral rules
 
@@ -100,3 +104,4 @@ Kubernetes API surface for the agentic operator. **Lifecycle and gates** are in
 - [OLS-3328] Add `spec.templog` to `AgenticOLSConfig` CRD for temporary audit log storage.
 - [DONE: OLS-3295] Renamed `Proposal` → `AgenticRun`, `ProposalApproval` → `AgenticRunApproval` CRD kinds and all associated field names, RBAC resources, and label keys.
 - [PLANNED: OLS-3594] Optional `disableDefaultMCP` (and related auto-injection) — deferred; blocked by OLS-3526 and OLS-3572. Not near-term.
+- [DONE: OLS-3566] Added `AgenticOLSConfig.spec.lifecycle.terminalTTL` (cluster default) and `AgenticRun.spec.ttlAfterTerminal` / `status.terminalTime` (per-run override + stamp) for automatic terminal-run cleanup. Added `oc agentic run cleanup` CLI command for manual batch cleanup (see `how/cli.md`).

.ai/spec/what/run-lifecycle.md

@@ -42,9 +42,11 @@ Behavioral specification for the `AgenticRun` resource lifecycle. **Approval gat
 17. **Escalation failure**: `Escalated` with status `False` MUST yield `Failed` once rule 9 evaluates the `Escalated` presence branch (non-`True`, non-`Unknown`).
 18. **Result CR linkage**: Each analysis/execution/verification/escalation attempt SHOULD append a `status.steps.*.results[]` entry naming the corresponding result resource with an outcome matching agent success/failure for that attempt. **Exception:** when the execution agent reports `success=false` but all mutating actions succeeded (only inline verification checks failed), the controller MUST override the outcome to `Succeeded` and proceed to the verification step. Observation action types (`pre-check`, `post-check`, `verification`, `check`, `wait`) are not considered when determining mutation success.
 19. **Observed generation**: Conditions SHOULD carry `observedGeneration` aligned with `metadata.generation` when the controller updates them for the current spec generation, except revision completion MAY pin the analyzed condition to the generation that triggered the revision, per existing reconciliation behavior.
-20. **Immutable spec (excluding revision)**: Once set, `spec.request`, `spec.targetNamespaces`, `spec.analysisOutput`, `spec.tools`, `spec.analysis`, `spec.execution`, and `spec.verification` MUST NOT change; CEL on the CRD enforces this. Only `spec.revisionFeedback` is mutable for iterative feedback.
+20. **Immutable spec (excluding revision and TTL)**: Once set, `spec.request`, `spec.targetNamespaces`, `spec.analysisOutput`, `spec.tools`, `spec.analysis`, `spec.execution`, and `spec.verification` MUST NOT change; CEL on the CRD enforces this. `spec.revisionFeedback` (iterative feedback) and `spec.ttlAfterTerminal` (terminal-run TTL override, rule 23) are the two mutable spec fields; see rule 24 for how the controller keeps its own `ttlAfterTerminal` writes from being misread as a revision request.
 21. **Option trim after analysis**: When multiple remediation options exist, execution MUST use the option selected through the approval resource; non-selected options MAY be removed from the stored analysis result before execution (see `approval.md`).
 22. **Selected option for verification**: Verification MUST use the same selected remediation option as execution (latest trimmed analysis result).
+23. **Terminal-run TTL / auto-deletion**: On every reconcile of a terminal run (rule 4), the controller MUST: (a) stamp `status.terminalTime` once, the first time the run is observed terminal (this stamping is unconditional and independent of whether any cluster TTL config exists); (b) if `spec.ttlAfterTerminal` is unset, stamp it from `AgenticOLSConfig.spec.lifecycle.terminalTTL` when that cluster default exists (never overwriting a pre-set value) — when no cluster default exists, no default is stamped, but this does NOT suppress deletion of a run whose `ttlAfterTerminal` was already pre-set independently of the cluster config; (c) once `spec.ttlAfterTerminal` is a non-zero value, however it got set, delete the `AgenticRun` once `status.terminalTime + ttlAfterTerminal` has elapsed (Kubernetes GC cascades to owned resources), otherwise re-queue for the remaining duration. `ttlAfterTerminal == 0` explicitly disables auto-deletion for that run. When no TTL is ever configured (no cluster default and no per-run override), the run is never auto-deleted. The `AgenticOLSConfig`/`ApprovalPolicy`/watched-`ConfigMap` fan-out MUST re-enqueue terminal runs still missing `status.terminalTime` (unconditional), and MUST re-enqueue terminal runs missing `spec.ttlAfterTerminal` only when an effective cluster-wide TTL is currently configured — never when no cluster default exists, since there would be nothing to stamp and re-enqueuing every such run on every config-adjacent change would be pure churn. When a terminal run re-enters revision (rule 6), the revision handler MUST clear `status.terminalTime` so a later terminal phase gets a fresh timestamp rather than computing TTL expiry off the earlier terminal event. See `crd-api.md` rules 6a, 6b, 48.
+24. **TTL stamping MUST NOT desynchronize revision detection**: Stamping `spec.ttlAfterTerminal` (rule 23b) is a spec write and therefore advances `metadata.generation` like any other spec mutation. Because rule 6 (generation vs. `Analyzed.observedGeneration`) does not distinguish which spec field changed, the controller MUST advance `Analyzed.observedGeneration` to the post-stamp `metadata.generation` in the same operation, so this internal, non-user-initiated generation bump can never be misread as a new revision request (which would otherwise be possible via stale, previously-processed `spec.revisionFeedback`, since that field is never cleared — see rule 6).
 
 ## Configuration Surface
 
@@ -58,6 +60,7 @@ Behavioral specification for the `AgenticRun` resource lifecycle. **Approval gat
 - `status.conditions[*].type`, `status.conditions[*].status`, `status.conditions[*].reason`, `status.conditions[*].observedGeneration`
 - `status.steps.execution.retryCount`
 - `status.steps.*.results`, `status.steps.*.sandbox`
+- `spec.ttlAfterTerminal`, `status.terminalTime` (terminal-run auto-deletion, rules 23–24)
 
 ## Constraints
 
@@ -73,3 +76,4 @@ Behavioral specification for the `AgenticRun` resource lifecycle. **Approval gat
 - [PLANNED: OLS-3268] `NoActionRequired` terminal phase: when analysis returns `actionRequired=false`, the operator sets `Analyzed=True` with reason `NoActionRequired` and the run auto-completes, bypassing approval/execution/verification.
 - [DONE: OLS-3295] Renamed `Proposal` CRD kind to `AgenticRun`, `ProposalApproval` to `AgenticRunApproval`, and updated all associated API surface (labels, RBAC resources, CLI commands, audit events, OTEL spans).
 - [DONE: OLS-3558] Execution outcome override — controller no longer hard-fails when `success=false` but all mutating actions succeeded; defers outcome to the verification step. See `sandbox-execution.md` rule 21b.
+- [DONE: OLS-3566] Terminal-run TTL / auto-deletion added (rules 23–24); `oc agentic run cleanup` CLI command added for manual batch cleanup (see `how/cli.md`).

api/v1alpha1/agenticolsconfig_types.go

@@ -29,6 +29,26 @@ const (
 	AgenticOLSConfigConditionSuspended = "Suspended"
 )
 
+// LifecycleConfig controls automatic cleanup of terminal AgenticRun resources.
+//
+// +kubebuilder:validation:MinProperties=1
+type LifecycleConfig struct {
+	// terminalTTL is the default time-to-live in seconds for terminal
+	// AgenticRun resources (Completed, Failed, Denied, Escalated,
+	// EmergencyStopped, NoActionRequired). After a run reaches a terminal
+	// state and this many seconds elapse, the operator deletes the
+	// AgenticRun CR. Kubernetes garbage collection cascades deletion to
+	// owned resources via owner references.
+	//
+	// Per-run overrides via AgenticRun.spec.ttlAfterTerminal take
+	// precedence over this cluster-wide default.
+	//
+	// When omitted (nil), no automatic deletion occurs.
+	// +optional
+	// +kubebuilder:validation:Minimum=0
+	TerminalTTL *int32 `json:"terminalTTL,omitempty"`
+}
+
 // AgenticOLSConfigSpec defines the desired state of AgenticOLSConfig.
 //
 // +kubebuilder:validation:MinProperties=1
@@ -41,6 +61,11 @@ type AgenticOLSConfigSpec struct {
 	// +optional
 	// +default=false
 	Suspended bool `json:"suspended,omitempty"` //nolint:kubeapilinter // kill switch is genuinely binary; bool is the right type
+
+	// lifecycle controls automatic cleanup of terminal AgenticRun resources.
+	// When omitted, no automatic deletion occurs (backwards-compatible).
+	// +optional
+	Lifecycle LifecycleConfig `json:"lifecycle,omitzero"`
 }
 
 // AgenticOLSConfigStatus defines the observed state of AgenticOLSConfig.