feat: enhance insights with service context and custom priority scripting and rework health score by floreks · Pull Request #4514 · pluralsh/console

floreks · 2026-10-12T09:28:21Z

The agent's insight priority came from a fuzzy name match (concert-manager ranked like cert-manager), and the health score scaled with pod count and was diluted by healthy objects. A fully down third-party StatefulSet could zero a cluster. Neither could be customised, and managed failures were hidden from the list.

What changes

Priority is computed by fixed rules, never by an LLM

Class	Who	Result
Core	catalog (DNS, CNI, kube-proxy, CSI, mesh control plane, cert-manager, ingress controllers, metrics-server, autoscaler), `system-*` priority class, system namespaces (incl. `plrl-console`, `plrl-deploy-operator`), nodes	partly down is `HIGH`, mostly down is `CRITICAL`
App	everything else, incl. its Ingress and Certificate	at
most `MEDIUM`: partial is `LOW`, total outage is `MEDIUM`

Whole-word name matching replaces the fuzzy subsequence match.
Severity comes from the share of replicas that are not ready, plus failing time.
One root per failure: new root kinds (CronJob, Node, standalone Job and Pod, HPA, CustomHealth-degraded resources).

Health score

One penalty per root (LOW 2, MEDIUM 8, HIGH 25, CRITICAL 50), LOW capped at 10, MEDIUM at 30.
Independent of pod count and cluster size, Pending objects don't count, and the score and the list come from the same rows.

Customisation

deployments.plural.sh/insight-priority annotation (LOW to CRITICAL, IGNORE).
New CustomInsightPriority CRD with sandboxed Python scripts (timeouts, redacted errors, precedence, Overlapping condition).

Lifecycle

Grace window (--insight-grace-period, default 10m) so flapping workloads keep their component and insight.
The first ping waits for the store, and ownership of Flux children survives watch events.

Console

service_id on cluster_insight_components and serviceId in the GraphQL input. Managed components are listed with their service.
A service component and a cluster component for the same object share one insight row (one LLM call).
UI: service links and a board view for component insights.

Also fixed on the way

HPA health returned Healthy before reaching FailedGetResourceMetric.
A CronJob with successfulJobsHistoryLimit: 0 kept reading as failing from an old failed Job.
The profiled stores dropped the variadic args of SaveComponentAttributes.

Rollout notes

Deploy the Console first: it accepts serviceId before agents start sending it.
The agent's store schema version is bumped, so a file-backed store is rebuilt on upgrade, which is equivalent to a cold start.
The CustomInsightPriority CRD and the extraArgs passthrough in values.yaml.liquid ship with the chart in pluralsh/deployment-operator. An agent without the CRD starts without the feature and logs it.
Docs: insight-priority.md describes the model, the annotation and the CRD.

Testing

Go unit tests for classification, scoring, roots, ownership, grace and scripts (insights, store, insightpriority packages).
End-to-end on two kind clusters against one Console, one running master and one this branch, with the same fixtures on both: 39 scenarios in 7 groups (priority, roots, ownership, health score, lifecycle, CustomInsightPriority, compatibility), all passing.
Store benchmarks: ping read about 70% faster, single save about 60% faster, bulk save about 9% slower. At steady state the agent's memory and CPU are unchanged.

Report

REPORT.md

Test Plan

Test environment: https://console.plrl-dev-aws.onplural.sh/

Checklist

I have added a meaningful title and summary to convey the impact of this PR to a user.
If required, I have updated the Plural documentation accordingly.
I have added tests to cover my changes.
I have deployed the agent to a test environment and verified that it works as expected (required only when changing agent code).

Plural Flow: console

…ting - Add service info to GraphQL 'insights' query for better context. - Introduce `start_insight` and `await_insight` to reuse service insights in cluster evidence generation. - Add CustomInsightPriority CRD to define priority scripts for failing objects in deployments. - Implement sandboxed Python evaluation engine for priority scripting. - Update cluster insights UI to display associated service information. - Add tracing to `GetFailingRoots` in ProfiledStore. - Update insight tests to reflect new service associations.

linear · 2026-10-12T09:28:27Z

PROD-5282

soffi-ai · 2026-10-12T09:30:44Z

Soffi AI Summary

This PR significantly enhances the cluster insight system by introducing customizable, script-driven priority scoring for failing Kubernetes resources, improving insight context propagation, and refining health scoring logic throughout the deployment operator.

CustomInsightPriority CRD & Scripting Engine: A new CustomInsightPriority CRD (deployments.plural.sh) lets operators define Python scripts that evaluate failing Kubernetes objects and assign priorities (CRITICAL, HIGH, LOW, IGNORE, etc.) to them. Scripts are scoped by namespace, resource kind, and label selectors, and are evaluated in a sandboxed environment. A precedence system determines which script "owns" a root when multiple scripts match it, with ties broken deterministically by name. The evaluator, registry, and target-matching logic are all thoroughly unit- and integration-tested.

Insight Adoption Between Cluster and Service Components: A service_id field is added to cluster_insight_components, enabling insights to be adopted/reused between cluster-level and service-level components. The start_insight/await_insight pattern allows service insights to be incorporated into cluster evidence generation without duplication.

Health & Priority Scoring Improvements: A ScoreCore concept is introduced to distinguish core platform components from third-party ones during criticality scoring. HPA health conditions are now explicitly handled in GetResourceHealth. CronJob last-success tracking and improved failure evaluation logic are added. Failing time is reset correctly when a component is replaced with a new UID.

Observability & Operations: Prometheus metrics are added for the new insight priority evaluation pipeline. Tracing is added to GetFailingRoots in ProfiledStore. Debug logs are cleaned up from ClusterInsightComponent.

Updated: 2026-10-15 10:49 UTC

Deploy in Soffi

greptile-apps · 2026-10-12T09:38:33Z

Confidence Score: 5/5

[High impact] The PR appears safe to merge; no new blocking issue remains.

Summary

This PR connects cluster insights to managing services, shares component diagnoses, adds Python priority scripts, and reworks cluster health scoring.

Cluster insights and health scores come from the same prioritized failures.
Operators can set component priorities with sandboxed Python rules.
Cluster insights now link to services and share their fresh AI insights.
Cluster insights are easier to scan in List and Board views.
Horizontal pod autoscalers reflect their reported health conditions.

_{Reviews (8) · Last reviewed commit: "feat: add CronJob last success tracking ..." · Reviewed by Greptile}

…-evaluate-and-implement-heuristic-detection-for-insight

- Add `Generation` field to track script changes in `Evaluator` and `Registry`. - Improve `GetPriorityAnnotation` to retrieve normalized priority annotations for components. - Implement retry logic for failing or restarting synchronizers in `Supervisor`. - Add dedicated test cases for `Evaluator`, `Registry`, and `Supervisor` enhancements. - Update role-based access control (RBAC) to include `custominsightpriorities`. - Introduce `rootState` function for enriched priority annotations. - Refactor `isRootKind` and `replicas` with targeted workload handling constants. - Extend API docs with `CustomInsightPriority` and related schema details.

floreks · 2026-10-12T11:38:21Z

@greptileai

- Document `CustomInsightPriority` CRD for managing priority of failing objects in cluster insights. - Include `CustomInsightPrioritySpec`, `InsightTarget`, and `InsightEvaluation` schema details. - Define `InsightEngine` and enumerate supported scripting languages. - Extend API reference with validation rules and usage contexts for new fields.

- Eliminate reusing service insights for failing components to streamline logic. - Remove `with_insight` query and related service insight functions in `ServiceComponent`. - Adjust `Cluster.generate/1` to always generate insights independently without reusing service ones. - Simplify GraphQL resolver for `insight` to use a direct dataloader approach. - Update documentation to reflect changes in service-managed workload insights. - Remove associated tests for reused service insights and redundant query handling.

floreks · 2026-10-12T12:58:14Z

@greptileai

- Add `service_id` column in `cluster_insight_components` for improved service association. - Update migration script to include the new column.

- Add `Memoizer.adopt/1` to allow cluster and service components to share fresh insights for the same Kubernetes object. - Update `Memoizer.generate/1` to incorporate adoption logic. - Add query helpers `for_cluster/2`, `for_object/2`, and `with_insight/1` to `ClusterInsightComponent` and `ServiceComponent`. - Ensure stale insights are detached to prevent overwriting by unrelated components. - Add extensive tests for adopting, detaching, and retaining insights between cluster and service components.

floreks · 2026-10-13T10:00:01Z

@greptileai

- Update `Memoizer.adopt/1` to ensure cluster components consistently adopt fresh insights from service components. - Add `ServiceComponentIdentities` GraphQL query for accessing service-component relationships. - Introduce `ClusterInsightComponentCard` and `ClusterInsightComponentParts` for modular UI handling. - Update cluster insights UI with service-specific navigation and board/list view toggles. - Refactor tests to validate insight adoption behavior and UI updates.

- Update `failingSinceUpsert` to account for unique component `uid` and reset failing times appropriately for replacements. - Add logic in `Memoizer.adopt/1` to handle forced insight refresh, bypassing peer adoption. - Refine insights memoization check to ensure proper regeneration logic.

floreks · 2026-10-13T12:18:52Z

@greptileai

- Eliminate unnecessary `console.log` statements for cleaner code.

- Remove redundant clause for forced insights in `Memoizer.adopt/1`. - Ensure forced insights are detached rather than overwritten during adoption. - Update condition to properly link shared insights when fresh.

floreks · 2026-10-13T12:39:20Z

@greptileai

- Enhance `GetResourceHealth` to support HPA conditions from `autoscaling/v2` and `autoscaling/v2beta2`. - Add logic to prioritize `Degraded` status over `Healthy` in `checkConditions`. - Update `health_test.go` with new test cases for various HPA condition scenarios: - `Degraded` status for failed metrics or unreadable scale targets. - `Healthy` status for valid metrics and scaling. - `Progressing` status when no conditions are known yet. - Improve condition handling for clarity and accurate health evaluation.

- Add test case to ensure failing time is reset for replacements with the same name but a different UID. - Verify failing time for the replaced component is not inherited and starts fresh.

- Replace `ScoreHigh` and `ScoreCritical` with `ScoreCore` for essential cluster components. - Adjust scoring logic to cap third-party workloads below `ThresholdHigh` (max `MEDIUM`). - Update classification criteria for core components to prioritize based on criticality and replica counts. - Modify tests to align with new scoring logic and ensure apps are capped appropriately. - Improve documentation/comments to clarify differences between core and non-core component scoring.

- Update tests to cover critical priority scenarios for core DaemonSets and clarify remarks on replica-based criticality. - Ensure core workloads like `aws-node` are marked as `CRITICAL` when fully down. - Adjust comments for insight priority logic to distinguish core criticality from third-party workloads. - Expand documentation to clarify that only workloads with replicas can achieve `CRITICAL` priority independently.

…-evaluate-and-implement-heuristic-detection-for-insight

floreks · 2026-10-15T09:58:41Z

@greptileai

… logic - Add `last_success` column to database schema to track last successful CronJob run. - Update `failingRoots` query to exclude failures older than the last success. - Extend `Extract` function to compute `LastSuccessfulTime` from CronJob status. - Add `nullableUnix` and `sqlUnix` helpers for handling nullable Unix timestamps in database. - Introduce tests for failure evaluation considering previous CronJob successes. - Increment schema version to 4 to reflect CronJob changes.

floreks · 2026-10-15T10:30:42Z

@greptileai

- Replace `nullableUnix`, `nullableInt`, and `sqlUnix` helpers with generic `nullablePtr` and `sqlPtr` functions for consistency. - Set default `last_success` column value in database schema to `0` for clearer handling of unknown timestamps. - Refactor component classification logic to use `new()` for better pointer handling. - Remove redundant nullable-criticality logic in SQL queries and directly reference facts with default values. - Simplify SQL rendering logic in insert queries by standardizing the use of new helpers.

- Correct variadic argument handling in `SaveComponentAttributes` for `ProfiledStore` and `ProfiledStoreLocal` by appending `...` to `args`. - Add unit test to verify `CronJob` bulk save retains `lastSuccessfulTime` and excludes old failed jobs.

floreks self-assigned this Oct 12, 2026

floreks requested review from a team as code owners October 12, 2026 09:28

floreks added the enhancement New feature or request label Oct 12, 2026

greptile-apps Bot reviewed Oct 12, 2026

View reviewed changes

floreks added 2 commits October 12, 2026 12:55

Merge remote-tracking branch 'origin/master' into sebastian/prod-5282…

4ab5d64

…-evaluate-and-implement-heuristic-detection-for-insight

greptile-apps Bot reviewed Oct 12, 2026

View reviewed changes

Comment thread go/deployment-operator/pkg/streamline/store/db_queries.go Outdated

Comment thread lib/console/graphql/ai.ex Outdated

floreks added 2 commits October 12, 2026 14:11

floreks added 2 commits October 12, 2026 16:08

feat: add service_id to cluster_insight_components

66657e0

- Add `service_id` column in `cluster_insight_components` for improved service association. - Update migration script to include the new column.

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread lib/console/ai/memoizer.ex

floreks added 2 commits October 13, 2026 13:59

greptile-apps Bot reviewed Oct 13, 2026

View reviewed changes

Comment thread lib/console/ai/memoizer.ex Outdated

Comment thread js/console/src/components/cd/cluster/ClusterInsightComponent.tsx Outdated

floreks added 2 commits October 13, 2026 14:27

chore: remove debug logs from ClusterInsightComponent

8ba300e

- Eliminate unnecessary `console.log` statements for cleaner code.

feat: refine insight adoption logic for forced rows

88ddefa

- Remove redundant clause for forced insights in `Memoizer.adopt/1`. - Ensure forced insights are detached rather than overwritten during adoption. - Update condition to properly link shared insights when fresh.

floreks added 3 commits October 13, 2026 17:32

feat: reset failing time for component replacements with new UID

2c4eb10

- Add test case to ensure failing time is reset for replacements with the same name but a different UID. - Verify failing time for the replaced component is not inherited and starts fresh.

floreks added 2 commits October 15, 2026 11:57

Merge remote-tracking branch 'origin/master' into sebastian/prod-5282…

912bc33

…-evaluate-and-implement-heuristic-detection-for-insight

floreks changed the title ~~feat: enhance insights with service context and custom priority scripting~~ feat: enhance insights with service context and custom priority scripting and rework health score Oct 15, 2026

greptile-apps Bot reviewed Oct 15, 2026

View reviewed changes

Comment thread go/deployment-operator/pkg/streamline/store/db_store.go

Comment thread go/deployment-operator/pkg/streamline/store/db_queries.go Outdated

floreks added 2 commits October 15, 2026 12:46

Conversation

floreks commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Rollout notes

Testing

Report

Test Plan

Checklist

Uh oh!

linear Bot commented Oct 12, 2026

Uh oh!

soffi-ai Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

floreks commented Oct 12, 2026

Uh oh!

Uh oh!

Uh oh!

floreks commented Oct 12, 2026

Uh oh!

floreks commented Oct 13, 2026

Uh oh!

Uh oh!

floreks commented Oct 13, 2026

Uh oh!

Uh oh!

Uh oh!

floreks commented Oct 13, 2026

Uh oh!

floreks commented Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

floreks commented Oct 15, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

floreks commented Oct 12, 2026 •

edited

Loading

soffi-ai Bot commented Oct 12, 2026 •

edited

Loading

greptile-apps Bot commented Oct 12, 2026 •

edited

Loading