Skip to content

Add new tool for detecting OOMKilled and CrashloopBackoff PODs across any Konflux/Kubernetes cluster(s) #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jhutar merged 4 commits into from
Dec 17, 2025
Merged

Add new tool for detecting OOMKilled and CrashloopBackoff PODs across any Konflux/Kubernetes cluster(s) #17

jhutar merged 4 commits into from
Dec 17, 2025

Conversation

@smodak-rh
Copy link
Contributor

@smodak-rh smodak-rh commented Dec 12, 2025

Already described in 'README.md', but highlighted here as well:

What This Tool Does

  • Scans one or many OpenShift clusters (oc contexts)
  • Detects:
    • OOMKilled pods
    • CrashLoopBackOff pods
  • Looks back across multiple time windows:
    • 1h, 3h, 6h, 24h, 48h, 3d, 5d, 7d
  • Uses:
    • Kubernetes events first (fast)
    • Prometheus fallback for older history
  • Runs highly parallel:
    • Cluster-level batching
    • Namespace-level batching
  • Saves forensic artifacts:
    • oc describe pod
    • oc logs (or --previous)
  • Exports CSV + JSON with absolute paths to artifacts
  • Colorized terminal output

Sample output run:

bash-5.3$ ~/oc_get_ooms.py --ns-batch-size 100 --ns-workers 100 --timeout 120 --batch 4
Using contexts: ['default/api-kflux-ocp-p01-7ayg-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-osp-p01-yt45-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-es01-1ion-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-rh02-0fk9-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-rh03-nnv1-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-rhel-p01-fc38-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-stg-es01-21tc-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prd-rh01-pg1f-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prod-p01-wcfb-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prod-p02-hjvn-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stage-p01-hpmt-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stg-rh01-l2vh-p1-openshiftapps-com:6443/smodak']
Cluster-batch: 4  NS-batch-size: 100  NS-workers: 100  Prom-workers: 100
Retries: 3  OC timeout(s): 120s  Skip-prometheus: False

=== Cluster batch 1: ['default/api-kflux-ocp-p01-7ayg-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-osp-p01-yt45-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-es01-1ion-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-rh02-0fk9-p1-openshiftapps-com:6443/smodak'] ===

→ Processing cluster: kflux-ocp-p01

→ Processing cluster: kflux-osp-p01
→ Processing cluster: kflux-prd-es01


→ Processing cluster: kflux-prd-rh02
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 2: 70 namespaces
    Namespace clusters-611e937f-5577-47db-a301-cd73c98d6e99: 5 pod(s) found via oc
    Namespace clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57: 5 pod(s) found via oc
    Namespace clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3: 5 pod(s) found via oc
  Namespace batch 2: 100 namespaces
    Namespace clusters-2ecdf824-659a-43b4-8432-507e3398f8d0: 5 pod(s) found via oc
  Namespace batch 2: 48 namespaces
    Namespace pulp-access-controller: 1 pod(s) found via oc
  Running Prometheus fallback for 144 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 44 namespaces
Completed cluster kflux-prd-es01
  Running Prometheus fallback for 170 namespace(s) in batches
    Prom batch 1: 100 namespaces
  Namespace batch 3: 23 namespaces
    Prom batch 2: 70 namespaces
Completed cluster kflux-osp-p01
  Running Prometheus fallback for 222 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 22 namespaces
Completed cluster kflux-prd-rh02
  Namespace batch 2: 100 namespaces
  Namespace batch 3: 23 namespaces
  Running Prometheus fallback for 223 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 23 namespaces
Completed cluster kflux-ocp-p01

=== Cluster batch 2: ['default/api-kflux-prd-rh03-nnv1-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-rhel-p01-fc38-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-stg-es01-21tc-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prd-rh01-pg1f-p1-openshiftapps-com:6443/smodak'] ===

→ Processing cluster: kflux-prd-rh03

→ Processing cluster: kflux-rhel-p01
→ Processing cluster: kflux-stg-es01


→ Processing cluster: stone-prd-rh01
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 2: 35 namespaces
  Running Prometheus fallback for 135 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 35 namespaces
Completed cluster kflux-stg-es01
  Namespace batch 2: 100 namespaces
  Namespace batch 2: 100 namespaces
  Namespace batch 3: 100 namespaces
  Namespace batch 3: 100 namespaces
  Namespace batch 4: 74 namespaces
  Running Prometheus fallback for 374 namespace(s) in batches
    Prom batch 1: 100 namespaces
  Namespace batch 2: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 100 namespaces
    Prom batch 4: 74 namespaces
Completed cluster kflux-rhel-p01
  Namespace batch 4: 100 namespaces
  Namespace batch 5: 100 namespaces
  Namespace batch 3: 13 namespaces
  Running Prometheus fallback for 213 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 13 namespaces
Completed cluster kflux-prd-rh03
  Namespace batch 6: 100 namespaces
  Namespace batch 7: 100 namespaces
  Namespace batch 8: 100 namespaces
  Namespace batch 9: 100 namespaces
  Namespace batch 10: 100 namespaces
  Namespace batch 11: 100 namespaces
  Namespace batch 12: 100 namespaces
  Namespace batch 13: 100 namespaces
  Namespace batch 14: 100 namespaces
  Namespace batch 15: 100 namespaces
  Namespace batch 16: 100 namespaces
  Namespace batch 17: 100 namespaces
  Namespace batch 18: 100 namespaces
  Namespace batch 19: 100 namespaces
  Namespace batch 20: 100 namespaces
  Namespace batch 21: 97 namespaces
  Running Prometheus fallback for 2097 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 100 namespaces
    Prom batch 4: 100 namespaces
    Prom batch 5: 100 namespaces
    Prom batch 6: 100 namespaces
    Prom batch 7: 100 namespaces
    Prom batch 8: 100 namespaces
    Prom batch 9: 100 namespaces
    Prom batch 10: 100 namespaces
    Prom batch 11: 100 namespaces
    Prom batch 12: 100 namespaces
    Prom batch 13: 100 namespaces
    Prom batch 14: 100 namespaces
    Prom batch 15: 100 namespaces
    Prom batch 16: 100 namespaces
    Prom batch 17: 100 namespaces
    Prom batch 18: 100 namespaces
    Prom batch 19: 100 namespaces
    Prom batch 20: 100 namespaces
    Prom batch 21: 97 namespaces
Completed cluster stone-prd-rh01

=== Cluster batch 3: ['default/api-stone-prod-p01-wcfb-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prod-p02-hjvn-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stage-p01-hpmt-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stg-rh01-l2vh-p1-openshiftapps-com:6443/smodak'] ===

→ Processing cluster: stone-prod-p01

→ Processing cluster: stone-prod-p02

→ Processing cluster: stone-stage-p01

→ Processing cluster: stone-stg-rh01
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 1: 100 namespaces
  Namespace batch 2: 100 namespaces
  Namespace batch 2: 100 namespaces
  Namespace batch 2: 100 namespaces
  Namespace batch 3: 100 namespaces
  Namespace batch 3: 100 namespaces
  Namespace batch 2: 100 namespaces
  Namespace batch 4: 100 namespaces
  Namespace batch 5: 100 namespaces
  Namespace batch 3: 100 namespaces
  Namespace batch 6: 100 namespaces
  Namespace batch 7: 100 namespaces
  Namespace batch 4: 100 namespaces
  Namespace batch 5: 28 namespaces
  Running Prometheus fallback for 428 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 100 namespaces
    Prom batch 4: 100 namespaces
  Namespace batch 3: 40 namespaces
    Prom batch 5: 28 namespaces
Completed cluster stone-prod-p01
  Running Prometheus fallback for 240 namespace(s) in batches
    Prom batch 1: 100 namespaces
  Namespace batch 4: 100 namespaces
    Prom batch 2: 100 namespaces
    Namespace pulp-access-controller: 1 pod(s) found via oc
    Prom batch 3: 40 namespaces
Completed cluster stone-stage-p01
  Namespace batch 5: 100 namespaces
  Namespace batch 6: 100 namespaces
  Namespace batch 7: 100 namespaces
  Namespace batch 8: 100 namespaces
  Namespace batch 9: 29 namespaces
  Running Prometheus fallback for 828 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 100 namespaces
    Prom batch 4: 100 namespaces
    Prom batch 5: 100 namespaces
    Prom batch 6: 100 namespaces
    Prom batch 7: 100 namespaces
    Prom batch 8: 100 namespaces
    Prom batch 9: 28 namespaces
Completed cluster stone-stg-rh01
  Namespace batch 8: 100 namespaces
  Namespace batch 9: 100 namespaces
    Namespace pulp-access-controller: 1 pod(s) found via oc
  Namespace batch 10: 100 namespaces
  Namespace batch 11: 100 namespaces
    Namespace rhoai-tenant: 3 pod(s) found via oc
  Namespace batch 12: 100 namespaces
  Namespace batch 13: 86 namespaces
  Running Prometheus fallback for 1284 namespace(s) in batches
    Prom batch 1: 100 namespaces
    Prom batch 2: 100 namespaces
    Prom batch 3: 100 namespaces
    Prom batch 4: 100 namespaces
    Prom batch 5: 100 namespaces
    Prom batch 6: 100 namespaces
    Prom batch 7: 100 namespaces
    Prom batch 8: 100 namespaces
    Prom batch 9: 100 namespaces
    Prom batch 10: 100 namespaces
    Prom batch 11: 100 namespaces
    Prom batch 12: 100 namespaces
    Prom batch 13: 84 namespaces
Completed cluster stone-prod-p02
JSON written → oom_results.json
CSV written → oom_results.csv

Cluster: kflux-prd-es01
  Namespace: clusters-611e937f-5577-47db-a301-cd73c98d6e99
    Pod: catalog-operator-866d64458-dmk2x
      - CrashLoopBackOff event at: 2025-12-12T06:05:46Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__log.txt
    Pod: certified-operators-catalog-79757bd797-wtthq
      - CrashLoopBackOff event at: 2025-12-12T06:05:36Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__log.txt
    Pod: community-operators-catalog-6844d5696f-lj68d
      - CrashLoopBackOff event at: 2025-12-12T06:05:37Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__log.txt
    Pod: redhat-marketplace-catalog-556cf7f5fd-b72cg
      - CrashLoopBackOff event at: 2025-12-12T06:05:37Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__log.txt
    Pod: redhat-operators-catalog-6b8bd8fc8f-mg227
      - CrashLoopBackOff event at: 2025-12-12T06:05:38Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__log.txt
  Namespace: clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57
    Pod: catalog-operator-5948b97f7d-85bzm
      - CrashLoopBackOff event at: 2025-12-12T06:44:59Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__log.txt
    Pod: certified-operators-catalog-74dbb8f6c5-tb6nz
      - CrashLoopBackOff event at: 2025-12-12T06:44:51Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__log.txt
    Pod: community-operators-catalog-575fbfb7c5-9qr4p
      - CrashLoopBackOff event at: 2025-12-12T06:44:51Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__log.txt
    Pod: redhat-marketplace-catalog-5fff54d476-4hppf
      - CrashLoopBackOff event at: 2025-12-12T06:44:51Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__log.txt
    Pod: redhat-operators-catalog-7cd6f8cddc-2bk5k
      - CrashLoopBackOff event at: 2025-12-12T06:44:51Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__log.txt
  Namespace: clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3
    Pod: catalog-operator-74999f6f44-wv46s
      - CrashLoopBackOff event at: 2025-12-12T06:35:38Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__log.txt
    Pod: certified-operators-catalog-d5b7b896b-q5s2r
      - CrashLoopBackOff event at: 2025-12-12T06:35:28Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__log.txt
    Pod: community-operators-catalog-66f496b886-v6bk2
      - CrashLoopBackOff event at: 2025-12-12T06:35:28Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__log.txt
    Pod: redhat-marketplace-catalog-db87499f5-w9qgc
      - CrashLoopBackOff event at: 2025-12-12T06:35:28Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__log.txt
    Pod: redhat-operators-catalog-7fdf966868-lld94
      - CrashLoopBackOff event at: 2025-12-12T06:35:28Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__log.txt
  Namespace: clusters-2ecdf824-659a-43b4-8432-507e3398f8d0
    Pod: catalog-operator-68f979975b-ls94l
      - CrashLoopBackOff event at: 2025-12-12T06:05:58Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__log.txt
    Pod: certified-operators-catalog-79795d58-87w4n
      - CrashLoopBackOff event at: 2025-12-12T06:05:40Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__log.txt
    Pod: community-operators-catalog-798596ff98-cjvxg
      - CrashLoopBackOff event at: 2025-12-12T06:05:40Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__log.txt
    Pod: redhat-marketplace-catalog-5c7bc7b49f-cs8sr
      - CrashLoopBackOff event at: 2025-12-12T06:05:40Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__log.txt
    Pod: redhat-operators-catalog-7f6767649b-4wr8q
      - CrashLoopBackOff event at: 2025-12-12T06:05:40Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__log.txt

Cluster: kflux-osp-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: kflux-prd-rh02
  Namespace: pulp-access-controller
    Pod: pulp-access-operator-5dbc87f4-9964v
      - CrashLoopBackOff event at: 2025-12-12T06:49:17Z
      - description_file: /private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__log.txt

Cluster: kflux-ocp-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: kflux-stg-es01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: kflux-rhel-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: kflux-prd-rh03
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-prd-rh01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-prod-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-stage-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-stg-rh01
  Namespace: pulp-access-controller
    Pod: pulp-access-operator-5dbc87f4-lb8jf
      - CrashLoopBackOff event at: 2025-12-12T07:16:09Z
      - description_file: /private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__log.txt

Cluster: stone-prod-p02
  Namespace: pulp-access-controller
    Pod: pulp-access-operator-5dbc87f4-qdmv6
      - CrashLoopBackOff event at: 2025-12-12T07:18:05Z
      - description_file: /private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__log.txt
  Namespace: rhoai-tenant
    Pod: mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod
      - CrashLoopBackOff event at: 2025-12-12T04:54:29Z
      - description_file: /private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__log.txt
    Pod: mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod
      - CrashLoopBackOff event at: 2025-12-12T04:53:18Z
      - CrashLoopBackOff event at: 2025-12-12T04:53:19Z
      - description_file: /private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__log.txt
    Pod: rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod
      - CrashLoopBackOff event at: 2025-12-12T05:09:33Z
      - description_file: /private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__log.txt

Per-cluster logs written to /tmp/<cluster>/ (if any findings were found)

Files generated (oom_results.csv):

bash-5.3$ cat oom_results.csv 
cluster,namespace,pod,type,timestamps,sources,description_file,pod_log_file
kflux-prd-es01,clusters-611e937f-5577-47db-a301-cd73c98d6e99,catalog-operator-866d64458-dmk2x,CrashLoopBackOff,2025-12-12T06:05:46Z,events,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__log.txt
kflux-prd-es01,clusters-611e937f-5577-47db-a301-cd73c98d6e99,certified-operators-catalog-79757bd797-wtthq,CrashLoopBackOff,2025-12-12T06:05:36Z,events,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__log.txt
kflux-prd-es01,clusters-611e937f-5577-47db-a301-cd73c98d6e99,community-operators-catalog-6844d5696f-lj68d,CrashLoopBackOff,2025-12-12T06:05:37Z,events,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__log.txt
kflux-prd-es01,clusters-611e937f-5577-47db-a301-cd73c98d6e99,redhat-marketplace-catalog-556cf7f5fd-b72cg,CrashLoopBackOff,2025-12-12T06:05:37Z,events,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__log.txt
kflux-prd-es01,clusters-611e937f-5577-47db-a301-cd73c98d6e99,redhat-operators-catalog-6b8bd8fc8f-mg227,CrashLoopBackOff,2025-12-12T06:05:38Z,events,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__log.txt
kflux-prd-es01,clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57,catalog-operator-5948b97f7d-85bzm,CrashLoopBackOff,2025-12-12T06:44:59Z,events,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__log.txt
kflux-prd-es01,clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57,certified-operators-catalog-74dbb8f6c5-tb6nz,CrashLoopBackOff,2025-12-12T06:44:51Z,events,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__log.txt
kflux-prd-es01,clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57,community-operators-catalog-575fbfb7c5-9qr4p,CrashLoopBackOff,2025-12-12T06:44:51Z,events,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__log.txt
kflux-prd-es01,clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57,redhat-marketplace-catalog-5fff54d476-4hppf,CrashLoopBackOff,2025-12-12T06:44:51Z,events,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__log.txt
kflux-prd-es01,clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57,redhat-operators-catalog-7cd6f8cddc-2bk5k,CrashLoopBackOff,2025-12-12T06:44:51Z,events,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__log.txt
kflux-prd-es01,clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3,catalog-operator-74999f6f44-wv46s,CrashLoopBackOff,2025-12-12T06:35:38Z,events,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__log.txt
kflux-prd-es01,clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3,certified-operators-catalog-d5b7b896b-q5s2r,CrashLoopBackOff,2025-12-12T06:35:28Z,events,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__log.txt
kflux-prd-es01,clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3,community-operators-catalog-66f496b886-v6bk2,CrashLoopBackOff,2025-12-12T06:35:28Z,events,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__log.txt
kflux-prd-es01,clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3,redhat-marketplace-catalog-db87499f5-w9qgc,CrashLoopBackOff,2025-12-12T06:35:28Z,events,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__log.txt
kflux-prd-es01,clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3,redhat-operators-catalog-7fdf966868-lld94,CrashLoopBackOff,2025-12-12T06:35:28Z,events,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__log.txt
kflux-prd-es01,clusters-2ecdf824-659a-43b4-8432-507e3398f8d0,catalog-operator-68f979975b-ls94l,CrashLoopBackOff,2025-12-12T06:05:58Z,events,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__log.txt
kflux-prd-es01,clusters-2ecdf824-659a-43b4-8432-507e3398f8d0,certified-operators-catalog-79795d58-87w4n,CrashLoopBackOff,2025-12-12T06:05:40Z,events,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__log.txt
kflux-prd-es01,clusters-2ecdf824-659a-43b4-8432-507e3398f8d0,community-operators-catalog-798596ff98-cjvxg,CrashLoopBackOff,2025-12-12T06:05:40Z,events,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__log.txt
kflux-prd-es01,clusters-2ecdf824-659a-43b4-8432-507e3398f8d0,redhat-marketplace-catalog-5c7bc7b49f-cs8sr,CrashLoopBackOff,2025-12-12T06:05:40Z,events,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__log.txt
kflux-prd-es01,clusters-2ecdf824-659a-43b4-8432-507e3398f8d0,redhat-operators-catalog-7f6767649b-4wr8q,CrashLoopBackOff,2025-12-12T06:05:40Z,events,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__desc.txt,/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__log.txt
kflux-prd-rh02,pulp-access-controller,pulp-access-operator-5dbc87f4-9964v,CrashLoopBackOff,2025-12-12T06:49:17Z,events,/private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__desc.txt,/private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__log.txt
stone-stg-rh01,pulp-access-controller,pulp-access-operator-5dbc87f4-lb8jf,CrashLoopBackOff,2025-12-12T07:16:09Z,events,/private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__desc.txt,/private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__log.txt
stone-prod-p02,pulp-access-controller,pulp-access-operator-5dbc87f4-qdmv6,CrashLoopBackOff,2025-12-12T07:18:05Z,events,/private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__desc.txt,/private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__log.txt
stone-prod-p02,rhoai-tenant,mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod,CrashLoopBackOff,2025-12-12T04:54:29Z,events,/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__desc.txt,/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__log.txt
stone-prod-p02,rhoai-tenant,mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod,CrashLoopBackOff,2025-12-12T04:53:18Z;2025-12-12T04:53:19Z,events,/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__desc.txt,/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__log.txt
stone-prod-p02,rhoai-tenant,rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod,CrashLoopBackOff,2025-12-12T05:09:33Z,events,/private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__desc.txt,/private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__log.txt

Files generated (oom_results.json):

bash-5.3$ cat oom_results.json 
{
  "kflux-prd-es01": {
    "clusters-611e937f-5577-47db-a301-cd73c98d6e99": {
      "catalog-operator-866d64458-dmk2x": {
        "pod": "catalog-operator-866d64458-dmk2x",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:46Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__catalog-operator-866d64458-dmk2x__20251212T065132Z__log.txt"
      },
      "certified-operators-catalog-79757bd797-wtthq": {
        "pod": "certified-operators-catalog-79757bd797-wtthq",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:36Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__certified-operators-catalog-79757bd797-wtthq__20251212T065140Z__log.txt"
      },
      "community-operators-catalog-6844d5696f-lj68d": {
        "pod": "community-operators-catalog-6844d5696f-lj68d",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:37Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__community-operators-catalog-6844d5696f-lj68d__20251212T065142Z__log.txt"
      },
      "redhat-marketplace-catalog-556cf7f5fd-b72cg": {
        "pod": "redhat-marketplace-catalog-556cf7f5fd-b72cg",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:37Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-marketplace-catalog-556cf7f5fd-b72cg__20251212T065143Z__log.txt"
      },
      "redhat-operators-catalog-6b8bd8fc8f-mg227": {
        "pod": "redhat-operators-catalog-6b8bd8fc8f-mg227",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:38Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-611e937f-5577-47db-a301-cd73c98d6e99__redhat-operators-catalog-6b8bd8fc8f-mg227__20251212T065144Z__log.txt"
      }
    },
    "clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57": {
      "catalog-operator-5948b97f7d-85bzm": {
        "pod": "catalog-operator-5948b97f7d-85bzm",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:44:59Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__catalog-operator-5948b97f7d-85bzm__20251212T065148Z__log.txt"
      },
      "certified-operators-catalog-74dbb8f6c5-tb6nz": {
        "pod": "certified-operators-catalog-74dbb8f6c5-tb6nz",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:44:51Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__certified-operators-catalog-74dbb8f6c5-tb6nz__20251212T065156Z__log.txt"
      },
      "community-operators-catalog-575fbfb7c5-9qr4p": {
        "pod": "community-operators-catalog-575fbfb7c5-9qr4p",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:44:51Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__community-operators-catalog-575fbfb7c5-9qr4p__20251212T065158Z__log.txt"
      },
      "redhat-marketplace-catalog-5fff54d476-4hppf": {
        "pod": "redhat-marketplace-catalog-5fff54d476-4hppf",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:44:51Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-marketplace-catalog-5fff54d476-4hppf__20251212T065159Z__log.txt"
      },
      "redhat-operators-catalog-7cd6f8cddc-2bk5k": {
        "pod": "redhat-operators-catalog-7cd6f8cddc-2bk5k",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:44:51Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-dff42627-b5e9-46f1-b161-bdf5fc302c57__redhat-operators-catalog-7cd6f8cddc-2bk5k__20251212T065200Z__log.txt"
      }
    },
    "clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3": {
      "catalog-operator-74999f6f44-wv46s": {
        "pod": "catalog-operator-74999f6f44-wv46s",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:35:38Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__catalog-operator-74999f6f44-wv46s__20251212T065201Z__log.txt"
      },
      "certified-operators-catalog-d5b7b896b-q5s2r": {
        "pod": "certified-operators-catalog-d5b7b896b-q5s2r",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:35:28Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__certified-operators-catalog-d5b7b896b-q5s2r__20251212T065201Z__log.txt"
      },
      "community-operators-catalog-66f496b886-v6bk2": {
        "pod": "community-operators-catalog-66f496b886-v6bk2",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:35:28Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__community-operators-catalog-66f496b886-v6bk2__20251212T065203Z__log.txt"
      },
      "redhat-marketplace-catalog-db87499f5-w9qgc": {
        "pod": "redhat-marketplace-catalog-db87499f5-w9qgc",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:35:28Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-marketplace-catalog-db87499f5-w9qgc__20251212T065203Z__log.txt"
      },
      "redhat-operators-catalog-7fdf966868-lld94": {
        "pod": "redhat-operators-catalog-7fdf966868-lld94",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:35:28Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-05444e73-dd68-48b6-899e-0a807a4eb1b3__redhat-operators-catalog-7fdf966868-lld94__20251212T065204Z__log.txt"
      }
    },
    "clusters-2ecdf824-659a-43b4-8432-507e3398f8d0": {
      "catalog-operator-68f979975b-ls94l": {
        "pod": "catalog-operator-68f979975b-ls94l",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:58Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__catalog-operator-68f979975b-ls94l__20251212T065205Z__log.txt"
      },
      "certified-operators-catalog-79795d58-87w4n": {
        "pod": "certified-operators-catalog-79795d58-87w4n",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:40Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__certified-operators-catalog-79795d58-87w4n__20251212T065206Z__log.txt"
      },
      "community-operators-catalog-798596ff98-cjvxg": {
        "pod": "community-operators-catalog-798596ff98-cjvxg",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:40Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__community-operators-catalog-798596ff98-cjvxg__20251212T065206Z__log.txt"
      },
      "redhat-marketplace-catalog-5c7bc7b49f-cs8sr": {
        "pod": "redhat-marketplace-catalog-5c7bc7b49f-cs8sr",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:40Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-marketplace-catalog-5c7bc7b49f-cs8sr__20251212T065207Z__log.txt"
      },
      "redhat-operators-catalog-7f6767649b-4wr8q": {
        "pod": "redhat-operators-catalog-7f6767649b-4wr8q",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:05:40Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-es01/clusters-2ecdf824-659a-43b4-8432-507e3398f8d0__redhat-operators-catalog-7f6767649b-4wr8q__20251212T065215Z__log.txt"
      }
    }
  },
  "kflux-osp-p01": {},
  "kflux-prd-rh02": {
    "pulp-access-controller": {
      "pulp-access-operator-5dbc87f4-9964v": {
        "pod": "pulp-access-operator-5dbc87f4-9964v",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T06:49:17Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__desc.txt",
        "pod_log_file": "/private/tmp/kflux-prd-rh02/pulp-access-controller__pulp-access-operator-5dbc87f4-9964v__20251212T065224Z__log.txt"
      }
    }
  },
  "kflux-ocp-p01": {},
  "kflux-stg-es01": {},
  "kflux-rhel-p01": {},
  "kflux-prd-rh03": {},
  "stone-prd-rh01": {},
  "stone-prod-p01": {},
  "stone-stage-p01": {},
  "stone-stg-rh01": {
    "pulp-access-controller": {
      "pulp-access-operator-5dbc87f4-lb8jf": {
        "pod": "pulp-access-operator-5dbc87f4-lb8jf",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T07:16:09Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__desc.txt",
        "pod_log_file": "/private/tmp/stone-stg-rh01/pulp-access-controller__pulp-access-operator-5dbc87f4-lb8jf__20251212T071736Z__log.txt"
      }
    }
  },
  "stone-prod-p02": {
    "pulp-access-controller": {
      "pulp-access-operator-5dbc87f4-qdmv6": {
        "pod": "pulp-access-operator-5dbc87f4-qdmv6",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T07:18:05Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__desc.txt",
        "pod_log_file": "/private/tmp/stone-prod-p02/pulp-access-controller__pulp-access-operator-5dbc87f4-qdmv6__20251212T072016Z__log.txt"
      }
    },
    "rhoai-tenant": {
      "mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod": {
        "pod": "mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T04:54:29Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__desc.txt",
        "pod_log_file": "/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-kks5p-sast-coverity-check-pod__20251212T072108Z__log.txt"
      },
      "mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod": {
        "pod": "mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T04:53:18Z",
          "2025-12-12T04:53:19Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__desc.txt",
        "pod_log_file": "/private/tmp/stone-prod-p02/rhoai-tenant__mlflow-poc-on-pull-request-s4f77-sast-coverity-check-pod__20251212T072109Z__log.txt"
      },
      "rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod": {
        "pod": "rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod",
        "oom_timestamps": [],
        "crash_timestamps": [
          "2025-12-12T05:09:33Z"
        ],
        "sources": [
          "events"
        ],
        "description_file": "/private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__desc.txt",
        "pod_log_file": "/private/tmp/stone-prod-p02/rhoai-tenant__rhoai-health-on-pull-request-pdbgb-sast-coverity-check-pod__20251212T072110Z__log.txt"
      }
    }
  }

We have been using a prior version (shell based) of this tool to raise Jiras on various components so far. E.g:

  1. https://issues.redhat.com/browse/RELEASE-1916
  2. https://issues.redhat.com/browse/KONFLUX-10806
  3. https://issues.redhat.com/browse/KONFLUX-10918
  4. https://issues.redhat.com/browse/KONFLUX-10955
  5. https://issues.redhat.com/browse/KONFLUX-10957
  6. https://issues.redhat.com/browse/KONFLUX-10969
  7. https://issues.redhat.com/browse/KONFLUX-11032
  8. https://issues.redhat.com/browse/KONFLUX-11055
  9. https://issues.redhat.com/browse/KONFLUX-11056
  10. https://issues.redhat.com/browse/KONFLUX-11068
  11. https://issues.redhat.com/browse/KONFLUX-11178
  12. https://issues.redhat.com/browse/KONFLUX-11179
  13. https://issues.redhat.com/browse/KONFLUX-11365

Cc: @jhutar @psuriset

@smodak-rh
Add new tool for detecting OOMKilled and CrashloopBackoff PODs across…
f5e0a43 
… any Konflux/Kubernetes cluster(s)
@smodak-rh
Improve code quality, security, and error handling in oc_get_ooms.py
70ce6fe 
- Applied black/flake8 formatting and fixed linting issues
- Prevented shell injection by using subprocess argument lists
- Added specific exception handling and input validation
- Added logging module and improved error messages
- Enhanced code documentation and type hints
- All changes maintain backward compatibility

Generated-by: Cursor-AI
@smodak-rh
Optimize events fetching and replace Prometheus exec
157fcb6 
- Optimized events fetching: 3 API calls -> 1 call per namespace
- Replaced Prometheus pod exec with route-based HTTP access (no exec perms needed)
- Enhanced Prometheus query validation and error handling
- Improved performance and reliability while maintaining backward compatibility

Generated-by: Cursor-AI
@smodak-rh
Fix CSV/JSON consistency, add time range filtering, and improve detec…
2e8d8db 
…tion

CSV/JSON Consistency & Type Fixes:
- Fixed Prometheus query results to properly set OOMKilled/CrashLoopBackOff types
- Prometheus queries now track query type and set appropriate timestamps
- Removed FoundBy type from CSV - all rows now have proper type values
- CSV and JSON outputs are now fully consistent

OOMKilled Detection Improvements:
- Added oomkilled_via_pods_oc() to check pod status directly
- Now detects OOMKilled pods visible in UI that were missed by events only
- Checks lastState.terminated.reason == OOMKilled in pod status
- Extracts timestamps from finishedAt field for accurate timing

Time Range Filtering:
- Added --time-range option with configurable lookback window (default: 1d)
- Supports formats: 1s, 1m, 1h, 1d, 1M (30 days)
- Filters events by time range to focus on recent incidents
- Added time_range column to CSV output
- Added _metadata.time_range to JSON output
- Updated help documentation with time range examples

Constant Parallelism Model:
- Improved batch processing to maintain constant parallelism
- When one cluster finishes, immediately starts the next one
- --batch N now maintains N workers throughout execution (no waiting for batches)
- Better resource utilization and faster overall execution
- Updated architecture diagram in README to reflect new model

Documentation Updates:
- Updated README.md with all enhancements
- Updated architecture diagram showing constant parallelism
- Added time range filtering examples
- Updated output format documentation
- Added detection methods explanation

All changes maintain backward compatibility and improve reliability.

Generated-by: Cursor-AI
@smodak-rh
Copy link
Contributor Author

smodak-rh commented Dec 16, 2025

After the last commit: Fix CSV/JSON consistency, add time range filtering, and improve detec…,

Ran the test again for all clusters and below is the result:

$ ./oc_get_ooms.py --ns-batch-size 250 --ns-workers 250 --timeout 200 --batch 4 --time-range 2d
Using contexts: ['default/api-kflux-ocp-p01-7ayg-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-osp-p01-yt45-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-es01-1ion-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-rh02-0fk9-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-prd-rh03-nnv1-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-rhel-p01-fc38-p1-openshiftapps-com:6443/smodak', 'default/api-kflux-stg-es01-21tc-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prd-rh01-pg1f-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prod-p01-wcfb-p1-openshiftapps-com:6443/smodak', 'default/api-stone-prod-p02-hjvn-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stage-p01-hpmt-p1-openshiftapps-com:6443/smodak', 'default/api-stone-stg-rh01-l2vh-p1-openshiftapps-com:6443/smodak']
Cluster-parallelism: 4  NS-batch-size: 250  NS-workers: 250  Prom-workers: 250
Retries: 3  OC timeout(s): 200s  Time-range: 2d  Skip-prometheus: False

→ Processing cluster: kflux-ocp-p01
Started processing cluster: kflux-ocp-p01

→ Processing cluster: kflux-osp-p01
Started processing cluster: kflux-osp-p01

→ Processing cluster: kflux-prd-es01
Started processing cluster: kflux-prd-es01

→ Processing cluster: kflux-prd-rh02
Started processing cluster: kflux-prd-rh02
  Namespace batch 1: 142 namespaces
  Namespace batch 1: 171 namespaces
  Namespace batch 1: 224 namespaces
  Namespace batch 1: 224 namespaces
    Namespace release-service: 1 pod(s) found via oc
    Namespace openshift-ocm-agent-operator: 1 pod(s) found via oc
    Namespace release-service: 1 pod(s) found via oc
    Namespace clusters-653016ff-f619-47a2-92b3-1e8da6b45963: 5 pod(s) found via oc
  Running Prometheus fallback for 170 namespace(s) in batches
    Prom batch 1: 170 namespaces
    Namespace product-kubearchive-logging: 1 pod(s) found via oc
Completed cluster kflux-osp-p01
Started processing cluster: kflux-prd-rh03

→ Processing cluster: kflux-prd-rh03
    Namespace clusters-1912b582-73db-4d01-a34e-02c87a6b7556: 5 pod(s) found via oc
  Namespace batch 1: 215 namespaces
    Namespace release-service: 1 pod(s) found via oc
    Namespace clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d: 5 pod(s) found via oc
  Running Prometheus fallback for 139 namespace(s) in batches
    Prom batch 1: 139 namespaces
    Namespace aws-load-balancer-operator-tenant: 1 pod(s) found via oc
Completed cluster kflux-prd-es01
Started processing cluster: kflux-rhel-p01

→ Processing cluster: kflux-rhel-p01
  Namespace batch 1: 250 namespaces
    Namespace release-service: 1 pod(s) found via oc
    Namespace openshift-machine-config-operator: 1 pod(s) found via oc
    Namespace openshift-machine-config-operator: 23 pod(s) found via oc
  Running Prometheus fallback for 222 namespace(s) in batches
    Prom batch 1: 222 namespaces
  Running Prometheus fallback for 213 namespace(s) in batches
    Prom batch 1: 213 namespaces
    Namespace product-kubearchive-logging: 1 pod(s) found via oc
  Namespace batch 2: 125 namespaces
  Running Prometheus fallback for 372 namespace(s) in batches
    Prom batch 1: 250 namespaces
  Running Prometheus fallback for 222 namespace(s) in batches
    Prom batch 1: 222 namespaces
    Prom batch 2: 122 namespaces
Completed cluster kflux-prd-rh02
Started processing cluster: kflux-stg-es01

→ Processing cluster: kflux-stg-es01
  Namespace batch 1: 135 namespaces
Completed cluster kflux-ocp-p01
Started processing cluster: stone-prd-rh01
Completed cluster kflux-prd-rh03
Started processing cluster: stone-prod-p01

→ Processing cluster: stone-prd-rh01

→ Processing cluster: stone-prod-p01
Completed cluster kflux-rhel-p01
Started processing cluster: stone-prod-p02

→ Processing cluster: stone-prod-p02
  Namespace batch 1: 250 namespaces
  Running Prometheus fallback for 135 namespace(s) in batches
    Prom batch 1: 135 namespaces
  Namespace batch 1: 250 namespaces
  Namespace batch 1: 250 namespaces
Completed cluster kflux-stg-es01
Started processing cluster: stone-stage-p01

→ Processing cluster: stone-stage-p01
  Namespace batch 1: 242 namespaces
  Namespace batch 2: 179 namespaces
  Running Prometheus fallback for 429 namespace(s) in batches
    Prom batch 1: 250 namespaces
    Namespace openshift-sandboxed-containers-operator: 1 pod(s) found via oc
  Namespace batch 2: 250 namespaces
    Namespace openshift-machine-config-operator: 3 pod(s) found via oc
  Running Prometheus fallback for 240 namespace(s) in batches
    Prom batch 1: 240 namespaces
    Prom batch 2: 179 namespaces
  Namespace batch 2: 250 namespaces
  Namespace batch 3: 250 namespaces
  Namespace batch 3: 250 namespaces
  Namespace batch 4: 250 namespaces
  Namespace batch 4: 250 namespaces
    Namespace openshift-custom-domains-operator: 1 pod(s) found via oc
    Namespace openshift-deployment-validation-operator: 1 pod(s) found via oc
    Namespace release-service: 1 pod(s) found via oc
    Namespace openshift-machine-config-operator: 4 pod(s) found via oc
  Namespace batch 5: 250 namespaces
    Namespace release-service: 1 pod(s) found via oc
    Namespace product-kubearchive-logging: 4 pod(s) found via oc
  Namespace batch 6: 250 namespaces
    Namespace product-kubearchive-logging: 4 pod(s) found via oc
  Namespace batch 7: 250 namespaces
    Namespace openshift-multus: 14 pod(s) found via oc
  Namespace batch 5: 250 namespaces
  Namespace batch 8: 250 namespaces
  Namespace batch 6: 37 namespaces
  Namespace batch 9: 98 namespaces
  Running Prometheus fallback for 1281 namespace(s) in batches
    Prom batch 1: 250 namespaces
  Running Prometheus fallback for 2096 namespace(s) in batches
    Prom batch 1: 250 namespaces
    Prom batch 2: 250 namespaces
    Prom batch 2: 250 namespaces
    Prom batch 3: 250 namespaces
    Prom batch 3: 250 namespaces
    Prom batch 4: 250 namespaces
    Prom batch 4: 250 namespaces
    Prom batch 5: 250 namespaces
    Prom batch 5: 250 namespaces
    Prom batch 6: 31 namespaces
    Prom batch 6: 250 namespaces
    Prom batch 7: 250 namespaces
    Prom batch 8: 250 namespaces
    Prom batch 9: 96 namespaces
Completed cluster stone-prd-rh01
Started processing cluster: stone-stg-rh01
Completed cluster stone-prod-p02
Completed cluster stone-prod-p01
Completed cluster stone-stage-p01

→ Processing cluster: stone-stg-rh01
  Namespace batch 1: 250 namespaces
    Namespace dynatrace: 3 pod(s) found via oc
  Namespace batch 2: 250 namespaces
    Namespace release-service: 1 pod(s) found via oc
    Namespace openshift-pipelines: 1 pod(s) found via oc
    Namespace product-kubearchive: 1 pod(s) found via oc
    Namespace openshift-machine-config-operator: 10 pod(s) found via oc
  Namespace batch 3: 250 namespaces
  Namespace batch 4: 250 namespaces
  Namespace batch 5: 250 namespaces
  Namespace batch 6: 250 namespaces
  Namespace batch 7: 30 namespaces
  Running Prometheus fallback for 1525 namespace(s) in batches
    Prom batch 1: 250 namespaces
    Prom batch 2: 250 namespaces
    Prom batch 3: 250 namespaces
    Prom batch 4: 250 namespaces
    Prom batch 5: 250 namespaces
    Prom batch 6: 250 namespaces
    Prom batch 7: 25 namespaces
Completed cluster stone-stg-rh01
JSON written → oom_results.json
CSV written → oom_results.csv

Cluster: kflux-osp-p01
  Namespace: release-service
    Pod: release-service-monitor-deployment-756d845998-rrwpj
      - OOMKilled at: 2025-12-14T16:15:52Z
      - description_file: /private/tmp/kflux-osp-p01/release-service__release-service-monitor-deployment-756d845998-rrwpj__20251216T222305Z__desc.txt
      - pod_log_file: /private/tmp/kflux-osp-p01/release-service__release-service-monitor-deployment-756d845998-rrwpj__20251216T222305Z__log.txt

Cluster: kflux-prd-es01
  Namespace: clusters-653016ff-f619-47a2-92b3-1e8da6b45963
    Pod: catalog-operator-5b988b7694-67htb
      - CrashLoopBackOff event at: 2025-12-16T21:31:52Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__catalog-operator-5b988b7694-67htb__20251216T222222Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__catalog-operator-5b988b7694-67htb__20251216T222222Z__log.txt
    Pod: certified-operators-catalog-68bd788887-4cf9z
      - CrashLoopBackOff event at: 2025-12-16T21:31:45Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__certified-operators-catalog-68bd788887-4cf9z__20251216T222235Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__certified-operators-catalog-68bd788887-4cf9z__20251216T222235Z__log.txt
    Pod: community-operators-catalog-75648c6f9-q94j9
      - CrashLoopBackOff event at: 2025-12-16T21:31:45Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__community-operators-catalog-75648c6f9-q94j9__20251216T222252Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__community-operators-catalog-75648c6f9-q94j9__20251216T222252Z__log.txt
    Pod: redhat-marketplace-catalog-8698686b4d-cpgsh
      - CrashLoopBackOff event at: 2025-12-16T21:31:45Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__redhat-marketplace-catalog-8698686b4d-cpgsh__20251216T222311Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__redhat-marketplace-catalog-8698686b4d-cpgsh__20251216T222311Z__log.txt
    Pod: redhat-operators-catalog-7b8d47544c-r9qx8
      - CrashLoopBackOff event at: 2025-12-16T21:32:10Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__redhat-operators-catalog-7b8d47544c-r9qx8__20251216T222321Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-653016ff-f619-47a2-92b3-1e8da6b45963__redhat-operators-catalog-7b8d47544c-r9qx8__20251216T222321Z__log.txt
  Namespace: clusters-1912b582-73db-4d01-a34e-02c87a6b7556
    Pod: catalog-operator-77cf76c754-jmpkq
      - CrashLoopBackOff event at: 2025-12-16T21:31:01Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__catalog-operator-77cf76c754-jmpkq__20251216T222324Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__catalog-operator-77cf76c754-jmpkq__20251216T222324Z__log.txt
    Pod: certified-operators-catalog-dc8dd7d45-s7wt4
      - CrashLoopBackOff event at: 2025-12-16T21:30:52Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__certified-operators-catalog-dc8dd7d45-s7wt4__20251216T222324Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__certified-operators-catalog-dc8dd7d45-s7wt4__20251216T222324Z__log.txt
    Pod: community-operators-catalog-86d467f44f-6w5tx
      - CrashLoopBackOff event at: 2025-12-16T21:30:53Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__community-operators-catalog-86d467f44f-6w5tx__20251216T222338Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__community-operators-catalog-86d467f44f-6w5tx__20251216T222338Z__log.txt
    Pod: redhat-marketplace-catalog-68b5884d66-vphj9
      - CrashLoopBackOff event at: 2025-12-16T21:30:53Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__redhat-marketplace-catalog-68b5884d66-vphj9__20251216T222344Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__redhat-marketplace-catalog-68b5884d66-vphj9__20251216T222344Z__log.txt
    Pod: redhat-operators-catalog-74cf9dc99c-r54zb
      - CrashLoopBackOff event at: 2025-12-16T21:30:53Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__redhat-operators-catalog-74cf9dc99c-r54zb__20251216T222348Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-1912b582-73db-4d01-a34e-02c87a6b7556__redhat-operators-catalog-74cf9dc99c-r54zb__20251216T222348Z__log.txt
  Namespace: clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d
    Pod: catalog-operator-7ff98454b4-ms6qb
      - CrashLoopBackOff event at: 2025-12-16T21:58:01Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__catalog-operator-7ff98454b4-ms6qb__20251216T222348Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__catalog-operator-7ff98454b4-ms6qb__20251216T222348Z__log.txt
    Pod: certified-operators-catalog-7c59d7cc6b-64wjc
      - CrashLoopBackOff event at: 2025-12-16T21:57:56Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__certified-operators-catalog-7c59d7cc6b-64wjc__20251216T222353Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__certified-operators-catalog-7c59d7cc6b-64wjc__20251216T222353Z__log.txt
    Pod: community-operators-catalog-575d77f8bb-cvpcf
      - CrashLoopBackOff event at: 2025-12-16T21:57:55Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__community-operators-catalog-575d77f8bb-cvpcf__20251216T222415Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__community-operators-catalog-575d77f8bb-cvpcf__20251216T222415Z__log.txt
    Pod: redhat-marketplace-catalog-5f66485776-78rrk
      - CrashLoopBackOff event at: 2025-12-16T21:57:55Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__redhat-marketplace-catalog-5f66485776-78rrk__20251216T222423Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__redhat-marketplace-catalog-5f66485776-78rrk__20251216T222423Z__log.txt
    Pod: redhat-operators-catalog-666dd479d9-7gdvz
      - CrashLoopBackOff event at: 2025-12-16T21:58:16Z
      - description_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__redhat-operators-catalog-666dd479d9-7gdvz__20251216T222424Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-es01/clusters-a14e24b1-dfd5-4960-b4e6-4e5afa82f13d__redhat-operators-catalog-666dd479d9-7gdvz__20251216T222424Z__log.txt

Cluster: kflux-prd-rh02
  Namespace: release-service
    Pod: release-service-monitor-deployment-756d845998-fr44c
      - OOMKilled at: 2025-12-16T21:51:22Z
      - description_file: /private/tmp/kflux-prd-rh02/release-service__release-service-monitor-deployment-756d845998-fr44c__20251216T222322Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/release-service__release-service-monitor-deployment-756d845998-fr44c__20251216T222322Z__log.txt
  Namespace: openshift-machine-config-operator
    Pod: kube-rbac-proxy-crio-ip-10-208-0-177.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:15Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-177.ec2.internal__20251216T222323Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-177.ec2.internal__20251216T222323Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-0-180.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T21:26:53Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-180.ec2.internal__20251216T222324Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-180.ec2.internal__20251216T222324Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-0-190.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:16Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-190.ec2.internal__20251216T222329Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-190.ec2.internal__20251216T222329Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-0-28.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:08:56Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-28.ec2.internal__20251216T222342Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-28.ec2.internal__20251216T222342Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-0-61.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:08Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-61.ec2.internal__20251216T222347Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-0-61.ec2.internal__20251216T222347Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-108.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:12Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-108.ec2.internal__20251216T222348Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-108.ec2.internal__20251216T222348Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-120.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:24Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-120.ec2.internal__20251216T222349Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-120.ec2.internal__20251216T222349Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-124.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:24Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-124.ec2.internal__20251216T222359Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-124.ec2.internal__20251216T222359Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-141.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:00Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-141.ec2.internal__20251216T222419Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-141.ec2.internal__20251216T222419Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-149.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:07:47Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-149.ec2.internal__20251216T222423Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-149.ec2.internal__20251216T222423Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-226.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:21Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-226.ec2.internal__20251216T222424Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-226.ec2.internal__20251216T222424Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-245.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:02Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-245.ec2.internal__20251216T222425Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-245.ec2.internal__20251216T222425Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-27.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:31Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-27.ec2.internal__20251216T222429Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-27.ec2.internal__20251216T222429Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-32.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:33Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-32.ec2.internal__20251216T222439Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-32.ec2.internal__20251216T222439Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-42.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:04Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-42.ec2.internal__20251216T222443Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-42.ec2.internal__20251216T222443Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-57.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:11Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-57.ec2.internal__20251216T222444Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-57.ec2.internal__20251216T222444Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-8.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:54Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-8.ec2.internal__20251216T222447Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-8.ec2.internal__20251216T222447Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-87.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:07:41Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-87.ec2.internal__20251216T222500Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-87.ec2.internal__20251216T222500Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-1-97.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:01Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-97.ec2.internal__20251216T222510Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-1-97.ec2.internal__20251216T222510Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-2-106.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:00Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-106.ec2.internal__20251216T222523Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-106.ec2.internal__20251216T222523Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-2-202.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:07:36Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-202.ec2.internal__20251216T222524Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-202.ec2.internal__20251216T222524Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-2-242.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:07:36Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-242.ec2.internal__20251216T222525Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-242.ec2.internal__20251216T222525Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-208-2-64.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:06:12Z
      - description_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-64.ec2.internal__20251216T222526Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-208-2-64.ec2.internal__20251216T222526Z__log.txt

Cluster: kflux-ocp-p01
  Namespace: openshift-ocm-agent-operator
    Pod: ocm-agent-operator-668c457c8c-9n6gz
      - OOMKilled at: 2025-11-27T11:47:18Z
      - description_file: /private/tmp/kflux-ocp-p01/openshift-ocm-agent-operator__ocm-agent-operator-668c457c8c-9n6gz__20251216T222318Z__desc.txt
      - pod_log_file: /private/tmp/kflux-ocp-p01/openshift-ocm-agent-operator__ocm-agent-operator-668c457c8c-9n6gz__20251216T222318Z__log.txt
  Namespace: product-kubearchive-logging
    Pod: loki-querier-84f998ff96-hh7rd
      - OOMKilled at: 2025-12-15T11:53:15Z
      - description_file: /private/tmp/kflux-ocp-p01/product-kubearchive-logging__loki-querier-84f998ff96-hh7rd__20251216T222324Z__desc.txt
      - pod_log_file: /private/tmp/kflux-ocp-p01/product-kubearchive-logging__loki-querier-84f998ff96-hh7rd__20251216T222324Z__log.txt

Cluster: kflux-prd-rh03
  Namespace: release-service
    Pod: release-service-monitor-deployment-756d845998-ft5k9
      - OOMKilled at: 2025-12-16T21:15:52Z
      - description_file: /private/tmp/kflux-prd-rh03/release-service__release-service-monitor-deployment-756d845998-ft5k9__20251216T222422Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh03/release-service__release-service-monitor-deployment-756d845998-ft5k9__20251216T222422Z__log.txt
  Namespace: aws-load-balancer-operator-tenant
    Pod: aws952f341509e9bfd408e9dbd2ac09843cf4e49d8fa7a84ee09c6a4da7-pod
      - CrashLoopBackOff event at: 2025-12-16T20:08:58Z
      - description_file: /private/tmp/kflux-prd-rh03/aws-load-balancer-operator-tenant__aws952f341509e9bfd408e9dbd2ac09843cf4e49d8fa7a84ee09c6a4da7-pod__20251216T222428Z__desc.txt
      - pod_log_file: /private/tmp/kflux-prd-rh03/aws-load-balancer-operator-tenant__aws952f341509e9bfd408e9dbd2ac09843cf4e49d8fa7a84ee09c6a4da7-pod__20251216T222428Z__log.txt

Cluster: kflux-rhel-p01
  Namespace: release-service
    Pod: release-service-monitor-deployment-756d845998-cf49w
      - OOMKilled at: 2025-12-08T22:39:38Z
      - description_file: /private/tmp/kflux-rhel-p01/release-service__release-service-monitor-deployment-756d845998-cf49w__20251216T222522Z__desc.txt
      - pod_log_file: /private/tmp/kflux-rhel-p01/release-service__release-service-monitor-deployment-756d845998-cf49w__20251216T222522Z__log.txt
  Namespace: openshift-machine-config-operator
    Pod: kube-rbac-proxy-crio-ip-10-28-3-96.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T22:19:29Z
      - description_file: /private/tmp/kflux-rhel-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-28-3-96.ec2.internal__20251216T222524Z__desc.txt
      - pod_log_file: /private/tmp/kflux-rhel-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-28-3-96.ec2.internal__20251216T222524Z__log.txt
  Namespace: product-kubearchive-logging
    Pod: loki-query-frontend-779f87484b-sqq56
      - OOMKilled at: 2025-12-11T10:53:50Z
      - description_file: /private/tmp/kflux-rhel-p01/product-kubearchive-logging__loki-query-frontend-779f87484b-sqq56__20251216T222525Z__desc.txt
      - pod_log_file: /private/tmp/kflux-rhel-p01/product-kubearchive-logging__loki-query-frontend-779f87484b-sqq56__20251216T222525Z__log.txt

Cluster: kflux-stg-es01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-prd-rh01
  Namespace: release-service
    Pod: release-service-monitor-deployment-7697d75d64-nrgcz
      - OOMKilled at: 2025-12-16T22:14:18Z
      - description_file: /private/tmp/stone-prd-rh01/release-service__release-service-monitor-deployment-7697d75d64-nrgcz__20251216T223803Z__desc.txt
      - pod_log_file: /private/tmp/stone-prd-rh01/release-service__release-service-monitor-deployment-7697d75d64-nrgcz__20251216T223803Z__log.txt
  Namespace: product-kubearchive-logging
    Pod: loki-querier-7556c54664-4wq7r
      - OOMKilled at: 2025-12-16T20:00:21Z
      - CrashLoopBackOff event at: 2025-12-16T20:00:33Z
      - description_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-querier-7556c54664-4wq7r__20251216T223806Z__desc.txt
      - pod_log_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-querier-7556c54664-4wq7r__20251216T223806Z__log.txt
    Pod: loki-ingester-zone-a-0
      - OOMKilled at: 2025-12-16T00:45:54Z
      - description_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-a-0__20251216T223806Z__desc.txt
      - pod_log_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-a-0__20251216T223806Z__log.txt
    Pod: loki-ingester-zone-b-0
      - OOMKilled at: 2025-12-16T00:47:13Z
      - description_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-b-0__20251216T223808Z__desc.txt
      - pod_log_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-b-0__20251216T223808Z__log.txt
    Pod: loki-ingester-zone-c-0
      - OOMKilled at: 2025-12-16T00:45:58Z
      - description_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-c-0__20251216T223809Z__desc.txt
      - pod_log_file: /private/tmp/stone-prd-rh01/product-kubearchive-logging__loki-ingester-zone-c-0__20251216T223809Z__log.txt

Cluster: stone-prod-p02
  Namespace: openshift-custom-domains-operator
    Pod: custom-domains-operator-6f7cfcd596-5vcts
      - OOMKilled at: 2025-12-16T15:42:22Z
      - description_file: /private/tmp/stone-prod-p02/openshift-custom-domains-operator__custom-domains-operator-6f7cfcd596-5vcts__20251216T223659Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-custom-domains-operator__custom-domains-operator-6f7cfcd596-5vcts__20251216T223659Z__log.txt
  Namespace: openshift-deployment-validation-operator
    Pod: deployment-validation-operator-64df68b687-qrcw9
      - OOMKilled at: 2025-12-15T16:29:25Z
      - description_file: /private/tmp/stone-prod-p02/openshift-deployment-validation-operator__deployment-validation-operator-64df68b687-qrcw9__20251216T223706Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-deployment-validation-operator__deployment-validation-operator-64df68b687-qrcw9__20251216T223706Z__log.txt
  Namespace: release-service
    Pod: release-service-monitor-deployment-7697d75d64-lzn5b
      - OOMKilled at: 2025-12-16T21:09:15Z
      - description_file: /private/tmp/stone-prod-p02/release-service__release-service-monitor-deployment-7697d75d64-lzn5b__20251216T223712Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/release-service__release-service-monitor-deployment-7697d75d64-lzn5b__20251216T223712Z__log.txt
  Namespace: openshift-machine-config-operator
    Pod: kube-rbac-proxy-crio-ip-10-29-67-140.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:51:33Z
      - description_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-140.ec2.internal__20251216T223713Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-140.ec2.internal__20251216T223713Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-29-67-143.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:51:35Z
      - description_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-143.ec2.internal__20251216T223714Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-143.ec2.internal__20251216T223714Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-29-67-165.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:50:18Z
      - description_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-165.ec2.internal__20251216T223714Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-165.ec2.internal__20251216T223714Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-29-67-177.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T22:36:11Z
      - description_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-177.ec2.internal__20251216T223715Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-67-177.ec2.internal__20251216T223715Z__log.txt
  Namespace: product-kubearchive-logging
    Pod: loki-ingester-zone-b-0
      - OOMKilled at: 2025-12-09T17:30:24Z
      - description_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-ingester-zone-b-0__20251216T223716Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-ingester-zone-b-0__20251216T223716Z__log.txt
    Pod: loki-ingester-zone-c-0
      - OOMKilled at: 2025-12-09T17:22:16Z
      - description_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-ingester-zone-c-0__20251216T223720Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-ingester-zone-c-0__20251216T223720Z__log.txt
    Pod: loki-querier-f4d86f99-7s5cj
      - OOMKilled at: 2025-12-15T07:25:00Z
      - description_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-querier-f4d86f99-7s5cj__20251216T223843Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-querier-f4d86f99-7s5cj__20251216T223843Z__log.txt
    Pod: loki-query-frontend-65984fc57f-jmvbs
      - OOMKilled at: 2025-12-10T17:01:58Z
      - description_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-query-frontend-65984fc57f-jmvbs__20251216T223901Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/product-kubearchive-logging__loki-query-frontend-65984fc57f-jmvbs__20251216T223901Z__log.txt
  Namespace: openshift-multus
    Pod: multus-4fmqt
      - CrashLoopBackOff event at: 2025-12-16T20:49:36Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-4fmqt__20251216T223904Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-4fmqt__20251216T223904Z__log.txt
    Pod: multus-bk62k
      - CrashLoopBackOff event at: 2025-12-16T20:49:35Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-bk62k__20251216T223904Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-bk62k__20251216T223904Z__log.txt
    Pod: multus-cgq52
      - CrashLoopBackOff event at: 2025-12-16T20:49:16Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-cgq52__20251216T223905Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-cgq52__20251216T223905Z__log.txt
    Pod: multus-hhgl2
      - CrashLoopBackOff event at: 2025-12-16T20:49:03Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-hhgl2__20251216T223906Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-hhgl2__20251216T223906Z__log.txt
    Pod: multus-j9slq
      - CrashLoopBackOff event at: 2025-12-16T20:49:44Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-j9slq__20251216T223907Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-j9slq__20251216T223907Z__log.txt
    Pod: multus-kbd9s
      - CrashLoopBackOff event at: 2025-12-16T20:49:38Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-kbd9s__20251216T223907Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-kbd9s__20251216T223907Z__log.txt
    Pod: multus-kg4p9
      - CrashLoopBackOff event at: 2025-12-16T20:49:02Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-kg4p9__20251216T223908Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-kg4p9__20251216T223908Z__log.txt
    Pod: multus-mzk8z
      - CrashLoopBackOff event at: 2025-12-16T20:49:35Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-mzk8z__20251216T223909Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-mzk8z__20251216T223909Z__log.txt
    Pod: multus-nsz68
      - CrashLoopBackOff event at: 2025-12-16T20:50:08Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-nsz68__20251216T223921Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-nsz68__20251216T223921Z__log.txt
    Pod: multus-qv4th
      - CrashLoopBackOff event at: 2025-12-16T20:49:39Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-qv4th__20251216T223938Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-qv4th__20251216T223938Z__log.txt
    Pod: multus-r299x
      - CrashLoopBackOff event at: 2025-12-16T20:49:59Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-r299x__20251216T223944Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-r299x__20251216T223944Z__log.txt
    Pod: multus-rqml8
      - CrashLoopBackOff event at: 2025-12-16T20:49:46Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-rqml8__20251216T223945Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-rqml8__20251216T223945Z__log.txt
    Pod: multus-sh7kb
      - CrashLoopBackOff event at: 2025-12-16T20:49:37Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-sh7kb__20251216T223946Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-sh7kb__20251216T223946Z__log.txt
    Pod: multus-t7w9k
      - CrashLoopBackOff event at: 2025-12-16T20:49:11Z
      - description_file: /private/tmp/stone-prod-p02/openshift-multus__multus-t7w9k__20251216T223947Z__desc.txt
      - pod_log_file: /private/tmp/stone-prod-p02/openshift-multus__multus-t7w9k__20251216T223947Z__log.txt

Cluster: stone-prod-p01
  (no namespaces with OOM/CrashLoopBackOff found)

Cluster: stone-stage-p01
  Namespace: openshift-sandboxed-containers-operator
    Pod: controller-manager-5c8d7f4698-mcffp
      - OOMKilled at: 2025-12-16T20:41:12Z
      - CrashLoopBackOff event at: 2025-12-16T20:45:45Z
      - description_file: /private/tmp/stone-stage-p01/openshift-sandboxed-containers-operator__controller-manager-5c8d7f4698-mcffp__20251216T223100Z__desc.txt
      - pod_log_file: /private/tmp/stone-stage-p01/openshift-sandboxed-containers-operator__controller-manager-5c8d7f4698-mcffp__20251216T223100Z__log.txt
  Namespace: openshift-machine-config-operator
    Pod: kube-rbac-proxy-crio-ip-10-29-64-118.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:04:49Z
      - description_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-118.ec2.internal__20251216T223110Z__desc.txt
      - pod_log_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-118.ec2.internal__20251216T223110Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-29-64-145.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T19:40:05Z
      - description_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-145.ec2.internal__20251216T223122Z__desc.txt
      - pod_log_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-145.ec2.internal__20251216T223122Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-29-64-187.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T21:55:53Z
      - description_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-187.ec2.internal__20251216T223132Z__desc.txt
      - pod_log_file: /private/tmp/stone-stage-p01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-29-64-187.ec2.internal__20251216T223132Z__log.txt

Cluster: stone-stg-rh01
  Namespace: dynatrace
    Pod: konflux-stage-cluster-oneagent-l4nx5
      - CrashLoopBackOff event at: 2025-12-16T20:12:31Z
      - description_file: /private/tmp/stone-stg-rh01/dynatrace__konflux-stage-cluster-oneagent-l4nx5__20251216T225017Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/dynatrace__konflux-stage-cluster-oneagent-l4nx5__20251216T225017Z__log.txt
    Pod: konflux-stage-cluster-oneagent-wwr5m
      - CrashLoopBackOff event at: 2025-12-16T20:12:28Z
      - description_file: /private/tmp/stone-stg-rh01/dynatrace__konflux-stage-cluster-oneagent-wwr5m__20251216T225019Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/dynatrace__konflux-stage-cluster-oneagent-wwr5m__20251216T225019Z__log.txt
    Pod: dynatrace-operator-78679cc5f5-vb4vb
      - OOMKilled at: 2025-12-16T21:41:53Z
      - description_file: /private/tmp/stone-stg-rh01/dynatrace__dynatrace-operator-78679cc5f5-vb4vb__20251216T225019Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/dynatrace__dynatrace-operator-78679cc5f5-vb4vb__20251216T225019Z__log.txt
  Namespace: release-service
    Pod: release-service-monitor-deployment-5ff7db8c7c-wfbmn
      - OOMKilled at: 2025-12-16T21:30:05Z
      - description_file: /private/tmp/stone-stg-rh01/release-service__release-service-monitor-deployment-5ff7db8c7c-wfbmn__20251216T225043Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/release-service__release-service-monitor-deployment-5ff7db8c7c-wfbmn__20251216T225043Z__log.txt
  Namespace: openshift-pipelines
    Pod: pipeline-metrics-exporter-564ff8489f-77z9h
      - OOMKilled at: 2025-12-15T03:11:27Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-pipelines__pipeline-metrics-exporter-564ff8489f-77z9h__20251216T225051Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-pipelines__pipeline-metrics-exporter-564ff8489f-77z9h__20251216T225051Z__log.txt
  Namespace: product-kubearchive
    Pod: kubearchive-operator-6bdd786d7d-gt4l8
      - OOMKilled at: 2025-12-13T08:46:21Z
      - description_file: /private/tmp/stone-stg-rh01/product-kubearchive__kubearchive-operator-6bdd786d7d-gt4l8__20251216T225054Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/product-kubearchive__kubearchive-operator-6bdd786d7d-gt4l8__20251216T225054Z__log.txt
  Namespace: openshift-machine-config-operator
    Pod: kube-rbac-proxy-crio-ip-10-202-27-215.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T22:16:29Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-27-215.ec2.internal__20251216T225055Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-27-215.ec2.internal__20251216T225055Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-28-119.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:59Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-119.ec2.internal__20251216T225056Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-119.ec2.internal__20251216T225056Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-28-157.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:10:04Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-157.ec2.internal__20251216T225056Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-157.ec2.internal__20251216T225056Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-28-182.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:10:22Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-182.ec2.internal__20251216T225057Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-28-182.ec2.internal__20251216T225057Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-124.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:10:48Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-124.ec2.internal__20251216T225057Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-124.ec2.internal__20251216T225057Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-136.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:11:02Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-136.ec2.internal__20251216T225058Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-136.ec2.internal__20251216T225058Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-203.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:10:19Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-203.ec2.internal__20251216T225059Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-203.ec2.internal__20251216T225059Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-247.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T22:13:36Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-247.ec2.internal__20251216T225059Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-247.ec2.internal__20251216T225059Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-248.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:10:37Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-248.ec2.internal__20251216T225100Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-248.ec2.internal__20251216T225100Z__log.txt
    Pod: kube-rbac-proxy-crio-ip-10-202-29-4.ec2.internal
      - CrashLoopBackOff event at: 2025-12-16T20:09:49Z
      - description_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-4.ec2.internal__20251216T225100Z__desc.txt
      - pod_log_file: /private/tmp/stone-stg-rh01/openshift-machine-config-operator__kube-rbac-proxy-crio-ip-10-202-29-4.ec2.internal__20251216T225100Z__log.txt

Per-cluster logs written to /tmp/<cluster>/ (if any findings were found)

Cc: @jhutar

@smodak-rh
Copy link
Contributor Author

smodak-rh commented Dec 16, 2025

oom_results.csv
oom_results.json

@jhutar jhutar merged commit ccc50b6 into redhat-appstudio:main Dec 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@smodak-rh @jhutar