Skip to content

Fail health check on document processing deadlock/CPU starvation - #37591

Draft
hmusum wants to merge 1 commit into
masterfrom
hmusum/docproc-liveness-health-check
Draft

hmusum wants to merge 1 commit into
masterfrom
hmusum/docproc-liveness-health-check

Conversation

@hmusum

@hmusum hmusum commented Aug 17, 2026

Copy link
Copy Markdown
Member

Summary

  • Health checks (/state/v1/health, status.html) previously only reflected backend cluster availability, with no signal for a deadlocked or CPU-starved container.
  • Adds a liveness gate to VipStatus, independent of cluster status, driven by a new DocprocLivenessWatchdog that watches whether document processing requests are actually completing.
  • The watchdog samples submitted-vs-completed request counters on the live docproc request path and flags a stall (outstanding requests, zero completions for a sustained period), with hysteresis to trip/clear without flapping.
  • New config in config.docproc.docproc.def: livenessCheckEnabled (default true), livenessSampleIntervalSeconds, livenessStalledThresholdSeconds.

Test plan

  • New unit tests for the watchdog's stall-detection logic (idle, progressing, stalled-under/over-threshold, recovery)
  • Extended VipStatusTestCase for the new liveness gate interactions
  • mvn test passes for docproc and container-disc modules

Generated with Claude Code.

- Add a liveness gate to VipStatus, independent of cluster status,
  so /state/v1/health and status.html reflect a stalled container
- Add DocprocLivenessWatchdog: detects when requests are outstanding
  with no completions for a sustained period, with hysteresis to
  avoid flapping
- Wire submitted/completed counters into the live docproc request
  path (DocumentProcessingHandler/DocumentProcessingTask)
- New config: livenessCheckEnabled (default true), sample interval
  and stalled threshold
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant