feat(system-update): add progress ETA for entity-consistency scans - #19388
Open
david-leifker wants to merge 4 commits into
Open
feat(system-update): add progress ETA for entity-consistency scans#19388david-leifker wants to merge 4 commits into
david-leifker wants to merge 4 commits into
Conversation
Introduce reusable ProgressTracker and ConsistencyScanRunner so long entity-consistency upgrade runs can log rate and ETA without log spam, and expose additive totalEstimate on OpenAPI consistency responses. Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
product
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 23 files
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
maggiehays
added
the
needs-review
Correct resume rate/ETA baseline, cap onStart totals with limit, preserve zero totals, default warmup, narrow countMatching failures, stop scan on delay interrupt, and clear per-batch orphan state while keeping shared caches. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 23d8b03. Configure here.
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
…upted Propagate cancelled state from ConsistencyScanRunner through FixEntityConsistencyStep so interrupt during batch delay does not mark entity types or the job SUCCEEDED. Also soft-fail countMatching on ES errors only and normalize negative ProgressTracker inputs. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 9 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Return Action.ABORT on interrupt so DefaultUpgradeManager does not mark the upgrade SUCCEEDED, cap batch size to honor scan limits exactly, log countMatching failures with stack traces, and assert cancellation on the test thread after join. Co-authored-by: Cursor <cursoragent@cursor.com>
david-leifker
enabled auto-merge (squash)
August 24, 2026 22:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ProgressTrackerandConsistencyScanRunnerso long-running entity-consistency upgrade scans can report rate and human-readable ETA (~1 INFO line/min after warmup) without per-batch log spam.FixEntityConsistencySteponto the runner with configurableprogressLogIntervalMs/progressWarmupMs.totalEstimate(matching entity/doc count) on OpenAPI consistency check/fix responses for the first page.Test plan
:metadata-iounit tests forProgressTracker,ConsistencyScanRunner,ConsistencyService:datahub-upgradeFixEntityConsistencyStepTest:metadata-service:openapi-servletConsistencyControllerTestMade with Cursor
Summary by cubic
Adds ETA-based, throttled progress to entity-consistency scans and exposes a total estimate on first-page OpenAPI responses. Previously we logged per batch without ETA and always marked success; now we log rate and human-readable ETA after warmup, persist silent checkpoints, cap batches to respect limits, and keep IN_PROGRESS with Action.ABORT when a scan is interrupted.
metadata-io:ProgressTracker,ConsistencyScanRunner, andConsistencyService.countMatchingbacked byESSystemMetadataDAO.count; soft-fails on ES errors, logs failures, caps totals by limit (including zero), and only enables entity ETA when key-aspect-only is true.FixEntityConsistencyStepuses the runner: preserves batch delay/limit, caps batch size to honor limits, stops on interrupt and returns IN_PROGRESS/ABORT, clears per-batch orphan URNs, silently checkpoints scroll/counters/last ETA, and records totalEstimate in the final state./consistency/checkand/consistency/fixinclude totalEstimate on the first page only; resume pages skip counting.Written for commit 29e1d20. Summary will update on new commits.