fix(config): raise ci00/ci01 CosmosDB resourceContainerMaxScale to 20000 (AROSLSRE-2037) - #6844
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: raelga The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
There was a problem hiding this comment.
🟢 Approval recommended
The change is a scoped config-only bump for ci00/ci01 with correctly regenerated rendered outputs and no schema constraints preventing the new value.
Pull request overview
Raises the Cosmos DB Resources container autoscale ceiling for the ci00 and ci01 dev environments to reduce 429 throttling during e2e-parallel and tide batch load, by updating the source config and the corresponding rendered environment outputs.
Changes:
- Bump
frontend.cosmosDB.resourceContainerMaxScalefrom 10000 → 20000 forci00andci01defaults inconfig/config.yaml. - Regenerate the rendered configs for
dev/ci00/centralusanddev/ci01/centralusto reflect the new value.
File summaries
| File | Description |
|---|---|
| config/config.yaml | Updates ci00 and ci01 defaults to set frontend.cosmosDB.resourceContainerMaxScale: 20000. |
| config/rendered/dev/ci00/centralus.yaml | Regenerated rendered config reflecting resourceContainerMaxScale: 20000 for ci00. |
| config/rendered/dev/ci01/centralus.yaml | Regenerated rendered config reflecting resourceContainerMaxScale: 20000 for ci01. |
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
/test e2e-parallel Retesting: pull-ci-Azure-ARO-HCP-main-e2e-parallel/2096308372608389120 failed on a single spec, "SRE can pause schedules to stop backup execution for an HCP cluster", timing out after 660s waiting for a backup schedule's LastBackupTime to populate. This PR only bumps a CosmosDB config value for the ci00/ci01 accounts, it has no code path affecting backup scheduling. Scope check across concurrently running PRs confirms this is shared-CI contention, not a regression from this diff: e2e-parallel failed at the same time on unrelated PRs #6842 (different spec: cluster-health check timeout) and #6838, each with a different failing spec, none touching backups. |
|
/test e2e-parallel Retesting: pull-ci-Azure-ARO-HCP-main-e2e-parallel/2096341884518535168 failed only in the post-step |
|
/test e2e-parallel Retesting: pull-ci-Azure-ARO-HCP-main-e2e-parallel/2096376658683498496 failed on "Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API" with the HyperShift hosted-control-plane router deployment reporting unavailable replicas, an isolated provisioning flake. Scope check confirms e2e-parallel is failing concurrently across 5 unrelated open PRs (#6845, #6842, #6838, #6836) right now, confirming shared-CI contention rather than a regression from this config-only diff. Switching to a periodic backoff retest loop given the sustained fleet-wide episode. |
|
/test e2e-parallel Periodic backoff retest: this job has been failing repeatedly today with a fleet-wide infra flake (post-step gather-observability / isolated provisioning issues), unrelated to this PR's config-only diff. Retesting per the 20-min backoff cadence. |
1 similar comment
|
/test e2e-parallel Periodic backoff retest: this job has been failing repeatedly today with a fleet-wide infra flake (post-step gather-observability / isolated provisioning issues), unrelated to this PR's config-only diff. Retesting per the 20-min backoff cadence. |
|
/test e2e-parallel Periodic backoff retest: fleet-wide infra flake, unrelated to this PR's config-only diff. |
…000 (AROSLSRE-2037) Ephemeral per-job CosmosDB accounts in ci00/ci01 hit Cosmos DB Throttled Requests (429) alerts during e2e-parallel and tide batch runs. Raises the autoscale ceiling on the Resources container from 10000 to 20000 RU/s for these two CI environments only.
29071c8 to
ec3a2c3
Compare
|
Rebased onto main to resolve the conflict. Main's cosmosdb-container-maxscale bot bumped the global default resourceContainerMaxScale from 10000 to 15000 (#6629), which no longer matches the base this branch was cut from. Rebased cleanly, kept the ci00/ci01 override at 20000 as intended by this PR, and re-ran config materialize to confirm the rendered output matches. |
There was a problem hiding this comment.
🟢 Approval recommended
Config-only change with rendered artifacts updated; no functional or security issues found beyond a minor PR-description inconsistency.
Review details
- Files reviewed: 3/3 changed files
- Comments generated: 1
- Review effort level: Lite
| private: false | ||
| resourceContainerMaxScale: 20000 | ||
| zoneRedundantMode: 'Disabled' |
|
The fix was adapting the alert. Throttling at the beginning of an E2E Test is expected because coamosdb needs to realize it needs to scale RUs. The new alert honors that. |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/close per Gerd Oberlechner (@geoberle) comment |
|
/close |
|
Rael Garcia (@raelga): Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
AROSLSRE-2037
What
Raises
frontend.cosmosDB.resourceContainerMaxScalefrom 10000 to 20000 forthe
ci00andci01environments inconfig/config.yaml, and regeneratesconfig/rendered/dev/ci00/centralus.yamlandconfig/rendered/dev/ci01/centralus.yamlviamake materialize.Why
Each e2e-parallel test job provisions its own ephemeral CosmosDB account in
the shared
ci00/ci01CI underlay regions (e.g.arohcpci01-rp-j5814272). These accounts are hitting Cosmos DB ThrottledRequests (429) alerts on the Resources container during e2e-parallel runs
and tide batch jobs. Raising the autoscale ceiling gives the container more
headroom to absorb load spikes; autoscale billing only charges for RU/s
actually consumed, so this has no cost impact unless the higher ceiling is
actually needed.
Testing
cd config && make materializeandconfirmed the diff is scoped to
config/rendered/dev/ci00/centralus.yamland
config/rendered/dev/ci01/centralus.yamlonly.Special notes for your reviewer
Companion change (enabling CosmosDB burst capacity for the same
environments) is tracked separately in
AROSLSRE-2038 as its
own PR, kept independent per one-PR-one-change discipline.