Skip to content

Record the VM region, zone and capacity wait in Engine e2e results - #451

Draft
anle2024 wants to merge 1 commit into
mainfrom
anle/e2e-result-placement
Draft

anle2024 wants to merge 1 commit into
mainfrom
anle/e2e-result-placement

Conversation

@anle2024

@anle2024 anle2024 commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Each Engine e2e case result now records where its VM ran and how long the pool waited for that VM:

  • placement: the region and the zone of the VM, for example {"region": "eu-west-2", "zone": "eu-west-2a"}.
  • capacity_wait_seconds: the time from the pool start to the VM boot, in seconds, never below 0.

Results of cases that no VM ran, such as capacity and bootstrap failures, carry the same fields with null. So every result has one shape.

Why

The runner pools move to several regions in the multi-region sweep PR. Two capacity campaigns in November asked for 96 and 48 g7e VMs. Every VM came from a region other than us-west-2. The median wait ranged from minutes to hours by region and VM size. Today a case result does not record the region of its VM or the wait for it. So a slow region or a long wait does not show in the reports.

Design

The picture shows one run of the multi-region pools. This PR changes step 9: the result of each case records its region, zone and capacity wait. The other steps come from the multi-region sweep PR.

flowchart LR
  subgraph GHA["GitHub run"]
    PJ["Pool job<br/>starts the pools"]
  end
  subgraph HOME["AWS us-west-2 (home)"]
    SCH["Scheduler<br/>sweep and report"]
    BR["Broker Lambda<br/>sweep, tokens, report"]
    S3[("S3 bucket<br/>data, results, records")]
  end
  subgraph REG["AWS, each of 9 regions"]
    EXP["Expiry schedule<br/>one per group"]
    ASG["Auto Scaling group<br/>per pool, Launch suspended"]
    VM["g7e VMs<br/>one runner each"]
  end
  subgraph GHB["GitHub runners"]
    API["GitHub API<br/>jobs, runner tokens"]
    CJ["Case jobs<br/>wait for a runner"]
  end
  PJ -- "1 create" --> SCH
  PJ -- "1 create" --> EXP
  PJ -- "1 create" --> ASG
  SCH -- "2 sweep every minute" --> BR
  BR == "3 LaunchInstances" ==> ASG
  BR -- "4 sweep record" --> S3
  ASG -- "5 start VM" --> VM
  VM -- "6 token request" --> BR
  BR -- "7 jobs, tokens, failures" --> API
  VM -- "8 take a case" --> CJ
  VM -- "9 data, result" --> S3
  EXP -- "10 delete at expiry" --> ASG
Loading

Before and now

Part Before (main) Now (this PR)
Runner boot, engine-e2e-runner.sh Reads the instance type from the instance metadata service (IMDS, the local information endpoint of the VM) Also reads the region and the zone, and computes the capacity wait from the boot time in /proc/stat
Runner .env Instance type and capacity deadline Adds ENGINE_E2E_REGION, ENGINE_E2E_ZONE and ENGINE_E2E_CAPACITY_WAIT_SECONDS
Case result, run() in common.py hardware Adds placement and capacity_wait_seconds
Failure results in kumo-engine-e2e.yml and engine-e2e-lifecycle.yml No placement fields The same fields. If no VM ran, they are null

The pool start is the capacity deadline minus the tier window: 4 hours for nightly runs and 24 hours for weekly runs, as in engine-e2e-pool.cjs. A test runs the runner formula against a deadline from start(), so the two windows cannot drift apart.

Why this is better

  • Each result answers "which region and zone" and "how long did the pool wait" without a search in AWS records.
  • A bad metadata value stops the boot with a named check (instance region, instance zone or capacity wait) instead of a wrong result.
  • The cost is two metadata reads at boot. The largest launch template user data (48 cases) grows by about 1 KB, to 12,991 of 16,384 bytes.

Trade-off

Nightly runners take several cases until #319 merges. Until then, a nightly result records the boot wait of its VM, not the queue time of its case. Example: a VM that boots 30 minutes after the pool start and runs three cases reports 1800 for all three. If a pool has fewer VMs than cases, the wait of a later VM also includes the run time of earlier cases.

The multi-region sweep PR also changes engine-e2e-runner.sh, engine-e2e-lifecycle.yml, the cases README, test-engine-e2e-pool.cjs and test_engine_e2e_case_hardware.py. A trial merge conflicts in engine-e2e-runner.sh and in both test files. The PR that merges second needs a rebase.

Change map

File-by-file change map

Testing Done

  • node --test test/scripts/test-engine-e2e-pool.cjs: 30 of 30 pass on Node 24 and on Node 22.
  • The pytest list of oss-readiness.yml: 214 pass. The only failure is test_history_scan_explicitly_selects_all_commits, which needs a full clone. CI uses a full clone.
  • check_oss_readiness.sh --diff, check_secrets.sh --diff (gitleaks 8.30.1), yamllint, shellcheck, flake8 and isort pass.
  • New runner tests use fake metadata and a fake /proc/stat. They cover nightly, weekly, clock skew, a bad region, a bad zone, a Local Zone name, a missing boot time and the exact .env lines.
  • Other new tests cover each result writer, the token lifetime, and the runner formula against a start() deadline for both tiers.
  • Mutation checks: I broke each rule on purpose in a scratch copy. Examples: the 4-hour window, the floor at 0, the zone check, the token lifetime and a swapped .env field. A test failed for each one.
  • Real metadata: on an EC2 VM, the exact runner commands returned us-west-2 and us-west-2b. Both values pass the new checks, and btime equals the boot time from /proc/uptime.
  • Three independent review passes. This PR fixes every finding that a reproduction proved.
  • No cloud run of the pool. The scheduler flags stay off.

Hotfix Context

Not a hotfix.

Checklist

  • I listed all the breaking changes introduced by the PR: none. The new result fields are additive, and compare_reference ignores them.
  • I made sure to update examples and documentation with my changes (if necessary)
  • I have added and modified unit tests required to cover changes made by this PR.

🤖 Generated with Claude Code

The runner reads the region and zone from the instance metadata service
and its boot time from /proc/stat. It writes them, with the seconds from
the pool start to the boot, to the runner .env file. Case results and
setup failure results now record placement and capacity_wait_seconds.

Admission failure results and lifecycle runner failure results record
null values for both fields, because no VM ran those cases. The VM that
rejects a case at admission did not run it, and a bootstrap error comes
from another VM. If the wait value is not a non-negative integer, the
case and setup failure writers record null and still write the result
file.

The runner now requests a metadata token that lives 300 seconds. Three
reads use the token, and each read can retry for close to a minute. The
zone check accepts every zone name that starts with the region and has
only lowercase letters, digits and hyphens, so Local Zones pass. A bad
region, zone or boot time fails the boot with its own check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@copy-pr-bot

copy-pr-bot Bot commented Nov 21, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@anle2024
anle2024 temporarily deployed to gha-kumo-workflow-tests November 21, 2026 00:52 — with GitHub Actions Inactive
@coderabbitai

coderabbitai Bot commented Nov 21, 2026 •

Copy link
Copy Markdown

Note

.coderabbit.yaml has unrecognized properties

CodeRabbit is using all valid settings from your configuration. Unrecognized properties (listed below) have been ignored and may indicate typos or deprecated fields that can be removed.

Parsing warnings: 1

Validation error: Unrecognized key: "tone_instructions" at "reviews"
Configuration instructions

  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
Pre-merge checks: Passed 2 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Change Map Design Review Inconclusive The custom check reviews the PR conversation comment that starts with <!-- kumo-pr-change-map -->. The PR description links to that comment (issuecomment-7054551738), but I could not retrieve it. My… To complete this check, provide the body of the PR conversation comment https://github.com/NVIDIA/kumo-engine/pull/451#issuecomment-7054551738, or grant a tool that reads PR comments. Then re-run the check against these points: 1. Each ro…
Passed Passed checks: 2 passed

Check name Status Explanation
Pr Size Passed The PR changes 7 files, which does not exceed the threshold of 40, so the check passes. Evidence: - The PR head commit 626b818 ("Record the VM region, zone and capacity wait in Engine e2e results") c…
Public Resource Access Passed The check applies only to external resource references added or modified by this pull request. I compared the PR's own commit (626b818, parent 11fe5e0) with its parent. It changes 7 files: `.github/…
Full details: Pr Size

Explanation

The PR changes 7 files, which does not exceed the threshold of 40, so the check passes. Evidence: - The PR head commit 626b818 ("Record the VM region, zone and capacity wait in Engine e2e results") changes these 7 files relative to its parent 11fe5e0 (rename-aware, git show --name-status -M): - .github/scripts/engine-e2e-runner.sh (M) - .github/workflows/engine-e2e-lifecycle.yml (M) - .github/workflows/kumo-engine-e2e.yml (M) - packages/kumo-engine/test/cases/README.md (M) - packages/kumo-engine/test/cases/common.py (M) - test/scripts/test-engine-e2e-pool.cjs (M) - test/scripts/test_engine_e2e_case_hardware.py (M) - These files match the scope in the PR description: runner, workflows, run() in common.py, the cases README, and the pool and hardware tests. Caveat on the base: - The local origin/main (b53781b, dated 2026-11-17) does not contain the PR's parent 11fe5e0 (dated 2026-11-20). The local origin/main is therefore stale. - A diff against that stale ref shows 311 files. Those files come from other merged PRs (for example #427, #376, #423) and are not part of this PR. I did not count them. - The provider-reported target is main. The head branch contains 11fe5e0, which is a main-line commit. The PR diff is therefore the single 7-file commit. - The commit subjects were unavailable, so I could not confirm the commit count from the provider. The 7-file count is the diff of the PR's own commit. Result: 7 files, which is at or below 40. No failure message is required. Line counts were ignored as the check requires.

Full details: Public Resource Access

Explanation

The check applies only to external resource references added or modified by this pull request. I compared the PR's own commit (626b818, parent 11fe5e0) with its parent. It changes 7 files: .github/scripts/engine-e2e-runner.sh, engine-e2e-lifecycle.yml, kumo-engine-e2e.yml, packages/kumo-engine/test/cases/README.md, packages/kumo-engine/test/cases/common.py, test/scripts/test-engine-e2e-pool.cjs and test/scripts/test_engine_e2e_case_hardware.py. Scanning every added line for URL schemes, git/S3/package sources, and public hosts found no new public URL, repository location, or package source. The only added URL-like references are four lines in engine-e2e-runner.sh (lines 6, 7, 10 and 13 of the diff). They point to http://169.254.169.254/latest/api/token and .../meta-data/placement/region and .../placement/availability-zone. This is the EC2 Instance Metadata Service. It is a link-local endpoint that exists only inside the running instance. It is runtime infrastructure, not an external resource that an anonymous public user could or should access, so it falls outside the "external resource reference" scope. The test fixtures use fake metadata, which the check excludes. The PR description links to other NVIDIA PRs and a comment. Those links are not reference text in the changed repository files, so they are outside the scope of this check. Note: the local origin/main has moved past the PR base, so a full origin/main...HEAD diff shows about 311 unrelated files. I limited the review to the PR's own commit.

Full details: Change Map Design Review

Explanation

The custom check reviews the PR conversation comment that starts with &lt;!-- kumo-pr-change-map --&gt;. The PR description links to that comment (issuecomment-7054551738), but I could not retrieve it. My tools read only the local checkout and the primary PR author record. The checkout has no copy of the comment. A search of the repository found only the policy text in docs/development/review.md, which requires one such comment per PR. So I could not check the change map's necessity, abstraction layer, behavior preservation, observed failure, verification, or stopping condition. I did not assess the code changes against the map. This is a missing-evidence result, not a pass. The check does not establish a defect. Context from the PR description: the change adds placement and capacity_wait_seconds to each Engine e2e case result, and it changes engine-e2e-runner.sh, common.py, and the two workflow files. The author states that the design depends on PR #455, and that PR #319 affects nightly queue-time semantics. The description also names a known trade-off: nightly results record boot wait, not case queue time, until #319 merges. A reviewer should check that the map states this trade-off.

Resolution

To complete this check, provide the body of the PR conversation comment https://github.com/NVIDIA/kumo-engine/pull/451#issuecomment-7054551738, or grant a tool that reads PR comments. Then re-run the check against these points: 1. Each row states a failure, a release need, or a stated requirement that the change addresses. 2. The map states the existing behavior that the author inspected. 3. Each change is the smallest change that meets the requirement, and it sits at the correct layer. Check whether the runner .env fields and the run() result fields are placed in the right layer. 4. The map names the focused verification and the stopping condition. 5. The map explains the nightly trade-off (boot wait versus queue time until #319 merges). It also explains the dependency on PR #455 and the rebase conflicts that the description names. 6. The map covers the omitted material behavior, including the null placement on failure results and the extra launch-template user data size.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@anle2024

Copy link
Copy Markdown
Collaborator Author

Change map

Requested behavior: each Engine e2e case result records the region and zone of its VM and the capacity wait of the pool for that VM.
Requirement source: the multi-region runner design of 20 November 2026. Without these fields, a result does not show which region ran a case or how long the pool waited for capacity.
Existing behavior: a result records only hardware.instance_type. Failure results from the capacity, setup and lifecycle writers have no placement.
Smallest change: the runner reads the region and zone from the instance metadata service and the boot time from /proc/stat. It writes three values to its .env. run() and every failure writer add placement and capacity_wait_seconds. Results of cases that no VM ran record null.
Verification: runner tests with fake metadata and a fake /proc/stat, tests for each result writer and for the pool window, a mutation check for each rule, and the exact metadata commands on a real EC2 VM. All CI-equivalent checks pass locally.
Stopping condition: every writer of result.json produces the same fields. If a new rule breaks, a test fails.

File Function / scope What changed Why
.github/scripts/engine-e2e-runner.sh Boot checks and .env Reads the region and zone with the metadata token and rejects bad values. Computes the capacity wait from btime. Writes ENGINE_E2E_REGION, ENGINE_E2E_ZONE and ENGINE_E2E_CAPACITY_WAIT_SECONDS. The metadata token lasts 300 seconds. The case job gets the VM placement from the runner.
packages/kumo-engine/test/cases/common.py run() Adds placement and capacity_wait_seconds next to hardware. Missing or invalid values become null. Every case result records where it ran.
.github/workflows/kumo-engine-e2e.yml Admission step, setup-failure writer The admission failure result records null placement, because another VM failed or no VM ran the case. The setup-failure result records the runner values. All results share one shape.
.github/workflows/engine-e2e-lifecycle.yml failed-case job The runner failure result records null placement and capacity wait. No VM ran these cases.
packages/kumo-engine/test/cases/README.md Results section Describes the new fields and the null cases. Matches the code.
test/scripts/test_engine_e2e_case_hardware.py Runner and writer tests Fake metadata and btime cases, every result writer, and the token lifetime. Covers each new rule.
test/scripts/test-engine-e2e-pool.cjs Capacity wait test Runs the runner formula against a deadline from start() for both tiers. The runner and the pool must use the same windows.

This branch was previously deployed

1 inactive deployment
gha-kumo-workflow-tests — 626b818b Deployed Nov 21, 2026 by anle2024 via import #2864
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant