Repository navigation
Conversation
The runner reads the region and zone from the instance metadata service and its boot time from /proc/stat. It writes them, with the seconds from the pool start to the boot, to the runner .env file. Case results and setup failure results now record placement and capacity_wait_seconds. Admission failure results and lifecycle runner failure results record null values for both fields, because no VM ran those cases. The VM that rejects a case at admission did not run it, and a bootstrap error comes from another VM. If the wait value is not a non-negative integer, the case and setup failure writers record null and still write the result file. The runner now requests a metadata token that lives 300 seconds. Three reads use the token, and each read can retry for close to a minute. The zone check accepts every zone name that starts with the region and has only lowercase letters, digits and hyphens, so Local Zones pass. A bad region, zone or boot time fails the boot with its own check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
Note
|
| Check name | Status | Explanation |
|---|---|---|
| Pr Size | The PR changes 7 files, which does not exceed the threshold of 40, so the check passes. Evidence: - The PR head commit 626b818 ("Record the VM region, zone and capacity wait in Engine e2e results") c… | |
| Public Resource Access | The check applies only to external resource references added or modified by this pull request. I compared the PR's own commit (626b818, parent 11fe5e0) with its parent. It changes 7 files: `.github/… |
Full details: Pr Size
-
Explanation
The PR changes 7 files, which does not exceed the threshold of 40, so the check passes. Evidence: - The PR head commit 626b818 ("Record the VM region, zone and capacity wait in Engine e2e results") changes these 7 files relative to its parent 11fe5e0 (rename-aware,
git show --name-status -M): -.github/scripts/engine-e2e-runner.sh(M) -.github/workflows/engine-e2e-lifecycle.yml(M) -.github/workflows/kumo-engine-e2e.yml(M) -packages/kumo-engine/test/cases/README.md(M) -packages/kumo-engine/test/cases/common.py(M) -test/scripts/test-engine-e2e-pool.cjs(M) -test/scripts/test_engine_e2e_case_hardware.py(M) - These files match the scope in the PR description: runner, workflows,run()incommon.py, the cases README, and the pool and hardware tests. Caveat on the base: - The localorigin/main(b53781b, dated 2026-11-17) does not contain the PR's parent 11fe5e0 (dated 2026-11-20). The localorigin/mainis therefore stale. - A diff against that stale ref shows 311 files. Those files come from other merged PRs (for example#427,#376,#423) and are not part of this PR. I did not count them. - The provider-reported target ismain. The head branch contains 11fe5e0, which is a main-line commit. The PR diff is therefore the single 7-file commit. - The commit subjects were unavailable, so I could not confirm the commit count from the provider. The 7-file count is the diff of the PR's own commit. Result: 7 files, which is at or below 40. No failure message is required. Line counts were ignored as the check requires.
Full details: Public Resource Access
-
Explanation
The check applies only to external resource references added or modified by this pull request. I compared the PR's own commit (626b818, parent 11fe5e0) with its parent. It changes 7 files:
.github/scripts/engine-e2e-runner.sh,engine-e2e-lifecycle.yml,kumo-engine-e2e.yml,packages/kumo-engine/test/cases/README.md,packages/kumo-engine/test/cases/common.py,test/scripts/test-engine-e2e-pool.cjsandtest/scripts/test_engine_e2e_case_hardware.py. Scanning every added line for URL schemes, git/S3/package sources, and public hosts found no new public URL, repository location, or package source. The only added URL-like references are four lines inengine-e2e-runner.sh(lines 6, 7, 10 and 13 of the diff). They point tohttp://169.254.169.254/latest/api/tokenand.../meta-data/placement/regionand.../placement/availability-zone. This is the EC2 Instance Metadata Service. It is a link-local endpoint that exists only inside the running instance. It is runtime infrastructure, not an external resource that an anonymous public user could or should access, so it falls outside the "external resource reference" scope. The test fixtures use fake metadata, which the check excludes. The PR description links to other NVIDIA PRs and a comment. Those links are not reference text in the changed repository files, so they are outside the scope of this check. Note: the localorigin/mainhas moved past the PR base, so a fullorigin/main...HEADdiff shows about 311 unrelated files. I limited the review to the PR's own commit.
Full details: Change Map Design Review
-
Explanation
The custom check reviews the PR conversation comment that starts with
<!-- kumo-pr-change-map -->. The PR description links to that comment (issuecomment-7054551738), but I could not retrieve it. My tools read only the local checkout and the primary PR author record. The checkout has no copy of the comment. A search of the repository found only the policy text indocs/development/review.md, which requires one such comment per PR. So I could not check the change map's necessity, abstraction layer, behavior preservation, observed failure, verification, or stopping condition. I did not assess the code changes against the map. This is a missing-evidence result, not a pass. The check does not establish a defect. Context from the PR description: the change addsplacementandcapacity_wait_secondsto each Engine e2e case result, and it changesengine-e2e-runner.sh,common.py, and the two workflow files. The author states that the design depends on PR#455, and that PR#319affects nightly queue-time semantics. The description also names a known trade-off: nightly results record boot wait, not case queue time, until#319merges. A reviewer should check that the map states this trade-off.Resolution
To complete this check, provide the body of the PR conversation comment
https://github.com/NVIDIA/kumo-engine/pull/451#issuecomment-7054551738, or grant a tool that reads PR comments. Then re-run the check against these points: 1. Each row states a failure, a release need, or a stated requirement that the change addresses. 2. The map states the existing behavior that the author inspected. 3. Each change is the smallest change that meets the requirement, and it sits at the correct layer. Check whether the runner.envfields and therun()result fields are placed in the right layer. 4. The map names the focused verification and the stopping condition. 5. The map explains the nightly trade-off (boot wait versus queue time until#319merges). It also explains the dependency on PR#455and the rebase conflicts that the description names. 6. The map covers the omitted material behavior, including thenullplacement on failure results and the extra launch-template user data size.
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Change mapRequested behavior: each Engine e2e case result records the region and zone of its VM and the capacity wait of the pool for that VM.
|
Summary
Each Engine e2e case result now records where its VM ran and how long the pool waited for that VM:
placement: the region and the zone of the VM, for example{"region": "eu-west-2", "zone": "eu-west-2a"}.capacity_wait_seconds: the time from the pool start to the VM boot, in seconds, never below 0.Results of cases that no VM ran, such as capacity and bootstrap failures, carry the same fields with
null. So every result has one shape.Why
The runner pools move to several regions in the multi-region sweep PR. Two capacity campaigns in November asked for 96 and 48 g7e VMs. Every VM came from a region other than
us-west-2. The median wait ranged from minutes to hours by region and VM size. Today a case result does not record the region of its VM or the wait for it. So a slow region or a long wait does not show in the reports.Design
The picture shows one run of the multi-region pools. This PR changes step 9: the result of each case records its region, zone and capacity wait. The other steps come from the multi-region sweep PR.
flowchart LR subgraph GHA["GitHub run"] PJ["Pool job<br/>starts the pools"] end subgraph HOME["AWS us-west-2 (home)"] SCH["Scheduler<br/>sweep and report"] BR["Broker Lambda<br/>sweep, tokens, report"] S3[("S3 bucket<br/>data, results, records")] end subgraph REG["AWS, each of 9 regions"] EXP["Expiry schedule<br/>one per group"] ASG["Auto Scaling group<br/>per pool, Launch suspended"] VM["g7e VMs<br/>one runner each"] end subgraph GHB["GitHub runners"] API["GitHub API<br/>jobs, runner tokens"] CJ["Case jobs<br/>wait for a runner"] end PJ -- "1 create" --> SCH PJ -- "1 create" --> EXP PJ -- "1 create" --> ASG SCH -- "2 sweep every minute" --> BR BR == "3 LaunchInstances" ==> ASG BR -- "4 sweep record" --> S3 ASG -- "5 start VM" --> VM VM -- "6 token request" --> BR BR -- "7 jobs, tokens, failures" --> API VM -- "8 take a case" --> CJ VM -- "9 data, result" --> S3 EXP -- "10 delete at expiry" --> ASGBefore and now
engine-e2e-runner.sh/proc/stat.envENGINE_E2E_REGION,ENGINE_E2E_ZONEandENGINE_E2E_CAPACITY_WAIT_SECONDSrun()incommon.pyhardwareplacementandcapacity_wait_secondskumo-engine-e2e.ymlandengine-e2e-lifecycle.ymlnullThe pool start is the capacity deadline minus the tier window: 4 hours for nightly runs and 24 hours for weekly runs, as in
engine-e2e-pool.cjs. A test runs the runner formula against a deadline fromstart(), so the two windows cannot drift apart.Why this is better
instance region,instance zoneorcapacity wait) instead of a wrong result.Trade-off
Nightly runners take several cases until #319 merges. Until then, a nightly result records the boot wait of its VM, not the queue time of its case. Example: a VM that boots 30 minutes after the pool start and runs three cases reports 1800 for all three. If a pool has fewer VMs than cases, the wait of a later VM also includes the run time of earlier cases.
The multi-region sweep PR also changes
engine-e2e-runner.sh,engine-e2e-lifecycle.yml, the cases README,test-engine-e2e-pool.cjsandtest_engine_e2e_case_hardware.py. A trial merge conflicts inengine-e2e-runner.shand in both test files. The PR that merges second needs a rebase.Change map
File-by-file change map
Testing Done
node --test test/scripts/test-engine-e2e-pool.cjs: 30 of 30 pass on Node 24 and on Node 22.oss-readiness.yml: 214 pass. The only failure istest_history_scan_explicitly_selects_all_commits, which needs a full clone. CI uses a full clone.check_oss_readiness.sh --diff,check_secrets.sh --diff(gitleaks 8.30.1), yamllint, shellcheck, flake8 and isort pass./proc/stat. They cover nightly, weekly, clock skew, a bad region, a bad zone, a Local Zone name, a missing boot time and the exact.envlines.start()deadline for both tiers..envfield. A test failed for each one.us-west-2andus-west-2b. Both values pass the new checks, andbtimeequals the boot time from/proc/uptime.Hotfix Context
Not a hotfix.
Checklist
compare_referenceignores them.🤖 Generated with Claude Code