[workflows] Run extended ROCgdb tests in parallel on PRs - #245
Conversation
290e6e7 to
c7d4877
Compare
|
Blocked on failures fixed by #247. |
249a25a to
c8c9c33
Compare
|
Blocked on #261. |
c8c9c33 to
8ea9ee5
Compare
|
I think this is a good first cut. Once we merge #296, we should be able to run this one clean. I'm still unsure if we should always run these or if we should put the extra tests behind a selectable label. Given we want to run these optimization tests, I think we should do it for every ROCgdb PR. |
8ea9ee5 to
4493ce5
Compare
4493ce5 to
0a7623b
Compare
|
@spatrang Could you please eyeball this one? |
0a7623b to
a06e86a
Compare
spatrang
left a comment
There was a problem hiding this comment.
Reviewed at head a06e86a. Good direction and the mechanism is sound, but there is one blocker plus some redundancy against options TheRock already sets. Details inline.
Highlights:
- Blocker: the
hip boardjob runs 1 of its 10 tests, and that one reportsunsupported, so it passes while testing nothing. - CI is red:
check-read1andcheck-readmoreboth fail ongdb.rocm/addr-bp-gpu-no-deb-info.exp. base_test_optionsandbase_gpu_test_optionsduplicate--parallel -f 0.25 --toolchain llvm, which TheRock already puts in every rocgdbtest_script.
Not line-anchorable: the PR description is stale. It lists six configurations (there are five, since --parallel became the shared default) and says the hip board job runs only gdb.base/break.exp when the code lists ten.
TheRock's fetch_test_configurations.py now emits the installed script path (python ./build/tests/rocgdb/test_rocgdb.py ...) directly, so the gsub that rewrote build_tools/.../test_rocgdb.py to the artifact path was already a no-op at the pinned commit (f438028). Remove it and the OUTPUT_ARTIFACTS_DIR env var that fed it.
a06e86a to
2931b76
Compare
|
Updated with the fixes suggested. I'll let CI run again. |
2931b76 to
94425ff
Compare
spatrang
left a comment
There was a problem hiding this comment.
Second round, reviewed at head 94425ff. Thanks for the thorough turnaround — the blocker is genuinely fixed, and dropping base_test_options / base_gpu_test_options outright was better than what I asked for, since it also reverts the unintended change to the three pre-existing test jobs.
I verified the new patch-scripts step against the real component JSON and the hip board case now produces a single --tests with the full list. I also confirmed that removing OUTPUT_ARTIFACTS_DIR is safe: therock-test-component.yml defines its own copy (line 42) and passes it to the test step (line 71), and workflow-level env: never propagated into the called workflow anyway.
Two remaining items inline, both of which I think should block merge.
Add five extended test configurations that run in parallel with the standard tests on every pull request. All jobs reuse the single build from therock-build-linux (no extra build cost) and fan out together once the build completes. The extended configurations are: | Job | Options | Architectures | |-----|---------|---------------| | Test GPU (-O3) | --optimization=-O3 | all (expands with ci:run-all-archs) | | Test GPU (-O3 -flto) | --optimization='-O3 -flto' | all (expands with ci:run-all-archs) | | Test GPU (check-read1) | --check-type check-read1 | gfx94X only | | Test GPU (check-readmore) | --check-type check-readmore | gfx94X only | | Test GPU (hip board) | --runtestflags=--target_board=hip | gfx94X only (GPU only) | The two optimization jobs follow the same test_families matrix as test-gpu, so with ci:run-all-archs each adds up to six GPU jobs (one per architecture family), for twelve optimization jobs total on top of the regular fan-out. To support per-config scoping, therock-test-packages.yml gains two new inputs: - test_options: appended to each component's test_script (passed via TEST_OPTIONS env var to avoid injection). When test_options contains --tests, any pre-existing --tests in the upstream test_script is stripped first so the caller's value wins explicitly. - projects_to_test: restricts the component matrix; optimization and hip board jobs use rocgdb-gpu,rocgdb-corefile only. Job names are stable identifiers rather than flag strings to avoid silently breaking branch-protection rules when options are tuned.
94425ff to
1282d32
Compare
spatrang
left a comment
There was a problem hiding this comment.
LGTM — approving at 1282d323. Both blocking items are resolved, and I verified them rather than relying on the replies.
Verified the patching logic. I replayed the new patch-scripts step against all three real rocgdb components across five test_options cases (empty, --check-type, --optimization=-O3, --optimization='-O3 -flto', and the hip board case), then fed every resulting command line through a mirror of test_rocgdb.py's parser. All parse cleanly:
- Empty
test_optionsis a proper no-op, so the three standard jobs are untouched. rocgdb-corefilekeeps its full five-file--testslist when the caller does not pass--tests.- The hip board case strips the upstream
--testscompletely, including the multi-value corefile form that broke the previous regex, and lands a single--testswith all ten files.
Verified on hardware. Run 33069089574 is green across all 26 jobs including TheRock CI Summary. The hip board job now reports:
Number of tests to run: 10
# of expected passes 38
# of unsupported tests 3
[✓] OVERALL STATUS: PASS
Up from the single skipped test that started this review. check-read1 and check-readmore also passed across all three components in this run, which supports the flaky-runner explanation over a systematic failure.
Thanks for the fast and thorough turnaround, particularly for dropping base_test_options / base_gpu_test_options outright instead of just documenting them — that removed the unintended behaviour change to the three pre-existing jobs and made the whole diff smaller.
One non-blocking question inline about the break.exp swap, plus two cosmetic notes below. None of these need to hold the merge.
Cosmetic, for a follow-up whenever convenient:
Test GPU (check-read1)andTest GPU (check-readmore)do not setprojects_to_test, so they also runrocgdb-cpuandrocgdb-corefile— visible in the run asTest GPU (check-read1) / Test rocgdb-cpu. The names claim GPU. Either scope them torocgdb-gpuor rename toTest check-read1. This one is partly my fault for suggesting theTest GPU (...)pattern.- The description says the hip board job uses
rocgdb-gpu,rocgdb-corefile(the code usesrocgdb-gpuonly) and that the workflow "gains two new inputs", thoughprojects_to_testalready existed and only gained a description.
These jobs run all three rocgdb components (rocgdb-gpu, rocgdb-cpu, rocgdb-corefile) since they do not set projects_to_test. The 'Test GPU' prefix implied GPU-only coverage.
Addressed these cosmetic issues as well in a new push. |
Summary
Add five extended test configurations that run in parallel with the
standard tests on every pull request. All jobs reuse the single build
from
therock-build-linux(no extra build cost) and fan out togetheronce the build completes.
The extended configurations are:
Test GPU (-O3)--optimization=-O3ci:run-all-archs)Test GPU (-O3 -flto)--optimization='-O3 -flto'ci:run-all-archs)Test check-read1--check-type check-read1Test check-readmore--check-type check-readmoreTest GPU (hip board)--runtestflags=--target_board=hip --tests gdb.base/...GPU runner budget: without
ci:run-all-archs, all five extended jobsrun on gfx94X only (5 extra GPU jobs). With
ci:run-all-archs, the twooptimization jobs each fan out to all six architecture families (12 extra
GPU jobs from optimization alone, plus 3 for check-type/hip-board). The
multi-arch coverage for optimization jobs is intentional: the goal is to
catch compiler-specific failures across all supported architectures.
To support per-config scoping,
therock-test-packages.ymlgains onenew input and updates another:
test_options— appended to each component'stest_script(passedvia a
TEST_OPTIONSenv var in the jq patch step to avoid injection).When
test_optionscontains--tests, any pre-existing--testsinthe upstream
test_scriptis stripped first to make the overrideexplicit rather than relying on argparse last-wins.
projects_to_test— restricts the component matrix (already existed;gains a description). The two optimization jobs use
rocgdb-gpu,rocgdb-corefilesince the options are compiler-specific;the hip board job uses
rocgdb-gpuonly.Commits
[workflows] Drop now-redundant test_rocgdb.py path rewrite— TheRocknow emits the installed script path directly; the gsub rewrite was
already a no-op at the pinned commit. Removes
OUTPUT_ARTIFACTS_DIR.[workflows] Run extended ROCgdb tests in parallel on PRs— theextended jobs and updated
therock-test-packages.ymlinputs.