Skip to content

[workflows] Run extended ROCgdb tests in parallel on PRs - #245

Merged
lumachad merged 3 commits into
amd-stagingfrom
users/lumachad/amd-staging/extended-testing-v2
Aug 27, 2026
Merged

[workflows] Run extended ROCgdb tests in parallel on PRs#245
lumachad merged 3 commits into
amd-stagingfrom
users/lumachad/amd-staging/extended-testing-v2

Conversation

@lumachad

@lumachad lumachad commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add five extended test configurations that run in parallel with the
standard tests on every pull request. All jobs reuse the single build
from therock-build-linux (no extra build cost) and fan out together
once the build completes.

The extended configurations are:

Job Options Architectures
Test GPU (-O3) --optimization=-O3 all (expands with ci:run-all-archs)
Test GPU (-O3 -flto) --optimization='-O3 -flto' all (expands with ci:run-all-archs)
Test check-read1 --check-type check-read1 gfx94X only
Test check-readmore --check-type check-readmore gfx94X only
Test GPU (hip board) --runtestflags=--target_board=hip --tests gdb.base/... gfx94X only (GPU only)

GPU runner budget: without ci:run-all-archs, all five extended jobs
run on gfx94X only (5 extra GPU jobs). With ci:run-all-archs, the two
optimization jobs each fan out to all six architecture families (12 extra
GPU jobs from optimization alone, plus 3 for check-type/hip-board). The
multi-arch coverage for optimization jobs is intentional: the goal is to
catch compiler-specific failures across all supported architectures.

To support per-config scoping, therock-test-packages.yml gains one
new input and updates another:

  • test_options — appended to each component's test_script (passed
    via a TEST_OPTIONS env var in the jq patch step to avoid injection).
    When test_options contains --tests, any pre-existing --tests in
    the upstream test_script is stripped first to make the override
    explicit rather than relying on argparse last-wins.
  • projects_to_test — restricts the component matrix (already existed;
    gains a description). The two optimization jobs use
    rocgdb-gpu,rocgdb-corefile since the options are compiler-specific;
    the hip board job uses rocgdb-gpu only.

Commits

  1. [workflows] Drop now-redundant test_rocgdb.py path rewrite — TheRock
    now emits the installed script path directly; the gsub rewrite was
    already a no-op at the pinned commit. Removes OUTPUT_ARTIFACTS_DIR.
  2. [workflows] Run extended ROCgdb tests in parallel on PRs — the
    extended jobs and updated therock-test-packages.yml inputs.

@lumachad
lumachad requested a review from a team as a code owner August 4, 2026 10:46
@lumachad
lumachad marked this pull request as draft August 4, 2026 10:47
@lumachad lumachad assigned lumachad and unassigned aktemur Aug 4, 2026
@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch 3 times, most recently from 290e6e7 to c7d4877 Compare August 4, 2026 11:46
@lumachad

lumachad commented Aug 5, 2026

Copy link
Copy Markdown
Collaborator Author

Blocked on failures fixed by #247.

@lumachad lumachad added the ci:skip Skip all pre-commit / CI jobs while the label is up label Aug 5, 2026
@lumachad lumachad assigned lumachad and unassigned lumachad Aug 5, 2026
@lumachad lumachad removed the ci:skip Skip all pre-commit / CI jobs while the label is up label Aug 7, 2026
@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch 2 times, most recently from 249a25a to c8c9c33 Compare August 7, 2026 10:36
@lumachad

Copy link
Copy Markdown
Collaborator Author

Blocked on #261.

@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from c8c9c33 to 8ea9ee5 Compare August 20, 2026 07:35
@lumachad
lumachad marked this pull request as ready for review August 21, 2026 07:47
@lumachad lumachad assigned aktemur and lancesix and unassigned lumachad Aug 21, 2026
@lumachad

Copy link
Copy Markdown
Collaborator Author

I think this is a good first cut. Once we merge #296, we should be able to run this one clean.

I'm still unsure if we should always run these or if we should put the extra tests behind a selectable label. Given we want to run these optimization tests, I think we should do it for every ROCgdb PR.

@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from 8ea9ee5 to 4493ce5 Compare August 25, 2026 15:03
@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from 4493ce5 to 0a7623b Compare August 26, 2026 10:31
@lumachad

Copy link
Copy Markdown
Collaborator Author

@spatrang Could you please eyeball this one?

@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from 0a7623b to a06e86a Compare August 26, 2026 10:46
@lumachad

Copy link
Copy Markdown
Collaborator Author

@spatrang Could you please eyeball this one?

@spatrang ping?

@spatrang spatrang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head a06e86a. Good direction and the mechanism is sound, but there is one blocker plus some redundancy against options TheRock already sets. Details inline.

Highlights:

  • Blocker: the hip board job runs 1 of its 10 tests, and that one reports unsupported, so it passes while testing nothing.
  • CI is red: check-read1 and check-readmore both fail on gdb.rocm/addr-bp-gpu-no-deb-info.exp.
  • base_test_options and base_gpu_test_options duplicate --parallel -f 0.25 --toolchain llvm, which TheRock already puts in every rocgdb test_script.

Not line-anchorable: the PR description is stale. It lists six configurations (there are five, since --parallel became the shared default) and says the hip board job runs only gdb.base/break.exp when the code lists ten.

Comment thread .github/workflows/therock-ci.yml Outdated
Comment thread .github/workflows/therock-ci.yml Outdated
Comment thread .github/workflows/therock-ci.yml Outdated
Comment thread .github/workflows/therock-ci.yml
Comment thread .github/workflows/therock-ci.yml Outdated
Comment thread .github/workflows/therock-test-packages.yml Outdated
Comment thread .github/workflows/therock-test-packages.yml Outdated
Comment thread .github/workflows/therock-test-packages.yml
Comment thread .github/workflows/therock-test-packages.yml Outdated
Comment thread .github/workflows/therock-test-packages.yml Outdated
TheRock's fetch_test_configurations.py now emits the installed script
path (python ./build/tests/rocgdb/test_rocgdb.py ...) directly, so the
gsub that rewrote build_tools/.../test_rocgdb.py to the artifact path
was already a no-op at the pinned commit (f438028).

Remove it and the OUTPUT_ARTIFACTS_DIR env var that fed it.
@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from a06e86a to 2931b76 Compare August 27, 2026 11:02
@lumachad

Copy link
Copy Markdown
Collaborator Author

Updated with the fixes suggested. I'll let CI run again.

@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from 2931b76 to 94425ff Compare August 27, 2026 11:26

@spatrang spatrang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second round, reviewed at head 94425ff. Thanks for the thorough turnaround — the blocker is genuinely fixed, and dropping base_test_options / base_gpu_test_options outright was better than what I asked for, since it also reverts the unintended change to the three pre-existing test jobs.

I verified the new patch-scripts step against the real component JSON and the hip board case now produces a single --tests with the full list. I also confirmed that removing OUTPUT_ARTIFACTS_DIR is safe: therock-test-component.yml defines its own copy (line 42) and passes it to the test step (line 71), and workflow-level env: never propagated into the called workflow anyway.

Two remaining items inline, both of which I think should block merge.

Comment thread .github/workflows/therock-test-packages.yml Outdated
Comment thread .github/workflows/therock-ci.yml
Add five extended test configurations that run in parallel with the
standard tests on every pull request. All jobs reuse the single build
from therock-build-linux (no extra build cost) and fan out together
once the build completes.

The extended configurations are:

| Job | Options | Architectures |
|-----|---------|---------------|
| Test GPU (-O3) | --optimization=-O3 | all (expands with ci:run-all-archs) |
| Test GPU (-O3 -flto) | --optimization='-O3 -flto' | all (expands with ci:run-all-archs) |
| Test GPU (check-read1) | --check-type check-read1 | gfx94X only |
| Test GPU (check-readmore) | --check-type check-readmore | gfx94X only |
| Test GPU (hip board) | --runtestflags=--target_board=hip | gfx94X only (GPU only) |

The two optimization jobs follow the same test_families matrix as
test-gpu, so with ci:run-all-archs each adds up to six GPU jobs (one
per architecture family), for twelve optimization jobs total on top of
the regular fan-out.

To support per-config scoping, therock-test-packages.yml gains two
new inputs:
- test_options: appended to each component's test_script (passed via
  TEST_OPTIONS env var to avoid injection). When test_options contains
  --tests, any pre-existing --tests in the upstream test_script is
  stripped first so the caller's value wins explicitly.
- projects_to_test: restricts the component matrix; optimization and
  hip board jobs use rocgdb-gpu,rocgdb-corefile only.

Job names are stable identifiers rather than flag strings to avoid
silently breaking branch-protection rules when options are tuned.
@lumachad
lumachad force-pushed the users/lumachad/amd-staging/extended-testing-v2 branch from 94425ff to 1282d32 Compare August 27, 2026 11:49
@lumachad
lumachad requested a review from spatrang August 27, 2026 12:42

@spatrang spatrang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — approving at 1282d323. Both blocking items are resolved, and I verified them rather than relying on the replies.

Verified the patching logic. I replayed the new patch-scripts step against all three real rocgdb components across five test_options cases (empty, --check-type, --optimization=-O3, --optimization='-O3 -flto', and the hip board case), then fed every resulting command line through a mirror of test_rocgdb.py's parser. All parse cleanly:

  • Empty test_options is a proper no-op, so the three standard jobs are untouched.
  • rocgdb-corefile keeps its full five-file --tests list when the caller does not pass --tests.
  • The hip board case strips the upstream --tests completely, including the multi-value corefile form that broke the previous regex, and lands a single --tests with all ten files.

Verified on hardware. Run 33069089574 is green across all 26 jobs including TheRock CI Summary. The hip board job now reports:

Number of tests to run: 10
# of expected passes		38
# of unsupported tests		3
[✓] OVERALL STATUS: PASS

Up from the single skipped test that started this review. check-read1 and check-readmore also passed across all three components in this run, which supports the flaky-runner explanation over a systematic failure.

Thanks for the fast and thorough turnaround, particularly for dropping base_test_options / base_gpu_test_options outright instead of just documenting them — that removed the unintended behaviour change to the three pre-existing jobs and made the whole diff smaller.

One non-blocking question inline about the break.exp swap, plus two cosmetic notes below. None of these need to hold the merge.

Cosmetic, for a follow-up whenever convenient:

  • Test GPU (check-read1) and Test GPU (check-readmore) do not set projects_to_test, so they also run rocgdb-cpu and rocgdb-corefile — visible in the run as Test GPU (check-read1) / Test rocgdb-cpu. The names claim GPU. Either scope them to rocgdb-gpu or rename to Test check-read1. This one is partly my fault for suggesting the Test GPU (...) pattern.
  • The description says the hip board job uses rocgdb-gpu,rocgdb-corefile (the code uses rocgdb-gpu only) and that the workflow "gains two new inputs", though projects_to_test already existed and only gained a description.

Comment thread .github/workflows/therock-ci.yml
These jobs run all three rocgdb components (rocgdb-gpu, rocgdb-cpu,
rocgdb-corefile) since they do not set projects_to_test. The 'Test GPU'
prefix implied GPU-only coverage.
@lumachad

Copy link
Copy Markdown
Collaborator Author

LGTM — approving at 1282d323. Both blocking items are resolved, and I verified them rather than relying on the replies.

Verified the patching logic. I replayed the new patch-scripts step against all three real rocgdb components across five test_options cases (empty, --check-type, --optimization=-O3, --optimization='-O3 -flto', and the hip board case), then fed every resulting command line through a mirror of test_rocgdb.py's parser. All parse cleanly:

  • Empty test_options is a proper no-op, so the three standard jobs are untouched.
  • rocgdb-corefile keeps its full five-file --tests list when the caller does not pass --tests.
  • The hip board case strips the upstream --tests completely, including the multi-value corefile form that broke the previous regex, and lands a single --tests with all ten files.

Verified on hardware. Run 33069089574 is green across all 26 jobs including TheRock CI Summary. The hip board job now reports:

Number of tests to run: 10
# of expected passes		38
# of unsupported tests		3
[✓] OVERALL STATUS: PASS

Up from the single skipped test that started this review. check-read1 and check-readmore also passed across all three components in this run, which supports the flaky-runner explanation over a systematic failure.

Thanks for the fast and thorough turnaround, particularly for dropping base_test_options / base_gpu_test_options outright instead of just documenting them — that removed the unintended behaviour change to the three pre-existing jobs and made the whole diff smaller.

One non-blocking question inline about the break.exp swap, plus two cosmetic notes below. None of these need to hold the merge.

Cosmetic, for a follow-up whenever convenient:

  • Test GPU (check-read1) and Test GPU (check-readmore) do not set projects_to_test, so they also run rocgdb-cpu and rocgdb-corefile — visible in the run as Test GPU (check-read1) / Test rocgdb-cpu. The names claim GPU. Either scope them to rocgdb-gpu or rename to Test check-read1. This one is partly my fault for suggesting the Test GPU (...) pattern.
  • The description says the hip board job uses rocgdb-gpu,rocgdb-corefile (the code uses rocgdb-gpu only) and that the workflow "gains two new inputs", though projects_to_test already existed and only gained a description.

Addressed these cosmetic issues as well in a new push.

@lumachad lumachad added the ci:run-all-archs Run ROCgdb tests across more gfx architectures (whenever available) label Aug 27, 2026
@lumachad
lumachad merged commit 49b6bc3 into amd-staging Aug 27, 2026
58 of 69 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:run-all-archs Run ROCgdb tests across more gfx architectures (whenever available)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants