Add nightly build workflow for ROCgdb master branch by lumachad · Pull Request #120 · ROCm/ROCgdb

lumachad · 2026-05-08T11:06:01Z

Adds a scheduled nightly build that tests ROCgdb master against the current TheRock dependency pins, with full GPU, CPU, and corefile test coverage.

Changes:

Add nightly-build.yml: runs at 2:07 AM UTC daily; manually dispatchable with a configurable ROCgdb branch (defaults to master)
Resolve TheRock refs (therock_commit_ref, build_image) from configs.json, matching the existing PR CI pattern
Enable upstream builds with THEROCK_ROCGDB_UPSTREAM_BUILD=ON so TheRock fetches test scripts for the master branch
Run GPU, CPU, and corefile tests against the nightly artifacts
Add rocgdb_ref input to therock-ci-linux.yml to allow nightly builds to check out a specific ROCgdb branch or ref

PR-based CI is unchanged.

lumachad · 2026-05-19T15:37:21Z

Blocked on:

Migrate to multi-arch CI #130

spatrang

Review summary

Nightly CI for master against current TheRock pins is the right idea, and the workflow shape (resolve → reusable build → GPU/CPU/corefile → summary) matches PR CI. A few issues should be fixed before merge; one looks like a hard configure failure on master.

PR CI on this branch is green, but that only exercises therock-ci.yml with rocgdb_ref unset. nightly-build.yml is schedule/dispatch only and has not been run.

Blocker

Missing -DTHEROCK_ROCGDB_DOWNLOAD_CI_SCRIPT=ON. THEROCK_ROCGDB_UPSTREAM_BUILD does not fetch test_rocgdb.py. master has no .github/ tree, so configure will FATAL_ERROR with THEROCK_BUILD_TESTING=ON.

High

Dispatch this workflow from the PR branch after the CMake fix and confirm it actually checks out master.
rocgdb_ignore_list.json is also missing on master (TheRock only warns and skips it). Decide whether to fetch/install it or accept raw xfails on nightly.

Medium

Build still uses pre-#300 gfx94X-dcgpu while PR CI builds multi-arch.
No build_runs_on / select_runner.py (falls back to aws-linux-scale-rocm-prod).
Checkout fallback should be github.sha, not github.ref.
No concurrency group.

Low

Extra CMake flags will drift from therock-ci.yml; a “keep in sync except …” comment would help.
Tests should take therock_ref from the build job output.
AWS configure if is PR-shaped (!github.event.pull_request.head.repo.fork). Empty pull_request on schedule should still be truthy for !fork, but confirm artifact push on the first dispatch.
Echo rocgdb_ref in the build log so it is obvious which branch was built.
The May 19 “blocked on #130” comment is stale (#300 landed); please resolve that thread.

Merge bar

Add -DTHEROCK_ROCGDB_DOWNLOAD_CI_SCRIPT=ON.
Decide ignore-list behavior for master.
workflow_dispatch nightly from this branch; confirm configure, master checkout, artifact push, and GPU/CPU/corefile.
Confirm or document gfx94X-dcgpu vs multi-arch, and whether build_runs_on should match PR CI.

Adds a scheduled nightly build that tests ROCgdb master against the current TheRock dependency pins, with GPU, CPU, and corefile test coverage. Changes: - Add nightly-build.yml: scheduled (2:07 AM UTC) and manually dispatchable workflow that builds ROCgdb master via TheRock and runs the full test suite (GPU on gfx94X, CPU on gfx94X, corefile on gfx942) - Mirror PR CI build topology (dcgpu-all;dgpu-all;igpu-all, multiarch bundle name, select_runner.py for runner selection) - Add concurrency group to prevent overlapping runs - Add rocgdb_ref input to therock-ci-linux.yml to allow nightly builds to check out a specific ROCgdb branch or ref; fallback preserves PR CI's pinned-SHA checkout semantics

Temporary commit to allow manual dispatch of the nightly workflow from this branch via the GitHub Actions UI. Drop before merging.

lumachad · 2026-08-27T17:22:13Z

therock-ci.yml temporarily overwritten for validation purposes. If this looks OK, we'll drop the second commit.

spatrang

Round 2 review

Round-1 workflow comments are addressed in ac903dcd (CMake flags, multi-arch, runner selection, concurrency, github.sha, therock_ref from the build). A dispatch of the nightly path (run 33095027788) built master successfully — so the previous configure blocker is gone.

That run also showed the nightly as designed will be red every night unless a few test-policy gaps are fixed. The WIP commit is still on HEAD and must not land.

Round-1 follow-up

Item	Status
`THEROCK_ROCGDB_DOWNLOAD_CI_SCRIPT=ON`	Fixed and validated (build succeeded, `test_rocgdb.py` ran from artifacts)
Multi-arch + `select_runner.py`	Fixed
Concurrency	Fixed
Checkout fallback `github.sha`	Fixed
`therock_ref` from build output	Fixed (run output matched the pin)
Ignore list	Not fixed; now evidenced — CPU is red only on known amd-staging xfails

Blocker

Drop 012946ee (WIP: overwrite therock-ci.yml…) before merge. It replaces PR CI with the nightly workflow and removes the pull_request trigger (this push only ran pre-commit). Keep the useful Show rocGDB ref step from that commit in therock-ci-linux.yml.

High

CPU will fail every night without rocgdb_ignore_list.json. The 6 unexpected failures are exactly the amd-staging LLVM xfail list. Fetch/install that file with the CI script.
Corefile will fail every night on master: those gdb.rocm/corefile*.exp files exist only on amd-staging. Dispatch error: Missing or invalid test file: gdb.rocm/corefile.exp. Skip test-corefile for master (or only invoke tests that exist in the tree).
GPU had a real 1-test fail (gdb.rocm/displaced-stepping.exp). That is useful nightly signal; it is not on the ignore list. Fine to leave failing, or xfail it explicitly.

Merge bar

Drop the WIP commit; restore therock-ci.yml; keep Show rocGDB ref.
Install the ignore list (CPU) and stop running missing corefile tests on master.
Re-dispatch (or wait for a PR CI run of the restored workflow) and confirm: build of master green, CPU green with xfails, corefile skipped or green, GPU = the displaced-stepping result you want.

spatrang · 2026-08-27T18:26:54Z

@@ -1,63 +1,31 @@
-name: TheRock CI for rocgdb
+name: Nightly ROCgdb Build


Blocker (merge): this file must not ship as the nightly workflow. 012946ee (WIP: overwrite therock-ci.yml with nightly for dispatch testing) replaces PR CI and drops on.pull_request, which is why the latest push only ran pre-commit.

Dispatching via this workaround was reasonable (nightly-build.yml is new and not on the default branch, so it does not show in the Actions UI). Please drop this commit before merge and restore therock-ci.yml.

When dropping it, keep the Show rocGDB ref step that this WIP also added in therock-ci-linux.yml (that step is not in ac903dcd).

spatrang · 2026-08-27T18:26:54Z

+  test-corefile:
+    name: Test corefile (gfx942)
+    needs: [nightly-rocgdb-build-linux]
+    if: ${{ needs.nightly-rocgdb-build-linux.result == 'success' }}
+    permissions:
+      contents: read
+    uses: ./.github/workflows/therock-test-packages.yml
+    with:
+      amdgpu_families: gfx942
+      artifact_group: nightly-rocgdb-multiarch
+      therock_ref: ${{ needs.nightly-rocgdb-build-linux.outputs.therock_ref }}
+      artifact_run_id: ${{ github.run_id }}
+      projects_to_test: rocgdb-corefile


High: this job cannot succeed against default master. Those tests are not in upstream/master gdb/testsuite/gdb.rocm/ (they exist on amd-staging only). Dispatch run 33095027788 failed immediately with:

Error: Missing or invalid test file: gdb.rocm/corefile.exp

TheRock’s rocgdb-corefile entry always passes that file list into test_rocgdb.py, which treats a missing .exp as a hard error.

Please skip test-corefile when rocgdb_ref is master (or a ref that lacks those files), or teach the runner to skip missing tests. As written, corefile will fail every nightly.

spatrang · 2026-08-27T18:26:54Z

+  test-cpu:
+    name: Test CPU
+    needs: [nightly-rocgdb-build-linux]
+    if: ${{ needs.nightly-rocgdb-build-linux.result == 'success' }}
+    permissions:
+      contents: read
+    uses: ./.github/workflows/therock-test-packages.yml
+    with:
+      # CPU tests are gfx-independent; gfx94X selects the artifact set to fetch.
+      amdgpu_families: gfx94X
+      artifact_group: nightly-rocgdb-multiarch
+      therock_ref: ${{ needs.nightly-rocgdb-build-linux.outputs.therock_ref }}
+      artifact_run_id: ${{ github.run_id }}
+      projects_to_test: rocgdb-cpu


High: dispatch CPU failed only on ignore-listed gdb.dwarf2 tests (6 unexpected failures; 3067/2717 PASSes otherwise). Log:

Ignore list file not found: .../build/tests/rocgdb/rocgdb_ignore_list.json

THEROCK_ROCGDB_DOWNLOAD_CI_SCRIPT fetches test_rocgdb.py from amd-staging, not rocgdb_ignore_list.json. Without that file, this job stays red every night and the workflow is not a smoke signal.

Please install the ignore list into tests/rocgdb as well (TheRock wrapper change, or a small extra fetch in this workflow).

lumachad self-assigned this May 8, 2026

lumachad added the ci:skip Skip all pre-commit / CI jobs while the label is up label May 8, 2026

lumachad force-pushed the users/lumachad/amd-staging/github/build_master branch from 939e4d6 to 81b8002 Compare August 27, 2026 12:37

lumachad changed the title ~~Add nightly build workflow and refactor CI for branch flexibility~~ Add nightly build workflow for ROCgdb master branch Aug 27, 2026

lumachad force-pushed the users/lumachad/amd-staging/github/build_master branch from 81b8002 to 2348c39 Compare August 27, 2026 12:41

lumachad marked this pull request as ready for review August 27, 2026 12:45

lumachad requested a review from a team as a code owner August 27, 2026 12:45

lumachad force-pushed the users/lumachad/amd-staging/github/build_master branch 3 times, most recently from 17d71a9 to 8221158 Compare August 27, 2026 12:56

lumachad removed the ci:skip Skip all pre-commit / CI jobs while the label is up label Aug 27, 2026

lumachad assigned spatrang and unassigned lumachad Aug 27, 2026

spatrang requested changes Aug 27, 2026

View reviewed changes

Comment thread .github/workflows/nightly-build.yml

Comment thread .github/workflows/nightly-build.yml Outdated

Comment thread .github/workflows/nightly-build.yml

Comment thread .github/workflows/therock-ci-linux.yml Outdated

Comment thread .github/workflows/nightly-build.yml Outdated

lumachad force-pushed the users/lumachad/amd-staging/github/build_master branch from 8221158 to ac903dc Compare August 27, 2026 16:32

WIP: overwrite therock-ci.yml with nightly for dispatch testing

012946e

Temporary commit to allow manual dispatch of the nightly workflow from this branch via the GitHub Actions UI. Drop before merging.

lumachad force-pushed the users/lumachad/amd-staging/github/build_master branch from 131321d to 012946e Compare August 27, 2026 17:20

spatrang requested changes Aug 27, 2026

View reviewed changes

Navigation Menu

Uh oh!

Add nightly build workflow for ROCgdb master branch - #120

Add nightly build workflow for ROCgdb master branch#120
lumachad wants to merge 2 commits into
amd-stagingfrom
users/lumachad/amd-staging/github/build_master

lumachad commented May 8, 2026 •

edited

Loading

Uh oh!

lumachad commented May 19, 2026

Uh oh!

spatrang left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lumachad commented Aug 27, 2026

Uh oh!

spatrang left a comment

Uh oh!

spatrang Aug 27, 2026

Uh oh!

spatrang Aug 27, 2026

Uh oh!

spatrang Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -1,63 +1,31 @@
		name: TheRock CI for rocgdb
		name: Nightly ROCgdb Build

Uh oh!

Conversation

lumachad commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lumachad commented May 19, 2026

Uh oh!

spatrang left a comment

Choose a reason for hiding this comment

Review summary

Blocker

High

Medium

Low

Merge bar

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lumachad commented Aug 27, 2026

Uh oh!

spatrang left a comment

Choose a reason for hiding this comment

Round 2 review

Round-1 follow-up

Blocker

High

Merge bar

Uh oh!

spatrang Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

spatrang Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

spatrang Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lumachad commented May 8, 2026 •

edited

Loading