Skip to content

Removed examples from TB and added a tag to build TB from that repo t… #240

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
urtiwari merged 2 commits into from
Jun 26, 2026
Merged

Removed examples from TB and added a tag to build TB from that repo t… #240

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
62567e0
Removed examples from TB and added a tag to build TB from that repo t…
Jun 24, 2026
f8f967a
Addressed the review comments
Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions cvs/input/config_file/health/mi300_health_config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
},
"transferbench": {
"path": "/home/{user-id}/INSTALL/TransferBench",
"example_tests_path": "/home/{user-id}/INSTALL/TransferBench/examples",
"git_install_path": "/home/{user-id}/INSTALL/",
"git_url": "https://github.com/ROCm/TransferBench.git",
"git_tag": "v1.67.00",
"_comment_rocm_path": "ROCm installation path. Set the placeholder changeme to auto-detect from /opt/rocm or /opt/rocm/core-*",
"rocm_path": "<changeme>",
"results": {
Expand All @@ -26,15 +26,7 @@
"32_cu_local_copy": "1250.0",
"32_cu_rem_read": "48.0",
"32_cu_rem_write": "48.0",
"32_cu_rem_copy": "48.0",
"example_results": {
"test1": "47.1",
"test2": "48.4",
"test3_0_to_1": "31.9",
"test3_1_to_0": "38.9",
"test4": "1264",
"test6": "48.6"
}
"32_cu_rem_copy": "48.0"
}
},
"rvs": {
Expand Down
1 change: 0 additions & 1 deletion cvs/tests/health/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,6 @@ pytest -vvv -log-file=/tmp/agfhc_test.log -s ./tests/health/agfhc_cvs.py --clust
```
In the config file, cvs/input/config_file/health/mi300_health_config.json, change the value of parameters(/tmp/cvs):
"path": "/tmp/cvs/INSTALL/TransferBench",
"example_tests_path": "/tmp/cvs/INSTALL/TransferBench/examples",
"git_install_path": "/tmp/cvs/INSTALL/", to desired location.

pytest -vvv --log-file=/tmp/test.log -s ./tests/health/install/install_agfhc.py --cluster_file input/cluster_file/cluster.json --config_file input/config_file/health/mi300_health_config.json --html=/var/www/html/cvs/agfhc.html --capture=tee-sys --self-contained-html
Expand Down
29 changes: 28 additions & 1 deletion cvs/tests/health/install/install_transferbench.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,7 @@ def test_install_transferbench(orch, config_dict):
Steps:
- Ensure git_install_path exists on all nodes.
- Clone TransferBench repo under git_install_path on every node.
- Checkout the configured git tag.
- Build on all nodes using detected ROCM_PATH and HIPCC.
- Verify the build artifact is present on each node.

Expand All @@ -193,13 +194,19 @@ def test_install_transferbench(orch, config_dict):
config_dict (dict): Includes:
- git_install_path: directory to clone/build
- git_url: repository URL
- git_tag: git tag to checkout after clone
"""

globals.error_list = []

log.info('Testcase install transferbench')
git_install_path = config_dict['git_install_path']
git_url = config_dict['git_url']
git_tag = config_dict.get('git_tag')
if not git_tag:
fail_test('TransferBench config is missing the required "git_tag" field')
update_test_result()
return

out_dict = orch.exec(f'ls -ld {git_install_path}')
for node in out_dict.keys():
Expand All @@ -213,6 +220,27 @@ def test_install_transferbench(orch, config_dict):
timeout=120,
)

tb_src = f'{git_install_path}/TransferBench'
out_dict = orch.exec(
f"bash -c 'cd {tb_src} && git checkout {git_tag}'",
timeout=120,
)
Comment on lines +224 to +227

@solaiys solaiys Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking (correctness): the git checkout result isn't verified, so the version pin can silently fail open.

  • The out_dict returned by this git checkout is discarded — there's no per-node error check.
  • The only post-install verification is ls -l {git_install_path}/TransferBench (line 239), which confirms the clone directory exists, not that the tag was actually checked out.
  • So if git checkout {git_tag} fails (tag missing, dirty/detached tree, fetch issue), the build below proceeds on the default branch, the test still passes, and you get a false-green that looks like v1.67.00 but isn't.
  • Pinning to v1.67.00 is the whole purpose of this PR, so an unverified checkout makes the pin a no-op exactly when it matters.

Suggested fix — scan the checkout output the same way the install step does:

Suggested change
out_dict = orch.exec(
f"bash -c 'cd {tb_src} && git checkout {git_tag}'",
timeout=120,
)
out_dict = orch.exec(
f"bash -c 'cd {tb_src} && git checkout {git_tag}'",
timeout=120,
)
# Fail loudly if the tag checkout did not succeed; otherwise the build below
# silently falls back to the default branch and the version pin is a no-op.
for node in out_dict.keys():
if re.search(r'error:|fatal:|did not match any', out_dict[node] or '', re.I):
fail_test(f'TransferBench checkout of tag {git_tag} failed on node {node}: {out_dict[node]}')
  • Even stronger (optional): assert the resolved HEAD matches the tag — e.g. run git describe --tags/git rev-parse HEAD after checkout and compare against {git_tag} — so a wrong-but-non-erroring checkout is also caught.

scan_test_results(out_dict)
for node, output in out_dict.items():
if re.search(r'error:|fatal:', output, re.I):
fail_test(f'git checkout {git_tag} failed on node {node}: {output.strip()}')

out_dict = orch.exec(
f"bash -c 'cd {tb_src} && git describe --tags --exact-match'",
timeout=60,
)
for node, output in out_dict.items():
actual_tag = output.strip()
if actual_tag != git_tag:
fail_test(
f'TransferBench not at git tag {git_tag} on node {node} after checkout (git describe: {actual_tag!r})'
)

# Detect ROCm path and compiler
rocm_path = detect_rocm_path(orch, config_dict.get('rocm_path', ''))
hip_compiler = detect_hip_compiler(orch, rocm_path)
Expand All @@ -221,7 +249,6 @@ def test_install_transferbench(orch, config_dict):
# uses cwd-relative paths so cwd MUST be the source tree; we wrap the
# build in `bash -c` explicitly to make the cwd dependency visible
# at the call site rather than via an implicit `cd X; cmd` chain.
tb_src = f'{git_install_path}/TransferBench'
out_dict = orch.exec(
f"bash -c 'cd {tb_src} && ROCM_PATH={rocm_path} HIPCC={hip_compiler} make'",
timeout=500,
Expand Down
142 changes: 16 additions & 126 deletions cvs/tests/health/transferbench_cvs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,21 @@ def parse_tb_p2p_bw(out_dict, exp_dict):
def parse_tb_scaling_bw(out_dict, exp_dict):
for node in out_dict.keys():
log.info(f"^^^^^ {out_dict[node]} ^^^^^^")
match = re.search('Best\s+[0-9\.]+\(\s[0-9]+\)\s+[0-9\.]+\(\s[0-9]+\)\s+([0-9\.]+)', out_dict[node])
# Best summary row: skip CPU00/CPU01 columns, capture GPU00 peak bandwidth.
# CU counts in parentheses may be padded (e.g. "( 8)" vs "( 32)").
match = re.search(
r'(?m)^[ \t]*Best\s+'
r'(?:[0-9\.]+\(\s*[0-9]+\)\s+){2}'
r'([0-9\.]+)',
out_dict[node],
)
if not match:
out = out_dict[node]
tail = 6000
if len(out) > tail:
out = f"...[truncated {len(out) - tail} chars; Best row is usually near the end]...\n{out[-tail:]}"
fail_test(f"TransferBench scaling Best row GPU00 bandwidth not found on node {node}. Output: {out}")
continue
gpu0_bw = float(match.group(1))
if float(gpu0_bw) < float(exp_dict['best_gpu0_bw']):
fail_test(
Expand Down Expand Up @@ -208,130 +222,6 @@ def parse_tb_schmoo_bw(out_dict, exp_dict):
)


def parse_tb_example_test_results(out_dict, exp_dict):
for node in out_dict.keys():
# Test 1 results parse
match = re.search('(Test\s1:\n.*\n.*\n.*\n)', out_dict[node])
if not match:
fail_test(f"Test 1 output section not found in TransferBench output on node {node}")
continue
test1_out = match.group(1)
match = re.search(r'(?:Transfer|Executor:\s+GPU)\s+[0-9]+\s+[|│]\s*([0-9\.]+)\s+GB/s', test1_out, re.I)
if not match:
fail_test(f"Transfer bandwidth pattern not found in Test 1 output on node {node}. Output: {test1_out}")
continue
test1_res = match.group(1)
if float(test1_res) < float(exp_dict['test1']):
fail_test(
f"Transfer Bench example test1 failed actual value {test1_res} is less than expected {exp_dict['test1']}"
)

# Test 2 results parse
match = re.search('(Test\s2:\n.*\n.*\n.*\n)', out_dict[node])
if not match:
fail_test(f"Test 2 output section not found in TransferBench output on node {node}")
continue
test2_out = match.group(1)
match = re.search(r'(?:Transfer|Executor:\s+(?:GPU|DMA))\s+[0-9]+\s+[|│]\s*([0-9\.]+)\s+GB/s', test2_out, re.I)
if not match:
fail_test(f"Transfer bandwidth pattern not found in Test 2 output on node {node}. Output: {test2_out}")
continue
test2_res = match.group(1)
if float(test2_res) < float(exp_dict['test2']):
fail_test(
f"Transfer Bench example test2 failed actual value {test2_res} is less than expected {exp_dict['test2']}"
)

# Test 3 results parse
match = re.search('(Test\s3:.*?)(?=Test\s[456]:|##|$)', out_dict[node], re.DOTALL)
if not match:
fail_test(f"Test 3 output section not found in TransferBench output on node {node}")
continue
test3_out = match.group(1)

match = re.search(
r'(?:Transfer|Executor:\s+GPU)\s+[0-9]+\s+[|│]\s*([0-9\.]+)\s+GB/s(?:[\s0-9\.\|a-z]+\s+G0\s*-\>)?',
test3_out,
re.I,
)
if not match:
fail_test(
f"G0->G1 transfer bandwidth pattern not found in Test 3 output on node {node}. Output: {test3_out}"
)
continue
test3_res_0_1 = match.group(1)
if float(test3_res_0_1) < float(exp_dict['test3_0_to_1']):
fail_test(
f"Transfer Bench example test3 failed actual value {test3_res_0_1} is less than expected {exp_dict['test3_0_to_1']}"
)

match = re.search(
r'(?:Transfer|Executor:\s+GPU)\s+[0-9]+\s+[|│]\s*([0-9\.]+)\s+GB/s(?:[\s0-9\.\|a-z]+\s+G1\s*-\>)?',
test3_out,
re.I,
)
if not match:
fail_test(
f"G1->G0 transfer bandwidth pattern not found in Test 3 output on node {node}. Output: {test3_out}"
)
continue
test3_res_1_0 = match.group(1)
if float(test3_res_1_0) < float(exp_dict['test3_1_to_0']):
fail_test(
f"Transfer Bench example test3 failed actual value {test3_res_1_0} is less than expected {exp_dict['test3_1_to_0']}"
)

# Test 4 results parse
match = re.search('(Test\s4:\n.*\n.*\n.*\n)', out_dict[node])
if not match:
fail_test(f"Test 4 output section not found in TransferBench output on node {node}")
continue
test4_out = match.group(1)
match = re.search(r'(?:Transfer|Executor:\s+GPU)\s+[0-9]+\s+[|│]\s*([0-9\.]+)\s+GB/s', test4_out, re.I)
if not match:
fail_test(f"Transfer bandwidth pattern not found in Test 4 output on node {node}. Output: {test4_out}")
continue
test4_res = match.group(1)
if float(test4_res) < float(exp_dict['test4']):
fail_test(
f"Transfer Bench example test4 failed actual value {test4_res} is less than expected {exp_dict['test4']}"
)

# Test 5 CPU only .. skip
# Test 6 results parse
match = re.search('(Test\s6:\n.*\n.*\n.*\n)', out_dict[node])
if not match:
fail_test(f"Test 6 output section not found in TransferBench output on node {node}")
continue
test6_out = match.group(1)
match = re.search(r'Executor:\s+GPU\s+[0-9]+\s+│\s*([0-9\.]+)\s+GB/s', test6_out, re.I)
if not match:
fail_test(f"Transfer bandwidth pattern not found in Test 6 output on node {node}. Output: {test6_out}")
continue
test6_res = match.group(1)
if float(test6_res) < float(exp_dict['test6']):
fail_test(
f"Transfer Bench example test6 failed actual value {test6_res} is less than expected {exp_dict['test6']}"
)


def test_transfer_bench_example_tests_1_6_t(orch, config_dict):
globals.error_list = []
log.info('Testcase Run TransferBench example tests 1-6')
path = config_dict['path']
rocm_path = detect_rocm_path(orch, config_dict.get('rocm_path', ''))
log.info("%s", config_dict)
example_path = config_dict['example_tests_path']
out_dict = orch.exec(
f"sudo bash -c 'export LD_LIBRARY_PATH={rocm_path}/lib:$LD_LIBRARY_PATH && echo \"LD_LIBRARY_PATH: $LD_LIBRARY_PATH\" && {path}/TransferBench {example_path}/example.cfg'",
timeout=(60 * 5),
)
print_test_output(log, out_dict)
scan_test_results(out_dict)
parse_tb_example_test_results(out_dict, config_dict['results']['example_results'])
update_test_result()


def test_transfer_bench_a2a(
orch,
config_dict,
Expand Down Expand Up @@ -412,7 +302,7 @@ def test_transfer_bench_scaling(
path = config_dict['path']
rocm_path = detect_rocm_path(orch, config_dict.get('rocm_path', ''))
out_dict = orch.exec(
f"sudo bash -c 'export LD_LIBRARY_PATH={rocm_path}/lib:$LD_LIBRARY_PATH && echo \"LD_LIBRARY_PATH: $LD_LIBRARY_PATH\" && {path}/TransferBench scaling'",
f"sudo bash -c 'export LD_LIBRARY_PATH={rocm_path}/lib:$LD_LIBRARY_PATH && echo \"LD_LIBRARY_PATH: $LD_LIBRARY_PATH\" && GFX_TEMPORAL=3 GFX_UNROLL=32 {path}/TransferBench scaling'",
timeout=(60 * 10),
)
print_test_output(log, out_dict)
Expand Down
1 change: 0 additions & 1 deletion docs/how-to/run-cvs-tests.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -267,7 +267,6 @@ You can list all available TransferBench test cases using the CLI:
.. code:: text

Available tests in transferbench_cvs:
- test_transfer_bench_example_tests_1_6_t
- test_transfer_bench_a2a
- test_transfer_bench_p2p
- test_transfer_bench_healthcheck
Expand Down
1 change: 0 additions & 1 deletion docs/install/cvs-install.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,6 @@ Health

- Under ``transferbench``:

- ``example_tests_path``
- ``git_install_path``

- Under ``rvs``:
Expand Down
37 changes: 5 additions & 32 deletions docs/reference/configuration-files/health.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,9 +32,9 @@ Here's a code snippet of the ``mi300_health_config.json`` file for reference:
"transferbench":
{
"path": "/home/{user-id}/INSTALL/TransferBench",
"example_tests_path": "/home/{user-id}/INSTALL/TransferBench/examples",
"git_install_path": "/home/{user-id}/INSTALL/",
"git_url": "https://github.com/ROCm/TransferBench.git",
"git_tag": "v1.67.00",
"_comment_rocm_path": "ROCm installation path. Set the placeholder changeme to auto-detect from /opt/rocm or /opt/rocm/core-*",
"rocm_path": "<changeme>",
"results":
Expand All @@ -50,16 +50,7 @@ Here's a code snippet of the ``mi300_health_config.json`` file for reference:
"32_cu_local_copy": "1250.0",
"32_cu_rem_read": "48.0",
"32_cu_rem_write": "48.0",
"32_cu_rem_copy": "48.0",
"example_results":
{
"test1": "47.1",
"test2": "48.4",
"test3_0_to_1": "31.9",
"test3_1_to_0": "38.9",
"test4": "1264",
"test6": "48.6"
}
"32_cu_rem_copy": "48.0"

}
},
Expand Down Expand Up @@ -198,15 +189,15 @@ TransferBench
* - ``path``
- ``/opt/amd/transferbench``
- Path where TransferBench is installed
* - ``example_tests_path``
- ``/home/{user-id}/INSTALL`` |br| ``/TransferBench/examples``
- Path where TransferBench examples are installed
* - ``git_install_path``
- ``/home/{user-id}/INSTALL/``
- Path where the Git repo is installed
* - ``git_url``
- `https://github.com/ROCm/TransferBench.git <https://github.com/ROCm/TransferBench.git>`_
- URL for Git repo
* - ``git_tag``
- ``v1.67.00``
- Git tag to checkout after cloning TransferBench
* - ``rocm_path``
- ``<changeme>``
- Set the path of rocm
Expand Down Expand Up @@ -243,24 +234,6 @@ TransferBench
* - ``32_cu_rem_copy``
- 48.0
- Remote memory copy bandwidth (GB/s) using 32 CUs
* - ``test1``
- 47.1
- Specific benchmark result (likely bandwidth in GB/s)
* - ``test2``
- 48.4
- Another benchmark result
* - ``test3_0_to_1``
- 31.9
- Directional test from GPU 0 to GPU 1
* - ``test3_1_to_0``
- 38.9
- Directional test from GPU 1 to GPU 0
* - ``test4``
- 1264
- High-performance test result (possibly local memory)
* - ``test6``
- 48.6
- Additional benchmark result

ROCm Validation Suite (RVS)
---------------------------
Expand Down
Loading