[CI/Build] Add self-hosted MI350X ROCm external PD tier

[andyluo7]

Description

Add a trusted self-hosted ROCm external prefill/decode CI tier for the two-node MI350X cluster.

This revision replaces the previously staged T-One controller because T-One does not provide AMD GPUs. The change:

• builds the Python 3.10 and 3.12 ROCm wheels in the same Actions run and downloads them directly on the controller runner;
• schedules the GPU job only for a trusted workflow_dispatch, a push to main, or the maintainer-controlled run-e2e-ci pull_request_target path;
• runs one non-root Actions runner on the controller and reaches the worker through a dedicated, pinned SSH identity;
• preserves the reserved NUMA0 allocation with GPU, render-device, CPU, memory, and RoCE allowlists;
• fixes the RoCE device set to ionic_0-ionic_3, selects GID index 1, and forces HCA transport so a missing RDMA path cannot silently fall back to TCP;
• adds a cluster-wide concurrency group and host lock, container-scoped cleanup, and per-run log artifacts;
• consumes local Actions artifacts while retaining the existing GitHub artifact path for the CUDA T-One job; and
• preserves all existing CUDA defaults and behavior.

The host profile is documented in scripts/tone_tests/rocm_runner.env.example. No T-One credential or long-lived GitHub token is required by the ROCm job.

The organization runner mooncake-rocm-mi350x-controller is registered with labels amd, rocm, gfx950, and mooncake-pd, but its service is intentionally disabled. Before activation, an organization administrator must place it in a runner group restricted to kvcache-ai/Mooncake and the trusted CI/E2E workflows. This PR remains a draft until a trusted end-to-end run and burn-in complete.

Module

• Transfer Engine (mooncake-transfer-engine)
• Mooncake Store (mooncake-store)
• Mooncake EP (mooncake-ep)
• Mooncake PG (mooncake-pg)
• Integration (mooncake-integration)
• P2P Store (mooncake-p2p-store)
• Python Wheel (mooncake-wheel)
• Common (mooncake-common)
• CI/CD
• Docs
• Other

Type of Change

• Bug fix
• New feature
• Refactor
• Breaking change
• Documentation update
• Performance improvement
• Other: CI/infrastructure

How Has This Been Tested?

Static validation:

actionlint -ignore 'label ".*" is unknown' \
  .github/workflows/ci.yml \
  .github/workflows/ci_rocm.yml \
  .github/workflows/e2e-ci.yml \
  .github/workflows/integration-test.yml

bash -n scripts/tone_tests/scripts/common.sh \
  scripts/tone_tests/scripts/run_test.sh \
  scripts/tone_tests/scripts/test_1p1d_erdma.sh \
  scripts/tone_tests/scripts/test_epd_sglang.sh \
  scripts/tone_tests/scripts/test_vllm_1p1d_erdma.sh

SKIP=trailing-whitespace,end-of-file-fixer,codespell \
  pre-commit run --files <touched-files>

git diff --check origin/main

The targeted Actionlint, YAML parser, shell syntax, codespell, pre-commit, local-wheel input, and whitespace checks pass. The skipped pre-commit hooks would report unrelated pre-existing text or normalize legacy shell files outside this change.

Host validation:

• controller and worker preflight passes as the dedicated mooncake-ci user;
• the selected fabric links are up with MTU 9000 and IPv4 RoCE GID index 1;
• the pinned SGLang container sees exactly four MI350X devices and only the selected render/RDMA devices on both nodes;
• ibv_devinfo opens ionic_0 through ionic_3 inside the least-privilege controller container; and
• all three public test-model snapshots are complete on both nodes.

The two-node Mooncake/SGLang/vLLM end-to-end Actions job remains pending the trusted upstream activation described above.

Test results:

• Unit tests pass
• Integration tests pass (pending trusted self-hosted activation)
• Manual static and host validation completed as described above

Checklist

• I have performed a human self-review of every changed line
• I have formatted my code using ./scripts/code_format.sh (not applicable; no C/C++ files changed)
• I have run pre-commit run --all-files and all hooks pass (targeted relevant hooks pass)
• I have updated the runner configuration example
• I have added tests to prove my changes are effective (live E2E pending)
• For changes >500 LOC: not a major product architectural change; this is isolated CI/test infrastructure

AI Assistance Disclosure

• No AI tools were used
• AI tools were used

OpenAI Codex helped analyze the CUDA/ROCm CI gap, adapt the integration scripts and workflow for a self-hosted MI350X controller/worker pair, validate resource isolation, and prepare this draft. The human submitter will review and defend every changed line before the PR is marked ready.

[staryxchen]

Hi @andyluo7 Is this PR ready to merge?

[andyluo7]

@staryxchen waiting for Wenjie's confirm.

[staryxchen]

@staryxchen waiting for Wenjie's confirm.

Hi @andyluo7, thanks for the work on this. I'd like to see this PR merged — the MI350X external-PD tier fills a real gap in our CI coverage for the ROCm path.

Separately, since you already added the ROCm release pipeline in #3184: could you help push the mooncake-transfer-engine-rocm wheel to PyPI as well? CUDA/EFA/MUSA/NPU already have their variants published there, but the pipeline hasn't been triggered yet.

[Aionw]

Hello, Stary. I've been validating this PR on a separate branch, where most of the issues have already been fixed. I'll port those changes back to this PR once the vLLM-related issue is resolved.