test: Add HSTU (Generative Recommenders) Torch AOTI CI test

[yinggeh]

Summary

Adds qa/L0_torch_aoti_hstu/test.sh, an end-to-end HSTU (Generative Recommenders) Torch AOTI test. One script, one container, matching the other L0 tests:

1. exports the HSTU ranking model to an AOTI package from the KuaiRand-1K checkpoint,
2. produces a reference output dump with the C++ KV-cache runtime,
3. assembles the platform: "torch_aoti" model repository,
4. serves it with tritonserver and runs the HSTU client against the dump.

The command sequence mirrors the DevTech reference test (Devtech-Compute/distributed-recommender: ci/tritonserver_test.sh), which splits these phases across two images. No Docker here: the CI job runs this in a single container built from recsys-examples' Dockerfile.tritonserver with BASE_IMAGE set to the server image from the pipeline under test (internal tritonserver MR), so /opt/tritonserver is the Triton being tested.

Two things worth review attention:

• Errexit is cleared explicitly. The harness invokes tests as bash -ex, and run_server → wait_for_server_ready leaves set -e enabled on return, so every EXIT_CODE=$? handler here would be dead code and a client failure would abort before the FlexKV and tritonserver teardown. set +e is applied at the top and again after run_server. L0_torch_aoti has the same latent pattern, which I've left alone.
• The server is run once before serving. The test image LD_PRELOADs nine HSTU ops libraries into every process, so tritonserver --version runs first and a link error surfaces directly instead of as a startup timeout.

All artifacts (logs, exported package, reference dump, model repository) are written under the test directory so CI collects them and reruns start clean.

Validation

• bash -n passes.
• Ran the script end-to-end against stubbed recsys tooling and a fake tritonserver (invoked exactly as CI does, bash -ex ./test.sh) with a deliberately failing client: the failure is caught, the client log and Test Suite FAILED banner are printed, teardown of both tritonserver and FlexKV runs, and the script exits non-zero. Also confirmed the model repository is assembled with the exported package as version 1 alongside config.pbtxt.
• Command sequence reconciled line-by-line against the DevTech reference.
• Full pipeline execution depends on a stable tag for DevTech's train image, registry access, and the computelab scratch mounts, all tracked in the internal tritonserver CI MR (job gated on TRITON_HSTU, allow_failure: true until confirmed).

Refs: TRI-1615

[greptile-apps]

Greptile Summary

Adds an end-to-end HSTU Torch AOTI CI test.

• Exports the HSTU ranking model and generates reference output with the C++ KV-cache runtime.
• Builds a Torch AOTI model repository, starts FlexKV and Triton, waits for server readiness, and runs the comparison client.
• Uses the pipeline-staged Triton installation and preserves logs and generated artifacts under the test directory.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
qa/L0_torch_aoti_hstu/test.sh	Adds the complete HSTU export, reference-generation, Triton-serving, client-validation, and cleanup workflow; the previously reported startup race is resolved by the readiness-aware run_server helper.

Sequence Diagram

sequenceDiagram
    participant Test as test.sh
    participant Exporter as HSTU Exporter
    participant FlexKV
    participant Reference as C++ Runtime
    participant Triton
    participant Client
    Test->>Exporter: Export AOTI package
    Test->>FlexKV: Start reference cache
    Test->>Reference: Generate reference dump
    Test->>FlexKV: Stop and restart clean cache
    Test->>Triton: Start with generated model repository
    Test->>Triton: Wait for readiness
    Test->>Client: Compare Triton output with reference dump
    Test->>Triton: Stop server
    Test->>FlexKV: Stop cache

Loading

_{Reviews (3): Last reviewed commit: "test: Serve HSTU AOTI test from the imag..." | Re-trigger Greptile}