feature request: enforce RAPIDS `dependencies.yaml` conventions

[jameslamb]

Description

RAPIDS projects use https://github.com/rapidsai/dependency-file-generator to centralize dependency lists, usually in a single root-level file called dependencies.yaml.

Some features of those files are standardized only by convention... we should introduce a hook to put those conventions into code.

Benefits of this work

• reduces the risk of accidental regressions

Acceptance Criteria

• a hook exists which fails if a dependencies.yaml file does not follow any of RAPIDS' preferred conventions
• at a minimum, the hook correctly catches this case: Update CuPy and NumPy dependency specs NVIDIA/raft#3058 (comment)
• that hook has been rolled out to main of every RAPIDS repo using rapids-dependency-file-generator

Approach

Indifferent on the name. verify-dependencies is the best balance of informative and future-proof I can think of. It doesn't reference a specific file type (-yaml) or even rapids-dependency-file-generator at all, so I could see it being expanded to enforce conventions on pyproject.toml, conda/recipes/*/recipe.yaml, etc.

See https://github.com/rapidsai/build-planning/blob/f6dcd28ccfd0faabeb17c67c7bab0fecb0f9a971/docs/docs/packaging.md?plain=1#L85 for a list of RAPIDS dependencies.yaml conventions.

The dependencies.yaml path should be configurable... at least 1 project has multiple: https://github.com/rapidsai/cudf/blob/78c0fd3076da8955351680d705986aa58dc1d657/python/cudf/cudf_pandas_tests/third_party_integration_tests/dependencies.yaml

rapids-dependency-file-generator has a Python API that could be used in the hook, see https://github.com/rapidsai/rapids-build-backend for an example of how it's used to read dependency lists.

Notes

This request is inspired by the migration to bump RAPIDS to cupy>14. That introduced the cupy[ctk] requirement for wheels, which would have violated our convention that anything that pulls in cuda-toolkit or nvidia-{something} wheels is guarded by use_cuda_wheels: {true,false}:

• feat(deps): raise cupy lower bound to >=14 build-planning#279 (comment)
• Update CuPy and NumPy dependency specs NVIDIA/raft#3058 (comment)

Nothing in CI would have caught that change... it would have just been merged and then needed to be patch in RAPIDS DLFW and devcontainers builds.

A hook like the one described in this request would have caught it, without needing any particular reviewer's institutional knowledge.

[KyleFromNVIDIA]

We already have verify-alpha-spec which reads dependencies.yaml. If we don't roll that into this check, then at the very least, we can share code between them.