[logging] Publish the training loop phase as a Prometheus gauge

[eicherseiji]

What does this PR do?

Publishes which macro-phase the fully async training loop is in as a Prometheus gauge. Today, joining "what was the loop doing" to "how busy were the GPUs" means hand-correlating the experiment tracker against Prometheus by wall-clock.

New metric:

• skyrl_training_phase{phase=...}: 1.0 for the active phase, 0.0 for the rest. Phases match the loop's Timer() keys (wait_for_generation_buffer, convert_to_training_input, run_training, sync_weights, eval, save_checkpoints) plus the default generating, so the wandb timing/* keys and the Prometheus label share one vocabulary.

Ray exports it to the same Prometheus that scrapes node GPU metrics, so per-phase GPU utilization is a single-store query:

avg(ray_node_gpus_utilization) and on() (ray_skyrl_training_phase{phase="eval"} == 1)

Prometheus also survives a cluster restart, when the tracker may be unreachable.

How

New module skyrl/train/utils/phase_metrics.py with TrainingPhaseGauge, following the one-module-per-concern layout of ray_gpu_monitor.py. The trainer constructs it in __init__ and sets the phase at each existing Timer() boundary. Each transition writes only the two changed series; construction seeds all series so PromQL selectors never hit a missing one. Best-effort: it no-ops when Ray metrics are unavailable and warns on unknown phase names, so it never breaks training or tests.

Tests

• tests/train/utils/test_phase_metrics.py (new): exactly one phase active, context-manager restore, unknown-phase rejection, no-op when Ray is unavailable. 5 passed.
• ruff and black clean on changed files. End to end needs a GPU cluster.

[gemini-code-assist]

Code Review

This pull request introduces a TrainingPhaseGauge to publish the training loop's current macro-phase (such as eval, training, weight synchronization, and checkpointing) to Prometheus via Ray metrics. This allows training loop phases to be overlaid with cluster-wide GPU utilization metrics. The feedback suggests validating phase names against the defined list of allowed phases to prevent developer typos and silent failures, along with adding a corresponding unit test for this validation.