[logging] Rollout time split + async-trainer GPU/phase/buffer observability

[eicherseiji]

What does this PR do?

Adds observability for the async RL trainer. After a run, working out where GPU time went meant lining up two places that do not share an axis: Prometheus, which holds GPU and disk metrics by wall-clock time, and W&B, which holds phase timings by training step. Four changes each fill part of that gap.

1. Split rollout time into engine wait vs env.step()

Adds generate/trajectory_llm_time_{mean,p90,max}, generate/trajectory_env_time_{mean,p90,max}, and generate/frac_time_in_env. Until now only the total per-trajectory time existed, trajectory_completion_time_* from #1804, which cannot tell whether a slow rollout is waiting on the inference engine or running the environment. That is the first thing you want to know for multi-turn and agentic envs.

agent_loop adds up the two parts per trajectory. They are stored as raw per-trajectory lists on GeneratorOutput, the same as e2e_time, and re-aggregated in concatenate_generator_outputs. The raw lists have to reach that step. Every logging path goes through it, and its fallback aggregator sums any key it does not recognize, which is right for a total but wrong for a p90 or a ratio. frac_time_in_env is env time over total busy time, so long trajectories weigh proportionally. The two parts need not add up to e2e_time. The rest is tokenization, chat-template rendering, and other in-loop work. Both are omitted if any trajectory in the batch did not record them.

2. Fix: RayGpuMonitor never ran on the async trainer

RayGpuMonitor from #1712 is built in RayPPOTrainer.__init__ and flushed by the base loop. FullyAsyncRayPPOTrainer overrides train() and never called .start(), .flush(), or .stop(). So async runs logged no ray/ GPU keys at all, even though enable_ray_gpu_monitor=True by default. Confirmed on real runs. It now starts at loop entry, flushes each step into the logged payload, and stops in the finally.

3. skyrl_training_phase gauge to Prometheus

New module skyrl/train/utils/phase_metrics.py. It reports which phase the loop is in (waiting_for_buffer, converting, training, weight_sync, eval, checkpoint, or generating) through ray.util.metrics. Ray sends that to the same Prometheus that already scrapes node GPU metrics, under the ray_ prefix. So ray_skyrl_training_phase{phase="eval"} == 1 marks the eval windows on the GPU timeline, and you can read GPU use per phase from one place. It also keeps working after a cluster restart, when the experiment tracker may be gone. The gauge is set at each existing Timer() boundary. It is best-effort and does nothing when Ray metrics are unavailable, so it never breaks training or tests.

4. Log the rollout buffer depth

Generation and training run at the same time. Generators produce rollout batches into a buffer, and the trainer pulls mini_batch_size batches per step. The buffer depth, how many batches are waiting, tells you which side is the bottleneck. Near zero means the trainer is waiting on generation. Near the cap means generation is paused at the staleness limit. Same idle-GPU symptom, opposite cause. This was only visible in a live tqdm bar before and then lost.

Logs async/gen_buffer_qsize, the current depth, and async/gen_buffer_maxsize, the cap, to W&B each step. The same two values, plus skyrl_mini_batch_size and skyrl_gen_group_keep_rate, also go to Prometheus as gauges through ScalarGauges, so buffer depth sits next to the phase gauge and GPU metrics. keep_rate is kept / (kept + dropped) for the last batch, the share of rollouts not thrown out as zero-variance under sample_full_batch. A deep buffer with a low keep rate is mostly rollouts that will be dropped.

Scope

Item 1 was the original scope. Items 2 through 4 grow it into a small async-observability PR. I can split them into a follow-up if preferred. A skyrl_node_role{train|inference} gauge is left out because inference node ids are not cleanly reachable from the trainer. Noted as a follow-up.

Tests

• tests/train/utils/test_phase_metrics.py, new: one active phase at a time, context-manager restore, no-op when Ray is unavailable, and the ScalarGauges create-once and disable-on-failure behavior.
• tests/train/generators/test_skyrl_gym_generator.py: adds test_llm_vs_env_time_split_metrics. It checks the split on a single generate() output and on a concatenate_generator_outputs result, so frac_time_in_env stays in range and p90 is not inflated on the path every logger uses.
• uv run --isolated --extra dev --extra skyrl-train pytest tests/train/utils/test_phase_metrics.py tests/train/generators/test_skyrl_gym_generator.py passes, 36 tests. ruff and black clean on changed files.

Trainer wiring for items 2 through 4 follows the base-trainer pattern and is covered by unit tests and lint. It is not run end-to-end here, which needs a GPU cluster.