Jobs not being picked up occasionally

[mkannwischer]

I'm seeing a new failure mode recently.

In mldsa-native/mlkem-native, we start 4 relatively fast jobs on the RISE runners for each PR. Since recently (maybe yesterday), I often see 3 of the 4 jobs being picked up very quickly, but one does not get picked up for a long time even after the other 3 finish. Sometimes it helps to cancel the job and restart it.

For example here the 4th job wasn't picked up for almost 2 hours until I cancelled it and restarted it. The restarted one is not yet picked up after 30 minutes even though I think you are not under heavy load right now according to the dashboard.

The dashboard shows the job as pending and a corresponding worker as "active":

[mkannwischer]

The same pattern occured now here: https://github.com/pq-code-package/mlkem-native/actions/runs/27008198416/job/79726517694?pr=1648 with 3 jobs being picked up instantly, while the 4th one remained pending.

Eventually the mldsa-native job I mentioned above got picked up, so maybe this is a problem with the start-up of the runner being slow?

[mkannwischer]

Btw, we have recently changed our CI, so the RISE jobs only run if the base CI passes to reduce load for you.
If there are still load issues, we could reduce the number of tests.

[mkannwischer]

This one was waiting for almost 6 hours before I cancelled it: https://github.com/pq-code-package/mlkem-native/actions/runs/27063469189/job/79898328014?pr=1677

We are not the only ones affected by this: https://github.com/kairos-io/hadron/actions/runs/27069994409/job/79910245453?pr=406 has been waiting for 2.5h even though there is only one job running on the RISE runners.

[luhenry]

Just seeing you issue here, let me have a look.

[mkannwischer]

Just seeing you issue here, let me have a look.

Thanks. I have cut our CI down to run just one job for each PR on the rise runners. Since then I have not seen any more issues.