Give the gate build job room and stop a stalled friend-module build from eating it - #446
Conversation
…rom eating it The build_linux_amd64 job of every language gate is cancelled by the 60 minute default job timeout whenever anything runs slightly long. The job's cost is both high and variable. Three recent runs of the C gate, same steps, same agent image: 41m checkout 17m15s, language image 4m39s, friend module 18m06s 54m checkout 15m14s, language image 4m39s, friend module 33m02s 60m checkout 15m20s, language image 4m34s, friend module killed at 39m05s So a healthy run already spends 41-54 of its 60 minutes, and the friend module step alone varies by more than 15 minutes between runs that differ in nothing. There is no headroom, and a run at the slow end of the normal range ends with "ran longer than the maximum time of 60 minutes" and no error that points at anything. The change under test is irrelevant to it. Two changes: - timeoutInMinutes: 120 on build_linux_amd64 in all five gate templates, so the job is bounded by something above its real cost rather than just below it. All five share the same step template and the same near-cap cost. - Bound the friend-module docker build at 40 minutes and retry it once. The `lerna bootstrap --hoist` layer installs ~640 packages and prints nothing while it does so; it was silent for 7m47s in a run that passed and for 26m35s in the run that was killed, still silent at the end. Unbounded, that phase consumes whatever is left of the job and is reported as a job timeout rather than as a failed build. 40 minutes clears the slowest healthy run seen (33m) and fits the budget twice: ~22 minutes of checkout and language image plus two attempts and the push stays inside the 120 minute cap. The retry is cheap because layers completed by the first attempt are still in the local image cache, so the second attempt resumes at the layer that stalled. Raising the job timeout alone would very likely have carried the killed run; bounding the build is what turns a future stall into a reported stall instead of an unexplained job cancellation. Verified locally. All five templates parse and the extracted step script passes bash -n. The script was exercised against a stubbed docker and the real timeout binary: success on the first attempt builds once and pushes once; a build failure and a genuine stall each retry once, name the reason and exit 1; a stall followed by a success pushes. BUILD_TIMEOUT_SECONDS is overridable. The docker build and ACR push themselves need a hosted agent and cannot be reproduced outside one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR adjusts Azure Pipelines gate jobs to avoid unexplained cancellations by (1) increasing job-level timeouts to provide headroom and (2) bounding the friend-module Docker build with a timeout + retry so stalls fail fast with an explicit error.
Changes:
- Increase
build_linux_amd64job timeout to 120 minutes across all gate job templates. - Wrap the friend-module
docker buildin atimeoutand retry it once to convert silent stalls into actionable failures.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| vsts/templates/steps-build-docker-image.yaml | Add bounded/retried friend-module Docker build to prevent stalls from consuming the entire job timeout. |
| vsts/templates/jobs-gate-pythonv2.yaml | Raise build_linux_amd64 job timeout to 120 minutes for Python gate. |
| vsts/templates/jobs-gate-node.yaml | Raise build_linux_amd64 job timeout to 120 minutes for Node gate. |
| vsts/templates/jobs-gate-java.yaml | Raise build_linux_amd64 job timeout to 120 minutes for Java gate. |
| vsts/templates/jobs-gate-csharp.yaml | Raise build_linux_amd64 job timeout to 120 minutes for C# gate. |
| vsts/templates/jobs-gate-c.yaml | Raise build_linux_amd64 job timeout to 120 minutes for C gate. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…the budget The timeout branch reported "made no progress", but the bound is wall clock and does not measure output, so the message was misleading when reading a failed run. It now says the build did not finish in time. The comment also claimed a 40 minute bound per attempt. --kill-after gives docker a further minute to tear down after SIGTERM, so an attempt can reach 41 minutes and two attempts ~82 minutes. Stated, and still inside the 120 minute job timeout alongside the ~22 minutes of checkout and language image. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Fresh occurrence of the failure mode this PR bounds: c gate build 162681 lost |
Failure
build_linux_amd64in the C gate is cancelled by the default 60 minute job timeout:No step reports an error of its own. The change under test is unrelated to it.
Cause
The job's cost is high and variable. Three recent runs of the same gate, same steps, same agent image:
A healthy run already spends 41-54 of its 60 minutes, and the friend-module step alone varies by more than 15 minutes between runs that differ in nothing. There is no headroom, so a run at the slow end of the normal range is cancelled.
Inside that step, the
lerna bootstrap --hoistlayer installs ~640 packages and prints nothing while it does so. It was silent for 7m47s in a run that passed, and for 26m35s in the run that was killed, still silent at the end. Agent CPU was 0.5-1% throughout and disk was not exhausted.Fix
timeoutInMinutes: 120onbuild_linux_amd64in all five gate templates. They share the same step template and the same near-cap cost.docker buildat 40 minutes and retry it once. 40 minutes clears the slowest healthy run seen (33m) and fits the budget twice: ~22 minutes of checkout and language image plus two attempts and the push stays inside the 120 minute cap. The retry is cheap because layers completed by the first attempt are still in the local image cache, so the second attempt resumes at the layer that stalled.BUILD_TIMEOUT_SECONDSoverrides the bound.Raising the job timeout alone would very likely have carried the cancelled run. Bounding the build is what turns a future stall into a reported stall instead of an unexplained job cancellation.
Validation
Local. All five templates parse. The step script was extracted from the template, passes
bash -n, and was exercised against a stubbeddockerand the realtimeoutbinary:Both failure paths name the reason (
Build made no progress within Ns and was stopped./Build failed with exit code N.). The happy path is unchanged and adds no latency.The docker build and ACR push themselves need a hosted agent and cannot be reproduced outside one.