Skip to content

Give the gate build job room and stop a stalled friend-module build from eating it - #446

Merged
Ewerton Scaboro da Silva (ewertons) merged 2 commits into
masterfrom
fix/gate-build-job-timeout-and-friend-module-stall
Aug 27, 2026
Merged

Ewerton Scaboro da Silva (ewertons) merged 2 commits into
masterfrom
fix/gate-build-job-timeout-and-friend-module-stall

Conversation

@ewertons

Copy link
Copy Markdown
Contributor

Failure

build_linux_amd64 in the C gate is cancelled by the default 60 minute job timeout:

The job running on agent Azure Pipelines 17 ran longer than the maximum time of 60 minutes.

No step reports an error of its own. The change under test is unrelated to it.

Cause

The job's cost is high and variable. Three recent runs of the same gate, same steps, same agent image:

job total checkout language image friend module
41m 17m15s 4m39s 18m06s
54m 15m14s 4m39s 33m02s
60m (cancelled) 15m20s 4m34s killed at 39m05s

A healthy run already spends 41-54 of its 60 minutes, and the friend-module step alone varies by more than 15 minutes between runs that differ in nothing. There is no headroom, so a run at the slow end of the normal range is cancelled.

Inside that step, the lerna bootstrap --hoist layer installs ~640 packages and prints nothing while it does so. It was silent for 7m47s in a run that passed, and for 26m35s in the run that was killed, still silent at the end. Agent CPU was 0.5-1% throughout and disk was not exhausted.

Fix

  • timeoutInMinutes: 120 on build_linux_amd64 in all five gate templates. They share the same step template and the same near-cap cost.
  • Bound the friend-module docker build at 40 minutes and retry it once. 40 minutes clears the slowest healthy run seen (33m) and fits the budget twice: ~22 minutes of checkout and language image plus two attempts and the push stays inside the 120 minute cap. The retry is cheap because layers completed by the first attempt are still in the local image cache, so the second attempt resumes at the layer that stalled. BUILD_TIMEOUT_SECONDS overrides the bound.

Raising the job timeout alone would very likely have carried the cancelled run. Bounding the build is what turns a future stall into a reported stall instead of an unexplained job cancellation.

Validation

Local. All five templates parse. The step script was extracted from the template, passes bash -n, and was exercised against a stubbed docker and the real timeout binary:

scenario exit build attempts pushes
succeeds first time 0 1 1
build fails 1 2 0
build stalls 1 2 0
stalls, then succeeds 0 2 1

Both failure paths name the reason (Build made no progress within Ns and was stopped. / Build failed with exit code N.). The happy path is unchanged and adds no latency.

The docker build and ACR push themselves need a hosted agent and cannot be reproduced outside one.

…rom eating it

The build_linux_amd64 job of every language gate is cancelled by the 60
minute default job timeout whenever anything runs slightly long.

The job's cost is both high and variable. Three recent runs of the C gate,
same steps, same agent image:

  41m  checkout 17m15s, language image 4m39s, friend module 18m06s
  54m  checkout 15m14s, language image 4m39s, friend module 33m02s
  60m  checkout 15m20s, language image 4m34s, friend module killed at 39m05s

So a healthy run already spends 41-54 of its 60 minutes, and the friend
module step alone varies by more than 15 minutes between runs that differ in
nothing. There is no headroom, and a run at the slow end of the normal range
ends with "ran longer than the maximum time of 60 minutes" and no error that
points at anything. The change under test is irrelevant to it.

Two changes:

- timeoutInMinutes: 120 on build_linux_amd64 in all five gate templates, so
  the job is bounded by something above its real cost rather than just below
  it. All five share the same step template and the same near-cap cost.

- Bound the friend-module docker build at 40 minutes and retry it once. The
  `lerna bootstrap --hoist` layer installs ~640 packages and prints nothing
  while it does so; it was silent for 7m47s in a run that passed and for
  26m35s in the run that was killed, still silent at the end. Unbounded, that
  phase consumes whatever is left of the job and is reported as a job timeout
  rather than as a failed build. 40 minutes clears the slowest healthy run
  seen (33m) and fits the budget twice: ~22 minutes of checkout and language
  image plus two attempts and the push stays inside the 120 minute cap. The
  retry is cheap because layers completed by the first attempt are still in
  the local image cache, so the second attempt resumes at the layer that
  stalled.

Raising the job timeout alone would very likely have carried the killed run;
bounding the build is what turns a future stall into a reported stall instead
of an unexplained job cancellation.

Verified locally. All five templates parse and the extracted step script
passes bash -n. The script was exercised against a stubbed docker and the
real timeout binary: success on the first attempt builds once and pushes
once; a build failure and a genuine stall each retry once, name the reason
and exit 1; a stall followed by a success pushes. BUILD_TIMEOUT_SECONDS is
overridable. The docker build and ACR push themselves need a hosted agent
and cannot be reproduced outside one.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts Azure Pipelines gate jobs to avoid unexplained cancellations by (1) increasing job-level timeouts to provide headroom and (2) bounding the friend-module Docker build with a timeout + retry so stalls fail fast with an explicit error.

Changes:

  • Increase build_linux_amd64 job timeout to 120 minutes across all gate job templates.
  • Wrap the friend-module docker build in a timeout and retry it once to convert silent stalls into actionable failures.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
vsts/templates/steps-build-docker-image.yaml Add bounded/retried friend-module Docker build to prevent stalls from consuming the entire job timeout.
vsts/templates/jobs-gate-pythonv2.yaml Raise build_linux_amd64 job timeout to 120 minutes for Python gate.
vsts/templates/jobs-gate-node.yaml Raise build_linux_amd64 job timeout to 120 minutes for Node gate.
vsts/templates/jobs-gate-java.yaml Raise build_linux_amd64 job timeout to 120 minutes for Java gate.
vsts/templates/jobs-gate-csharp.yaml Raise build_linux_amd64 job timeout to 120 minutes for C# gate.
vsts/templates/jobs-gate-c.yaml Raise build_linux_amd64 job timeout to 120 minutes for C gate.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread vsts/templates/steps-build-docker-image.yaml Outdated
Comment thread vsts/templates/steps-build-docker-image.yaml
…the budget

The timeout branch reported "made no progress", but the bound is wall clock
and does not measure output, so the message was misleading when reading a
failed run. It now says the build did not finish in time.

The comment also claimed a 40 minute bound per attempt. --kill-after gives
docker a further minute to tear down after SIGTERM, so an attempt can reach
41 minutes and two attempts ~82 minutes. Stated, and still inside the 120
minute job timeout alongside the ~22 minutes of checkout and language image.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

@ewertons
Ewerton Scaboro da Silva (ewertons) merged commit cd29a8e into master Aug 27, 2026
3 checks passed
@ewertons
Ewerton Scaboro da Silva (ewertons) deleted the fix/gate-build-job-timeout-and-friend-module-stall branch August 27, 2026 19:42
@ewertons

Copy link
Copy Markdown
Contributor Author

Fresh occurrence of the failure mode this PR bounds: c gate build 162681 lost build and push default-friend-module to npm error network read ECONNRESET (lerna bootstrap, exit 152) after ~355s. A plain re-run passed, which is exactly the retry this PR automates.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants