Skip to content

fix: reliable, non-flooding container log streaming during deploys (retry + replicas budget + stop on pod ready) - #408

Open
zoulja wants to merge 3 commits into
werf:mainfrom
zoulja:fix/pod-log-stream-retry
Open

zoulja wants to merge 3 commits into
werf:mainfrom
zoulja:fix/pod-log-stream-retry

Conversation

@zoulja

@zoulja zoulja commented Nov 19, 2026 •

Copy link
Copy Markdown

Summary

Three fixes that make container log output reliable and sane during Kubernetes deploys. All were found while investigating intermittent "no pod logs" / "too many logs" symptoms when deploying with nelm (and thus werf, which uses nelm as its deployment engine).

Fix 1 — single-shot, non-retrying log stream

The pod tracker followed each container's logs with a single GetLogs(Follow=true) stream and gave up on the first attach error, silently dropping all logs of that container (the error was only reported at debug level). A transient attach failure — a restarting pod, a connection reset, a container not yet running because an init container is still starting — could produce zero logs even for a crashing pod.

Fix: trackContainer retries followContainerLogs with a short backoff on error, for as long as the tracking context is active, resetting the since-time to now on each retry so already-captured lines are not re-emitted as duplicates.

Fix 2 — old ReplicaSet pods exhaust the log-replicas budget

During a Deployment rollout the old ReplicaSet is scaled down while its pods are still being tracked. runPodTracker incremented savingLogsReplicas for every tracked pod regardless of ReplicaSet, so old scaling-down pods could exhaust SaveLogsOnlyForNumberOfReplicas and the actual new pods (e.g. the second new replica) silently got no logs at all — not even init-container logs.

Fix: runPodTracker now receives rsNew and only pods of the new ReplicaSet consume the budget:

ignoreLogs := d.ignoreLogs || !rsNew || d.savingLogsReplicas >= d.SaveLogsOnlyForNumberOfReplicas
if !ignoreLogs {
    d.savingLogsReplicas++
}

This mirrors the IsNew guard in handleReplicaSetPodLogChunk, which already drops log chunks of non-new ReplicaSets.

Fix 3 — stop streaming container logs once the pod is Ready

A pod that becomes Ready but keeps running (e.g. a webserver already serving traffic) kept streaming its container logs until the whole Deployment converged, flooding the deploy output with runtime/access logs (thousands of lines) while other pods of the rollout were still starting.

The naive "stop on Ready" (transitioning the container to ContainerTrackerDone) was not enough: followContainerLogs blocks inside req.Stream and does not react to the state change until the tracking context is canceled at deploy end.

Fix: each container follow now runs on a per-container cancellable context (containerLogFollowCancel). handleContainersState cancels it the moment the container is marked done (pod Ready), which interrupts the in-flight stream and stops access logs at Ready. trackContainer survives the Ready-cancel (labeled break + ctx.Done in the outer select) so it can resume if the pod becomes not-ready again, avoiding a deadlock. Crash-looping / not-ready pods keep streaming, so failure logs are still shown.

Why all three are needed

They fix independent, complementary failure modes:

  • Fix 1 makes capture reliable for pods that are followed (transient attach failures).
  • Fix 2 makes sure the right (new) pods are followed, not old scaling-down ones.
  • Fix 3 prevents the deploy output from being flooded by a ready pod's runtime/access logs.

How it was found / verified

  • Reported: on a successful Deployment update with werf.io/show-logs-only-for-number-of-replicas=2 and 2 replicas, logs appeared from only the first pod; the second new pod showed zero logs (not even init-container logs). Root cause: old ReplicaSet pods consumed the budget (Fix 2).
  • Reported: a Caddy webserver flooded the deploy output with access logs after it became Ready. Verified against the live cluster: the pod is Ready=True, readiness probe httpGet /readyz:8080 with initialDelaySeconds: 30; the pod became Ready at 10:28:54 and the access logs ran from 10:28:55 (after Ready); the Service has no publishNotReadyAddresses. The flood was post-Ready access logs that the follow kept streaming (Fix 3).
  • The retry fix (Fix 1) was verified separately (reliable single-pod logs on a failing deploy).
  • Combined build verified against real Deployment rollouts: both new pods' logs appear, and access logs stop once each pod is Ready.

Files changed

  • pkg/tracker/pod/tracker.go — Fix 1 (retry stream) and Fix 3 (per-container cancel on Ready).
  • pkg/tracker/deployment/tracker.go — Fix 2 (only new ReplicaSet pods count towards the budget).

Notes

  • Deployment tracker tests are integration-style (harness + fake clients); without mocking GetLogs they hang locally, so the full deployment test suite requires a real cluster / the kubedog CI environment. go build ./... and go test ./pkg/tracker/pod/ pass locally.
  • After merge, bump the kubedog version in nelm/go.mod and rebuild the deployer image.

a.v.chumak added 2 commits November 19, 2026 21:12
The pod tracker followed container logs with a single GetLogs stream per
container and gave up on the first attach error, silently dropping all logs
of that container (the error was only logged in debug mode). A transient
attach failure (pod restarting, connection reset) could therefore produce
zero logs even for a crashing pod.

Retry followContainerLogs with a short backoff on error until the tracking
context is canceled, resetting the since-time to now on each retry so already
captured lines are not re-emitted as duplicates.
During a Deployment rollout the old ReplicaSet is scaled down while its pods
are still tracked. runPodTracker incremented savingLogsReplicas for every
tracked pod regardless of which ReplicaSet it belonged to, so old scaling-down
pods could exhaust SaveLogsOnlyForNumberOfReplicas and the actual new pods
(e.g. the second new replica) silently got no logs at all.

Now only pods of the new ReplicaSet consume the budget, mirroring the IsNew
guard already used by handleReplicaSetPodLogChunk to drop log chunks of
non-new ReplicaSets.
@zoulja zoulja changed the title fix: retry container log stream on error instead of dropping logs fix: reliable container log streaming during deploys (retry stream + replicas budget) Nov 20, 2026
A pod that becomes Ready but keeps running (e.g. a webserver already serving
traffic) continued to stream its container logs until the whole Deployment
converged, flooding the deploy output with runtime/access logs (thousands of
lines) while other pods of the rollout were still starting.

Now handleContainersState transitions containers to ContainerTrackerDone as
soon as the pod is Ready, stopping the log follow once startup is complete.
Crash-looping/not-ready pods keep streaming, so failure logs are still shown.
@zoulja zoulja changed the title fix: reliable container log streaming during deploys (retry stream + replicas budget) fix: reliable, non-flooding container log streaming during deploys (retry + replicas budget + stop on pod ready) Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant