Repository navigation
Conversation
added 2 commits
November 19, 2026 21:12
The pod tracker followed container logs with a single GetLogs stream per container and gave up on the first attach error, silently dropping all logs of that container (the error was only logged in debug mode). A transient attach failure (pod restarting, connection reset) could therefore produce zero logs even for a crashing pod. Retry followContainerLogs with a short backoff on error until the tracking context is canceled, resetting the since-time to now on each retry so already captured lines are not re-emitted as duplicates.
During a Deployment rollout the old ReplicaSet is scaled down while its pods are still tracked. runPodTracker incremented savingLogsReplicas for every tracked pod regardless of which ReplicaSet it belonged to, so old scaling-down pods could exhaust SaveLogsOnlyForNumberOfReplicas and the actual new pods (e.g. the second new replica) silently got no logs at all. Now only pods of the new ReplicaSet consume the budget, mirroring the IsNew guard already used by handleReplicaSetPodLogChunk to drop log chunks of non-new ReplicaSets.
A pod that becomes Ready but keeps running (e.g. a webserver already serving traffic) continued to stream its container logs until the whole Deployment converged, flooding the deploy output with runtime/access logs (thousands of lines) while other pods of the rollout were still starting. Now handleContainersState transitions containers to ContainerTrackerDone as soon as the pod is Ready, stopping the log follow once startup is complete. Crash-looping/not-ready pods keep streaming, so failure logs are still shown.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three fixes that make container log output reliable and sane during Kubernetes deploys. All were found while investigating intermittent "no pod logs" / "too many logs" symptoms when deploying with nelm (and thus werf, which uses nelm as its deployment engine).
Fix 1 — single-shot, non-retrying log stream
The pod tracker followed each container's logs with a single
GetLogs(Follow=true)stream and gave up on the first attach error, silently dropping all logs of that container (the error was only reported at debug level). A transient attach failure — a restarting pod, a connection reset, a container not yet running because an init container is still starting — could produce zero logs even for a crashing pod.Fix:
trackContainerretriesfollowContainerLogswith a short backoff on error, for as long as the tracking context is active, resetting the since-time tonowon each retry so already-captured lines are not re-emitted as duplicates.Fix 2 — old ReplicaSet pods exhaust the log-replicas budget
During a Deployment rollout the old ReplicaSet is scaled down while its pods are still being tracked.
runPodTrackerincrementedsavingLogsReplicasfor every tracked pod regardless of ReplicaSet, so old scaling-down pods could exhaustSaveLogsOnlyForNumberOfReplicasand the actual new pods (e.g. the second new replica) silently got no logs at all — not even init-container logs.Fix:
runPodTrackernow receivesrsNewand only pods of the new ReplicaSet consume the budget:This mirrors the
IsNewguard inhandleReplicaSetPodLogChunk, which already drops log chunks of non-new ReplicaSets.Fix 3 — stop streaming container logs once the pod is Ready
A pod that becomes Ready but keeps running (e.g. a webserver already serving traffic) kept streaming its container logs until the whole Deployment converged, flooding the deploy output with runtime/access logs (thousands of lines) while other pods of the rollout were still starting.
The naive "stop on Ready" (transitioning the container to
ContainerTrackerDone) was not enough:followContainerLogsblocks insidereq.Streamand does not react to the state change until the tracking context is canceled at deploy end.Fix: each container follow now runs on a per-container cancellable context (
containerLogFollowCancel).handleContainersStatecancels it the moment the container is marked done (pod Ready), which interrupts the in-flight stream and stops access logs at Ready.trackContainersurvives the Ready-cancel (labeled break +ctx.Donein the outer select) so it can resume if the pod becomes not-ready again, avoiding a deadlock. Crash-looping / not-ready pods keep streaming, so failure logs are still shown.Why all three are needed
They fix independent, complementary failure modes:
How it was found / verified
werf.io/show-logs-only-for-number-of-replicas=2and 2 replicas, logs appeared from only the first pod; the second new pod showed zero logs (not even init-container logs). Root cause: old ReplicaSet pods consumed the budget (Fix 2).Ready=True, readiness probehttpGet /readyz:8080withinitialDelaySeconds: 30; the pod became Ready at 10:28:54 and the access logs ran from 10:28:55 (after Ready); the Service has nopublishNotReadyAddresses. The flood was post-Ready access logs that the follow kept streaming (Fix 3).Files changed
pkg/tracker/pod/tracker.go— Fix 1 (retry stream) and Fix 3 (per-container cancel on Ready).pkg/tracker/deployment/tracker.go— Fix 2 (only new ReplicaSet pods count towards the budget).Notes
GetLogsthey hang locally, so the full deployment test suite requires a real cluster / the kubedog CI environment.go build ./...andgo test ./pkg/tracker/pod/pass locally.nelm/go.modand rebuild the deployer image.