Repository navigation
Conversation
anxkhn
force-pushed
the
fix/kube-exec-all-output-race
branch
from
August 12, 2026 05:24
d5ec205 to
6f82851
Compare
execAll spawned one goroutine per pod/container that appended each command's stdout to a shared `output` string via `output = output + "\n" + stdout` without synchronization. That unsynchronized read-modify-write races between the goroutines and can silently drop concatenations when appends interleave. The write also happened after the send on errChan, so draining the error channel established no happens-before with it; the main goroutine could read `output` in parseLogAndCreateOutput while goroutines were still writing, racing the read against the writes. Give each goroutine a dedicated slot in a preallocated slice keyed by index, so writes never overlap, and join the slots into the output string only after every goroutine has reported on errChan. The channel receives now order the slot writes before the join, removing both the write/write and read/write races. Output ordering is now the deterministic pod/container iteration order. Extract the concurrent core into execAllWith with an injectable exec seam and add a race regression test that fans out many pods/containers with a fake exec; it fails under `go test -race` on the old code and passes after the fix. Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
anxkhn
force-pushed
the
fix/kube-exec-all-output-race
branch
from
August 13, 2026 07:43
6f82851 to
71df615
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Overview
KubeExecAllruns a command across every pod/container in parallel.execAllspawned one goroutine per pod/container, and each goroutine appended its command
stdout to a single shared string:
That is an unsynchronized read-modify-write of
outputshared across all Ngoroutines, which is a data race on the string and can silently drop
concatenations when two appends interleave (lost update). It also writes
outputafter the send onerrChan, so draining the error channel in the maingoroutine establishes no happens-before with that write: that same goroutine can
read
outputinparseLogAndCreateOutputwhile goroutines are still writing,racing the read against the writes. Any
KubeExecAllphase that targets morethan one pod or container (its whole purpose) can therefore drop or corrupt exec
output.
The fix keeps the change confined to
execAll:outputsslice keyedby index. Distinct indices never overlap, so the concurrent writes are
race-free by construction (no mutex needed).
errChan. The main goroutinereceives all N errors before joining any slot, so every per-container write
happens-before the read in
parseLogAndCreateOutput. Both the write/write andthe read/write races are removed.
previously nondeterministic goroutine-completion order). This does not change
semantics:
parseLogAndCreateOutputparses order-independent phase-outputkey/value lines into a map.
To make the concurrent path testable without a live cluster, the core is
extracted into
execAllWith(..., exec execFunc)with an injectable exec seamthat defaults to
KubeExecAndLog.execAll's signature and its only caller areunchanged.
A release note is included under
releasenotes/notes/.Issues
Test Plan
Added
TestExecAllCollectsEveryContainerOutput(newKubeExecAllOutputTestgocheck suite) in
pkg/function/kube_exec_all_test.go. It fans out 4 pods x 3containers through a fake exec that emits a distinct phase-output line per
pod/container, then asserts the parsed output map contains all 12 keys, so a lost
or corrupted concatenation shows up as a missing key. Run under the race
detector:
The test fails under
-raceon the pre-fix body (the race detector fires on theshared
outputstring) and passes after the fix.Also verified locally:
go build -race ./pkg/function/...,go vet ./pkg/function/, and golangci-lint with the repo config onpkg/function(0issues). The existing live-cluster suites (
TestKubeExecAllDeployment,TestKubeExecAllStatefulSet) still compile; they need a kubeconfig and were notrun locally.