Skip to content

fix(function): avoid data race on shared output in KubeExecAll - #4121

Open
anxkhn wants to merge 1 commit into
kanisterio:masterfrom
anxkhn:fix/kube-exec-all-output-race
Open

anxkhn wants to merge 1 commit into
kanisterio:masterfrom
anxkhn:fix/kube-exec-all-output-race

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 5, 2026

Copy link
Copy Markdown

Change Overview

KubeExecAll runs a command across every pod/container in parallel. execAll
spawned one goroutine per pod/container, and each goroutine appended its command
stdout to a single shared string:

output := ""
...
go func(p string, c string) {
    stdout, _, err := KubeExecAndLog(ctx, cli, namespace, p, c, cmd, nil)
    errChan <- err
    output = output + "\n" + stdout   // shared, unsynchronized
}(p, c)

That is an unsynchronized read-modify-write of output shared across all N
goroutines, which is a data race on the string and can silently drop
concatenations when two appends interleave (lost update). It also writes
output after the send on errChan, so draining the error channel in the main
goroutine establishes no happens-before with that write: that same goroutine can
read output in parseLogAndCreateOutput while goroutines are still writing,
racing the read against the writes. Any KubeExecAll phase that targets more
than one pod or container (its whole purpose) can therefore drop or corrupt exec
output.

The fix keeps the change confined to execAll:

  • Each goroutine writes into its own slot of a preallocated outputs slice keyed
    by index. Distinct indices never overlap, so the concurrent writes are
    race-free by construction (no mutex needed).
  • The slot write now happens before the send on errChan. The main goroutine
    receives all N errors before joining any slot, so every per-container write
    happens-before the read in parseLogAndCreateOutput. Both the write/write and
    the read/write races are removed.
  • Output is joined in the deterministic pod/container iteration order (it was
    previously nondeterministic goroutine-completion order). This does not change
    semantics: parseLogAndCreateOutput parses order-independent phase-output
    key/value lines into a map.

To make the concurrent path testable without a live cluster, the core is
extracted into execAllWith(..., exec execFunc) with an injectable exec seam
that defaults to KubeExecAndLog. execAll's signature and its only caller are
unchanged.

A release note is included under releasenotes/notes/.

  • 🚧 Work in Progress
  • 🌈 Refactoring (no functional changes, no api changes)
  • 🐹 Trivial/Minor
  • 🐛 Bugfix
  • 🌻 Feature
  • 🗺️ Documentation
  • 🤖 Test
  • 🏗️ Build

Issues

  • N/A

Test Plan

  • 💪 Manual
  • ⚡ Unit test
  • 💚 E2E

Added TestExecAllCollectsEveryContainerOutput (new KubeExecAllOutputTest
gocheck suite) in pkg/function/kube_exec_all_test.go. It fans out 4 pods x 3
containers through a fake exec that emits a distinct phase-output line per
pod/container, then asserts the parsed output map contains all 12 keys, so a lost
or corrupted concatenation shows up as a missing key. Run under the race
detector:

go test -race -run 'Test/TestExecAllCollectsEveryContainerOutput' \
  ./pkg/function/ -check.f 'KubeExecAllOutputTest'

The test fails under -race on the pre-fix body (the race detector fires on the
shared output string) and passes after the fix.

Also verified locally: go build -race ./pkg/function/..., go vet ./pkg/function/, and golangci-lint with the repo config on pkg/function (0
issues). The existing live-cluster suites (TestKubeExecAllDeployment,
TestKubeExecAllStatefulSet) still compile; they need a kubeconfig and were not
run locally.

@anxkhn
anxkhn force-pushed the fix/kube-exec-all-output-race branch from d5ec205 to 6f82851 Compare August 12, 2026 05:24
execAll spawned one goroutine per pod/container that appended each command's
stdout to a shared `output` string via `output = output + "\n" + stdout`
without synchronization. That unsynchronized read-modify-write races between
the goroutines and can silently drop concatenations when appends interleave.

The write also happened after the send on errChan, so draining the error
channel established no happens-before with it; the main goroutine could read
`output` in parseLogAndCreateOutput while goroutines were still writing,
racing the read against the writes.

Give each goroutine a dedicated slot in a preallocated slice keyed by index,
so writes never overlap, and join the slots into the output string only after
every goroutine has reported on errChan. The channel receives now order the
slot writes before the join, removing both the write/write and read/write
races. Output ordering is now the deterministic pod/container iteration order.

Extract the concurrent core into execAllWith with an injectable exec seam and
add a race regression test that fans out many pods/containers with a fake exec;
it fails under `go test -race` on the old code and passes after the fix.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@anxkhn
anxkhn force-pushed the fix/kube-exec-all-output-race branch from 6f82851 to 71df615 Compare August 13, 2026 07:43

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant