[8.19](backport #7815) test: E2E regression test for fleet-server parsed policy race condition (#7794) by mergify[bot] · Pull Request #7839 · elastic/fleet-server

mergify · 2026-09-17T11:37:01Z

Warning

Merge after #7840 ([8.19] backport of #7794). This PR is a regression test for that fix — without the fix in place the test will fail or produce misleading results.

Summary

Regression test for #7794 (the ParsedPolicy shared-state race when secret_references is non-empty), moved from elastic/elastic-agent#16582 into fleet-server's own E2E suite where it belongs.

Adds TestRemoteESOutputWithSecrets to AgentContainerSuite in testing/e2e/remote_es_output_test.go
- Starts a fleet-server container (current commit's build via AGENT_E2E_IMAGE)
- Creates a remote_elasticsearch Fleet output whose service_token is stored as a Fleet secret (using the "secrets" wrapper), which populates secret_references in the generated policy — the trigger condition for the race
- Creates an agent policy using that output and enrolls a second agent container
- Verifies the agent reaches the online state (fleet-server did not crash)
- Verifies fleet-server's status endpoint is still healthy
- Verifies the agent's applied policy revision matches Fleet's latest revision
Adds four scaffold helpers to testing/e2e/scaffold/scaffold.go: CreateServiceToken, CreateFleetOutput, CreateAgentPolicy, GetAgentPolicyRevision

Test plan

mage test:e2e runs TestRemoteESOutputWithSecrets and it passes
Reverting the fix from refactor: clone ParsedPolicy in processPolicy to prevent shared-state races #7794 (ParsedPolicy.Clone() at dispatch site) causes the test to fail (fleet-server panics with index-out-of-range, agent never reaches online)

🤖 Generated with Claude Code

This is an automatic backport of pull request #7815 done by Mergify.

mergify · 2026-09-17T11:37:03Z

Cherry-pick of 552fb78 has failed:

On branch mergify/bp/8.19/pr-7815
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 552fb78.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   testing/e2e/remote_es_output_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   testing/e2e/scaffold/scaffold.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

github-actions · 2026-09-17T17:29:07Z

TL;DR

Buildkite E2E Test fails at compile time because TestRemoteESOutputWithSecrets calls suite.GetAgent(...), but that method is not available on this backport branch’s AgentContainerSuite/embedded scaffold. Replace that call with an existing helper (or backport GetAgent) and rerun the E2E step.

Remediation

In testing/e2e/remote_es_output_test.go:237, change suite.GetAgent(ctx, firstEnrolledAgentID) to a helper present on 8.19 (for example suite.WaitForAgentDoc(ctx, firstEnrolledAgentID)) and keep the revision comparison logic unchanged.
Alternatively, backport GetAgent(ctx, id) into testing/e2e/scaffold/scaffold.go so the new test compiles as written.
Re-run .buildkite/scripts/e2e_test.sh (or mage test:e2e) to confirm compile and test execution pass.

Investigation details

Root Cause

This is a code/backport mismatch, not infrastructure flakiness. The newly added test in this PR references suite.GetAgent at testing/e2e/remote_es_output_test.go:237, and the E2E package build fails because that method is missing for *AgentContainerSuite in the backported code.

Evidence

Build: https://buildkite.com/elastic/fleet-server/builds/16962
Job/step: E2E Test (.buildkite/scripts/e2e_test.sh)
Key log excerpt:
- e2e/remote_es_output_test.go:237:21: suite.GetAgent undefined (type *AgentContainerSuite has no field or method GetAgent)
- FAIL github.com/elastic/fleet-server/testing/e2e [build failed]
PR change context:
- testing/e2e/remote_es_output_test.go introduces the suite.GetAgent(...) call.
- testing/e2e/scaffold/scaffold.go additions in this PR include GetAgentPolicyRevision and WaitForAgentDocsInIndex, but not GetAgent.

Verification

Not run in this detective workflow (read-only); conclusions are from Buildkite logs plus PR diff inspection.

Follow-up

Checked for open flaky-test issues matching TestRemoteESOutputWithSecrets; none found.

What is this? | From workflow: Observability Agentic Workflow — Status

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

…on (#7794) (#7815) * test: E2E regression test for fleet-server parsed policy race condition fix Adds TestRemoteESOutputWithSecrets to AgentContainerSuite in testing/e2e/remote_es_output_test.go. The test verifies that fleet-server correctly serves a policy with a remote_elasticsearch output whose service_token is stored as a Fleet secret (non-empty secret_references) without crashing, and that an enrolled agent successfully applies the policy. This is the trigger condition for the race fixed in #7794. Also adds four scaffold helpers to testing/e2e/scaffold/scaffold.go: CreateServiceToken, CreateFleetOutput, CreateAgentPolicy, and GetAgentPolicyRevision. Closes elastic/elastic-agent#16582 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: address Copilot review feedback on TestRemoteESOutputWithSecrets - Add CreateEnrollmentAPIKey scaffold helper and use it in the test instead of GetEnrollmentTokenForPolicyID; CreateAgentPolicy does not guarantee an auto-generated key is immediately available, so creating one explicitly is more reliable - Enroll two agents concurrently under the same policy so fleet-server dispatches the policy to multiple subscribers simultaneously — the scenario that exercises the shared-ParsedPolicy race fixed in #7794 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: remove GOCOVERDIR from enrolled agent containers fleet-server never starts in FLEET_ENROLL mode, so the env var and /cover mount have no effect on the agent containers. Only the fleet-server container (FLEET_SERVER_ENABLE=1) needs GOCOVERDIR. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: verify secret_references is populated before enrolling agents Add WaitForPolicySecretReferences scaffold helper that polls .fleet-policies until the latest revision for a given policy ID has a non-empty secret_references array. Call it in TestRemoteESOutputWithSecrets after CreateAgentPolicy so the test fails fast — with a clear message — if the "secrets" wrapper on the Fleet output did not produce a secret reference. Without this check, Kibana silently dropping the wrapper would let the test pass against an ordinary (non-secret-referencing) policy that never touches the #7794 code path. Serial agent container startup is intentional: the bug manifests even sequentially because agent 1 receives the already-mutated cached ParsedPolicy that agent 0's processPolicy corrupted. >= in the revision check is intentional: the policy revision is monotonic and can advance between the two reads, so == would be flaky. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: verify agent data lands in remote ES over the output connection Add Outputs field to ESAgentDoc and WaitForAgentDocsInIndex scaffold helper. In TestRemoteESOutputWithSecrets, set monitoring_output_id to the remote ES output and monitoring_enabled to ["logs", "metrics"] so the agent writes its self-monitoring metrics there. Assert those documents appear in metrics-elastic_agent.* in the remote ES, proving the full path: secret reference resolved → API key created in remote ES → agent authenticated → data indexed. CreateAgentPolicy now accepts variadic extra body fields so callers can set optional Fleet API fields without changing the base signature. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: start agent containers concurrently to reliably trigger the race Replace the sequential for-loop with a sync.WaitGroup that launches all five agent containers in parallel goroutines. This maximises the window during which multiple agents are receiving and applying the same policy simultaneously in fleet-server, reliably triggering the shared- ParsedPolicy mutation when the fix is absent. Verified: 10/10 test runs fail (agent panic: index out of range [-1]) against code with #7794 and #7740 reverted; 0/1 fail with the fix in place. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: extend timeouts to survive 5-minute long-poll retry cycle Agents that miss the initial policy dispatch (due to the Subscribe→kickDeploy cancellation race) must wait for their 5-minute long-poll to expire before retrying. The previous 3-minute Eventually and 5-minute outer context were too short for this retry cycle. Bump the outer context to 12 minutes and the Eventually to 9 minutes to give agents time to exhaust their long-poll and succeed on the second attempt. Also drop the monitoring_output_id / WaitForAgentDocsInIndex assertions: they require inter-container network access that is not set up in this test environment, so they always failed. The core regression (no panic, all agents reach online) is covered by the remaining assertions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: include name in CreateEnrollmentAPIKey request body The Fleet enrollment_api_keys API accepts a name field alongside policy_id. Including it makes the request unambiguous and avoids any Kibana validation that may require it in stricter versions. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: register agent container cleanup before checking startup errors If Require().NoError aborted on agent N's startup error, the cleanup callbacks for agents N+1 through numAgents-1 were never registered, leaving those containers running and interfering with subsequent E2E runs. Fix: iterate agentContainers first to register a Cleanup for every non-nil container, then iterate agentErrs to fail on any error. This guarantees all created containers are terminated regardless of which agent failed to start. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: use sync.WaitGroup.Go instead of manual Add/goroutine/Done Go 1.25 added (*sync.WaitGroup).Go, which combines Add(1), go func(), and defer Done() into a single call. The project is on Go 1.26, so we can use it here to simplify the concurrent agent container launch loop. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: verify agent monitoring data lands in remote ES output Route both the data and monitoring outputs to the remote_elasticsearch output. After all agents come online, wait for metrics-elastic_agent.* documents attributed to an enrolled agent to appear in Elasticsearch. This end-to-end check confirms the full path: secret_references resolved → fleet-server creates API key in remote ES → agent receives key and authenticates → self-monitoring data indexed. Extend WaitForAgentDocsInIndex timeout from 2 to 4 minutes so two full 60-second metric periods are covered before giving up. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (cherry picked from commit 552fb78) # Conflicts: # testing/e2e/scaffold/scaffold.go

Mergify left unresolved conflict markers when backporting #7815 to 8.19. GetPolicy did not exist on 8.19, so the cherry-pick conflicted. Resolved by keeping the incoming version (GetPolicy + new helpers) and removing the <<<<<<< / ======= / >>>>>>> markers. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…e fix The E2E test added in #7815 calls suite.GetAgent() which was added to main after the 8.19 branch was cut. Add the missing ESAgentDoc type and GetAgent method to the 8.19 scaffold so the backport compiles. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

mergify Bot added backport conflicts There is a conflict in the backported pull request labels Sep 17, 2026

mergify Bot requested a review from a team as a code owner September 17, 2026 11:37

mergify Bot added the backport label Sep 17, 2026

mergify Bot requested review from macdewee and samuelvl and removed request for a team September 17, 2026 11:37

mergify Bot added the conflicts There is a conflict in the backported pull request label Sep 17, 2026

mergify Bot assigned ycombinator Sep 17, 2026

mergify Bot mentioned this pull request Sep 17, 2026

test: E2E regression test for fleet-server parsed policy race condition (#7794) #7815

Merged

2 tasks

This comment has been minimized.

Sign in to view

ycombinator previously approved these changes Sep 17, 2026

View reviewed changes

ycombinator and others added 2 commits September 17, 2026 13:02

ycombinator enabled auto-merge (squash) September 17, 2026 20:02

ycombinator force-pushed the mergify/bp/8.19/pr-7815 branch from c06d1d5 to 181d2e6 Compare September 17, 2026 20:02

ycombinator dismissed their stale review via 453e734 September 18, 2026 00:40

ycombinator approved these changes Sep 18, 2026

View reviewed changes

Navigation Menu

[8.19](backport #7815) test: E2E regression test for fleet-server parsed policy race condition (#7794) - #7839

[8.19](backport #7815) test: E2E regression test for fleet-server parsed policy race condition (#7794)#7839
mergify[bot] wants to merge 3 commits into
8.19from
mergify/bp/8.19/pr-7815

mergify Bot commented Sep 17, 2026 •

edited by ycombinator

Loading

Uh oh!

mergify Bot commented Sep 17, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 17, 2026

Root Cause

Evidence

Verification

Follow-up

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

mergify Bot commented Sep 17, 2026 • edited by ycombinator Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

mergify Bot commented Sep 17, 2026

Uh oh!

This comment has been minimized.

github-actions Bot commented Sep 17, 2026

TL;DR

Remediation

Root Cause

Evidence

Verification

Follow-up

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mergify Bot commented Sep 17, 2026 •

edited by ycombinator

Loading