[8.19](backport #7571) fix: gracefully handle missing secrets in policy monitor

[mergify]

What is the problem this PR solves?

Fleet Server crash-loops when any secret referenced in an agent policy's secret_references cannot be found in the Fleet secrets store. A single missing secret — from a deleted or orphaned integration credential — causes the entire policy monitor to abort with:

Error: failed to get secret values: unexpected status 404 from fleet secret read: ...
Fleet Server failed

Fleet Server then restarts, re-reads the same stale policy document from the policy monitor (the checkpoint hasn't advanced), encounters the same 404, and crashes again — indefinitely. Every agent on that fleet-server instance goes offline for the duration.

This is a disproportionate blast radius: one deleted integration credential (unrelated to fleet-server's own connectivity) can take the entire Fleet Server offline.

The root causes that produce missing secrets are tracked in Kibana (elastic/kibana#282911 and elastic/kibana#282912) and in fleet-server (#7533, #7534). This PR is a defense-in-depth fix: even after those root causes are resolved, fleet-server should not treat a single missing package-policy credential as a reason to crash-loop and take all agents offline.

How does this PR solve the problem?

Two changes:

1. bulk/secret.go — ExtendedAPI.Read: Returns a typed ErrSecretNotFound sentinel when the Fleet secrets API responds with 404. This lets callers distinguish "this secret document doesn't exist" from network/auth failures (403, 500, etc.), which should still be fatal.

2. bulk/engine.go — ReadSecrets: When ReadSecret returns ErrSecretNotFound, logs a warning with the secret ID and skips it (continues the loop) rather than aborting. The returned map contains only the secrets that were successfully fetched.

With this change, a missing secret causes the policy to load without that secret value — the raw $co.elastic.secret{id} placeholder remains unsubstituted in the affected integration's config. That integration's credentials won't resolve (it will fail to authenticate), but fleet-server itself stays up and continues serving all other agents.

Callers that need hard failure for missing secrets (e.g. output API key resolution, where an empty key would silently break agent→ES connectivity) can check whether the expected ID is present in the returned map — as the second commit in #7416 already does.

How to test this PR locally

1. Start a local stack with fleet-server
2. Create an integration that uses secrets (e.g. any OAuth-based integration)
3. Directly delete the secret document from Elasticsearch: DELETE /_fleet/secret/<id>
4. Restart fleet-server and observe that it:
   • Logs WARN secret not found; policy will load without it secret_id=<id>
   • Continues to start successfully
   • Serves other agents normally
5. Verify (via agent diagnostics) that the affected integration's credential field contains the raw $co.elastic.secret{id} placeholder rather than the secret value

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Each instance evaluates ReadSecrets independently per policy load; skipping is per-request with no shared state.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected. (The change reduces work on failure — skipping a secret is cheaper than returning an error — so scale impact is neutral or positive.)
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (No new load introduced; 404s are now fast-path skips.)

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Add graceful handling for missing secrets in policy monitor #7536
• Root cause (how secrets become dangling in the first place):
  • [Fleet] Orphaned .fleet-policies documents survive agent policy deletion and can crash-loop fleet-server kibana#282911 — agentPolicyService.delete() can leave orphaned .fleet-policies documents when the cleanup call after package-policy deletion fails silently
  • [Fleet] Secret cleanup only checks package policy saved objects, can delete a secret still referenced by a compiled policy kibana#282912 — deleteSecretsIfNotReferenced only checks live package-policy saved objects, not .fleet-policies, so secrets are deleted even when an orphaned policy document still references them
• Mitigation for the agent-update-failure source of dangling secrets: Retain output secret when agent update fails #7533, Reconcile orphaned output secrets after agent update failures #7534
• Context: the >= 400 check in ExtendedAPI.Read was introduced in Store output API key secrets in .fleet-secrets instead of .fleet-agents #7416. That PR's own second commit described the original tolerant design — ReadSecrets was not expected to fail on a missing secret; hard failure for the output API key was added at the caller level. This PR restores that layered design by making 404 non-fatal in ReadSecrets while preserving fatal behavior for other 4xx/5xx responses.

  ---

  This is an automatic backport of pull request fix: gracefully handle missing secrets in policy monitor #7571 done by Mergify.

[mergify]

Cherry-pick of 5ece0ba has failed:

On branch mergify/bp/8.19/pr-7571
Your branch is up to date with 'origin/8.19'.

You are currently cherry-picking commit 5ece0ba.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1786053515-graceful-missing-secrets.yaml
	modified:   internal/pkg/bulk/engine.go
	new file:   internal/pkg/bulk/secret_test.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/secret.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[ebeahan]

@ycombinator checking with you if we need to backport this or not. If not, let's close the PR.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏

[github-actions]

TL;DR

This Buildkite run has one confirmed infra failure (Run unit tests), while the other failed jobs only provide truncated tail logs, so the exact failing test/package cannot be identified from the provided artifacts.

Remediation

• Re-run Run unit tests; it failed before tests started because bin/gvm returned an HTML 504 Gateway Time-out, then .buildkite/scripts/common.sh failed with go: command not found.
• Re-run the other failed steps with full raw test output (from command start) and capture the first --- FAIL: / FAIL\t stanza; the current tail-only logs only show the wrapper go test ... exit code 1 line.
• Once the first failing package/test is visible, map it to code and check for an existing flaky-test issue. I checked current open flaky-test issues and found no direct match based on the available log signatures.

Investigation details

Root Cause

1. Infrastructure (Run unit tests, exit 127): environment setup failed before Go tests began.
2. Unresolved from available artifacts (Run fips140=on, Run fips140=only, Run unit tests: MacOS 13, Run integration tests, all exit 1): logs end with harness wrapper failure and do not include first failing test/package output.

I also checked the current PR head code path that was previously suspected:

• internal/pkg/bulk/secret.go already contains non-404 error handling at L50-L53 (if res.StatusCode >= 400 { ... return error ... }).
• internal/pkg/bulk/secret_test.go still enforces that behavior at L50-L55 in TestExtendedAPIRead_ServerError_ReturnsGenericError.

So, from current code + available logs, there is no direct evidence that missing non-404 handling is the active failure cause in this run.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/17021
• Job/step (confirmed infra): Run unit tests
• Key log excerpt (/tmp/gh-aw/buildkite-logs/fleet-server-smartbear-testexecute-run-unit-tests.txt):
  • L133: .../bin/gvm: line 1: syntax error near unexpected token '<'
  • L135: .../bin/gvm: line 1: '<html><body><h1>504 Gateway Time-out</h1>'
  • L137: .buildkite/scripts/common.sh: line 49: go: command not found
  • L142: user command error: exit status 127
• Other failed job logs terminate at wrapper messages like:
  • Error: running "go test ..." failed with exit code 1
  • without a preceding package/test-level failure stanza.

Verification

• Local reproduction in this detective run: not completed (dependency downloads did not finish in time in the ephemeral runner).
• Analysis is based on provided Buildkite artifacts plus current PR branch source inspection.

Follow-up

• If reruns continue to upload tail-only logs, include the full stdout artifact (or first-failure excerpt) so the failing package/test can be attributed confidently.

---

What is this? | From workflow: Observability Agentic Workflow — Status

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.