monitor: checkpoint must not advance when search returns nil due to unavailable index or shard

[juliaElastic]

Summary

On a fresh serverless (stateless ES) deployment, the first action written to .fleet-actions is silently dropped when a second action is submitted before the catch-up path rescues it.

Root cause

monitor.go:search() swallows index_not_found_exception and returns nil, nil — treating a failed search identically to a genuinely empty result. Zero hits falls through to m.storeCheckpoint(newCheckpoint), advancing the checkpoint past _seq_no: 0 even though nothing was dispatched.

Why it happens on stateless ES: When the first action is written it auto-creates .fleet-actions-7. The stateless ES search shard for a newly created index may not be STARTED yet — FleetSearch with wait_for_checkpoints=[0] fails. The monitor treats this as "no documents", advances the checkpoint, and the action is never dispatched.

Why catch-up also fails: FindAgentActions(seqno, checkpoint) queries _seq_no > seqno. If a second action (_seq_no: 1) is submitted and ACKd before the catch-up runs, the agent's seqno becomes 1. All subsequent catch-up queries use _seq_no > 1, permanently excluding the action at _seq_no: 0.

Evidence

Reproduced on two fresh serverless projects.

Endpoint response action (running-processes):

• Action c06b42b0 → _seq_no: 0, nbAgentsAck: 0, permanently stuck
• Action d401b044 → _seq_no: 1, nbAgentsAck: 1, delivered in ~2s
• Only 2 documents in .fleet-actions-7
• Endpoint log confirms c06b42b0 never received; d401b044 received within 2s of creation

REQUEST_DIAGNOSTICS (no Endpoint):

• Action 773d3d6f → _seq_no: 0, nbAgentsAck: 0, permanently stuck
• Action 43b26e52 → _seq_no: 1, nbAgentsAck: 1, delivered in 7s
• Only 2 documents in .fleet-actions-7

Single-action test: a REQUEST_DIAGNOSTICS at _seq_no: 0 with no retry was rescued by catch-up after ~3.5 minutes — confirming the permanent hang requires a second action to be ACKd first.

Related ES bug: elastic/elasticsearch#130555 ("Fleet search using wait_for_checkpoints can fail if the node executing the search is recovering").

Fix

monitor.go:search() must distinguish between a search that genuinely returned zero hits and one that failed because the index or search shard was unavailable. The checkpoint should only advance when the search genuinely completed — not when it returned nil, nil due to index_not_found or shard unavailability.

Kibana issue

elastic/kibana#281980

[juliaElastic]

@elastic/elastic-agent-control-plane hey, could you check how feasible is to fix this bug on fleet-server side? There is a suggested enhancement in kibana, but we might not have to do that if the fleet-server fix can be done first.

[cmacknz]

@ycombinator can you take a look at this? If the response from ES lets us distinguish the two cases where I imagine this would be straight forward on the FS side. I would also expect this also affects or could affect the scale tests.