Fix syscheck queue/restart races and honor FIM frequency under realtime. - #2295
Conversation
Bind analysisd MQ early, wake recv on shutdown with PID wait before restart, and use adaptive syscheck idle waits so short <frequency> is not capped by SYSCHECK_WAIT.
There was a problem hiding this comment.
Pull request overview
This PR addresses startup/shutdown race conditions around the analysisd message queue and improves syscheck daemon idling so configured FIM/rootcheck frequencies are honored even under realtime mode.
Changes:
- Add adaptive idle waiting in syscheckd so short
<frequency>values aren’t effectively capped bySYSCHECK_WAIT. - Make MQ connection readiness based on actual AF_UNIX connect success (not just socket-path existence) and bind analysisd’s queue earlier during startup.
- Improve restart/shutdown behavior by waiting for PIDs to exit and waking analysisd’s receive thread during shutdown.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| src/syscheckd/run_check.c | Computes an adaptive idle wait so daemon-loop sleep/select respects configured scan cadences. |
| src/shared/mq_op.c | Changes StartMQ(WRITE) to wait for successful connect and updates SendMSG retry/close behavior. |
| src/os_net/os_net.c | Fixes unlink error handling by checking errno correctly. |
| src/init/ossec-server.sh | Adds PID-exit waiting and queue-readiness gating before starting producers. |
| src/init/ossec-local.sh | Same PID-exit waiting and analysis queue readiness gating for local installs. |
| src/init/ossec-client.sh | Adds PID-exit waiting and agent queue readiness gating before starting local producers. |
| src/analysisd/pipeline.c | Adds poll-based recv loop and closes/wakes input queue during shutdown. |
| src/analysisd/analysisd.c | Binds analysisd’s queue earlier (post-chroot) so producers don’t race stale sockets. |
Suppressed comments (3)
src/init/ossec-server.sh:350
- This script uses a /bin/sh shebang, but
localis not POSIX and will fail under common /bin/sh implementations. Removelocaldeclarations in wait_for_analysis_queue().
local elapsed=0
local max=60
local qpath="${DIR}/queue/ossec/queue"
src/init/ossec-local.sh:335
- This script uses a /bin/sh shebang, but
localis not POSIX and will fail under common /bin/sh implementations. Removelocaldeclarations in wait_for_analysis_queue().
local elapsed=0
local max=60
local qpath="${DIR}/queue/ossec/queue"
src/init/ossec-client.sh:257
- This script uses a /bin/sh shebang, but
localis not POSIX and will fail under common /bin/sh implementations. Removelocaldeclarations in wait_for_agent_queue().
local elapsed=0
local max=60
local qpath="${DIR}/queue/ossec/queue"
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| while [ ${elapsed} -lt ${max} ]; do | ||
| if [ -S "${qpath}" ] || [ -e "${qpath}" ]; then | ||
| python3 - "${qpath}" <<'PY' 2>/dev/null |
Drop non-POSIX local from init helpers, probe queues with python3/python fail-fast, preserve connect errno in StartMQ, and serialize analysisd input-queue close against double-close races.
Bind the analysis queue after decoder/rule load so control-script connect waits do not start producers before recv can drain. Reconnect logcollector on ignored SendMSG failures after SendMSG closes the fd.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Suppressed comments (5)
src/init/ossec-server.sh:362
- Hard-requiring python here makes
ossec-control startfail on minimal systems even though the daemons themselves are C binaries. Since StartMQ(WRITE) already has its own connect-wait logic, consider degrading gracefully when python isn’t available (warn and continue) instead of aborting startup.
else
echo "ERROR: python3 or python required to probe analysis queue at ${qpath}"
return 1
fi
src/shared/mq_op.c:117
- This updated comment documents that SendMSG() may close the queue fd on failure, but a number of existing call sites don’t consistently handle a -1 return (some only log; others retry once and ignore the second result). With a closed fd, later SendMSG() calls can hit EBADF or (worse) send on a reused/unrelated fd number. If this close-on-error behavior is required, the PR should also update all SendMSG() callers to treat -1 as “fd is dead” and reconnect before any reuse (or change the API to take an
int *queueso SendMSG can invalidate the caller’s fd).
/* Five send attempts; after the first failure, sleep 1/3/5/10s
* between retries (total sleep budget 19s). OS_SOCKTERR is fatal
* only on the initial attempt; close(queue) so callers that StartMQ
* again do not leak the dead fd. Callers must not reuse queue after -1
* without StartMQ (pass-by-value cannot clear the caller's variable).
src/init/ossec-server.sh:312
- Before sending SIGKILL, re-validate the PID still belongs to an OSSEC process. With the new 45s grace period, a PID can exit and be reused, and
kill -9could hit an unrelated process.
This issue also appears on line 359 of the same file.
if [ $? = 0 ]; then
echo "Process ${pid} did not exit; sending SIGKILL .."
kill -9 ${pid} >/dev/null 2>&1
fi
src/init/ossec-local.sh:347
- Hard-requiring python here makes startup fail on systems without python even if the daemons can run. Since producers can already rely on StartMQ(WRITE) retry/connect behavior, consider warning and continuing when python isn’t available rather than aborting.
else
echo "ERROR: python3 or python required to probe analysis queue at ${qpath}"
return 1
fi
src/init/ossec-client.sh:269
- This introduces a hard dependency on python for agent startup. On minimal installations without python,
ossec-control startwill now fail even though the agent daemons are native binaries. Consider warning and continuing when python isn’t present (letting StartMQ retries handle readiness) instead of returning failure here.
else
echo "ERROR: python3 or python required to probe agent queue at ${qpath}"
return 1
fi
| static volatile sig_atomic_t analysisd_shutting_down = 0; | ||
| static volatile sig_atomic_t pipeline_m_queue = -1; |
sig_atomic_t only covers signal handlers; concurrent pthread access to analysisd_shutting_down and pipeline_m_queue needs _Atomic stores/loads.
Bind analysisd MQ early, wake recv on shutdown with PID wait before restart, and use adaptive syscheck idle waits so short is not capped by SYSCHECK_WAIT.