Live-replay: real users + varied scores via Simulate feed; mode-switcher & consumer fixes - #30
Merged
cheeseburden merged 8 commits intoJun 20, 2026
Conversation
…ne bugs - Add portal-only environment configs (docker-compose.portal.yml, run-portal.bat). - Add zeek-conn-* ES index wiping to confirm_reset to clear connection logs. - Conditionally skip write_zeek_log in /simulate when running in portal-only mode. - Retain event.event_id to enable frontend-side deduplication. - Restore ALLOW events to frontend pipeline logs and add green packet animations. - Abbreviate IPv6 addresses in Alert Cards and Event Logs using formatIpShort. - Remove StrictMode from main.jsx to prevent duplicate WS connections.
- Add PORTAL_ONLY_MODE environment variable configuration - Guard Zeek log writing in auth routes to skip during portal-only mode - Maintain WebSocket connection behavior in simulate routes based on portal flag - Refactor Kafka client connectivity checks to gracefully handle portal-only environments
simulate.py: stream test_events during live-replay, 0.5s delay, heartbeat to stop zombie loops; threat_engine.py: real latencies for Kafka + Cred Rotation stages; Header.jsx + index.css: MODE Live Replay/Portal toggle; mode_switcher.py: host agent for compose down/up
…itch - mode_switcher.py: catch ConnectionAborted/Reset/BrokenPipe in _json so browser-closed /status polls do not dump WinError 10053 tracebacks - docker-compose.portal.yml: add restart: unless-stopped to ngrok so it self-recovers from the ERR_NGROK_334 endpoint race on fast down->up switches
…+ seek-latest - simulate.py: single shared producer (was one per WS connection, so refresh compounded the event rate) - kafka_client.py: consumer seeks to latest on connect and throttles the UI feed, not the consumer (no more stale-backlog draining) - docker-compose.yml: gate zeek/filebeat behind the live-replay profile (raw Zeek events have no user identity) - mode_switcher.py: plain 'up -d' switch, detect mode via hpe-es-to-kafka, live switch no longer starts the Zeek replay - add generate_conn_log_from_test_events.py; document the mode switcher in README
…ulate-feed # Conflicts: # backend/app/routes/admin.py # frontend/src/App.jsx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the live-replay dashboard showing every event as
unknown / connectionwith aconstant score. Live-replay now streams the labelled
test_events.jsonthrough theexisting Simulate path, so the AI engine scores real users (
USR-XXXX) with varied,model-driven anomaly scores. Also hardens the dashboard mode switcher and the Kafka
consumer, and documents both. Merged with the latest
upstream/main(pipeline-resetwork) with conflicts resolved.
Why
The Zeek/pcap stream produces events with random Zeek UIDs and no user identity, so the
uid → event_idenrichment never matched → constant30.3%/unknown. The labelledSimulate stream carries identity with each event, which is what the model needs.
Changes
browser refresh compounded the event rate ~+60/refresh). Producer stops when no
dashboard is connected.
the UI feed instead of the consumer, so a restart/switch shows current events
instead of draining hours of stale backlog.
zeek+filebeatgated behind thelive-replayprofile(off by default); the identity-less raw Zeek stream is now opt-in.
up -dswitch (no rebuild),current_mode()detects viahpe-es-to-kafka, live switch no longer starts the Zeekreplay; silenced
/statuspoll tracebacks; ngrokrestart: unless-stopped.conn.logwithuid = event_idfor the opt-in raw replay path.Testing
docker compose -f docker-compose.yml up -d→ dashboard shows realUSR-XXXXwithvaried scores at ~2 events/sec; rate stays steady across browser refreshes.
upstream's Redis live-counter reset.
Merge notes
Merged
upstream/mainand resolved conflicts inadmin.py(kept both reset paths) andApp.jsx(kept the new animation + upstream'sresetInProgressRefguard).