feat(portal): introduce portal-only execution mode and resolve pipeli… - #28
Merged
Conversation
…ne bugs - Add portal-only environment configs (docker-compose.portal.yml, run-portal.bat). - Add zeek-conn-* ES index wiping to confirm_reset to clear connection logs. - Conditionally skip write_zeek_log in /simulate when running in portal-only mode. - Retain event.event_id to enable frontend-side deduplication. - Restore ALLOW events to frontend pipeline logs and add green packet animations. - Abbreviate IPv6 addresses in Alert Cards and Event Logs using formatIpShort. - Remove StrictMode from main.jsx to prevent duplicate WS connections.
- Add PORTAL_ONLY_MODE environment variable configuration - Guard Zeek log writing in auth routes to skip during portal-only mode - Maintain WebSocket connection behavior in simulate routes based on portal flag - Refactor Kafka client connectivity checks to gracefully handle portal-only environments
simulate.py: stream test_events during live-replay, 0.5s delay, heartbeat to stop zombie loops; threat_engine.py: real latencies for Kafka + Cred Rotation stages; Header.jsx + index.css: MODE Live Replay/Portal toggle; mode_switcher.py: host agent for compose down/up
…itch - mode_switcher.py: catch ConnectionAborted/Reset/BrokenPipe in _json so browser-closed /status polls do not dump WinError 10053 tracebacks - docker-compose.portal.yml: add restart: unless-stopped to ngrok so it self-recovers from the ERR_NGROK_334 endpoint race on fast down->up switches
…+ seek-latest - simulate.py: single shared producer (was one per WS connection, so refresh compounded the event rate) - kafka_client.py: consumer seeks to latest on connect and throttles the UI feed, not the consumer (no more stale-backlog draining) - docker-compose.yml: gate zeek/filebeat behind the live-replay profile (raw Zeek events have no user identity) - mode_switcher.py: plain 'up -d' switch, detect mode via hpe-es-to-kafka, live switch no longer starts the Zeek replay - add generate_conn_log_from_test_events.py; document the mode switcher in README
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Key Features & Changes
1. Portal-Only Setup & Launchers (Main Intention)
docker-compose.portal.yml: Defines the services stack running in isolated portal mode with thePORTAL_ONLY_MODE=trueenvironment flag enabled for the backend service.run-portal.bat: Adds a launcher script to easily initialize the database models, start the container services, and open the dashboard.2. Pipeline Reset & Clean Slate
zeek-conn-*index pattern deletion to the reset confirm endpoint inadmin.pyto ensure past connection logs are fully removed.failed_attempts,last_login,last_login_region, etc.) for all users to default/NULL values.dataset/zeek-live/conn.logto empty.3. Duplication & Ingestion Flow Fixes
/simulatewhen running inPORTAL_ONLY_MODE(which uses a case-insensitive check). This avoids double-processing simulated events.event.event_idinside the backend consumer/threat engine so the client'sseenEventIdsRefcan successfully filter out pipeline-delivered duplicates.StrictModefrommain.jsxto prevent the browser from mounting components twice and initializing redundant WebSocket connections.4. Frontend Visualization & UI Polish
ALLOWlogin events to the Pipeline Flow log, backed by a new green packet animation path.formatIpShort()inAdminConsoleandPipelineFlowto cleanly abbreviate long IPv6 addresses.word-breakproperties to detail panel IP values to prevent visual overflow.Verification Checklist
./run-portal.bat.USR-0005(Admin) from the public gateway and confirm they are not immediately flagged as a threat for impossible travel.