From 7b8d77e6b5555dbc8e9a50f8631deaa29e5ce413 Mon Sep 17 00:00:00 2001 From: TechHutTV Date: Tue, 11 Aug 2026 16:48:19 -0700 Subject: [PATCH 1/2] Improve CrowdSec setup, recovery, monitoring, and access-log documentation --- .../manage/reverse-proxy/access-logs.mdx | 20 +++- .../maintenance/crowdsec-dashboard.mdx | 107 +++++++++++++++++- src/pages/selfhosted/maintenance/crowdsec.mdx | 94 ++++++++++++++- .../migration/enable-reverse-proxy.mdx | 56 ++++++++- 4 files changed, 259 insertions(+), 18 deletions(-) diff --git a/src/pages/manage/reverse-proxy/access-logs.mdx b/src/pages/manage/reverse-proxy/access-logs.mdx index 508b721e4..c7abf16cc 100644 --- a/src/pages/manage/reverse-proxy/access-logs.mdx +++ b/src/pages/manage/reverse-proxy/access-logs.mdx @@ -9,7 +9,7 @@ NetBird logs every request and connection that passes through your reverse proxy ## Viewing access logs -Access logs are available in the NetBird dashboard under **Activity** > **Proxy Events**. This view displays a table of all HTTP requests and L4 connections that have passed through your reverse proxy services, with filters to narrow down results by time range, status, or other fields. +Access logs are available in the NetBird dashboard under **Reverse Proxy** > **Access Logs**. This view displays a table of all HTTP requests and L4 connections that have passed through your reverse proxy services, with filters to narrow down results by time range, status, or other fields.

Proxy Events table showing reverse proxy access log entries @@ -59,7 +59,7 @@ Denied L4 connections (blocked by access restrictions) are logged immediately wi ### Deny reasons -The following deny reasons can appear for both HTTP and L4 services: +The following deny reasons identify why a connection was rejected. Note that for HTTP services these values are not carried in the entry's `reason` field: see the note below the table. | Reason | Description | |--------|-------------| @@ -73,7 +73,21 @@ The following deny reasons can appear for both HTTP and L4 services: All CrowdSec decision types (ban, captcha, throttle) result in a connection denial in enforce mode. The proxy does not serve captcha challenges or apply rate limiting: the decision type is recorded for informational purposes only. -When CrowdSec is in **observe** mode, the verdict appears in the log metadata but the deny reason field is empty (the connection is allowed). In the dashboard, these entries render with an observe-mode badge on the reason cell and show the underlying decision type (ban, captcha, throttle, unavailable) on hover. This lets you audit what CrowdSec would block without affecting traffic. For a self-test workflow, see [Testing the integration](/selfhosted/maintenance/crowdsec#testing-the-integration). + +For HTTP services, the deny code from the table above is recorded in the `auth_method_used` field, and the entry's `reason` field carries a synthesized generic value rather than the specific code. This applies to every access restriction, not only CrowdSec: + +```json +{ "status_code": 403, "reason": "Authentication failed", "auth_method_used": "ip_restricted" } +{ "status_code": 403, "reason": "Authentication failed", "auth_method_used": "crowdsec_ban", + "metadata": { "crowdsec_verdict": "crowdsec_ban" } } +``` + +When reading entries through `GET /api/events/proxy`, match on `auth_method_used` (and `metadata.crowdsec_verdict` for CrowdSec specifically) rather than `reason`. + +Observe-mode entries carry the normal status code and record both `crowdsec_mode` and `crowdsec_verdict` in `metadata`. Because the connection is allowed, CrowdSec itself contributes no `reason`, but the field can still be populated by a later stage of the request such as an authentication or backend failure. Treat `metadata.crowdsec_mode` as the signal that an entry is an observe-mode verdict, not the absence of `reason`. + + +When CrowdSec is in **observe** mode, the verdict appears in the log metadata and CrowdSec adds no deny reason of its own (the connection is allowed). In the dashboard, these entries render with an observe-mode badge on the reason cell and show the underlying decision type (ban, captcha, throttle, unavailable) on hover. This lets you audit what CrowdSec would block without affecting traffic. For a self-test workflow, see [Testing the integration](/selfhosted/maintenance/crowdsec#testing-the-integration). ## Use cases diff --git a/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx b/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx index 219374217..1a21f0ef3 100644 --- a/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx +++ b/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx @@ -96,15 +96,16 @@ services: - '--experimental.plugins.bouncer.version=v1.6.0' ``` -If your generated Compose project network is named `netbird_netbird`, make sure Traefik uses that Docker network: +The generated Compose file already sets `--providers.docker.network` in the Traefik command block, normally to `netbird`, matching the network key in the Compose file. Leave it as generated and do not add a second copy of the flag. -```yaml -services: - traefik: - command: - - '--providers.docker.network=netbird_netbird' +Only change it if Traefik logs warnings about a missing Docker network. Compose prefixes the network name with the project name, which defaults to the install directory name, so the actual name varies by deployment. Check it before editing: + +```bash +docker network ls | grep netbird ``` +An install in `/root` produces `root_netbird`; one in `/opt/netbird` produces `netbird_netbird`. + Under `services.traefik.depends_on`, make Traefik wait until CrowdSec is healthy: ```yaml @@ -299,10 +300,104 @@ curl -ks -A 'Mozilla/5.0 NetBirdDashboardCheck' -o /dev/null -w '%{http_code}\n' The response should be `200`. +## Verifying the middleware stays attached + +This protection fails open. If the middleware is detached for any reason, most commonly a label edit applied with `docker compose restart` instead of `docker compose up -d`, which does not recreate the container, requests stop being inspected and nothing reports an error. The dashboard loads, authentication works, and all containers report healthy. The only symptom is that a known malicious probe returns `404` instead of `403`. + +After any change to the labels, recreate the affected containers and re-run the probe: + +```bash +docker compose up -d dashboard netbird-server +curl -ks -o /dev/null -w '%{http_code}\n' "https://$NETBIRD_DOMAIN/.env" +``` + +A `403` confirms the middleware is in the request path. A `404` means it is not. + +To surface this automatically, enable the Traefik API on the container's internal port and add a health check that asserts both protected routers still reference the middleware: + +```yaml +services: + traefik: + command: + - '--api=true' + - '--api.insecure=true' + healthcheck: + test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/api/http/routers/netbird-dashboard@docker | grep -q netbird-dashboard-crowdsec && wget -qO- http://127.0.0.1:8080/api/http/routers/netbird-dashboard-api@docker | grep -q netbird-dashboard-crowdsec"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 20s +``` + +Do not publish port `8080`. With the port unpublished the API is reachable only from inside the container and from other containers on the same Docker network. If exposing the API to the Docker network is unacceptable in your environment, omit it and monitor the `/.env` probe externally instead, treating `403` as the healthy response. + + + Traefik reporting `unhealthy` does not stop it serving traffic. The health + check makes the loss of protection visible in `docker compose ps` and to + external monitoring; it does not restore enforcement. + + +## Recovering from a dashboard lockout + +If AppSec or an IP decision blocks you from the dashboard, the block also covers `/api` and `/oauth2`, so it cannot be lifted from the UI. All recovery is performed over SSH on the host. + +First identify which of the two mechanisms is blocking you, because they are handled differently: + +```bash +docker compose exec crowdsec cscli decisions list +``` + +If your address is listed, an **IP decision** is blocking you. If it is not, the block is an **AppSec** match on the request itself: AppSec inspects requests and does not create decisions, so `cscli decisions delete` has no effect in that case. Confirm with `cscli metrics show appsec` and `cscli alerts list`, where AppSec matches appear with `kind` set to `waf`. + +Allowlisting the address resolves both cases. `cscli allowlists add` fails with `allowlist '' not found` if the list does not already exist, so create it first if you have not set one up: + +```bash +docker compose exec crowdsec cscli allowlists create netbird-admins -d "Addresses that should never be blocked" +docker compose exec crowdsec cscli allowlists add netbird-admins -d "admin" +``` + +Adding an address that currently carries a decision expires that decision immediately. Allowlisted addresses are also exempted from AppSec blocking, though that takes effect on the Traefik bouncer's next stream refresh rather than instantly, so allow up to a minute. See [Allowlisting addresses](/selfhosted/maintenance/crowdsec#allowlisting-addresses) for details. + +If an IP decision is present and you want it gone without allowlisting the address permanently, delete it directly: + +```bash +docker compose exec crowdsec cscli decisions delete --ip +``` + +The Traefik bouncer runs in `stream` mode and refreshes periodically, so allow up to a minute for access to return. + +If the dashboard is still unreachable and you need immediate access, detach the middleware. It is attached to **two** routers and both must be commented out: + +```yaml +services: + dashboard: + labels: + # - traefik.http.routers.netbird-dashboard.middlewares=netbird-dashboard-crowdsec@docker + + netbird-server: + labels: + # - traefik.http.routers.netbird-dashboard-api.middlewares=netbird-dashboard-crowdsec@docker +``` + +```bash +docker compose up -d dashboard netbird-server +``` + + + Commenting out only the `dashboard` label leaves the dashboard reachable while + `/api` and `/oauth2` continue to return `403`. The interface loads but cannot + authenticate or fetch data, which is easily mistaken for a different fault. + Detach both labels, or neither. + + +Restore both labels once the underlying issue is resolved, recreate the containers, and confirm with the `/.env` probe above. + ## Troubleshooting If all dashboard requests return `403` immediately after startup, Traefik may have started before CrowdSec LAPI and AppSec were ready. Confirm that the CrowdSec health check is present and that Traefik uses `depends_on.condition: service_healthy`. +If CrowdSec is stopped or unreachable while `crowdsecAppsecUnreachableBlock=true` is set, all dashboard requests are denied by design. The same applies to reverse proxy services in `enforce` mode, which fail closed when the LAPI is unavailable. + If Traefik logs warnings about a missing Docker network, check the actual network name: ```bash diff --git a/src/pages/selfhosted/maintenance/crowdsec.mdx b/src/pages/selfhosted/maintenance/crowdsec.mdx index fd6c66992..cf6d1f087 100644 --- a/src/pages/selfhosted/maintenance/crowdsec.mdx +++ b/src/pages/selfhosted/maintenance/crowdsec.mdx @@ -48,7 +48,7 @@ CrowdSec decisions include several remediation types (ban, captcha, throttle). T ### Reviewing observe-mode verdicts -Observe-mode verdicts are recorded in the NetBird proxy access logs, not in the CrowdSec Console. When a service is in observe mode and CrowdSec flags an IP, the connection is allowed but the verdict is attached to the log entry as metadata, while the `deny_reason` field stays empty. In the dashboard's reverse proxy event log, these entries render with an observe-mode badge on the reason cell and show the decision type (ban, captcha, throttle, unavailable) so you can audit what would have been blocked before switching the service to enforce. The CrowdSec Console shows the aggregate view of community decisions and scenarios but does not know which of your proxy requests the bouncer was consulted on. +Observe-mode verdicts are recorded in the NetBird proxy access logs, not in the CrowdSec Console. When a service is in observe mode and CrowdSec flags an IP, the connection is allowed and the verdict is attached to the log entry as metadata (`crowdsec_mode` and `crowdsec_verdict`), with no deny reason contributed by CrowdSec. In the dashboard's reverse proxy event log, these entries render with an observe-mode badge on the reason cell and show the decision type (ban, captcha, throttle, unavailable) so you can audit what would have been blocked before switching the service to enforce. The CrowdSec Console shows the aggregate view of community decisions and scenarios but does not know which of your proxy requests the bouncer was consulted on. ![CrowdSec observe-mode badge in proxy event logs](/docs-static/img/selfhosted/maintenance/crowdsec-observe-badge.png) @@ -56,7 +56,7 @@ Access restrictions are evaluated in a fixed order: CIDR, then country, then Cro ## Enroll with the CrowdSec Console (optional) -Enrolling your LAPI with the [CrowdSec Console](https://app.crowdsec.net) lets you view blocked IPs, manage scenarios, and opt into premium blocklists from a web UI. The quickstart script prompts for an enrollment key and registers it automatically. To enroll an existing deployment: +Enrolling your LAPI with the [CrowdSec Console](https://app.crowdsec.net) lets you view blocked IPs, manage scenarios, and opt into premium blocklists from a web UI. Enrollment is always a manual step: the quickstart script prints the command to run once setup finishes, but does not perform the enrollment for you. To enroll any deployment: ```bash docker compose exec crowdsec cscli console enroll @@ -71,13 +71,13 @@ Once CrowdSec is enabled on the proxy, the **CrowdSec IP Reputation** dropdown a ## Testing the integration -After enabling CrowdSec, confirm that the proxy bouncer connected to the LAPI and completed its initial decision sync: +After enabling CrowdSec on at least one service, confirm that the proxy bouncer connected to the LAPI and completed its initial decision sync: ```bash docker compose logs proxy | grep -i crowdsec ``` -A healthy startup looks like this: +A healthy bouncer looks like this: ```text netbird-proxy | INFO proxy/internal/crowdsec/bouncer.go:70: connecting to CrowdSec LAPI at http://crowdsec:8080 @@ -85,10 +85,94 @@ netbird-proxy | INFO proxy/internal/crowdsec/registry.go:94: CrowdSec bouncer s netbird-proxy | INFO proxy/internal/crowdsec/bouncer.go:187: CrowdSec bouncer synced initial decisions ``` + +These lines only appear once a service has CrowdSec set to `enforce` or `observe`. The bouncer starts lazily, so configuring `NB_PROXY_CROWDSEC_API_URL` and restarting the proxy produces no CrowdSec log output on its own. Empty output before the first service is enabled is expected and is not a failure. + + +### What the engine holds + +A newly registered CrowdSec instance receives an **empty** community blocklist and fetches the full list on its next scheduled pull, up to two hours later: + +```text +capi/community-blocklist : received 0 new entries (expected if you just installed crowdsec) +Start pull from CrowdSec Central API (interval: 1h59m7s once, then 2h0m0s) +``` + +Until that completes, a service in enforce mode has nothing to enforce. Use the manual decision test below to verify enforcement immediately. + +Once the blocklist has synced, note that `cscli decisions list` hides CAPI-sourced decisions by default, so it can show a handful of entries while the engine holds thousands. To inspect them, request the CAPI origin explicitly: + +```bash +docker compose exec crowdsec cscli decisions list --origin CAPI | head -20 +``` + +`--limit` bounds the number of *alerts* returned, not decisions. The community blocklist arrives as a single alert containing every address, so `--limit` does not meaningfully shorten this output; pipe it through `head` instead. + +To size the blocklist, read the `cs_active_decisions` gauge. It is reported per `origin`/`reason`/`action` combination rather than as a single total, so sum the series you care about: + +```bash +docker compose exec crowdsec sh -c "wget -qO- http://127.0.0.1:6060/metrics" | grep '^cs_active_decisions' +``` + +```text +cs_active_decisions{action="ban",origin="CAPI",reason="generic:scan"} 6547 +cs_active_decisions{action="ban",origin="CAPI",reason="ssh:bruteforce"} 8452 +cs_active_decisions{action="ban",origin="crowdsec",reason="crowdsecurity/ssh-slow-bf"} 1 +``` + +The `crowdsecurity/linux` collection installed by default provides syslog and SSH parsers. It only produces decisions if CrowdSec can read those logs, which requires mounting them into the container (for example `/var/log:/var/log:ro` plus an acquisition file in `crowdsec/acquis.d/`). Without a log source, the collection is inert and the community blocklist is the only source of decisions. + To verify end-to-end enforcement without waiting for a real malicious IP to hit the cache, add a short-lived decision for your own IP and then attempt a connection: ```bash docker compose exec crowdsec cscli decisions add --ip --duration 5m --reason "netbird test" ``` -With a service set to **enforce**, the connection from `` should be rejected and the event log should show a `crowdsec_ban` deny reason. With a service set to **observe**, the connection succeeds but the verdict appears on the event entry with an observe-mode badge. The decision expires automatically after 5 minutes, or you can remove it earlier with `cscli decisions delete --ip `. +With a service set to **enforce**, the connection from `` is rejected with a `403` within roughly 10 seconds (the bouncer poll interval), and the access log entry carries `crowdsec_ban` in its `auth_method_used` and `metadata.crowdsec_verdict` fields. With a service set to **observe**, the connection succeeds and the verdict appears on the event entry with an observe-mode badge. The decision expires automatically after 5 minutes, or you can remove it earlier with `cscli decisions delete --ip `. + + +If the dashboard is also protected by [CrowdSec AppSec](/selfhosted/maintenance/crowdsec-dashboard), banning the address you are browsing from blocks the NetBird dashboard and the `/api` path as well, and the block cannot be lifted from the UI. Run this test from an address you are not administering from, or be prepared to remove the decision over SSH. + + +`cscli decisions add` also accepts `--type captcha` and `--type throttle`. NetBird Proxy denies all three types identically with a `403`; only the recorded verdict string differs. + +## Allowlisting addresses + +To exempt an address or range from all decisions, including the community blocklist, use CrowdSec allowlists (available from CrowdSec 1.6.8): + +```bash +docker compose exec crowdsec cscli allowlists create netbird-admins -d "Addresses that should never be blocked" +docker compose exec crowdsec cscli allowlists add netbird-admins -d "admin" +docker compose exec crowdsec cscli allowlists check +``` + +The `-d` description flag is required on `create`. Changes apply immediately with no restart. + +Allowlisting applies to existing decisions as well as future ones. Adding an address that is already blocked expires the matching decisions immediately: + +```text +1 decisions deleted by allowlists +``` + +CrowdSec also refuses to create new decisions against an allowlisted address, which affects the manual test above: + +```text +Error: cscli decisions add: is allowlisted by item from netbird-admins (admin), +use --bypass-allowlist to add the decision anyway +``` + +The older `capi_whitelists_path` setting in `config.yaml` is deprecated upstream in favour of allowlists. + +## Recovering from an unwanted block + +If a legitimate address is blocked, remove the decision and, if it should never be blocked again, add it to an allowlist: + +```bash +docker compose exec crowdsec cscli decisions list +docker compose exec crowdsec cscli alerts list +docker compose exec crowdsec cscli decisions delete --ip +``` + +`cscli alerts list` includes a `kind` column that identifies which component generated the alert: `crowdsec` for local log detection, `waf` for an AppSec match, and `cscli` for a manually added decision. AppSec matches block the individual request but do not create an IP decision, so they never appear in `cscli decisions list`; use `cscli metrics show appsec` to inspect those instead. + +For dashboard lockouts specifically, see [Recovering from a dashboard lockout](/selfhosted/maintenance/crowdsec-dashboard#recovering-from-a-dashboard-lockout). diff --git a/src/pages/selfhosted/migration/enable-reverse-proxy.mdx b/src/pages/selfhosted/migration/enable-reverse-proxy.mdx index aa4e8cc29..788fd1d7e 100644 --- a/src/pages/selfhosted/migration/enable-reverse-proxy.mdx +++ b/src/pages/selfhosted/migration/enable-reverse-proxy.mdx @@ -403,18 +403,41 @@ Then restart the proxy: docker compose up -d proxy ``` -Verify the connection in the proxy logs: +At this point the proxy logs will **not** mention CrowdSec yet: ```bash docker compose logs proxy | grep -i crowdsec ``` -You should see `CrowdSec bouncer synced initial decisions` once the LAPI connection is established. +Empty output here is expected. The stream bouncer is started lazily: it does not connect to the LAPI until at least one service has CrowdSec enabled in step 7d. + +To confirm the proxy picked up the configuration, check that the cluster now advertises CrowdSec support instead: + +```bash +curl -s -H "Authorization: Token " \ + https:///api/reverse-proxies/clusters | jq '.[].supports_crowdsec' +``` + +This should return `true`. In the dashboard, the equivalent check is that a **CrowdSec IP Reputation** dropdown now appears on the **Access Control** tab of a reverse proxy service. #### 7d. Enable per service CrowdSec must be enabled individually on each service through the dashboard under **Access Control**. Set the CrowdSec mode to **enforce** or **observe**. +Once the first service is set to `enforce` or `observe`, the bouncer starts and the proxy logs the connection: + +```bash +docker compose logs proxy | grep -i crowdsec +``` + +```text +netbird-proxy | INFO proxy/internal/crowdsec/bouncer.go:70: connecting to CrowdSec LAPI at http://crowdsec:8080 +netbird-proxy | INFO proxy/internal/crowdsec/registry.go:94: CrowdSec bouncer started +netbird-proxy | INFO proxy/internal/crowdsec/bouncer.go:187: CrowdSec bouncer synced initial decisions +``` + +The bouncer re-polls the LAPI every 10 seconds. You can confirm from the CrowdSec side with `cscli bouncers list`, which shows a recent **Last API pull** for `netbird-proxy`. + ![CrowdSec IP Reputation Overview](/docs-static/img/selfhosted/maintenance/crowdsec-overview.png) @@ -450,12 +473,27 @@ If this fails, verify both containers are on the same Docker network. **All connections denied after enabling enforce mode**: This typically means the bouncer has not completed its initial sync. Check the proxy logs for the `CrowdSec bouncer synced initial decisions` message. If it's missing, the LAPI may be unreachable or the API key may be incorrect. Switch to **observe** mode on affected services until the issue is resolved. +Note that the message is only expected *after* a service has CrowdSec enabled (step 7d). Its absence before that point is normal and does not indicate a problem. + +**No decisions on a new installation**: a freshly registered CrowdSec instance receives an empty community blocklist and pulls the full list on its next scheduled sync, up to two hours later. The startup logs state this directly: + +```text +capi/community-blocklist : received 0 new entries (expected if you just installed crowdsec) +Start pull from CrowdSec Central API (interval: 1h59m7s once, then 2h0m0s) +``` + +Until that first pull completes, a service in `enforce` mode has nothing to enforce and traffic passes normally. Use a manual decision (below) to verify enforcement immediately, since local decisions apply straight away. + **Checking active decisions**: ```bash -# List current decisions +# List locally-generated and manually-added decisions docker compose exec crowdsec cscli decisions list +# List community blocklist decisions (hidden from the default view) +# --limit bounds alerts, not decisions, so pipe through head +docker compose exec crowdsec cscli decisions list --origin CAPI | head -20 + # Ban an IP for 1 hour (for testing) docker compose exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 1h --reason "manual test" @@ -463,6 +501,16 @@ docker compose exec crowdsec cscli decisions add --ip 1.2.3.4 --duration 1h --re docker compose exec crowdsec cscli decisions delete --ip 1.2.3.4 ``` +`cscli decisions list` excludes CAPI-sourced decisions by default, so it can report a handful of entries while the engine holds thousands. To size the blocklist, read the `cs_active_decisions` gauge, which is reported per `origin`/`reason`/`action` combination rather than as a single total: + +```bash +docker compose exec crowdsec sh -c "wget -qO- http://127.0.0.1:6060/metrics" | grep '^cs_active_decisions' +``` + + +`cscli decisions delete --all` removes every decision including the entire synced community blocklist. Restarting CrowdSec may not immediately re-fetch it: startup attempts a pull, but the request is skipped if the previous CAPI pull was recent, leaving restoration to the next scheduled pull up to two hours later. The instance runs without community reputation data until then. Prefer `--ip` for targeted removals. + + ## Configure SSO for external identity providers ### Who this applies to @@ -682,7 +730,7 @@ The proxy is configured through environment variables (each one maps to an equiv | `NB_PROXY_PRESHARED_KEY` | No | Pre-shared key for the tunnel between the proxy and peers. | - | | `NB_PROXY_SUPPORTS_CUSTOM_PORTS` | No | Whether the proxy can bind arbitrary ports for UDP/TCP passthrough. | `true` | | `NB_PROXY_REQUIRE_SUBDOMAIN` | No | Require a subdomain label in front of the cluster domain. | `false` | -| `NB_PROXY_PRIVATE` | No | Serve private services with NetBird-Only authentication, reachable exclusively over the WireGuard tunnel (also enables per-account inbound listeners). Advanced; intended for proxies embedded in a NetBird client rather than standalone deployments. | `false` | +| `NB_PROXY_PRIVATE` | No | Serve private services with NetBird-Only authentication, reachable exclusively over the WireGuard tunnel (also enables per-account inbound listeners). Required for the **Proxy Cluster** target type and **NetBird-Only Access**. The `netbirdio/reverse-proxy` image runs in embedded mode by default, so setting this on a standard self-hosted deployment is supported: the cluster then reports the `Private` capability and the dashboard **Clusters** page shows a **Private** badge. | `false` | | `NB_PROXY_MAX_DIAL_TIMEOUT` | No | Cap the per-service backend dial timeout (`0` = no cap), e.g. `10s`. | `0` | | `NB_PROXY_MAX_SESSION_IDLE_TIMEOUT` | No | Cap the per-service session idle timeout (`0` = no cap), e.g. `5m`. | `0` | From c5d45c39360f0c6c1de6cd869f09ae72fa9723c3 Mon Sep 17 00:00:00 2001 From: TechHutTV Date: Wed, 12 Aug 2026 08:07:53 -0700 Subject: [PATCH 2/2] Refine CrowdSec dashboard recovery and access-log field documentation --- .../manage/reverse-proxy/access-logs.mdx | 19 ++++++---- .../maintenance/crowdsec-dashboard.mdx | 36 +++++++++---------- 2 files changed, 31 insertions(+), 24 deletions(-) diff --git a/src/pages/manage/reverse-proxy/access-logs.mdx b/src/pages/manage/reverse-proxy/access-logs.mdx index c7abf16cc..d0a405082 100644 --- a/src/pages/manage/reverse-proxy/access-logs.mdx +++ b/src/pages/manage/reverse-proxy/access-logs.mdx @@ -37,9 +37,9 @@ Every log entry (HTTP and L4) shares a common set of fields. Some fields are onl | **Bytes Downloaded** | Bytes sent from backend to client | Yes | Yes | | **Source IP** | The client's IP address | Yes | Yes | | **Location** | Country, city, and subdivision based on source IP geolocation | Yes | Yes | -| **Auth Method** | Authentication method used (SSO, password, PIN, header, or none) | Yes | N/A | +| **Auth Method** | Authentication method used (SSO, password, PIN, header, or none). For denied requests, carries the restriction code instead (e.g. `ip_restricted`, `crowdsec_ban`) | Yes | N/A | | **User** | The authenticated user's ID (if SSO was used) | Yes | N/A | -| **Reason** | Reason for denial, if applicable | Yes | Yes | +| **Reason** | `Authentication failed` when authentication or an access restriction rejected the request, or `Request failed` when an authenticated request returned `4xx`/`5xx`. Not a specific denial code: see the note under [Deny reasons](#deny-reasons) | Yes | No | ## Understanding log entries @@ -48,14 +48,14 @@ Every log entry (HTTP and L4) shares a common set of fields. Some fields are onl HTTP log entries fall into three categories based on the status code: - **Allowed requests**: successful requests show a `2xx` status code along with the authentication method used to access the service. -- **Denied requests**: failed authentication or access restriction blocks show `401` or `403` status codes with a reason explaining why the request was denied (e.g., invalid password, missing SSO session, IP restricted, country restricted). +- **Denied requests**: failed authentication or access restriction blocks show `401` or `403` status codes with `reason` set to `Authentication failed`. The specific cause (invalid password, missing SSO session, IP restricted, country restricted, CrowdSec verdict) is carried in `auth_method_used`, not in `reason`. - **Errors**: backend errors or proxy issues show `5xx` status codes. These typically indicate that the target service is unreachable or returned an error. ### L4 log entries L4 entries are logged when the connection closes and record the total bytes transferred in each direction and the connection duration. L4 entries do not have HTTP status codes. -Denied L4 connections (blocked by access restrictions) are logged immediately with a deny reason. Since L4 services do not support authentication, denials come from access restrictions only. +Denied L4 connections (blocked by access restrictions) are logged immediately. L4 entries carry no `reason` value, so the restriction code identifies the denial. Since L4 services do not support authentication, denials come from access restrictions only. ### Deny reasons @@ -78,8 +78,15 @@ For HTTP services, the deny code from the table above is recorded in the `auth_m ```json { "status_code": 403, "reason": "Authentication failed", "auth_method_used": "ip_restricted" } -{ "status_code": 403, "reason": "Authentication failed", "auth_method_used": "crowdsec_ban", - "metadata": { "crowdsec_verdict": "crowdsec_ban" } } +``` + +```json +{ + "status_code": 403, + "reason": "Authentication failed", + "auth_method_used": "crowdsec_ban", + "metadata": { "crowdsec_verdict": "crowdsec_ban" } +} ``` When reading entries through `GET /api/events/proxy`, match on `auth_method_used` (and `metadata.crowdsec_verdict` for CrowdSec specifically) rather than `reason`. diff --git a/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx b/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx index 1a21f0ef3..946a3bc1e 100644 --- a/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx +++ b/src/pages/selfhosted/maintenance/crowdsec-dashboard.mdx @@ -313,28 +313,26 @@ curl -ks -o /dev/null -w '%{http_code}\n' "https://$NETBIRD_DOMAIN/.env" A `403` confirms the middleware is in the request path. A `404` means it is not. -To surface this automatically, enable the Traefik API on the container's internal port and add a health check that asserts both protected routers still reference the middleware: +To surface this automatically, monitor the same probe from outside the deployment and treat `403` as the healthy response. Point an uptime monitor or a cron job at: -```yaml -services: - traefik: - command: - - '--api=true' - - '--api.insecure=true' - healthcheck: - test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:8080/api/http/routers/netbird-dashboard@docker | grep -q netbird-dashboard-crowdsec && wget -qO- http://127.0.0.1:8080/api/http/routers/netbird-dashboard-api@docker | grep -q netbird-dashboard-crowdsec"] - interval: 30s - timeout: 5s - retries: 3 - start_period: 20s +```text +https:///.env ``` -Do not publish port `8080`. With the port unpublished the API is reachable only from inside the container and from other containers on the same Docker network. If exposing the API to the Docker network is unacceptable in your environment, omit it and monitor the `/.env` probe externally instead, treating `403` as the healthy response. +Alert when the response is anything other than `403`. This is an inverted check, alerting on the absence of a block rather than on an outage, but it is the only signal that confirms the protection layer is still in the request path. - Traefik reporting `unhealthy` does not stop it serving traffic. The health - check makes the loss of protection visible in `docker compose ps` and to - external monitoring; it does not restore enforcement. + Do not enable the Traefik API (`--api.insecure=true`) for this purpose. The + API serves the full dynamic configuration without authentication, including + middleware plugin settings such as `crowdsecLapiKey`. Even with port `8080` + unpublished, every container on the Docker network could then read the + CrowdSec bouncer key. + + + + This is detection only. Nothing here restores enforcement automatically: the + probe tells you protection has stopped, and reattaching the middleware + remains a manual step. ## Recovering from a dashboard lockout @@ -396,7 +394,9 @@ Restore both labels once the underlying issue is resolved, recreate the containe If all dashboard requests return `403` immediately after startup, Traefik may have started before CrowdSec LAPI and AppSec were ready. Confirm that the CrowdSec health check is present and that Traefik uses `depends_on.condition: service_healthy`. -If CrowdSec is stopped or unreachable while `crowdsecAppsecUnreachableBlock=true` is set, all dashboard requests are denied by design. The same applies to reverse proxy services in `enforce` mode, which fail closed when the LAPI is unavailable. +If CrowdSec is stopped or unreachable while `crowdsecAppsecUnreachableBlock=true` is set, all dashboard requests are denied by design. + +Separately, a reverse proxy service in `enforce` mode denies all connections while its bouncer has not completed its initial sync, for example when the proxy starts with the LAPI unavailable. See [Enforcement modes](/selfhosted/maintenance/crowdsec#enforcement-modes). If Traefik logs warnings about a missing Docker network, check the actual network name: