diff --git a/docs.json b/docs.json index 47a1fb3e..5611151d 100644 --- a/docs.json +++ b/docs.json @@ -412,7 +412,8 @@ "self-host/customize-deployment/enable-headless-browser-for-lightdash", "self-host/customize-deployment/sandboxes", "self-host/customize-deployment/secure-lightdash-with-https", - "self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash" + "self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash", + "self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash" ] }, { diff --git a/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash.mdx b/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash.mdx new file mode 100644 index 00000000..ed1b8082 --- /dev/null +++ b/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash.mdx @@ -0,0 +1,90 @@ +--- +title: "Configure OpenTelemetry tracing for self-hosted Lightdash" +sidebarTitle: OpenTelemetry tracing +description: "Export distributed traces from your Lightdash instance to any OpenTelemetry-compatible backend" +boost: 0.001 +doc-type: reference +--- + +import OtelEnv from '/snippets/self-host/otel-env.mdx'; + + +🛠 This page is for engineering teams self-hosting their own Lightdash instance. For instance health metrics, see [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash). + + +Lightdash can export distributed traces using the OpenTelemetry SDK, so you can follow a request or scheduled job across the API server, scheduler, and warehouse queries in any OpenTelemetry-compatible backend (for example Grafana Tempo, Jaeger, Honeycomb, or Datadog). + +Tracing runs in one of two exclusive modes: + +- **Sentry mode (default)**: spans are created and exported through Sentry, controlled by the [Sentry environment variables](/self-host/customize-deployment/environment-variables#sentry). +- **OpenTelemetry mode**: spans are created by the OpenTelemetry SDK and exported according to the standard `OTEL_*` environment variables. Sentry still captures errors, but receives no spans. + +## Enabling OpenTelemetry tracing + +By default, Lightdash traces through Sentry. To switch to OpenTelemetry mode, set the following environment variable on every Lightdash container (API server and scheduler): + +```bash +LIGHTDASH_OTEL_TRACES_ENABLED=true +``` + +Then point the OTLP exporter at your collector: + +```bash +OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318 +``` + +## Configuration options + + + +## Choosing an exporter + +Lightdash defers exporter and protocol selection to the OpenTelemetry Node SDK's standard environment variable handling: + +- `OTEL_TRACES_EXPORTER` supports `otlp` (default), `console`, `zipkin`, and `none`. You can combine exporters with a comma-separated list. If the list contains `none`, no traces are exported regardless of the other values. +- With the `otlp` exporter, `OTEL_EXPORTER_OTLP_PROTOCOL` (or the traces-specific `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`) selects `grpc`, `http/json`, or `http/protobuf` (default). +- Unsupported exporter or protocol values are ignored with a warning in the Lightdash logs at startup. + +For example, to export traces over gRPC with an authentication header: + +```bash +LIGHTDASH_OTEL_TRACES_ENABLED=true +OTEL_TRACES_EXPORTER=otlp +OTEL_EXPORTER_OTLP_PROTOCOL=grpc +OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com:4317 +OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer " +``` + +## Sampling + +`LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` sets the head-sampling ratio for trace roots, from `0.0` (nothing) to `1.0` (everything, the default). Child spans follow their root's decision, so a sampled request captures the whole waterfall. + +Two behaviours to be aware of: + +- **AI agent traces are always sampled** regardless of the ratio, so a broken agent run always has a trace to debug. Set `LIGHTDASH_OTEL_ALWAYS_SAMPLE_AI_TRACES=false` to make them follow the global ratio instead. +- **`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are overridden** by Lightdash's own sampler and have no effect. + +Requests to health checks (`/api/v1/health`, `livez`), status polling endpoints, `favicon.ico`, and `robots.txt` are never traced. + +## Database query tracing + +Set `LIGHTDASH_OTEL_DB_TRACES_ENABLED=true` to add a span for each application database (Postgres) query, so you can see where a request spends time inside Lightdash's own database: + +```bash +LIGHTDASH_OTEL_TRACES_ENABLED=true +LIGHTDASH_OTEL_DB_TRACES_ENABLED=true +``` + +- Database spans only appear inside an existing trace, so they follow the sampling decision of their parent request or job. +- Each span records the SQL statement, truncated to `LIGHTDASH_OTEL_DB_TRACES_MAX_QUERY_LENGTH` characters (default `1022`). Invalid values fall back to the default with a warning. +- This traces queries to Lightdash's application database only, not queries sent to your data warehouse. + +## Troubleshooting + +At startup, Lightdash logs a line confirming the tracing configuration, including the active exporters, OTLP protocol, and sampling ratio. This confirms configuration only; exporter connectivity is not validated at startup. + +If traces aren't arriving in your backend, set `OTEL_LOG_LEVEL=DEBUG` to enable OpenTelemetry SDK and exporter diagnostics in the Lightdash logs. Diagnostic output is redacted before logging: URL credentials and query strings are stripped, values containing tokens or API keys are omitted, and span payloads are not printed. + +## Metrics + +This page covers traces only. For instance health metrics (CPU, memory, event loop, query durations), see [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash), which can also feed an OpenTelemetry backend through the collector's Prometheus receiver. diff --git a/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash.mdx b/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash.mdx index fb9ce345..97099687 100644 --- a/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash.mdx +++ b/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash.mdx @@ -193,6 +193,8 @@ For example, you might want to create alerts for: Lightdash metrics are also compatible with OpenTelemetry. You can use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector) with the [Prometheus receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) to scrape Lightdash's Prometheus metrics endpoint and export them to any OpenTelemetry-compatible backend. +Lightdash can also export distributed traces directly over OTLP. See [Configure OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash). + Example OpenTelemetry Collector configuration: ```yaml diff --git a/self-host/customize-deployment/environment-variables.mdx b/self-host/customize-deployment/environment-variables.mdx index 8d5381e5..817d62ed 100644 --- a/self-host/customize-deployment/environment-variables.mdx +++ b/self-host/customize-deployment/environment-variables.mdx @@ -9,6 +9,7 @@ doc-type: reference import SmtpEnv from '/snippets/self-host/smtp-env.mdx'; import LoggingEnv from '/snippets/self-host/logging-env.mdx'; import PrometheusEnv from '/snippets/self-host/prometheus-env.mdx'; +import OtelEnv from '/snippets/self-host/otel-env.mdx'; import HeadlessBrowserEnv from '/snippets/self-host/headless-browser-env.mdx'; import McpEnv from '/snippets/self-host/mcp-env.mdx'; @@ -314,6 +315,12 @@ The variables below act as instance-wide defaults. Organization admins can overr +## OpenTelemetry tracing + +See [Configure OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash) for setup, exporter selection, and sampling behaviour. + + + ## Security | Variable | Description | diff --git a/self-host/data-flows-and-telemetry.mdx b/self-host/data-flows-and-telemetry.mdx index 31f700c8..ebe9680f 100644 --- a/self-host/data-flows-and-telemetry.mdx +++ b/self-host/data-flows-and-telemetry.mdx @@ -284,6 +284,7 @@ Community Edition (no `LIGHTDASH_LICENSE_KEY`) does not call `api.keygen.sh` or - [Production deployment checklist](/self-host/production-deployment-checklist) — includes the default egress policy - [Usage Analytics dashboards and query tags](/workspace-admin/usage-analytics) - [Prometheus and OpenTelemetry metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash) +- [OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash) - [AI agents and AI providers](/self-host/enterprise-features/ai-agents) - [Data apps sandboxes](/self-host/customize-deployment/sandboxes) - [MCP](/self-host/enterprise-features/mcp) diff --git a/self-host/production-deployment-checklist.mdx b/self-host/production-deployment-checklist.mdx index 532114a3..29533637 100644 --- a/self-host/production-deployment-checklist.mdx +++ b/self-host/production-deployment-checklist.mdx @@ -236,6 +236,7 @@ Enable Prometheus metrics and structured JSON logging on every pod, and scrape t - Metrics and alerting guidance: [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash). Scrape port **9090** on all pods labelled `app.kubernetes.io/name=lightdash` (the chart ships no ServiceMonitor/PodMonitoring — create one; Lightdash Cloud scrapes at a 30s interval). If you run NATS, its Prometheus exporter is on port **7777** (`nats.promExporter.enabled: true`). - Log configuration: [Configure logging](/self-host/customize-deployment/environment-variables#logging). Ship `LIGHTDASH_LOG_FORMAT: json` to your log platform; `LIGHTDASH_LOG_LEVEL: audit` adds an audit trail of user actions. - Alert on HTTP p95/error rate (`http_server_request_duration_seconds`), queue depth / scheduler job failures, Postgres pool saturation, and event-loop lag. +- Distributed traces (optional): [OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash). Set `LIGHTDASH_OTEL_TRACES_ENABLED: "true"` and point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector. ## Recommended resources diff --git a/snippets/self-host/otel-env.mdx b/snippets/self-host/otel-env.mdx new file mode 100644 index 00000000..bfb5fb2c --- /dev/null +++ b/snippets/self-host/otel-env.mdx @@ -0,0 +1,17 @@ +| Variable | Description | +| :---------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `LIGHTDASH_OTEL_TRACES_ENABLED` | Switches tracing to OpenTelemetry mode: spans are created by the OpenTelemetry SDK and Sentry receives errors only. When `false`, tracing runs through Sentry. (default=false) | +| `LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` | Trace sampling ratio from 0.0 to 1.0. Falls back to `SENTRY_TRACES_SAMPLE_RATE` (deprecated) when unset. (default=1) | +| `LIGHTDASH_OTEL_ALWAYS_SAMPLE_AI_TRACES` | Set to `false` to stop always-sampling AI agent traces, so they follow the global sampling ratio instead. (default=true) | +| `LIGHTDASH_OTEL_DB_TRACES_ENABLED` | Adds spans for application database (Postgres) queries to traces, including the SQL statement. Requires OpenTelemetry mode. (default=false) | +| `LIGHTDASH_OTEL_DB_TRACES_MAX_QUERY_LENGTH` | Maximum length of the SQL statement recorded on database spans; longer statements are truncated. Must be a non-negative integer. (default=1022) | +| `OTEL_SDK_DISABLED` | Standard OpenTelemetry kill switch. When `true`, OpenTelemetry mode is off even if `LIGHTDASH_OTEL_TRACES_ENABLED=true`. | +| `OTEL_TRACES_EXPORTER` | Comma-separated exporters: `otlp`, `console`, `zipkin`, or `none`. Unsupported values are ignored with a startup warning; `none` overrides any other value. (default=otlp) | +| `OTEL_EXPORTER_OTLP_PROTOCOL` | OTLP protocol: `grpc`, `http/json`, or `http/protobuf`. `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` takes precedence. Unsupported values fall back to the default. (default=http/protobuf) | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL of your OTLP collector. Handled by the OpenTelemetry Node SDK, along with `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS`. | +| `OTEL_SERVICE_NAME` | Service name attached to exported spans. (default=lightdash) | +| `OTEL_LOG_LEVEL` | Enables OpenTelemetry SDK and exporter diagnostics at the given level (e.g. `DEBUG`). Lightdash redacts credentials, tokens, and span payloads from diagnostic output. | + + + `OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are overridden by Lightdash. Control sampling with `LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` instead. +