Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs.json
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,8 @@
"self-host/customize-deployment/enable-headless-browser-for-lightdash",
"self-host/customize-deployment/sandboxes",
"self-host/customize-deployment/secure-lightdash-with-https",
"self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash"
"self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash",
"self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash"
]
},
{
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
---
title: "Configure OpenTelemetry tracing for self-hosted Lightdash"
sidebarTitle: OpenTelemetry tracing
description: "Export distributed traces from your Lightdash instance to any OpenTelemetry-compatible backend"
boost: 0.001
doc-type: reference
---

import OtelEnv from '/snippets/self-host/otel-env.mdx';

<Note>
🛠 This page is for engineering teams self-hosting their own Lightdash instance. For instance health metrics, see [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash).
</Note>

Lightdash can export distributed traces using the OpenTelemetry SDK, so you can follow a request or scheduled job across the API server, scheduler, and warehouse queries in any OpenTelemetry-compatible backend (for example Grafana Tempo, Jaeger, Honeycomb, or Datadog).

Tracing runs in one of two exclusive modes:

- **Sentry mode (default)**: spans are created and exported through Sentry, controlled by the [Sentry environment variables](/self-host/customize-deployment/environment-variables#sentry).
- **OpenTelemetry mode**: spans are created by the OpenTelemetry SDK and exported according to the standard `OTEL_*` environment variables. Sentry still captures errors, but receives no spans.

## Enabling OpenTelemetry tracing

By default, Lightdash traces through Sentry. To switch to OpenTelemetry mode, set the following environment variable on every Lightdash container (API server and scheduler):

```bash
LIGHTDASH_OTEL_TRACES_ENABLED=true
```

Then point the OTLP exporter at your collector:

```bash
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
```

## Configuration options

<OtelEnv />

## Choosing an exporter

Lightdash defers exporter and protocol selection to the OpenTelemetry Node SDK's standard environment variable handling:

- `OTEL_TRACES_EXPORTER` supports `otlp` (default), `console`, `zipkin`, and `none`. You can combine exporters with a comma-separated list. If the list contains `none`, no traces are exported regardless of the other values.
- With the `otlp` exporter, `OTEL_EXPORTER_OTLP_PROTOCOL` (or the traces-specific `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL`) selects `grpc`, `http/json`, or `http/protobuf` (default).
- Unsupported exporter or protocol values are ignored with a warning in the Lightdash logs at startup.

For example, to export traces over gRPC with an authentication header:

```bash
LIGHTDASH_OTEL_TRACES_ENABLED=true
OTEL_TRACES_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_EXPORTER_OTLP_ENDPOINT=https://otlp.example.com:4317
OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer <token>"
```

## Sampling

`LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` sets the head-sampling ratio for trace roots, from `0.0` (nothing) to `1.0` (everything, the default). Child spans follow their root's decision, so a sampled request captures the whole waterfall.

Two behaviours to be aware of:

- **AI agent traces are always sampled** regardless of the ratio, so a broken agent run always has a trace to debug. Set `LIGHTDASH_OTEL_ALWAYS_SAMPLE_AI_TRACES=false` to make them follow the global ratio instead.
- **`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are overridden** by Lightdash's own sampler and have no effect.

Requests to health checks (`/api/v1/health`, `livez`), status polling endpoints, `favicon.ico`, and `robots.txt` are never traced.

## Database query tracing

Set `LIGHTDASH_OTEL_DB_TRACES_ENABLED=true` to add a span for each application database (Postgres) query, so you can see where a request spends time inside Lightdash's own database:

```bash
LIGHTDASH_OTEL_TRACES_ENABLED=true
LIGHTDASH_OTEL_DB_TRACES_ENABLED=true
```

- Database spans only appear inside an existing trace, so they follow the sampling decision of their parent request or job.
- Each span records the SQL statement, truncated to `LIGHTDASH_OTEL_DB_TRACES_MAX_QUERY_LENGTH` characters (default `1022`). Invalid values fall back to the default with a warning.
- This traces queries to Lightdash's application database only, not queries sent to your data warehouse.

## Troubleshooting

At startup, Lightdash logs a line confirming the tracing configuration, including the active exporters, OTLP protocol, and sampling ratio. This confirms configuration only; exporter connectivity is not validated at startup.

If traces aren't arriving in your backend, set `OTEL_LOG_LEVEL=DEBUG` to enable OpenTelemetry SDK and exporter diagnostics in the Lightdash logs. Diagnostic output is redacted before logging: URL credentials and query strings are stripped, values containing tokens or API keys are omitted, and span payloads are not printed.

## Metrics

This page covers traces only. For instance health metrics (CPU, memory, event loop, query durations), see [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash), which can also feed an OpenTelemetry backend through the collector's Prometheus receiver.
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,8 @@ For example, you might want to create alerts for:

Lightdash metrics are also compatible with OpenTelemetry. You can use the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector) with the [Prometheus receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver) to scrape Lightdash's Prometheus metrics endpoint and export them to any OpenTelemetry-compatible backend.

Lightdash can also export distributed traces directly over OTLP. See [Configure OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash).

Example OpenTelemetry Collector configuration:

```yaml
Expand Down
7 changes: 7 additions & 0 deletions self-host/customize-deployment/environment-variables.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ doc-type: reference
import SmtpEnv from '/snippets/self-host/smtp-env.mdx';
import LoggingEnv from '/snippets/self-host/logging-env.mdx';
import PrometheusEnv from '/snippets/self-host/prometheus-env.mdx';
import OtelEnv from '/snippets/self-host/otel-env.mdx';
import HeadlessBrowserEnv from '/snippets/self-host/headless-browser-env.mdx';
import McpEnv from '/snippets/self-host/mcp-env.mdx';

Expand Down Expand Up @@ -314,6 +315,12 @@ The variables below act as instance-wide defaults. Organization admins can overr

<PrometheusEnv />

## OpenTelemetry tracing

See [Configure OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash) for setup, exporter selection, and sampling behaviour.

<OtelEnv />

## Security

| Variable | Description |
Expand Down
1 change: 1 addition & 0 deletions self-host/data-flows-and-telemetry.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,7 @@ Community Edition (no `LIGHTDASH_LICENSE_KEY`) does not call `api.keygen.sh` or
- [Production deployment checklist](/self-host/production-deployment-checklist) — includes the default egress policy
- [Usage Analytics dashboards and query tags](/workspace-admin/usage-analytics)
- [Prometheus and OpenTelemetry metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash)
- [OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash)
- [AI agents and AI providers](/self-host/enterprise-features/ai-agents)
- [Data apps sandboxes](/self-host/customize-deployment/sandboxes)
- [MCP](/self-host/enterprise-features/mcp)
Expand Down
1 change: 1 addition & 0 deletions self-host/production-deployment-checklist.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -236,6 +236,7 @@ Enable Prometheus metrics and structured JSON logging on every pod, and scrape t
- Metrics and alerting guidance: [Prometheus metrics](/self-host/customize-deployment/configure-prometheus-metrics-for-self-hosted-lightdash). Scrape port **9090** on all pods labelled `app.kubernetes.io/name=lightdash` (the chart ships no ServiceMonitor/PodMonitoring — create one; Lightdash Cloud scrapes at a 30s interval). If you run NATS, its Prometheus exporter is on port **7777** (`nats.promExporter.enabled: true`).
- Log configuration: [Configure logging](/self-host/customize-deployment/environment-variables#logging). Ship `LIGHTDASH_LOG_FORMAT: json` to your log platform; `LIGHTDASH_LOG_LEVEL: audit` adds an audit trail of user actions.
- Alert on HTTP p95/error rate (`http_server_request_duration_seconds`), queue depth / scheduler job failures, Postgres pool saturation, and event-loop lag.
- Distributed traces (optional): [OpenTelemetry tracing](/self-host/customize-deployment/configure-opentelemetry-tracing-for-self-hosted-lightdash). Set `LIGHTDASH_OTEL_TRACES_ENABLED: "true"` and point `OTEL_EXPORTER_OTLP_ENDPOINT` at your collector.

## Recommended resources

Expand Down
17 changes: 17 additions & 0 deletions snippets/self-host/otel-env.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
| Variable | Description |
| :---------------------------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `LIGHTDASH_OTEL_TRACES_ENABLED` | Switches tracing to OpenTelemetry mode: spans are created by the OpenTelemetry SDK and Sentry receives errors only. When `false`, tracing runs through Sentry. (default=false) |
| `LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` | Trace sampling ratio from 0.0 to 1.0. Falls back to `SENTRY_TRACES_SAMPLE_RATE` (deprecated) when unset. (default=1) |
| `LIGHTDASH_OTEL_ALWAYS_SAMPLE_AI_TRACES` | Set to `false` to stop always-sampling AI agent traces, so they follow the global sampling ratio instead. (default=true) |
| `LIGHTDASH_OTEL_DB_TRACES_ENABLED` | Adds spans for application database (Postgres) queries to traces, including the SQL statement. Requires OpenTelemetry mode. (default=false) |
| `LIGHTDASH_OTEL_DB_TRACES_MAX_QUERY_LENGTH` | Maximum length of the SQL statement recorded on database spans; longer statements are truncated. Must be a non-negative integer. (default=1022) |
| `OTEL_SDK_DISABLED` | Standard OpenTelemetry kill switch. When `true`, OpenTelemetry mode is off even if `LIGHTDASH_OTEL_TRACES_ENABLED=true`. |
| `OTEL_TRACES_EXPORTER` | Comma-separated exporters: `otlp`, `console`, `zipkin`, or `none`. Unsupported values are ignored with a startup warning; `none` overrides any other value. (default=otlp) |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | OTLP protocol: `grpc`, `http/json`, or `http/protobuf`. `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` takes precedence. Unsupported values fall back to the default. (default=http/protobuf) |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Base URL of your OTLP collector. Handled by the OpenTelemetry Node SDK, along with `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` and `OTEL_EXPORTER_OTLP_HEADERS`. |
| `OTEL_SERVICE_NAME` | Service name attached to exported spans. (default=lightdash) |
| `OTEL_LOG_LEVEL` | Enables OpenTelemetry SDK and exporter diagnostics at the given level (e.g. `DEBUG`). Lightdash redacts credentials, tokens, and span payloads from diagnostic output. |

<Note>
`OTEL_TRACES_SAMPLER` and `OTEL_TRACES_SAMPLER_ARG` are overridden by Lightdash. Control sampling with `LIGHTDASH_OTEL_TRACES_SAMPLE_RATE` instead.
</Note>
Loading