Skip to content

Commit 77b18b0

Browse files
committed
fix(ci): enable OTLP export only when a collector is requested
Kubernetes E2E layered ci/values-skaffold.yaml, which pointed the gateway at the openshell-collector service that only the local k3d workflow installs. Kind lanes logged a BatchSpanProcessor.ExportError every batch. The local gateway tasks probed 127.0.0.1:4317 instead, and the gateway task configured the endpoint unconditionally. Move the collector endpoint into ci/values-local-tracing.yaml, layered only by skaffold.yaml, and have every local gateway task enable OTLP export only when OPENSHELL_DEV_OTLP_ENDPOINT is set. Closes #5347 Refs #3767 Signed-off-by: Kris Hicks <khicks@nvidia.com>
1 parent e9ba8d9 commit 77b18b0

12 files changed

Lines changed: 62 additions & 91 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -266,8 +266,9 @@ task running while using those endpoints.
266266
### Viewing local traces
267267

268268
The gateway exports OTLP/gRPC to
269-
`http://openshell-collector.observability.svc.cluster.local:4317` through the
270-
default Skaffold values. Forward OTLP/gRPC and the trace UI to the host:
269+
`http://openshell-collector.observability.svc.cluster.local:4317` through
270+
`ci/values-local-tracing.yaml`, which only `skaffold.yaml` layers. The Kubernetes
271+
E2E wrapper omits it, so E2E clusters export no traces. Forward OTLP/gRPC and the trace UI to the host:
271272

272273
```bash
273274
mise run helm:k3s:forward
@@ -278,9 +279,10 @@ Kubernetes compute-driver spans under their distinct service names, along with
278279
Agent Sandbox controller reconciliation spans linked through the Sandbox
279280
trace-context annotation. The same command exposes OTLP/gRPC on
280281
`http://127.0.0.1:4317` and, when deployed, the Kubernetes gateway on
281-
`http://127.0.0.1:8090`. The local `gateway:docker`, `gateway:podman`, and
282-
`gateway:vm` tasks detect the collector listener at startup and enable trace
283-
export, OCSF log export, and the agent trace relay only while it is reachable.
282+
`http://127.0.0.1:8090`. The local `gateway`, `gateway:docker`,
283+
`gateway:podman`, and `gateway:vm` tasks enable trace export, OCSF log export,
284+
and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` is set, for
285+
example `OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317`.
284286
Spans an agent exports from inside a sandbox arrive through the supervisor's
285287
relay with `openshell.telemetry.source=agent` and share `openshell.sandbox.id`
286288
with that sandbox's supervisor spans.
@@ -642,6 +644,7 @@ for dependencies still declared in `Chart.yaml`.
642644
| `deploy/helm/openshell/skaffold.yaml` | Skaffold config — images, Helm releases, values overlays |
643645
| `deploy/helm/openshell/values.yaml` | Default Helm values |
644646
| `deploy/helm/openshell/ci/values-skaffold.yaml` | Dev overrides (image pull policy, TLS disabled for local Skaffold) |
647+
| `deploy/helm/openshell/ci/values-local-tracing.yaml` | OTLP export to the `helm:k3s:create` collector (Skaffold only) |
645648
| `deploy/helm/openshell/ci/values-cert-manager.yaml` | cert-manager PKI overlay (opt-in; disables pkiInitJob) |
646649
| `deploy/helm/openshell/ci/values-gateway.yaml` | Envoy Gateway GRPCRoute + Gateway overlay |
647650
| `deploy/helm/openshell/ci/values-high-availability.yaml` | HA test overlay (`replicaCount: 2` with external PostgreSQL Secret) |

‎crates/openshell-driver-vm/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -70,7 +70,7 @@ By default `mise run gateway:vm`:
7070
- Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`.
7171
- Places the VM driver state (per-sandbox `overlay.ext4`, image cache, and `run/compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`.
7272
- Writes `.cache/gateway-vm/gateway.toml` with `[openshell.drivers.vm].driver_dir = "$PWD/target/debug"` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`.
73-
- Enables OTLP trace export, OCSF log export, and the agent trace relay to `http://127.0.0.1:4317` only when a local collector is listening there. Otherwise, it omits the OTLP configuration to avoid repeated export failures.
73+
- Enables OTLP trace export, OCSF log export, and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` names a collector, for example `http://127.0.0.1:4317`. Otherwise, it omits the OTLP configuration.
7474

7575
For GPU passthrough (VFIO), pass `-- --gpu` and run with root privileges:
7676

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Export gateway traces to the collector that `mise run helm:k3s:create`
5+
# installs. Layer only onto clusters that run that collector.
6+
gatewayConfig:
7+
openshell.gateway.otlp:
8+
endpoint: http://openshell-collector.observability.svc.cluster.local:4317
9+
openshell.gateway.otlp.agent:
10+
enabled: true

‎deploy/helm/openshell/ci/values-skaffold.yaml‎

Lines changed: 0 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,10 @@
55
gatewayConfig:
66
openshell.drivers.kubernetes:
77
image_pull_policy: if_not_present
8-
openshell.gateway.otlp:
9-
endpoint: http://openshell-collector.observability.svc.cluster.local:4317
10-
openshell.gateway.otlp.agent:
11-
enabled: true
128
openshell.gateway.auth:
139
allow_unauthenticated_users: true
1410

1511
server:
16-
otlp:
17-
endpoint: http://openshell-collector.observability.svc.cluster.local:4317
1812
# Comment out to enforce mTLS (uses PKI secrets generated by pkiInitJob).
1913
disableTls: true
2014
auth:

‎deploy/helm/openshell/skaffold.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -124,6 +124,7 @@ deploy:
124124
valuesFiles:
125125
- values.yaml
126126
- ci/values-skaffold.yaml
127+
- ci/values-local-tracing.yaml
127128
# Add ci/values-cert-manager.yaml here (and uncomment the cert-manager
128129
# release above) to switch TLS generation to cert-manager.
129130
#- ci/values-cert-manager.yaml

‎docs/how-it-works/gateways/configuration.mdx‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -786,9 +786,10 @@ the trace UI to local port `18888`. When Skaffold has deployed a Kubernetes
786786
gateway, the task also forwards it to local port `8090`; otherwise it continues
787787
with the collector ports only. The Skaffold run task registers the forwarded
788788
gateway under the worktree-specific k3d cluster name; select it with
789-
`openshell gateway select <name>`. The local Podman, Docker, and VM gateway
790-
tasks export spans and OCSF log records to the forwarded receiver
791-
automatically.
789+
`openshell gateway select <name>`. To export spans and OCSF log records from
790+
the local `gateway`, `gateway:docker`, `gateway:podman`, or `gateway:vm` task to
791+
the forwarded receiver, set `OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317`
792+
when starting it. Without it, those tasks leave OTLP export disabled.
792793

793794
In-process compute drivers read their backend-specific settings from `[openshell.drivers.<name>]`. An external driver's gateway table supplies only its `socket_path`; configure the external driver process itself through that binary's flags or environment variables. A driver that advertises `supports_sandbox_authentication` may authenticate an opaque bootstrap credential through the compute-driver protocol. It must return the same non-empty runtime identity from sandbox creation and credential authentication. Start requests include the previously recorded opaque identity so the driver can preserve the stable resource while rotating generation-specific compute. For `IssueSandboxToken`, the gateway verifies the sandbox exists and the driver/runtime identity matches the durable provisioning record before returning a generation-bound session JWT. The in-process Kubernetes driver reads `service_account_name`, `workspace_mode`, and namespace discovery from `[openshell.drivers.kubernetes]`; an external Kubernetes driver receives the equivalent values through its own CLI or environment contract.
794795

‎skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1582,7 +1582,7 @@ credential failures.
15821582
| `openshell status` fails | Gateway endpoint unreachable or auth mismatch | `openshell gateway info`, gateway logs |
15831583
| Collector receives no OCSF log records, or `openshell.ocsf.gap` records | `[openshell.gateway.otlp.ocsf]` missing or without `enabled = true`, an unreachable endpoint or one that does not speak the table's `protocol`, or export failures and queue pressure | Check `[openshell.gateway.otlp.ocsf]` for `enabled = true` and the endpoint it resolves to, the gateway's `OTLP exporting enabled` startup log for `ocsf_endpoint` and `ocsf_protocol`, export warnings, and `openshell_ocsf_otlp_*` metrics. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
15841584
| Gateway OCSF JSONL stops growing, has gaps, or is rejected by a SIEM | File errors, queue pressure, or shipper falling behind retention | Inspect gateway warnings and `openshell_ocsf_log_*` metrics; check `[openshell.gateway.ocsf_log]`, directory permissions, free space, per-replica paths, and shipper rotation checkpoints. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
1585-
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
1585+
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with `OPENSHELL_DEV_OTLP_ENDPOINT` set but no collector is listening, for example because `helm:k3s:forward` stopped | Start the collector forwarding again, or restart the gateway task without `OPENSHELL_DEV_OTLP_ENDPOINT`; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
15861586
| Sandbox log shows `OTLP agent trace relay inactive: not enabled by the gateway` at startup, then `agent OTLP export attempted but the agent trace relay is not enabled` when the agent exports | The gateway does not enable `[openshell.gateway.otlp.agent]`, so the supervisor refuses the workload's OTLP relay address | Set `enabled = true` in `[openshell.gateway.otlp.agent]` (Helm: `gatewayConfig."openshell.gateway.otlp.agent".enabled`) with an endpoint from that table or `[openshell.gateway.otlp]`, and recreate the sandbox; `openshell logs <sandbox> --source sandbox` then shows `OTLP agent trace relay enabled` |
15871587
| Gateway starts but sandbox create fails | Compute driver cannot reach runtime | Docker/Podman/Kubernetes/VM driver logs |
15881588
| Docker or Podman sandbox never registers | Wrong gateway endpoint, unavailable host networking, or supervisor startup failure | Gateway logs and supervisor container logs |

‎tasks/scripts/gateway-docker.sh‎

Lines changed: 2 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
# OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:docker
1919
# OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:docker
2020
# OPENSHELL_SANDBOX_RUNTIME_IMAGE=ghcr.io/... mise run gateway:docker
21+
# OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317 mise run gateway:docker
2122
#
2223
# After the gateway is running, point the CLI at it with either:
2324
# openshell --gateway docker-dev <command>
@@ -83,27 +84,6 @@ ensure_docker_runtime_image() {
8384
fi
8485
}
8586

86-
append_local_otlp_config_if_available() {
87-
local config_path=$1
88-
if ! port_is_in_use 4317; then
89-
echo "OTLP collector not detected on 127.0.0.1:4317; trace export, OCSF log export, and the agent trace relay disabled."
90-
return
91-
fi
92-
93-
cat >>"${config_path}" <<'EOF'
94-
95-
[openshell.gateway.otlp]
96-
endpoint = "http://127.0.0.1:4317"
97-
98-
[openshell.gateway.otlp.ocsf]
99-
enabled = true
100-
101-
[openshell.gateway.otlp.agent]
102-
enabled = true
103-
EOF
104-
echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for http://127.0.0.1:4317."
105-
}
106-
10787
register_gateway_metadata() {
10888
local name=$1
10989
local endpoint=$2
@@ -231,7 +211,7 @@ if [[ -n "${OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET+x}" ]]; then
231211
printf 'provider_spiffe_workload_api_socket = "%s"\n' "$(toml_escape "${OPENSHELL_PROVIDER_SPIFFE_WORKLOAD_API_SOCKET}")" >>"${CONFIG_PATH}"
232212
fi
233213

234-
append_local_otlp_config_if_available "${CONFIG_PATH}"
214+
append_dev_otlp_config "${CONFIG_PATH}"
235215

236216
GATEWAY_ENDPOINT="http://127.0.0.1:${PORT}"
237217
register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}"

‎tasks/scripts/gateway-podman.sh‎

Lines changed: 2 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@
1919
# OPENSHELL_SANDBOX_IMAGE=ghcr.io/... mise run gateway:podman
2020
# OPENSHELL_SUPERVISOR_IMAGE=ghcr.io/... mise run gateway:podman
2121
# OPENSHELL_SANDBOX_RUNTIME_IMAGE=ghcr.io/... mise run gateway:podman
22+
# OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317 mise run gateway:podman
2223

2324
set -euo pipefail
2425

@@ -112,27 +113,6 @@ port_is_in_use() {
112113
(echo >/dev/tcp/127.0.0.1/"${port}") >/dev/null 2>&1
113114
}
114115

115-
append_local_otlp_config_if_available() {
116-
local config_path=$1
117-
if ! port_is_in_use 4317; then
118-
echo "OTLP collector not detected on 127.0.0.1:4317; trace export, OCSF log export, and the agent trace relay disabled."
119-
return
120-
fi
121-
122-
cat >>"${config_path}" <<'EOF'
123-
124-
[openshell.gateway.otlp]
125-
endpoint = "http://127.0.0.1:4317"
126-
127-
[openshell.gateway.otlp.ocsf]
128-
enabled = true
129-
130-
[openshell.gateway.otlp.agent]
131-
enabled = true
132-
EOF
133-
echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for http://127.0.0.1:4317."
134-
}
135-
136116
register_gateway_metadata() {
137117
local name=$1
138118
local endpoint=$2
@@ -273,7 +253,7 @@ if [[ -n "${OPENSHELL_SANDBOX_PROXY_CA_BUNDLE+x}" ]]; then
273253
printf 'proxy_ca_bundle = "%s"\n' "$(toml_escape "${OPENSHELL_SANDBOX_PROXY_CA_BUNDLE}")" >>"${CONFIG_PATH}"
274254
fi
275255

276-
append_local_otlp_config_if_available "${CONFIG_PATH}"
256+
append_dev_otlp_config "${CONFIG_PATH}"
277257

278258
GATEWAY_ENDPOINT="http://${CLI_ENDPOINT_HOST}:${PORT}"
279259
register_gateway_metadata "${GATEWAY_NAME}" "${GATEWAY_ENDPOINT}" "${PORT}"

‎tasks/scripts/gateway-toml.sh‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,3 +17,27 @@ toml_escape() {
1717
value=${value//$'\t'/\\t}
1818
printf '%s' "${value}"
1919
}
20+
21+
# Append OTLP export to a gateway config when OPENSHELL_DEV_OTLP_ENDPOINT is
22+
# set. Export stays off otherwise so no collector is assumed.
23+
append_dev_otlp_config() {
24+
local config_path=$1
25+
local endpoint=${OPENSHELL_DEV_OTLP_ENDPOINT:-}
26+
if [[ -z "${endpoint}" ]]; then
27+
echo "OTLP export disabled; set OPENSHELL_DEV_OTLP_ENDPOINT to enable it."
28+
return
29+
fi
30+
31+
cat >>"${config_path}" <<TOML
32+
33+
[openshell.gateway.otlp]
34+
endpoint = "$(toml_escape "${endpoint}")"
35+
36+
[openshell.gateway.otlp.ocsf]
37+
enabled = true
38+
39+
[openshell.gateway.otlp.agent]
40+
enabled = true
41+
TOML
42+
echo "OTLP trace export, OCSF log export, and the agent trace relay enabled for ${endpoint}."
43+
}

0 commit comments

Comments
 (0)