You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 77b18b0
Browse filesBrowse the repository at this point in the historyBrowse files
fix(ci): enable OTLP export only when a collector is requested
Kubernetes E2E layered ci/values-skaffold.yaml, which pointed the gateway
at the openshell-collector service that only the local k3d workflow
installs. Kind lanes logged a BatchSpanProcessor.ExportError every batch.
The local gateway tasks probed 127.0.0.1:4317 instead, and the gateway
task configured the endpoint unconditionally.
Move the collector endpoint into ci/values-local-tracing.yaml, layered only
by skaffold.yaml, and have every local gateway task enable OTLP export only
when OPENSHELL_DEV_OTLP_ENDPOINT is set.
Closes#5347
Refs #3767
Signed-off-by: Kris Hicks <khicks@nvidia.com>
Copy file name to clipboardExpand all lines: crates/openshell-driver-vm/README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -70,7 +70,7 @@ By default `mise run gateway:vm`:
70
70
- Persists the gateway SQLite DB under `.cache/gateway-vm/gateway.db`.
71
71
- Places the VM driver state (per-sandbox `overlay.ext4`, image cache, and `run/compute-driver.sock`) under `/tmp/openshell-vm-driver-$USER-vm-dev/` so the AF_UNIX socket path stays under macOS `SUN_LEN`.
72
72
- Writes `.cache/gateway-vm/gateway.toml` with `[openshell.drivers.vm].driver_dir = "$PWD/target/debug"` so the freshly built `openshell-driver-vm` is used instead of an older installed copy from `~/.local/libexec/openshell`, `/usr/libexec/openshell`, or `/usr/local/libexec`.
73
-
- Enables OTLP trace export, OCSF log export, and the agent trace relay to `http://127.0.0.1:4317` only when a local collector is listening there. Otherwise, it omits the OTLP configuration to avoid repeated export failures.
73
+
- Enables OTLP trace export, OCSF log export, and the agent trace relay only when `OPENSHELL_DEV_OTLP_ENDPOINT` names a collector, for example `http://127.0.0.1:4317`. Otherwise, it omits the OTLP configuration.
74
74
75
75
For GPU passthrough (VFIO), pass `-- --gpu` and run with root privileges:
Copy file name to clipboardExpand all lines: docs/how-it-works/gateways/configuration.mdx
+4-3Lines changed: 4 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -786,9 +786,10 @@ the trace UI to local port `18888`. When Skaffold has deployed a Kubernetes
786
786
gateway, the task also forwards it to local port `8090`; otherwise it continues
787
787
with the collector ports only. The Skaffold run task registers the forwarded
788
788
gateway under the worktree-specific k3d cluster name; select it with
789
-
`openshell gateway select <name>`. The local Podman, Docker, and VM gateway
790
-
tasks export spans and OCSF log records to the forwarded receiver
791
-
automatically.
789
+
`openshell gateway select <name>`. To export spans and OCSF log records from
790
+
the local `gateway`, `gateway:docker`, `gateway:podman`, or `gateway:vm` task to
791
+
the forwarded receiver, set `OPENSHELL_DEV_OTLP_ENDPOINT=http://127.0.0.1:4317`
792
+
when starting it. Without it, those tasks leave OTLP export disabled.
792
793
793
794
In-process compute drivers read their backend-specific settings from `[openshell.drivers.<name>]`. An external driver's gateway table supplies only its `socket_path`; configure the external driver process itself through that binary's flags or environment variables. A driver that advertises `supports_sandbox_authentication` may authenticate an opaque bootstrap credential through the compute-driver protocol. It must return the same non-empty runtime identity from sandbox creation and credential authentication. Start requests include the previously recorded opaque identity so the driver can preserve the stable resource while rotating generation-specific compute. For `IssueSandboxToken`, the gateway verifies the sandbox exists and the driver/runtime identity matches the durable provisioning record before returning a generation-bound session JWT. The in-process Kubernetes driver reads `service_account_name`, `workspace_mode`, and namespace discovery from `[openshell.drivers.kubernetes]`; an external Kubernetes driver receives the equivalent values through its own CLI or environment contract.
| Collector receives no OCSF log records, or `openshell.ocsf.gap` records | `[openshell.gateway.otlp.ocsf]` missing or without `enabled = true`, an unreachable endpoint or one that does not speak the table's `protocol`, or export failures and queue pressure | Check `[openshell.gateway.otlp.ocsf]` for `enabled = true` and the endpoint it resolves to, the gateway's `OTLP exporting enabled` startup log for `ocsf_endpoint` and `ocsf_protocol`, export warnings, and `openshell_ocsf_otlp_*` metrics. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
1584
1584
| Gateway OCSF JSONL stops growing, has gaps, or is rejected by a SIEM | File errors, queue pressure, or shipper falling behind retention | Inspect gateway warnings and `openshell_ocsf_log_*` metrics; check `[openshell.gateway.ocsf_log]`, directory permissions, free space, per-replica paths, and shipper rotation checkpoints. See the published [gateway configuration reference](https://docs.nvidia.com/openshell/latest/how-it-works/gateways/configuration). |
1585
-
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with OTLP configured but the collector forwarding task later stopped, or the config was created manually | Restart `gateway:docker`, `gateway:podman`, or `gateway:vm` so it re-detects the listener; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
1585
+
| `BatchSpanProcessor.ExportError` repeatedly reports connection refused on `127.0.0.1:4317` | The local gateway started with `OPENSHELL_DEV_OTLP_ENDPOINT` set but no collector is listening, for example because `helm:k3s:forward` stopped | Start the collector forwarding again, or restart the gateway task without `OPENSHELL_DEV_OTLP_ENDPOINT`; inspect the generated `gateway.toml` for `[openshell.gateway.otlp]` |
1586
1586
| Sandbox log shows `OTLP agent trace relay inactive: not enabled by the gateway` at startup, then `agent OTLP export attempted but the agent trace relay is not enabled` when the agent exports | The gateway does not enable `[openshell.gateway.otlp.agent]`, so the supervisor refuses the workload's OTLP relay address | Set `enabled = true` in `[openshell.gateway.otlp.agent]` (Helm: `gatewayConfig."openshell.gateway.otlp.agent".enabled`) with an endpoint from that table or `[openshell.gateway.otlp]`, and recreate the sandbox; `openshell logs <sandbox> --source sandbox` then shows `OTLP agent trace relay enabled` |
0 commit comments