Skip to content

fix(otlp-grpc): send generator resource and real severity on logs, metrics and traces - #329

Open
adnanrahic wants to merge 2 commits into
observIQ:mainfrom
adnanrahic:fix/otlp-grpc-log-resource
Open

adnanrahic wants to merge 2 commits into
observIQ:mainfrom
adnanrahic:fix/otlp-grpc-log-resource

Conversation

@adnanrahic

@adnanrahic adnanrahic commented Oct 7, 2026 •

Copy link
Copy Markdown

Proposed Change

The OTLP gRPC output dropped data every generator already provides, and the hostmetrics generator ignored its configured OS. Collectors couldn't route on the source of a record, and most severities arrived as INFO.

1. Resource was never sent (logs, metrics, traces)

Generators attach a resource (host.name, telemetry.source, apache.format, ...) in Metadata.Resource, but the OTLP gRPC output ignored it:

  • logs: every batch got a hard-coded service.name=blitz resource; the record carried only the body plus empty environment/location attributes. Per-record Metadata.Attributes were dropped too.
  • metrics: buildMetricRequest(metrics, nil), so host metrics arrived without host.name.
  • traces: hard-coded service.name=blitz resource.

Found while running Blitz into a Bindplane gateway that routes on resource.attributes["telemetry.source"]: no condition ever matched, so every log fell through to the catch-all route.

Each queued log record, metric and span now carries its generator's resource (generic entry[T] in resource.go). At send time the batch is split into one ResourceLogs / ResourceMetrics / ResourceSpans per distinct resource, in order of first appearance. service.name=blitz is still added unless the generator sets its own. Log records now also carry Metadata.Attributes.

2. Severity matching was uppercase-only

mapSeverityNumber only matched exact DEBUG/INFO/WARN/ERROR/FATAL, so these all became INFO:

Generator Levels
kubernetes warn, error (lowercase)
apache error crit, error, warn, notice
wel Warning, Error, Critical, Verbose
postgres WARNING, PANIC, LOG, NOTICE

Matching is now case-insensitive and covers those names. Unknown levels still map to INFO. Severity text is unchanged.

3. generator.hostmetrics.os was ignored

Since generators draw their host from the simulated environment (PIPE-1036), hostmetrics used whatever system SystemForKey("hostmetrics") selected, including that system's OS. With the default randomized fleet that is often Windows, so BLITZ_GENERATOR_HOSTMETRICS_OS=linux (also the default) still produced Windows metrics.

New Environment.SystemForKeyWithOS does the same key-hashed selection restricted to systems running a given OS. Hostmetrics uses it with the configured OS (empty means linux), and falls back to its synthetic host when no system runs that OS.

Verification

  • New tests: TestEnvironmentSystemForKeyWithOS and TestHostMetricsIdentityHonorsConfiguredOS (OS-restricted host selection), resource_test.go (resource reaches the OTLP resource for logs, metrics and traces; grouping by resource; no duplicate service.name) and severity_test.go (table of every level name above).
  • go test ./..., go vet, gofmt and revive -config .revive.toml are clean.
  • End to end: ran the hostmetrics, traces and nginx generators against an OTel Collector with the debug exporter. All three arrived with host.name and telemetry.source on the resource. Against a Bindplane gateway, the routing connector stopped sending everything to its catch-all route, and Kubernetes warn/error lines now match severity_number >= SEVERITY_NUMBER_WARN. The hostmetrics generator, previously started as os.type: windows despite BLITZ_GENERATOR_HOSTMETRICS_OS=linux, now starts as linux.
Checklist
  • Changes are tested
  • CI has passed

🤖 Generated with Claude Code

…trics and traces

The OTLP gRPC output dropped the resource every generator attaches
(Metadata.Resource): logs and traces were always sent with a hard-coded
service.name=blitz resource, and metrics with buildMetricRequest(metrics, nil).
host.name, telemetry.source, apache.format and the rest never reached the
collector, so pipelines could not route or filter on them. Per-record
Metadata.Attributes on logs were dropped too.

Each queued log record, metric and span now carries its generator's
resource. A batch is split into one ResourceLogs / ResourceMetrics /
ResourceSpans per distinct resource, in order of first appearance.
service.name=blitz is still added unless the generator sets its own.

mapSeverityNumber only matched exact uppercase DEBUG/INFO/WARN/ERROR/FATAL,
so lowercase levels (kubernetes, apache error), Windows Event Log level
names (Warning, Error, Critical), PostgreSQL WARNING/PANIC/LOG and syslog
notice/crit all became INFO. Matching is now case-insensitive and covers
those names.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@adnanrahic
adnanrahic requested review from a team as code owners October 7, 2026 11:52
…et host

Since generators draw their host identity from the simulated environment
(PIPE-1036), hostmetrics took whatever system SystemForKey("hostmetrics")
selected and used that system's OS, ignoring generator.hostmetrics.os. With
the default randomized fleet that is often a Windows host, so
BLITZ_GENERATOR_HOSTMETRICS_OS=linux (also the default) still produced
Windows metrics.

Add Environment.SystemForKeyWithOS, the same key-hashed selection restricted
to systems running a given OS, and use it for hostmetrics with the configured
OS (empty means linux). When no system runs that OS the generator falls back
to its synthetic host built from OS and Hostname.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@Dylan-M Dylan-M left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest closing. These fixes are already in progress.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants