Skip to content

Fingerprint sanitization removes the key required by issue-activity rollups #285

Description

@MicroMilo

Summary

The proxy stamps superlog.issue_fingerprint as authoritative data before selecting either delivery branch, and the ClickHouse activity views require that exact attribute. However, the direct ClickHouse mapper unconditionally removes every superlog.* log and event attribute, while the collector removes the stamped log attribute as though it were client-supplied. The collector's trace policy is also asymmetric: it strips span/resource attributes but not event attributes, so collector trace fingerprints survive while equivalent log fingerprints do not.

Impact: On direct-to-ClickHouse delivery, both error-log and exception-event rows reach ClickHouse without superlog.issue_fingerprint, so both activity materialized views reject them. On collector delivery, every stamped error log loses the key and is rejected by the log activity view, while trace-event fingerprints survive. Incident statistics then query issue_activity_daily; an empty successful query returns zero buckets and does not invoke the raw-telemetry or issue-derived fallback.

Code path

Checked against current main at commit 5db3e5842bff9f3b4df005542d80da4608cf6817.

  • apps/proxy/src/ingest-queue.ts:776-809
  • apps/proxy/src/ingest-fingerprints.ts:144-198
  • apps/proxy/src/otlp-clickhouse.ts:105-115
  • apps/proxy/src/otlp-clickhouse.ts:216-222
  • apps/proxy/src/otlp-clickhouse.ts:251-257
  • infra/collector/config.yaml:35-46

Steps to reproduce

Validation level: current HEAD dynamic witness reproduced.

node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const fp={key:"superlog.issue_fingerprint",value:{stringValue:"fp16"}}; const control={key:"exception.type",value:{stringValue:"Error"}}; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:[control,fp]}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:[control,fp]}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],issue_activity_log_admitted:Boolean(logs[0].LogAttributes["superlog.issue_fingerprint"]),issue_activity_trace_admitted:Boolean(traces[0]["Events.Attributes"][0]["superlog.issue_fingerprint"]),neighbor_control_exception_type_survives:logs[0].LogAttributes["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["exception.type"]==="Error"}; console.log(JSON.stringify(out)); if(out.issue_activity_log_admitted||out.issue_activity_trace_admitted||!out.neighbor_control_exception_type_survives)process.exit(1);'

Observed output:

{"log_attributes":{"exception.type":"Error"},"trace_event_attributes":{"exception.type":"Error"},"issue_activity_log_admitted":false,"issue_activity_trace_admitted":false,"neighbor_control_exception_type_survives":true}

Neighboring control:

node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const attrs=[{key:"exception.type",value:{stringValue:"Error"}},{key:"user.id",value:{stringValue:"u1"}}]; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:attrs}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:attrs}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],ordinary_log_attributes_survive:logs[0].LogAttributes["exception.type"]==="Error"&&logs[0].LogAttributes["user.id"]==="u1",ordinary_trace_event_attributes_survive:traces[0]["Events.Attributes"][0]["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["user.id"]==="u1"}; console.log(JSON.stringify(out)); if(!out.ordinary_log_attributes_survive||!out.ordinary_trace_event_attributes_survive)process.exit(1);'

Control output:

{"log_attributes":{"exception.type":"Error","user.id":"u1"},"trace_event_attributes":{"exception.type":"Error","user.id":"u1"},"ordinary_log_attributes_survive":true,"ordinary_trace_event_attributes_survive":true}

Expected behavior

The documented/source-grounded contract should hold without the drift described above.

Actual behavior

Post-stamping sanitization removes the fingerprint required by issue-activity materialized views; B0229 is the direct-ingest subset.

Existing coverage

I checked the current issue and PR lists for overlapping titles/root-cause keywords before filing this. I did not find an item covering this same root cause.

Suggested fix

Sanitize untrusted reserved attributes before trusted fingerprint enrichment, then preserve the proxy-authored fingerprint through direct and collector delivery.

Suggested tests

  • Add a regression test for the reproduction above.
  • Add a neighboring control assertion so the intended non-bug path remains covered.

Submitted with Codex.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions