Summary
The proxy stamps superlog.issue_fingerprint as authoritative data before selecting either delivery branch, and the ClickHouse activity views require that exact attribute. However, the direct ClickHouse mapper unconditionally removes every superlog.* log and event attribute, while the collector removes the stamped log attribute as though it were client-supplied. The collector's trace policy is also asymmetric: it strips span/resource attributes but not event attributes, so collector trace fingerprints survive while equivalent log fingerprints do not.
Impact: On direct-to-ClickHouse delivery, both error-log and exception-event rows reach ClickHouse without superlog.issue_fingerprint, so both activity materialized views reject them. On collector delivery, every stamped error log loses the key and is rejected by the log activity view, while trace-event fingerprints survive. Incident statistics then query issue_activity_daily; an empty successful query returns zero buckets and does not invoke the raw-telemetry or issue-derived fallback.
Code path
Checked against current main at commit 5db3e5842bff9f3b4df005542d80da4608cf6817.
apps/proxy/src/ingest-queue.ts:776-809
apps/proxy/src/ingest-fingerprints.ts:144-198
apps/proxy/src/otlp-clickhouse.ts:105-115
apps/proxy/src/otlp-clickhouse.ts:216-222
apps/proxy/src/otlp-clickhouse.ts:251-257
infra/collector/config.yaml:35-46
Steps to reproduce
Validation level: current HEAD dynamic witness reproduced.
node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const fp={key:"superlog.issue_fingerprint",value:{stringValue:"fp16"}}; const control={key:"exception.type",value:{stringValue:"Error"}}; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:[control,fp]}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:[control,fp]}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],issue_activity_log_admitted:Boolean(logs[0].LogAttributes["superlog.issue_fingerprint"]),issue_activity_trace_admitted:Boolean(traces[0]["Events.Attributes"][0]["superlog.issue_fingerprint"]),neighbor_control_exception_type_survives:logs[0].LogAttributes["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["exception.type"]==="Error"}; console.log(JSON.stringify(out)); if(out.issue_activity_log_admitted||out.issue_activity_trace_admitted||!out.neighbor_control_exception_type_survives)process.exit(1);'
Observed output:
{"log_attributes":{"exception.type":"Error"},"trace_event_attributes":{"exception.type":"Error"},"issue_activity_log_admitted":false,"issue_activity_trace_admitted":false,"neighbor_control_exception_type_survives":true}
Neighboring control:
node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const attrs=[{key:"exception.type",value:{stringValue:"Error"}},{key:"user.id",value:{stringValue:"u1"}}]; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:attrs}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:attrs}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],ordinary_log_attributes_survive:logs[0].LogAttributes["exception.type"]==="Error"&&logs[0].LogAttributes["user.id"]==="u1",ordinary_trace_event_attributes_survive:traces[0]["Events.Attributes"][0]["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["user.id"]==="u1"}; console.log(JSON.stringify(out)); if(!out.ordinary_log_attributes_survive||!out.ordinary_trace_event_attributes_survive)process.exit(1);'
Control output:
{"log_attributes":{"exception.type":"Error","user.id":"u1"},"trace_event_attributes":{"exception.type":"Error","user.id":"u1"},"ordinary_log_attributes_survive":true,"ordinary_trace_event_attributes_survive":true}
Expected behavior
The documented/source-grounded contract should hold without the drift described above.
Actual behavior
Post-stamping sanitization removes the fingerprint required by issue-activity materialized views; B0229 is the direct-ingest subset.
Existing coverage
I checked the current issue and PR lists for overlapping titles/root-cause keywords before filing this. I did not find an item covering this same root cause.
Suggested fix
Sanitize untrusted reserved attributes before trusted fingerprint enrichment, then preserve the proxy-authored fingerprint through direct and collector delivery.
Suggested tests
- Add a regression test for the reproduction above.
- Add a neighboring control assertion so the intended non-bug path remains covered.
Submitted with Codex.
Summary
The proxy stamps
superlog.issue_fingerprintas authoritative data before selecting either delivery branch, and the ClickHouse activity views require that exact attribute. However, the direct ClickHouse mapper unconditionally removes everysuperlog.*log and event attribute, while the collector removes the stamped log attribute as though it were client-supplied. The collector's trace policy is also asymmetric: it strips span/resource attributes but not event attributes, so collector trace fingerprints survive while equivalent log fingerprints do not.Impact: On direct-to-ClickHouse delivery, both error-log and exception-event rows reach ClickHouse without
superlog.issue_fingerprint, so both activity materialized views reject them. On collector delivery, every stamped error log loses the key and is rejected by the log activity view, while trace-event fingerprints survive. Incident statistics then queryissue_activity_daily; an empty successful query returns zero buckets and does not invoke the raw-telemetry or issue-derived fallback.Code path
Checked against current
mainat commit5db3e5842bff9f3b4df005542d80da4608cf6817.apps/proxy/src/ingest-queue.ts:776-809apps/proxy/src/ingest-fingerprints.ts:144-198apps/proxy/src/otlp-clickhouse.ts:105-115apps/proxy/src/otlp-clickhouse.ts:216-222apps/proxy/src/otlp-clickhouse.ts:251-257infra/collector/config.yaml:35-46Steps to reproduce
Validation level: current HEAD dynamic witness reproduced.
node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const fp={key:"superlog.issue_fingerprint",value:{stringValue:"fp16"}}; const control={key:"exception.type",value:{stringValue:"Error"}}; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:[control,fp]}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:[control,fp]}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],issue_activity_log_admitted:Boolean(logs[0].LogAttributes["superlog.issue_fingerprint"]),issue_activity_trace_admitted:Boolean(traces[0]["Events.Attributes"][0]["superlog.issue_fingerprint"]),neighbor_control_exception_type_survives:logs[0].LogAttributes["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["exception.type"]==="Error"}; console.log(JSON.stringify(out)); if(out.issue_activity_log_admitted||out.issue_activity_trace_admitted||!out.neighbor_control_exception_type_survives)process.exit(1);'Observed output:
Neighboring control:
node --input-type=module -e 'import { otlpLogsToRows, otlpTracesToRows } from "./apps/proxy/src/otlp-clickhouse.ts"; const attrs=[{key:"exception.type",value:{stringValue:"Error"}},{key:"user.id",value:{stringValue:"u1"}}]; const logs=otlpLogsToRows({resourceLogs:[{scopeLogs:[{logRecords:[{timeUnixNano:"1000000000",severityNumber:17,attributes:attrs}]}]}]},"p"); const traces=otlpTracesToRows({resourceSpans:[{scopeSpans:[{spans:[{startTimeUnixNano:"1000000000",endTimeUnixNano:"2000000000",events:[{name:"exception",timeUnixNano:"1000000000",attributes:attrs}]}]}]}]},"p"); const out={log_attributes:logs[0].LogAttributes,trace_event_attributes:traces[0]["Events.Attributes"][0],ordinary_log_attributes_survive:logs[0].LogAttributes["exception.type"]==="Error"&&logs[0].LogAttributes["user.id"]==="u1",ordinary_trace_event_attributes_survive:traces[0]["Events.Attributes"][0]["exception.type"]==="Error"&&traces[0]["Events.Attributes"][0]["user.id"]==="u1"}; console.log(JSON.stringify(out)); if(!out.ordinary_log_attributes_survive||!out.ordinary_trace_event_attributes_survive)process.exit(1);'Control output:
Expected behavior
The documented/source-grounded contract should hold without the drift described above.
Actual behavior
Post-stamping sanitization removes the fingerprint required by issue-activity materialized views; B0229 is the direct-ingest subset.
Existing coverage
I checked the current issue and PR lists for overlapping titles/root-cause keywords before filing this. I did not find an item covering this same root cause.
Suggested fix
Sanitize untrusted reserved attributes before trusted fingerprint enrichment, then preserve the proxy-authored fingerprint through direct and collector delivery.
Suggested tests
Submitted with Codex.