metrics/gcp: three silent-loss gaps #4267 leaves open — shared export timeout, unknown aggregation type, and a 95s startup block #4293

Three gaps in pkg/gofr/metrics/exporters/gcp that #4267 does not close. None of them blocks #4267 — it is a large improvement on what it replaces, and I approved it. They are grouped because they share a failure mode: each one loses metrics without anything failing loudly, which is the property #4266 was about in the first place.

Numbers below are measured, not estimated. The harness is the example app (examples/using-gcp-metrics, unmodified) run against a TLS gRPC OTLP receiver that counts data points and rejects any request over 200 with Google's own message, plus a local OAuth token endpoint so the exporter's real ADC path runs.

1 — Every chunk of one collection shares a single export timeout

PeriodicReader.collectAndExport wraps collect and export in one r.timeout (sdk/metric@v1.46.0 periodic_reader.go:244, default 30s at :24) and then calls Export once (:262). #4267's splitter loops inside that single call, so all N requests draw on one budget. Three lines above, the SDK's own batcher does the opposite and says why:

// periodic_reader.go:257-259
// The export timeout is applied individually to each batch by using
// the original context.
err = errors.Join(err, r.exportWithTimeout(originalCtx, batch))

This is reachable at defaults. otlpmetricgrpc@v1.44.0 retries by default — Enabled: true, InitialInterval: 5s, MaxInterval: 30s, MaxElapsedTime: 1m (internal/retry/retry.go:21-26) — on Unavailable, Aborted, DataLoss, OutOfRange, DeadlineExceeded, and on ResourceExhausted when the server returns RetryInfo (client.go:187-197). One retried chunk costs at least 5s of the shared 30s, and Google's ingest is exactly the kind of endpoint that answers RESOURCE_EXHAUSTED with RetryInfo.

Measured

Identical binary, identical load (700 distinct paths → ~504 points → 3 chunks), receiver taking 1s per request, OTEL_METRIC_EXPORT_TIMEOUT=2500. The only variable is where the split happens:

06:34:00 REQ #1  points=200  ACCEPT
06:34:01 REQ #2  points=200  ACCEPT
06:34:02 REQ (aborted by client after 1s): context canceled
06:34:10 REQ #3  points=200  ACCEPT
06:34:11 REQ #4  points=200  ACCEPT
06:34:12 REQ (aborted by client after 1s): context deadline exceeded

requests_total — the application's own counter — never arrived once, across the whole run. The control delivered it every cycle with zero errors.

Why it does not self-heal

Cumulative temporality normally rescues a dropped export: the next interval re-sends every cumulative value. That only works when the loss is random. The split is deterministic — stable scope, metric and point order — so a persistently short budget cuts the same tail every cycle, for as long as the condition lasts. The backend keeps receiving 80% of the metrics and looks healthy.

Constraint worth recording

The split cannot simply be moved to the Reader, where the SDK puts its own. metricSdk.Reader has four unexported methods — register, temporality, aggregation, cardinalityLimit — so no package outside sdk/metric can implement it. Wrapping the Exporter is the only layer available to us, and this gap is a consequence of that, not of a bad choice.

Options, cheapest first

• Turn off transport retry: otlpmetricgrpc.WithRetry(otlpmetricgrpc.RetryConfig{Enabled: false}) (exported, config.go:37,258). Under cumulative temporality an in-cycle retry recovers one sample while costing the tail of the same collection. One line, and the argument is specific to this exporter.
• Rotate the starting chunk between cycles, so a short budget loses a different slice each time rather than the same one.
• Document it in the wrapper's doc comment: the position forces one shared budget, the SDK's batcher does not share it, and the loss is ordered rather than random.

2 — An aggregation type the SDK adds later silently reopens #4266

chunk.go:165-167 counts an unhandled metricdata.Aggregation as one point, and sliceMetric passes it through whole. The nine cases are exhaustive today — metricdata.Aggregation is sealed by an unexported privateAggregation() (metricdata/data.go:47-48) and has exactly five implementing types, three of them generic over int64|float64.

If a future sdk/metric adds a tenth, a metric of N points is budgeted as 1, the chunk reaches up to 199 + N, and the Telemetry API rejects the whole request — #4266 again, arriving on a dependency bump with nothing to announce it.

Nothing in the repo catches it. Both directions checked:

• The linter does not. exhaustive is enabled (.golangci.yml:14) but covers enum switches, not type switches over an interface. Deleting the metricdata.Summary case gives golangci-lint run → 0 issues.
• The tests catch a removed case, not an added one. That same deletion fails Test_splitResourceMetrics/spans_metrics,_scopes_and_every_aggregation, so today's nine are genuinely pinned. There is no Go construct that fails when a tenth appears.

Cheapest bound: give an unknown aggregation a chunk to itself instead of a seat in someone else's, so the degradation stays one metric wide rather than taking the request it lands in. A metricPoints returning (n int, known bool) and a flush on !known is about six lines.

3 — Startup blocks for ~95 seconds when the metadata server is wedged

Booted with GCE_METADATA_HOST pointed at a listener that accepts and never answers, and no key file, so ADC must reach the metadata server. Timestamps are from the app's own log:

Identical, so #4267 neither causes nor worsens this — it is pre-existing. buildReader calls google.FindDefaultCredentials(ctx, cloudPlatformScope) with the app's startup context and nothing caps it (gcp.go:152), and cachingDetector.Detect (gcp.go:88-90) delegates to gcpdetect.NewDetector().Detect(ctx), whose context is discarded.

The whole process serves nothing for a minute and a half: no HTTP listener, no health endpoint. On a platform that health-checks a new revision, that is a failed deploy rather than a slow one.

#4207 has already solved exactly this for the traces twin, bounding both steps at 2 × metadataTimeout = 10 s with awaitWithin. Once it lands, the same shape transfers here nearly verbatim — including the reason a deadline on the context does not work: contrib/detectors/gcp@v1.46.0 detector.go:35 takes context.Context unnamed and discards it, and compute/metadata@v0.9.0 metadata.go:125-127 has OnGCE() call OnGCEWithContext(context.Background()).

Related, already tracked

docs/memory/scars.md:28-29 records that a scheme-bearing endpoint passed to WithEndpoint fails at export time rather than startup, and notes the pattern exists in both gcp exporter modules. It is still live here — buildReader defaults only on "" and hands the value to otlpmetricgrpc.WithEndpoint unvalidated (gcp.go:157-163). #4207 is closing the traces half, including the strings.TrimSpace case; after it lands this is the last place it survives.