feat: add trace stats computation by duncanpharvey · Pull Request #51 · DataDog/serverless-components

duncanpharvey · 2025-12-03T21:21:12Z

What does this PR do?

Add trace stats computation in the Serverless Compatibility Layer.
Disables tracer computed stats

Motivation

Some tracers support trace stats computation, but all don't support it, or have it disabled by default. Trace stats computation in the Serverless Compatibility Layer allows all environments supported by this package to have trace stats.

https://datadoghq.atlassian.net/browse/SVLS-8032

Additional Notes

Uses stats_concentrator_service with some refactors from bottlecap.
Enabled by default, can be disabled with DD_SERVERLESS_STATS_COMPUTATION_ENABLED=false
Span derived primary tags is on hold pending some updates to the spec
- feat(stats)!: implement span derived primary tags in libdd-trace-stats libdatadog#1815
Tracer computed stats disabled with client_drop_p0s: false in agent /info endpoint
- [SVLS-8796] fix: Change client_drop_p0s from true to false datadog-lambda-extension#1155
Agent computed stats sent with client_computed: false to stats intake

Describe how to test/QA your changes

Unit tests for stats flusher to flush if tracer computed stats or serverless computed stats are available
Integration tests to compute stats in the serverless compatibility layer by default and to skip computation if the Datadog-Client-Computed-Stats is present in the trace payload
- Corresponding configuration in Go Agent: https://github.com/DataDog/datadog-agent/blob/867176f1b6b6f87f3aadb7a248d79c5fe9cd0bac/pkg/trace/agent/agent.go#L522-L524

duncanpharvey · 2026-04-01T20:48:28Z

@codex review

Copilot

Pull request overview

This PR adds trace stats computation capabilities to the Serverless Compatibility Layer. It enables the system to compute and send trace statistics when tracers haven't already computed them, enhancing stats availability across all supported serverless environments.

Changes:

Adds a new StatsConcentratorService module that manages span concentration and stats bucket generation using the libdd-trace-stats library
Adds a new StatsGenerator module that extracts metadata and spans from traces and sends them to the stats concentrator
Extends ServerlessTraceProcessor to optionally call the stats generator when the tracer hasn't already computed stats
Extends ServerlessStatsFlusher to flush both tracer-computed and serverless-computed stats
Adds integration tests to verify stats computation and to verify stats are skipped when client has already computed them
Adds configuration options for service, env, and version fields, and an environment variable to enable/disable stats computation

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
`src/stats_concentrator_service.rs`	New module implementing stats concentration service with metadata tracking and flush logic
`src/stats_generator.rs`	New module for extracting trace information and sending to stats concentrator
`src/trace_processor.rs`	Modified to conditionally invoke stats generator based on client-computed-stats header
`src/stats_flusher.rs`	Modified to flush both channel and concentrator-based stats with proper flow control
`src/config.rs`	Added service, env, version fields and corresponding environment variable reading
`src/main.rs`	Added initialization of stats computation with environment variable toggle
`src/lib.rs`	Exported new modules
`src/mini_agent.rs`	Minor code style improvements (let-else pattern)
`Cargo.toml`	Added `thiserror` and `libdd-trace-stats` dependencies
`tests/integration_test.rs`	Added comprehensive integration tests for stats generation scenarios
`tests/common/mock_server.rs`	Updated to handle stats intake with proper 202 status code response
`tests/common/helpers.rs`	Extended send_tcp_request to support additional headers

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac8d049051

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Lewis-E · 2026-04-02T20:21:44Z

-                self.flush_stats(config.clone(), buffer.to_vec()).await;
-                buffer.clear();
+            // copy the batch for this flush
+            let channel_stats = buffer.to_vec();


Nit: LLM recommends let channel_stats = std::mem::take(&mut *buffer); to replace let channel_stats = buffer.to_vec(); buffer.clear(); but that's unrelated to the actual intent of the change here, so.

Applied this change!

Lewis-E · 2026-04-02T20:26:09Z

+                }
+                Ok(None) => {}
+                Err(e) => {
+                    error!("Failed to flush stats concentrator: {e}");


Do we want a log in a loop here or a error to be caught and have the whole process restarted/shut down? The trace flusher and stats flusher both have a a Err("<this code> task terminated unexpectedly".into());, would be consistent to have that for the concentrator as well.

I added a task terminated unexpectedly check for the span concentrator service to be consistent with these other background tasks.

…nt-computed-stats (#1900) # What does this PR do? Treat an empty string as a falsy value for the `Datadog-Client-Computed-Stats` header. # Motivation We want to support agent computed stats in the Serverless Compatibility Layer. Currently when the `Datadog-Client-Computed-Stats` header is sent it always disables agent computed stats, even when the value of the header is an empty string. https://datadoghq.atlassian.net/browse/SVLS-8789 # Additional Notes - [Java Tracer sends an empty string](https://github.com/DataDog/dd-trace-java/blob/6e28457d70c41bf847c4af7b3ba4e2f6c1371070/dd-trace-core/src/main/java/datadog/trace/common/writer/ddagent/DDAgentApi.java#L110-L117) for `Datadog-Client-Computed-Stats` header where other tracers omit the header altogether - Go Agent [treats an empty string as a falsy value](https://github.com/DataDog/datadog-agent/blob/76aff83162011a15e5ee50295ac835f708e8ffa9/pkg/trace/api/api.go#L1049) - See DataDog/serverless-components#51 for adding agent computed stats in the Serverless Compatibility Layer. # How to test the change? Added a debug log in a test build: Before change with `DD_TRACE_STATS_COMPUTATION_ENABLED=false` ``` DEBUG datadog_trace_agent::trace_processor: Resolved tracer header tags: TracerHeaderTags { lang: "java", lang_version: "21.0.6", lang_interpreter: "OpenJDK 64-Bit Server VM", lang_vendor: "Microsoft", tracer_version: "1.61.1~e32291a78b", container_id: "", client_computed_top_level: true, client_computed_stats: true, dropped_p0_traces: 0, dropped_p0_spans: 0 } ``` After change with `DD_TRACE_STATS_COMPUTATION_ENABLED=false` ``` DEBUG datadog_trace_agent::trace_processor: Resolved tracer header tags: TracerHeaderTags { lang: "java", lang_version: "21.0.6", lang_interpreter: "OpenJDK 64-Bit Server VM", lang_vendor: "Microsoft", tracer_version: "1.61.1~e32291a78b", container_id: "", client_computed_top_level: true, client_computed_stats: false, dropped_p0_traces: 0, dropped_p0_spans: 0 } ``` Co-authored-by: duncan.harvey <duncan.harvey@datadoghq.com>

apiarian-datadog · 2026-04-22T20:21:17Z

+                    }
+                    let _ = flusher_shutdown_tx.send(());
+                    if let Err(e) = stats_flusher_handle.await {
+                        error!("Stats flusher task failed during shutdown: {e:?}");


in this case we don't return an error?

I updated this stats flusher shutdown to return an error so it is with the other task terminated unexpectedly errors.

apiarian-datadog · 2026-04-22T20:21:36Z

                    PROFILING_ENDPOINT_PATH
                ],
-                "client_drop_p0s": true,
+                "client_drop_p0s": false,


what does this mean?

client_drop_p0s tells the tracer whether it should drop unsampled p0 traces before sending them. In order to actually support this, the Serverless Compatibility Layer needs to be able to send sampling rates back to the tracer. This feature is not currently supported today:

serverless-components/crates/datadog-trace-agent/src/http_utils.rs

Line 55 in 1e0c4af

let body = json!({"rate_by_service":{"service:,env:":1}}).to_string();

Essentially client_drop_p0s: true allows for stats computation in the tracer and client_drop_p0s: false disables stats computation in the tracer, with the expectation that stats computation will be handled in the agent.

might be worth adding a note in a comment about that.

…ce terminations

…before asserting the results

…UTATION_ENABLED

apiarian-datadog · 2026-04-24T20:14:57Z

+    pub fn new(config: Arc<Config>) -> (Self, StatsConcentratorHandle) {
+        let (tx, rx) = mpsc::unbounded_channel();
+        let handle = StatsConcentratorHandle::new(tx);
+        // TODO: set span_kinds_stats_computed and peer_tag_keys


when will we do this todo?

apiarian-datadog · 2026-04-24T20:20:16Z

+                // Drain client stats in buffer and stats from concentrator on interval
+                _ = interval.tick() => {
+                    let client_stats = std::mem::take(&mut buffer);
+                    if !client_stats.is_empty() || self.stats_concentrator.is_some() {


why do we check for the stats concentrator to be some here?

apiarian-datadog · 2026-04-24T20:22:00Z

+                    tracer_language: tracer_payload.language_name.clone(),
+                    tracer_version: tracer_payload.tracer_version.clone(),
+                    hostname: String::new(),
+                    service_name: None,


the payload doesn't have a service name to pull?

apiarian-datadog · 2026-04-24T20:23:35Z

+                }
+            }
+        } else {
+            error!("Unsupported tracer payload version. Failed to send trace stats.");


we should probably say which version we saw that we don't know how to handle. i saw a proposal for a new tracer payload version coming down the pike

duncanpharvey force-pushed the duncan-harvey/trace-stats branch 2 times, most recently from a7773cc to 070ae12 Compare December 11, 2025 14:51

duncanpharvey force-pushed the duncan-harvey/trace-stats branch from 2056c19 to d5a3f08 Compare December 17, 2025 22:16

duncanpharvey force-pushed the duncan-harvey/trace-stats branch 2 times, most recently from 006cb8e to 4142915 Compare March 27, 2026 14:16

duncanpharvey mentioned this pull request Mar 27, 2026

feat(stats)!: implement span derived primary tags in libdd-trace-stats DataDog/libdatadog#1815

Open

duncanpharvey force-pushed the duncan-harvey/trace-stats branch from 5a64507 to 3562c93 Compare March 31, 2026 17:50

duncanpharvey marked this pull request as ready for review April 1, 2026 20:48

duncanpharvey requested a review from a team as a code owner April 1, 2026 20:48

Copilot AI review requested due to automatic review settings April 1, 2026 20:48

duncanpharvey requested a review from a team as a code owner April 1, 2026 20:48

duncanpharvey requested review from Lewis-E and litianningdatadog and removed request for a team April 1, 2026 20:48

Copilot started reviewing on behalf of duncanpharvey April 1, 2026 20:48 View session

Copilot AI reviewed Apr 1, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread crates/datadog-trace-agent/src/stats_concentrator_service.rs

duncanpharvey requested a review from apiarian-datadog April 2, 2026 15:12

apiarian-datadog reviewed Apr 2, 2026

View reviewed changes

Lewis-E reviewed Apr 2, 2026

View reviewed changes

duncanpharvey force-pushed the duncan-harvey/trace-stats branch from 9e4544f to c2191f8 Compare April 7, 2026 15:50

duncanpharvey marked this pull request as draft April 9, 2026 20:51

duncanpharvey requested a review from lucaspimentel April 9, 2026 20:51

duncanpharvey force-pushed the duncan-harvey/trace-stats branch 3 times, most recently from df2f714 to 3b74133 Compare April 20, 2026 17:14

duncanpharvey marked this pull request as ready for review April 20, 2026 17:21

duncanpharvey mentioned this pull request Apr 21, 2026

feat(libdd-trace-utils): check for empty value in header datadog-client-computed-stats DataDog/libdatadog#1900

Merged

apiarian-datadog reviewed Apr 22, 2026

View reviewed changes

duncanpharvey added 10 commits April 24, 2026 11:18

implement trace stats for serverless compat

d54c799

remove channel depth debug log

005ef6b

fmt and clippy

10f92e6

add stats concentrator service handle for unexpected background servi…

2f15d51

…ce terminations

drain in flight handlers prior to shutdown

85c33e2

refactor StatsGenerator into ServerlessTraceProcessor

1bd18c0

remove should_flush_stats_buffer from stats_flusher

5132865

only return errors rather than logging one error and returning another

4bd780f

wait for the mini agent to finish shutting down in integration tests …

0755452

…before asserting the results

add StatsConcentratorComponents struct

e46f752

duncanpharvey force-pushed the duncan-harvey/trace-stats branch from 3b74133 to e46f752 Compare April 24, 2026 18:23

duncanpharvey added 3 commits April 24, 2026 14:25

update licenses

8e83906

rename DD_SERVERLESS_STATS_COMPUTATION_ENABLED to DD_AGENT_STATS_COMP…

16044f1

…UTATION_ENABLED

check for multiple tracers in stats concentrator service

8b2e49f

duncanpharvey mentioned this pull request Apr 24, 2026

feat: add PartialEq and Eq traits to TracerMetadata DataDog/libdatadog#1922

Open

apiarian-datadog self-requested a review April 24, 2026 20:03

apiarian-datadog approved these changes Apr 24, 2026

View reviewed changes

Conversation

duncanpharvey commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Describe how to test/QA your changes

Uh oh!

duncanpharvey commented Apr 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lewis-E Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

duncanpharvey commented Dec 3, 2025 •

edited

Loading

Lewis-E Apr 2, 2026 •

edited

Loading