Propagate trace context for gRPC calls

[mtaghavi2005]

Description

Propagates trace context on gRPC calls so Dapr sidecar spans can preserve the expected parent-child relationship with the application operation that initiated the call.

This adds grpc-trace-bin metadata from the current .NET activity when available, and includes tests for the shared gRPC call options path and DaprClient publish flow.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #1857

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• Code compiles correctly
• Created/updated tests
• Extended the documentation

[WhitWaldo]

@mtaghavi2005 I just noticed this PR. I'm working on an improvement to add integration testing for these spans so this stops becoming a recurring issue as part of your ask in #1867. Some of your code looks awfully similar to my own, so I'm going to put this PR on hold until I merge my changes (as I've already got a fix and a test for your issue). We can circle back afterwards to see whether this change is still necessary.

Sorry I didn't see it sooner to use instead as a starting point!

[mtaghavi2005]

No problem, and thanks a lot for the update! That makes sense. I’m also glad to hear that the other issue will be resolved soon. Looking forward to taking another look once your changes are merged.

[WhitWaldo]

I just merged #1870 into master and will kick off another patch release on 1.18 later this evening once I get #1872 and #1871 merged as well. I'm eager to see if we can put all the observability issues with workflows behind us!

[mtaghavi2005]

That’s great news! Thank you for the update. I'm also looking forward to seeing all the tracing issues, especially those related to workflows, resolved. I really appreciate all your effort and support!

I'll test the changes from master (or the upcoming patch release), and if everything looks good, I'll close this PR.

[mtaghavi2005]

I tested the latest changes from master and can confirm that the issue with disconnected gRPC spans is now resolved. Thanks again for the fix! So I close this PR.

Update: Unfortunately, the issue is still not resolved.

AddGrpcClientInstrumentation was already in place during my testing. If I remove the gRPC client instrumentation, all gRPC spans become disconnected again. So, unfortunately, the underlying issue still appears to exist.

[WhitWaldo]

I was able to repro this locally - going to poke at it later this and next week.

[WhitWaldo]

@mtaghavi2005 I just wanted to give an update - I'm still looking into this. Historically, none of the SDKs are instrumented at all and we instead direct anyone to consume the otel endpoint from the runtime to get this information.

This PR validates that workflows capture e2e (though I'll certainly update it to reflect the other SDK functionality as well) via that OTEL endpoint as expected, but it occurs to me that the issue might lie in how Aspire auto-instruments the application (e.g. it's not consuming from the Dapr OTEL endpoint, but rather capturing from the gRPC and HTTP requests leaving the SDK itself, which is why it has that incomplete picture.

Rather than just patching this on the SDK and opening up a can of worms of enabling at-dev-time telemetry capture via Aspire that's entirely unintended for production usage (again, should be using the rutnime's endpoint), I'm looking into whether I can configure Aspire to instead visualize the telemetry from the Dapr runtime as that would be more aligned with our best practices generally.

Also, I'll be out of the office next week, so I likely won't have another update until the first week of August.

[mtaghavi2005]

Thanks for the update. I agree with the distinction here: I am not trying to make the Dapr .NET SDK produce its own telemetry spans for normal client operations.

The issue I am trying to separate is context propagation, not SDK instrumentation. It is also not limited to workflows; it is a general gRPC tracing/context propagation issue for Dapr .NET SDK calls to the sidecar.

From my investigation, Aspire can show a correct-looking trace when OpenTelemetry.Instrumentation.GrpcNetClient is enabled, but I do not think that should be the expected production workaround:

• OpenTelemetry.Instrumentation.GrpcNetClient is still beta/pre-release.
• It adds an extra gRPC client span, which changes the trace shape.
• The trace becomes correct because that instrumentation changes the activity/propagation behavior for the outgoing gRPC call.

I also traced this further and found that the Dapr runtime can extract the incoming traceparent; the problem appears to be that the propagated parent can correspond to the hidden Grpc.Net.Client transport activity rather than the logical application activity. That seems related to grpc-dotnet behavior, and I opened an upstream PR here:

grpc/grpc-dotnet#2744

So I agree the ideal fix is probably upstream in Grpc.Net.Client.

The reason I still think SDK-side grpc-trace-bin propagation is worth considering is that it is not SDK instrumentation. It does not create spans. It only sends Dapr’s documented gRPC trace context explicitly to the sidecar, using the current logical Activity.Current. In that sense, I see it as a compatibility workaround until the upstream grpc-dotnet behavior is fixed.

Regarding Aspire: I agree it may be showing an incomplete picture if it is observing app HTTP/gRPC instrumentation instead of consuming the Dapr runtime OTEL endpoint. To avoid mixing Aspire-specific behavior with production behavior, I can also verify the same scenario with Application Insights without using aspire by comparing:

1. app telemetry
2. Dapr runtime telemetry
3. behavior with and without explicit grpc-trace-bin propagation

That should make it clearer whether the remaining issue is only Aspire visualization, runtime OTEL consumption/configuration, or actual context propagation from the .NET app to the sidecar.

[mtaghavi2005]

I verified this without Aspire in the telemetry path. The services were started with Dapr multi-app run, app and sidecar telemetry were exported via OTLP to an OpenTelemetry Collector, and the result was inspected in Application Insights.

The sample flow is:

sequenceDiagram
    participant Client
    participant Api as apiservice
    participant ApiDapr as apiservice daprd
    participant PubSub as Dapr pub/sub
    participant AccountingDapr as accountingservice daprd
    participant Accounting as accountingservice

    Client->>Api: POST /payments
    Api->>ApiDapr: PublishEvent(payment-authorization-requests)
    ApiDapr->>PubSub: publish payment-authorization-requests
    PubSub->>AccountingDapr: deliver payment-authorization-requests
    AccountingDapr->>Accounting: POST /payment-authorizations

    Accounting->>AccountingDapr: PublishEvent(payment-authorization-results)
    AccountingDapr->>PubSub: publish payment-authorization-results
    PubSub->>ApiDapr: deliver payment-authorization-results
    ApiDapr->>Api: POST /payments/authorization-results

Loading

Without the SDK change, all spans share the same operation/trace ID, but the Dapr runtime PublishEvent spans appear as separate sibling branches. The parent-child relationship is incorrect.

With explicit grpc-trace-bin propagation from the SDK (this PR) , the same flow is shown as one correct causal chain:

POST /payments
  PUBSUB /dapr.proto.runtime.v1.Dapr/PublishEvent
    PUBSUB pubsub/payment-authorization-requests
      POST /payment-authorizations
        PUBSUB /dapr.proto.runtime.v1.Dapr/PublishEvent
          PUBSUB pubsub/payment-authorization-results
            POST /payments/authorization-results

This confirms the issue is not SDK instrumentation or Aspire visualization. The SDK change does not create spans; it only propagates Dapr’s documented gRPC trace context so runtime spans attach to the correct logical parent.

traceparent alone is not enough because Grpc.Net.Client creates its own transport activity before sending the request. That activity becomes the parent in the outgoing traceparent, so the runtime gets the right trace ID but the wrong parent span. Sending grpc-trace-bin from the SDK avoids that until the grpc-dotnet behavior is fixed upstream.

Please let me know if any other test or information would help clarify the issue.

[WhitWaldo]

Fair enough - I appreciate the evidence backing your solution. If you can clean up the merge conflicts, I'm happy to merge your PR as-is.

[mtaghavi2005]

@WhitWaldo I updated the PR and the merge conflicts are resolved now.

There are still a few failing integration checks. I’m not sure yet whether they are related to this PR or CI/environment issues, so I’ll leave them for your review.

[WhitWaldo]

@mtaghavi2005 We have a few lingering issues with GitHub and the CI checks running in parallel (lots of port exhaustion) - it's a known issue, but really low on my priority list since that "re-run jobs" button is easy enough to just keep clicking).

I've got this merged and I'll see about getting a patch released this evening. Thank you very much for all your time and effort into this.