fix: propagate W3C Trace Context from controller to agent pods#1312
fix: propagate W3C Trace Context from controller to agent pods#1312opspawn wants to merge 3 commits intokagent-dev:mainfrom
Conversation
Two changes to enable end-to-end W3C TraceContext propagation: 1. Add AppProtocol "kgateway.dev/a2a" to agent Service port so AgentGateway can discover agent Services directly via kgateway protocol matching, rather than proxying through the controller. 2. Set up W3C TraceContext propagator in the Python agent SDK tracing configuration so agent pods correctly extract incoming traceparent headers and propagate them on outgoing requests. Fixes kagent-dev#1295 Signed-off-by: opspawn <opspawn@users.noreply.github.com>
opspawn
requested review from
EItanya,
ilackarms,
peterj and
yuval-k
as code owners
There was a problem hiding this comment.
Pull request overview
This PR attempts to fix distributed tracing by propagating W3C Trace Context headers from the controller to agent pods. The changes include configuring the W3C TraceContext propagator in the Python SDK and adding an appProtocol field to agent Service specs to enable gateway-based trace header forwarding.
Changes:
- Configured W3C TraceContext propagator in Python SDK's tracing utilities
- Added
appProtocol: "kgateway.dev/a2a"to agent Service port specifications - Updated 24 golden test output files to reflect the new service configuration
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| python/packages/kagent-core/src/kagent/core/tracing/_utils.py | Added W3C TraceContext propagator configuration for incoming/outgoing trace header extraction |
| go/internal/controller/translator/agent/adk_api_translator.go | Added appProtocol field to agent Service ports for gateway-based routing |
| go/internal/controller/translator/agent/testdata/outputs/*.json | Updated golden test outputs to include new appProtocol field and OTEL metrics environment variable |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Set up W3C TraceContext propagator so incoming traceparent headers | ||
| # are extracted and outgoing requests carry them forward. | ||
| set_global_textmap(CompositeHTTPPropagator([TraceContextTextMapPropagator()])) |
There was a problem hiding this comment.
The new propagator configuration code lacks test coverage. Given that other modules in kagent-core have test files (test_a2a_config.py, test_hitl_handlers.py, etc.), consider adding tests to verify that:
- The W3C TraceContext propagator is properly configured when OTEL_TRACING_ENABLED=true
- The propagator correctly extracts traceparent headers from incoming requests
- The propagator properly injects trace headers into outgoing requests
| Name: "http", | ||
| Port: dep.Port, | ||
| TargetPort: intstr.FromInt(int(dep.Port)), | ||
| AppProtocol: ptr.To("kgateway.dev/a2a"), |
There was a problem hiding this comment.
The PR description states this change "enables the gateway to forward trace headers to agent pods," suggesting reliance on an external kgateway component. However, issue #1295 identifies two independent problems:
- Go Controller: A2ARequestHandler drops incoming HTTP headers (needs code changes in go/internal/httpserver/auth/authn.go)
- Python Agent: No W3C TraceContext propagator configured (addressed in this PR)
The issue specifically recommends modifying A2ARequestHandler to capture and inject trace headers (traceparent/tracestate) from incoming requests to outgoing requests to agent pods. This PR does not include those changes.
If the approach has changed to rely on kgateway instead of modifying A2ARequestHandler, this should be documented. Otherwise, trace headers will still be lost when the controller forwards requests to agent pods in deployments without kgateway (which is listed as "optional tooling" in the documentation).
Signed-off-by: opspawn <opspawn@users.noreply.github.com>
Signed-off-by: opspawn <opspawn@users.noreply.github.com>
1 participant
Summary
Fixes #1295 — trace context (
traceparentheader) is not propagated from the controller to agent pods, breaking distributed tracing.Changes
Go Controller (
go/internal/controller/translator/agent/adk_api_translator.go): AddedAppProtocolto agent Service port spec, enabling the gateway to forward trace headers to agent pods.Python SDK (
python/packages/kagent-core/src/kagent/core/tracing/_utils.py): Set up W3C TraceContext propagator to ensure incomingtraceparentheaders are properly extracted and propagated.Golden test files: Updated all translator golden test outputs to include the new
appProtocolfield.Signed-off-by: opspawn agent@opspawn.com