Skip to content

fix(strands): allowlist metrics, add span_origin, fix agent metadata reads#612

Merged
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
fix/strands-skill-alignment
Jul 21, 2026
Merged

fix(strands): allowlist metrics, add span_origin, fix agent metadata reads#612
Abhijeet Prasad (AbhiPrasad) merged 1 commit into
mainfrom
fix/strands-skill-alignment

Conversation

@starfolkai

@starfolkai starfolkai Bot commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Summary

Aligns the Strands integration with .agents/skills/sdk-integrations/SKILL.md, following the pattern of #606 (openai), #604 (mistral), #603 (livekit), #594 (litellm), #583 (anthropic). Audit turned up several issues:

  • Missing span_origin instrumentation name. Every other integration has _INSTRUMENTATION = "<provider>-auto" + a module-level start_span wrapper; strands was the only one whose spans lacked context.span_origin.instrumentation.name. Added the standard wrapper and pass internal={"instrumentation": _INSTRUMENTATION} explicitly through the nested parent.start_span(...) path, which doesn't route through the module wrapper.

  • Denylist metrics extraction (real leak vector). _end_model_invoke_span_wrapper was doing {k: v for k, v in metrics.items() if _is_supported_metric_value(v)} — every numeric key strands emits (present + future) flowed into the span's top-level metrics bag. Replaced with an explicit allowlist (latencyMs, timeToFirstByteMs). Since neither key is spec-listed, they land under metadata.strands_metrics rather than top-level metrics.

  • Silently-broken agent metadata extraction. _agent_metrics_and_metadata did metrics = getattr(result, "metrics"); if not isinstance(metrics, dict): return {}, {} — but AgentResult.metrics is an EventLoopMetrics dataclass, so the guard discarded everything. Renamed to _agent_metadata_from_result and read cycle_count / accumulated_usage / accumulated_metrics off the dataclass through a single _pick_numeric(source, keys) allowlist helper. Moved cycles from metrics to metadata (not a spec metric key).

  • Positional-arg reads. _start_agent_span_wrapper / _start_model_invoke_span_wrapper were kwargs.get("tools") / kwargs.get("system_prompt") etc. Strands' signatures put those at fixed positional slots — switched to _arg(args, kwargs, N, name) so positional callers work.

  • Simplifications (via /simplify): collapsed twin allowlist helpers into _pick_numeric; collapsed the parent-vs-module if/else in _start_span_for_otel; replaced the _bt_parent_otel_span metadata back-channel with a direct parent_otel_span= kwarg; dropped the now-orphan metrics= parameter from _end_span_for_otel; trimmed the verbose wrap_strands_tracer and _InvalidOtelSpanKey docstrings.

Known SKILL gaps NOT fixed here

Each deserves its own PR/discussion — flagging so they don't get lost:

  1. metadata.provider on LLM spans. Strands' Tracer only receives model_id, not the model class — provider inference would require brittle id-pattern matching.
  2. Tokens under metadata.strands_usage rather than metrics.tokens / prompt_tokens / completion_tokens. Fixing correctly means deciding whether model-invoke should stay typed llm at all — Strands' OpenAIModel / AnthropicModel delegate to the underlying provider SDK, which per the SKILL's "framework llm-like spans that delegate" rule should NOT be typed llm.
  3. No test_auto_strands.py subprocess auto-instrument test. Strands is wired into auto.py but lacks the subprocess parity test other integrations have.

Test plan

  • No new mocks/fakes. Existing cassette-backed VCR test (test_strands_openai_agent_traces_native_otel_lifecycle) covers the behavior; added a span_origin.instrumentation.name == "strands-auto" assertion on every emitted span.
  • No cassette re-recording. HTTP behavior is unchanged; only in-process span shaping was touched.
  • cd py && nox -s "test_strands(latest)" — 2/2 pass
  • cd py && nox -s "test_strands(1.20.0)" — 2/2 pass
  • ruff check + ruff format --check on touched files — clean
  • nox -s pylint — clean

🤖 Generated with Claude Code

Created by abhijeet

Slack thread

Audit against `.agents/skills/sdk-integrations/SKILL.md`:

- Missing `_INSTRUMENTATION = "strands-auto"` and local `start_span`
  wrapper. Every other integration has it (adk, autogen, anthropic,
  mistral, etc.); strands was the only integration whose spans lacked
  `context.span_origin.instrumentation.name`. Added the module-level
  wrapper and pass `internal={"instrumentation": _INSTRUMENTATION}`
  explicitly through nested `parent.start_span(...)` calls, which don't
  route through the module wrapper.

- Denylist metrics extraction on the model-invoke span:
  `{k: v for k, v in metrics.items() if _is_supported_metric_value(v)}`
  passed every numeric key strands emits (present + future) into the
  span's top-level `metrics` bag. Replaced with an explicit allowlist
  (`latencyMs`, `timeToFirstByteMs`) landing in
  `metadata.strands_metrics` since neither key is spec-listed.

- `_agent_metrics_and_metadata` was reading `getattr(result, "metrics")`
  as a dict — but `AgentResult.metrics` is an `EventLoopMetrics`
  dataclass, so the `isinstance(metrics, dict)` guard silently discarded
  everything. Renamed to `_agent_metadata_from_result` and read
  `cycle_count`, `accumulated_usage`, `accumulated_metrics` off the
  dataclass through a unified `_pick_numeric(source, keys)` allowlist
  helper. Moved `cycles` from `metrics` to `metadata` (not a spec key).

- Positional-arg reads: `_start_agent_span_wrapper` and
  `_start_model_invoke_span_wrapper` did `kwargs.get("tools")` /
  `kwargs.get("system_prompt")` etc. Strands' signatures put these at
  fixed positional slots — switched to `_arg(args, kwargs, N, name)`
  so positional callers work.

- Simplifications: collapsed the parent-vs-module `if/else` in
  `_start_span_for_otel` by picking the callable once; replaced the
  `_bt_parent_otel_span` metadata back-channel with a direct
  `parent_otel_span=` kwarg on `_start_span_for_otel`; dropped the
  now-orphan `metrics=` parameter from `_end_span_for_otel`; trimmed
  the excessive `wrap_strands_tracer` and `_InvalidOtelSpanKey`
  docstrings.

No cassettes re-recorded — HTTP behavior is unchanged, only in-process
span shaping was touched. The existing VCR test now also asserts
`span_origin.instrumentation.name == "strands-auto"` on every emitted
span.

Test plan
- nox -s "test_strands(latest)" — 2/2 pass
- nox -s "test_strands(1.20.0)" — 2/2 pass
- ruff check + ruff format --check on touched files — clean
- nox -s pylint — clean

Known SKILL gaps NOT fixed here (each deserves its own discussion):
- `metadata.provider` on LLM spans: Strands' Tracer only receives
  `model_id`, no model-class hook. Provider inference would require
  brittle id-pattern matching.
- Tokens under `metadata.strands_usage` rather than `metrics.tokens`
  et al: fixing correctly means deciding whether model-invoke should
  stay `llm` at all — strands' `OpenAIModel`/`AnthropicModel` delegate
  to the underlying provider SDK, which the SKILL's "framework
  llm-like spans that delegate" rule says should NOT be typed `llm`.
- No `test_auto_strands.py` subprocess auto-instrument test.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
@starfolkai starfolkai Bot changed the title fix(strands): align with sdk-integrations SKILL, drop denylist metrics fix(strands): allowlist metrics, add span_origin, fix agent metadata reads Jul 21, 2026
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) merged commit a1a1b11 into main Jul 21, 2026
82 checks passed
@AbhiPrasad
Abhijeet Prasad (AbhiPrasad) deleted the fix/strands-skill-alignment branch July 21, 2026 19:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants