Fix routing smoke assertions: attribute agent traffic by model instead of nonexistent purpose values

[Copilot]

Important

Before merging, a maintainer must add the smoke-routing label. That runs Smoke Copilot Routed, Smoke Pi Routed and Smoke Copilot SDK Routed on this PR's code. They have not run on it yet.

All three routing smokes fail on main in the post-step only. Routing and sub-agent delegation were correct. smoke_model_routing_assertions.cjs counted only token-usage records with purpose: "agent" or "subagent", but AWF never writes those values. The only purpose AWF records is routing_classification. As a result R3, M1 and S3 had nothing to match, and R4 passed on zero requests.

Request classification (smoke_model_routing_assertions.cjs)

• Classifier vs. agent traffic: purpose: "routing_classification" is the classifier (R2). Every other record is agent traffic. x_initiator is only the billing class, so it is not used.
• Attribution: agent traffic on a declared sub-agent's model goes to that sub-agent (S3). Everything else goes to the main agent (R3, R4, M1).
• Correlation, if it ever exists: a session link takes precedence over model matching, either a request ID on subagent.* events or a sub-agent ID on proxy records. Today there is none, so model matching is what runs.
• Empty input: R3, R4, M1 and S3 now fail when there are zero requests to check. The message names the check and the requests observed.
• New C1 check: fails when a declared sub-agent model is also in allowedModels, because model-based attribution would then be ambiguous.
• Normalization is unchanged: dated IDs and query strings were already handled but had never run on real records, because the old filter emptied the request sets first. Examples: claude-haiku-4-5-20251001 becomes claude-haiku-4.5, and /v1/messages?beta=true becomes /v1/messages.

Records as AWF writes them (from Smoke Pi Routed):

{"model":"gpt-5.6-luna","path":"/responses","status":200,"purpose":"routing_classification","x_initiator":"agent"}
{"model":"gpt-5.6-luna","path":"/responses","status":200,"x_initiator":"agent"}
{"model":"claude-haiku-4-5-20251001","path":"/v1/messages?beta=true","status":200,"x_initiator":"agent"}
{"model":"gpt-5.4-mini-2026-03-17","path":"/responses","status":200,"x_initiator":"agent"}

These now resolve to: the classifier (R2), the main agent (R3, R4), haiku-whoami (S3) and mini-whoami (S3).

smoke-pi-routed.md

• allowed-models and the post-step allowedModels are reduced to [gpt-5.6-luna], matching smoke-copilot-sdk-routed. The sub-agent models can no longer be routing selections, which satisfies C1. The lock file is recompiled.
• smoke-copilot-routed.md and smoke-copilot-sdk-routed.md are unchanged.

Tests

• Fixtures are rebuilt from runs 38097181601, 38096990663 and 38100426674. Requests come from the agent artifact's token-usage.jsonl and events from usage/aw_session.jsonl. They include:
  • agent requests with no purpose
  • dated model IDs and query-string endpoints
  • SDK events keyed by agentId, with a second subagent.completed marked cancelled: true. S2 is deliberately not tightened.
• Removed: the test asserting that requests without purpose cannot satisfy R3.
• Negative variants, each asserting the check ID and the observed values:
  • wrong model
  • wrong endpoint
  • non-200 status
  • no agent requests
  • missing routing event
  • harness-only routing outcome
  • C1 overlap
  • request-ID and agent-ID correlation

Scope

Only the assertion helper, its tests and the Pi smoke config change. Routing, harness, AWF configuration and audit behavior are untouched.

• Fixes Routing smokes fail on main: assertion helper filters on purpose values AWF never writes #67594

[github-actions]

✅ Ponytail Reviewer completed successfully!

Generated by Ponytail Reviewer for #67595

[github-actions]

✅ PR Code Quality Reviewer completed the code quality review.

🔎 Code quality review by PR Code Quality Reviewer

[github-actions]

🔍 Design Decision Gate 🏗️ is checking for design decision records on this pull request...

[github-actions]

🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer

[github-actions]

✅ Test Quality Sentinel completed test quality analysis.

Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff

🧪 Test quality analysis by Test Quality Sentinel

[github-actions]

✅ Smoke Copilot Routed completed successfully!

Generated by Smoke Copilot Routed for #67595

[github-actions]

✅ Smoke Pi Routed completed successfully!

Generated by Smoke Pi Routed for #67595

[github-actions]

✅ Smoke Copilot SDK Routed completed successfully!

Generated by Smoke Copilot SDK Routed for #67595

[github-actions]

One removable speculative attribution layer found.

net: -20 lines possible.

Generated by ✂️ Ponytail Reviewer for #67595 · codex · gpt56 · 12 AIC · ⌖ 5.16 AIC · ⊞ 13.5K
_{Comment /ponytail to run again}

[Copilot]

🟡 Changes recommended

Duplicate sub-agent models remain ambiguous and can let one request satisfy multiple S3 checks.

1 open finding

• Reject duplicate normalized sub-agent models to prevent request misattribution · New

What changed in this PR

Fixes routing smoke assertions to classify real AWF traffic correctly and updates the Pi smoke configuration.

Changes:

• Attributes non-classifier requests by correlation or model.
• Adds real-shape fixtures and negative coverage.
• Restricts Pi routing candidates and recompiles its lock file.

File	Description
actions/​setup/​js/​smoke_model_routing_assertions.cjs	Revises request attribution and checks.
actions/​setup/​js/​smoke_model_routing_assertions.test.cjs	Expands realistic assertion coverage.
.github/​workflows/​smoke-pi-routed.md	Restricts routed models to Luna.
.github/​workflows/​smoke-pi-routed.lock.yml	Regenerates the compiled workflow.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[github-actions]

Unable to submit full review body from the sandbox.

🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 45.8 AIC · ⌖ 5.33 AIC · ⊞ 21.2K
_{Comment /review to run again}

[SivaKesava1]

Ran the three routing smokes on this PR's code (f08249c) by adding the smoke-routing label. They started without needing workflow approval, and all three pass with every job green:

• Smoke Copilot Routed 38111946761: R1-R4 pass (4 main-agent requests on gpt-5.6-luna /responses).
• Smoke Pi Routed 38111946869: C1, R1-R4, and S1-S3 for haiku-whoami (claude-haiku-4.5 /v1/messages 200) and mini-whoami (gpt-5.4-mini /responses 200).
• Smoke Copilot SDK Routed 38111946778: C1, R1-R4, M1 (all on /responses), and S1-S4 for haiku-whoami (claude-haiku-4.5 /chat/completions 200, declared name on events).

On the two open review threads:

• Follow-up, small: C1 and duplicate sub-agent models (copilot-pull-request-reviewer). Valid: if two declared sub-agents share a normalized model, one request can satisfy both S3 checks. Please extend C1 to also fail when two declared sub-agents share a normalized model, with a test. The current smokes use distinct models, so nothing else changes.
• Keep: request-ID/agent-ID correlation (yagni thread). The issue asked for it so that a real correlation takes precedence as soon as AWF or gh-aw records one, and it's covered by tests. Please reply on that thread and leave the code.

[github-actions]

Skills-Based Review 🧠

Applied /diagnosing-bugs (root-cause bug fix) and /tdd (test rebuild). The fix correctly diagnoses the root cause — AWF never writes purpose: "agent"/"subagent", only purpose: "routing_classification" for the classifier — and switches to model-based attribution with session-correlation fallback. Comprehensive negative-variant tests were rebuilt from real production fixtures (runs 38097181601/38096990663/38100426674), covering dated model IDs, query-string endpoints, empty-request failure modes, and the new C1 disjointness check.

📋 Key Themes & Highlights

Key Themes

• One actionable correctness gap: model-based attribution (attributed in smoke_model_routing_assertions.cjs) doesn't guard against two declared sub-agents sharing the same model — left an inline comment with a concrete fix suggestion (extend C1, or add a regression test documenting the behavior).

Positive Highlights

• ✅ Root cause correctly diagnosed against real AWF record shapes, not just symptom patched
• ✅ Test suite rebuilt against actual production fixtures rather than synthetic approximations, closing the gap that let the original bug ship
• ✅ New C1 check and empty-request failure modes (R3/R4/M1/S3) add real coverage that was previously silently passing on zero requests
• ✅ Doc comment at the top of the file clearly explains the new attribution model for future maintainers

No blocking issues; approving with one actionable suggestion for the author to consider.

🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 99.8 AIC · ⌖ 14.6 AIC · ⊞ 10.3K
_{Comment /matt to run again}

[SivaKesava1]

Re-ran the three routing smokes on d4d3b24 (C1 change) by dispatching them on this branch (label-triggered runs on the new commit stopped at "Approve and run workflows"). The helper's unit tests pass (46).

• Smoke Copilot Routed 38115103020: all checks pass.
• Smoke Pi Routed 38115104767: all checks pass, including the new C1 ("declared sub-agent models are distinct and disjoint").
• Smoke Copilot SDK Routed 38115106576: failed, but not because of the checks. C1, R1-R4 and M1 pass. The sub-agent never ran: gpt-5.6-luna called the task tool with {"description":"Ask haiku whoami","prompt":"who am i?","agent_type":"haiku-whoami","mode":"sync"}, without the required name argument. The tool failed with "name": Required, and the agent created the result issue without retrying. Earlier runs passed because the model included name.

Blocking (flakiness): the scheduled smoke will fail randomly whenever the model leaves out name. Please harden the task instructions in smoke-copilot-sdk-routed.md, and in smoke-pi-routed.md if it delegates the same way: tell the agent exactly how to call the sub-agent (the task tool with agent_type and name set to the sub-agent's name, mode sync), and to retry the call once with the arguments corrected if the tool returns an error. Recompile the lock files. Keep the assertions unchanged.

[github-actions]

✅ Smoke Copilot SDK Routed completed successfully!

Generated by Smoke Copilot SDK Routed for #67595

[github-actions]

✅ Smoke Copilot Routed completed successfully!

Generated by Smoke Copilot Routed for #67595

[github-actions]

✅ Smoke Pi Routed completed successfully!

Generated by Smoke Pi Routed for #67595

[Copilot]

Re-ran the three routing smokes on d4d3b24 (C1 change) by dispatching them on this branch (label-triggered runs on the new commit stopped at "Approve and run workflows"). The helper's unit tests pas...

Fixed in d424c96. In smoke-copilot-sdk-routed.md, task 1 now spells out the exact task call: {"description":"Ask haiku-whoami","prompt":"who am i?","agent_type":"haiku-whoami","name":"haiku-whoami","mode":"sync"}. If the tool returns an error, the agent fixes the arguments to match and retries once. I ran make recompile, and the lock file's body hash is the only change there. I didn't touch the assertions.

I left smoke-pi-routed.md alone. It doesn't use the task tool. It calls pi's managed subagent tool and already gives the exact arguments ({"agent":"haiku-whoami","task":"who am i?"}).

I haven't re-run the smokes on this commit.

[SivaKesava1]

Verified d424c96 for review. All three routing smokes run on this PR's code (dispatched on the branch) and pass with every job green:

• Smoke Copilot Routed 38117916646: routing selected, one classifier request, main-agent requests on the selected model and endpoint, all within allowed-models.
• Smoke Pi Routed 38117918081: the same, plus C1 (sub-agent models distinct and disjoint from allowed-models) and S1-S3 for haiku-whoami (claude-haiku-4.5 /v1/messages 200) and mini-whoami (gpt-5.4-mini /responses 200).
• Smoke Copilot SDK Routed 38117919739: the same, plus S1-S4 for haiku-whoami (claude-haiku-4.5 /chat/completions 200, declared name on events) under a gpt-5.6-luna /responses main session.

The helper's unit tests pass (46). Review threads are resolved. The earlier SDK smoke failure (the model called task without name) is addressed by the explicit delegation instructions in d424c96. On main, these three smokes fail on every scheduled run until this merges.