Select Copilot wire APIs from AWF metadata and surface model mismatches

[Copilot]

Name-based wire API selection could pair Copilot models with incompatible endpoints, causing mid-run failures that the harness retried. This change uses AWF endpoint metadata and reports mismatches as model misconfiguration.

• Endpoint selection

  • Prefer complete AWF supported_endpoints; retain catalog/name fallback when metadata is unavailable.
  • Preserve existing preferences when both APIs are supported.
  • Reject incompatible overrides or models with no CLI-compatible endpoint before spawn.
  • Warn when declared custom-agent models conflict with the session endpoint.

• Retry handling

  • Classify all four mismatch signatures as non-retryable, including failures after --continue.
  • Capture available model and endpoint details.

• Diagnostics

  • Persist startup and runtime failures as model_endpoint.mismatch unified-session events.
  • Surface cause, endpoints, and remediation in step summaries, grouped failure reports, and audit.
  • Update event types, schema, and specification; redact diagnostic content.

• Fixes Copilot harness: choose the wire API from AWF endpoint data, fail fast on a model/endpoint mismatch, and report mismatch errors clearly #67485

[github-actions]

🔍 Design Decision Gate 🏗️ is checking for design decision records on this pull request...

[github-actions]

🔬 Test Quality Sentinel is analyzing test quality on this pull request...

[github-actions]

✂️ Ponytail Reviewer has started processing this pull request

[github-actions]

🔎 PR Code Quality Reviewer is reviewing code quality for this pull request...

[github-actions]

🧠 Matt Pocock Skills Reviewer is reviewing this pull request using Matt Pocock's engineering skills...

[Copilot]

🟡 Changes recommended

The new shared signal and filter category are not fully wired through, and dual-endpoint source attribution is inaccurate.

3 open findings

• Propagate endpoint mismatch output to conclusion job reporting · New
• Preserve the retained preference source for endpoint diagnostics · New
• Allow model_endpoint_mismatch in report-failure-as-issue filters · New

What changed in this PR

Updates the Copilot harness to validate wire APIs using AWF metadata and report model/endpoint mismatches through the unified-session pipeline.

Changes:

• Adds metadata-driven endpoint selection, validation, warnings, and non-retryable mismatch detection.
• Persists mismatch diagnostics for summaries, failure reports, and audits.
• Adds schemas, documentation, rendering, and tests.

File	Description
pkg/​cli/​model_routing_session.go	Parses mismatch session events.
pkg/​cli/​model_endpoint_mismatch_test.go	Tests audit parsing and safety.
pkg/​cli/​audit_report.go	Adds mismatches to audit data.
pkg/​cli/​audit_report_render.go	Renders mismatch diagnostics.
docs/​src/​content/​docs/​specs/​unified-agent-session-specification.md	Documents the event contract.
docs/​public/​schemas/​unified-session.schema.json	Defines mismatch payload schema.
actions/​setup/​md/​agent_failure_issue.md	Adds mismatch issue context.
actions/​setup/​md/​agent_failure_comment.md	Adds mismatch comment context.
actions/​setup/​js/​unified_session.test.cjs	Tests mismatch collection.
actions/​setup/​js/​unified_session.cjs	Collects mismatch records.
actions/​setup/​js/​unified_session_render.test.cjs	Tests safe publication.
actions/​setup/​js/​unified_session_render.cjs	Renders mismatch events.
actions/​setup/​js/​types/​unified_session.d.ts	Adds mismatch event types.
actions/​setup/​js/​harness_retry_guard.test.cjs	Tests mismatch parsing.
actions/​setup/​js/​harness_error_patterns.cjs	Detects mismatch signatures.
actions/​setup/​js/​handle_agent_failure.test.cjs	Tests failure reporting.
actions/​setup/​js/​handle_agent_failure.cjs	Groups and reports mismatches.
actions/​setup/​js/​detect_agent_errors.test.cjs	Tests shared detection.
actions/​setup/​js/​detect_agent_errors.cjs	Emits mismatch diagnostics.
actions/​setup/​js/​copilot_wire_api.cjs	Selects and validates wire APIs.
actions/​setup/​js/​copilot_harness.test.cjs	Tests endpoint and retry behavior.
actions/​setup/​js/​copilot_harness.cjs	Integrates validation and persistence.
actions/​setup/​js/​agent_error_patterns.cjs	Adds shared mismatch classification.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

[SivaKesava1]

@copilot Two things before we test this in our sandbox:

1. TestGeneratedOutputSchemasAreCurrent fails on this branch and passes on main: ../../schemas/audit.schema.json is stale; run make recompile and the same for logs-jsonl.schema.json. The new audit report fields need the generated output schemas regenerated (impacted-go-tests check).
2. Please merge main. The other red checks are from main and are fixed there: the setup_sh_file_lists.test.cjs failure about memory_schema_contract.cjs (fixed by Restore safe-output server startup after memory schema change #67506), the docs link check failing on experimental/drive-memory.md, and the agentic workflow checks whose safe outputs server crashed for the same reason.

Please keep this PR to the #67485 scope; don't fix unrelated failures here. Once it's green we'll run the forced model/endpoint mismatch case in the sandbox and post the results.

[Copilot]

@copilot Two things before we test this in our sandbox:

1. TestGeneratedOutputSchemasAreCurrent fails on this branch and passes on main: `../../schemas/audit.schema.json is stale; run make recomp...

Regenerated schemas/audit.schema.json and schemas/logs-jsonl.schema.json in 745407b, and merged latest main in 14dd37c. Final change-scoped validation and impacted tests pass; no unrelated fixes were added.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Review (actions/setup/js/agent_error_patterns.cjs:323): The shared detector writes this new output, but addMainJobEngineErrorOutputs does not declare it as an agent-job output and the conclusion job does not consume it. Consequently, endpoint mismatches detected outside the Copilot harness disappear at the step boundary and cannot reach the unified-session failure reporting added here. Wire this signal through the compiler/conclusion path and persist the corresponding unified evidence. - Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 (comment)
3. Review (actions/setup/js/copilot_wire_api.cjs:54): This always reports AWF /reflect as the selector whenever metadata is complete, even when the catalog, model-name rule, or CLI default actually supplied the retained preference. That makes the new source diagnostic inaccurate for dual-endpoint models. Preserve the existing source when preferred is usable, and switch to /reflect only when metadata chooses a fallback. - Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 (comment)
4. Review (actions/setup/js/handle_agent_failure.cjs:313): The new grouping category cannot be used in safe-outputs.report-failure-as-issue filters: pkg/parser/schemas/main_workflow_schema.json:13113 rejects model_endpoint_mismatch (and its ! form). Add this category to the schema allowlist and cover both include/exclude filtering so users can configure reports for the category introduced here. - Select Copilot wire APIs from AWF metadata and surface model mismatches #67487 (comment)

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 14dd37c
Sous-chef work: 4f9388f403779290af3a80ff8d4e7963d8802853710382e5410d8e26629524d4 69f8bdde551be61f3ff26494e7870c65f2dbaaec470378620e43ddfa80b8a684 afa6b9cefd7be378829c4a8700c317ca4c609d7c16a5c14855219d8c76cc88ce
Sous-chef state: 5ed72f2ab83950e2fc755fe2b19ea1aa77edb5f73c80a8ffd3eb559708b21b7d

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 4.4 AIC · ⌖ 9.67 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

[SivaKesava1]

@copilot Tested 14dd37c in our sandbox (AWF v0.28.50, Copilot CLI 1.0.90, org billing; workflows compiled with --gh-aw-ref 14dd37caf748f08c08d5ac79fc4b15d63e47964b).

Case	Result
model: gpt-5.6-luna with engine.env.COPILOT_PROVIDER_WIRE_API: completions (forced mismatch)	✅ Fails at startup before the CLI is spawned, no retries: Model endpoint mismatch: configured model 'gpt-5.6-luna'; COPILOT_PROVIDER_WIRE_API=completions (override, /chat/completions); supported endpoints: [/responses, ws:/responses]. Pin a compatible model, remove the COPILOT_PROVIDER_WIRE_API override, or upgrade gh-aw.
model: auto	✅ Resolves to claude-sonnet-5; auto-configuring COPILOT_PROVIDER_WIRE_API=completions … source=AWF /reflect; succeeds on attempt 1
Routed gpt-5.6-luna main with a claude-haiku-4.5 custom agent (CLI mode)	✅ Startup warning custom agent 'file-summarizer.agent.md' model 'claude-haiku-4.5' supports [/chat/completions, /v1/messages], incompatible with main model 'gpt-5.6-luna' session endpoint /responses …; audit reports Sub-agent Failed … Routing model "claude-haiku-4.5" to /chat/completions is incompatible

One gap: the startup mismatch never reaches the unified session, the failure issue, or the artifacts. In the forced-mismatch run:

• usage/aw_session.jsonl has no model_endpoint.mismatch event.
• The agent artifact has no agent/model-endpoint-mismatch.json. Its upload list includes agent/awf-routing-outcome.json but not this file.
• The generated failure issue says engine_driver_failure … Failure classification: engine_outage … Last engine error type: unknown … Retry attempts observed: 0 and shows neither the mismatch detail nor the fix.
• gh aw audit (built from this PR) shows only generic errors.

In this case the CLI never started, so there is no Copilot session. That is probably the path the unit tests don't cover. Please make sure that a startup mismatch:

• is written where generate_usage_artifacts (collectUnifiedSession, root /tmp/gh-aw) picks it up even when no engine session exists;
• is included in the agent artifact upload;
• is classified by handle_agent_failure as a model/endpoint mismatch, with the detail and fix in the issue.

Please add a test for this no-session startup case. Also note that threat detection inherits the same engine.env override and fails with the same mismatch, so the run's errors also include Detection result file not found. The failure report should make clear that both come from the same misconfiguration.

Minor: in the routed run the harness logs COPILOT_PROVIDER_WIRE_API already set to responses — source=override (verified against AWF /reflect), but routing chose that value, not a user override. A source=routing label would avoid confusion when debugging.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.
2. Fix failing check build (FAILURE): https://github.com/github/gh-aw/actions/runs/38088579009/job/114320448507.
3. Fix failing check build-wasm (FAILURE): https://github.com/github/gh-aw/actions/runs/38088578984/job/114320525982.
4. Fix failing check impacted-go-tests (FAILURE): https://github.com/github/gh-aw/actions/runs/38088578984/job/114320526143.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: 698fd4b
Sous-chef work: 42e5d572d9d75b7168fe7d0aeda0717cc07e65ed240f6d299e6fec661214f1ac 8acb61a077717e3a460018aacdfde96fd489f77606cd36169e2328d9079b75bb f7883e8c8105930e955354caa741d783bcca067e66097fe9612fb72b90823998
Sous-chef state: 152af5586a9442e3dd54f7ed1abdc99dd4715c16c24c890b278abb6d864abdaa

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 4.38 AIC · ⌖ 8.5 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

[SivaKesava1]

Tested 698fd4b in our sandbox (AWF v0.28.50, Copilot CLI 1.0.90, org billing; compiled with --gh-aw-ref 698fd4b57481c28b7b30ef83c312d71f4ba35430).

Fixed, verified in real runs

• Forced mismatch (gpt-5.6-luna with engine.env.COPILOT_PROVIDER_WIRE_API: completions), run 38089386804: agent/model-endpoint-mismatch.json is in the agent artifact. usage/aw_session.jsonl has {"type":"model_endpoint.mismatch","data":{"category":"model_endpoint_mismatch","phase":"startup","configured_model":"gpt-5.6-luna",…,"wire_api_source":"override","supported_endpoints":["/responses","ws:/responses"],…}}. gh aw audit shows the mismatch with its detail and fix: Pin a compatible model, remove the COPILOT_PROVIDER_WIRE_API override, or upgrade gh-aw.
• Routed run, 38089392076: the log now says COPILOT_PROVIDER_WIRE_API already set to responses — source=routing (verified against AWF /reflect).
• The two review threads (detector output propagation, schema category) are fixed. We resolved them.

Blocking 1: the failure report for a real mismatch still shows no mismatch. In run 38089386804 the conclusion job has GH_AW_MODEL_ENDPOINT_MISMATCH_ERROR: true, and the unified session has the event. But the comment handle_agent_failure posted (on the existing issue for this workflow, githubnext/gh-aw-routing-sandbox#154) still contains only the generic incompletion block:

engine_driver_failure
Agent finished without emitting a terminal safe output; task completion could not be confirmed.
Driver exit code: 1. The engine driver exited before a terminal safe output was recorded.
Failure classification: engine_outage
Last engine error type: unknown
Retry attempts observed: 0

Expected: when the mismatch signal or the model_endpoint.mismatch event is present, the issue or comment shows classification model_endpoint_mismatch with the detail and fix text from the record, and doesn't call it an engine outage. The report_incomplete / engine_driver_failure block must not override it. Test: handle_agent_failure with the mismatch output set, the startup record present, no agent session and no terminal safe output. Assert the posted body contains the mismatch detail and fix, and does not contain engine_outage.

Blocking 2: a successful run filed a failure issue. In run 38089392076 every job succeeded. The CLI-mode claude-haiku-4.5 sub-agent got AWF's Routing model "claude-haiku-4.5" to /chat/completions is incompatible 400 and the main agent finished the task. The detector still set GH_AW_MODEL_ENDPOINT_MISMATCH_ERROR: true, and the conclusion job created githubnext/gh-aw-routing-sandbox#159, "[aw] Task has model/endpoint mismatch". It is titled "Workflow Failure" and has no details at all. Expected: a recovered sub-agent mismatch in a successful run does not create a failure issue. It is already reported by the startup warning and audit's "Sub-agent Failed" finding. The startup mismatch path (Blocking 1) and real failed runs still report. Test: detector output true, agent job succeeded, terminal safe output present. Assert no failure issue is created.

Observation: run 38089386804's report was added as a comment to #154, whose title is "[aw] Task experienced an engine outage" (created by the previous version for the same workflow). If the existing-issue lookup is meant to match by workflow regardless of category, that's fine. Otherwise it should match on the category too.

[Copilot]

Tested 698fd4b in our sandbox (AWF v0.28.50, Copilot CLI 1.0.90, org billing; compiled with --gh-aw-ref 698fd4b57481c28b7b30ef83c312d71f4ba35430).

Fixed, verified in real runs...

Fixed both blocking findings in 9c55f8f. Both are in handle_agent_failure.cjs.

Blocking 1: The conclusion job runs handle_agent_failure before generate_usage_artifacts writes usage/aw_session.jsonl. That file didn't exist yet when the report was built, so the mismatch was never found.

• If the session file is missing (and GH_AW_UNIFIED_SESSION isn't set), the mismatch event is now collected in memory from the downloaded agent/model-endpoint-mismatch.json. It still goes through the collector's validation and provenance checks.
• When a mismatch is present, it takes precedence over the collector's engine_driver_failure / engine_outage result. That covers the issue title, the failure categories, and the issue-matching cause. The collector-generated report_incomplete block is dropped. The engine-failure block is suppressed when the record's detail and fix are shown.
• Because the cause is no longer engine_outage, these reports should stop being added as comments to the old engine-outage issue (your observation about [claude-test] Pull request not found #154).
• New test: mismatch output set, startup record present, no agent session, collector engine_driver_failure output. It asserts the body has the record's detail and fix, the mismatch classification and title, and no engine_outage or Last engine error type: unknown.

Blocking 2: In a successful run, GH_AW_MODEL_ENDPOINT_MISMATCH_ERROR=true now triggers failure handling only when a startup-phase mismatch record exists. Runs where the agent job didn't succeed still report as before. A runtime record from a successful run is no longer used for the title or body.

• New test: detector output true, agent succeeded, terminal safe output present. It asserts no issue or comment is created and no issue search runs.

Both new tests fail without the fix and pass with it. All 484 tests in handle_agent_failure.test.cjs pass.

On checks: the pre-push gate still fails its custom Go linter on unchecked fmt.Fprintf calls in pkg/workflow/compiler_yaml_artifacts.go. This PR doesn't change those lines, so I left them alone. CodeQL timed out in validation, so it hasn't scanned this commit.

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Pushed 80438e8. make merge-main confirmed the branch already contains current main. I refreshed the WASM goldens and fixed the impacted Go tests by classifying model_endpoint.mismatch as requiring Aider attribution; make agent-report-progress now passes. The remaining build failure is the unchanged ../reference/cache-memory/ link in docs/src/content/docs/experimental/drive-memory.md on main, so I left it untouched to keep this PR in scope. The review threads were already replied to and resolved. No CI was triggered.

[SivaKesava1]

This PR now conflicts with main in actions/setup/js/handle_agent_failure.cjs and handle_agent_failure.test.cjs. The cause is #67471 ("Add root-cause diagnostics to generic failure issues", merged 22:42 UTC), which changed the same failure-report code: it adds the failing Agent step name and log tail (capped at 50 lines / 8,000 characters, masks kept, fenced) and an attribution fallback when job data is missing.

The merge needs to keep both behaviors:

• From Add root-cause diagnostics to generic failure issues #67471: step diagnostics, the attribution fallback, and safe rendering, with all of its tests.
• From this PR: a startup model/endpoint mismatch takes precedence over the generic report_incomplete / engine_driver_failure / engine_outage block and shows the mismatch detail and fix; a recovered sub-agent mismatch in a successful run creates no failure issue.

For a real startup mismatch, the issue may also show #67471's step diagnostics (the failing Execute GitHub Copilot CLI step with the Model endpoint mismatch: … log line), but its classification must stay model_endpoint_mismatch, not engine outage. We'll re-test the merged head in our sandbox once it's pushed.

[Copilot]

This PR now conflicts with main in actions/setup/js/handle_agent_failure.cjs and handle_agent_failure.test.cjs. The cause is #67471 ("Add root-cause diagnostics to generic failure issues", merge...

Merged latest main in d8b60d9 and resolved the conflicts.

• handle_agent_failure.cjs: I kept this PR's mismatch-filtered reportItems and combined them with Add root-cause diagnostics to generic failure issues #67471's includeAgentText: !needsFailureDiagnostics and metadataOnly options. Generic engine context is still suppressed when there is mismatch context. When a mismatch is present, the failed-step metadata is now also fetched, and buildFailureDiagnosticsContext renders for model_endpoint_mismatch. In that case it says "The model/endpoint mismatch above is the classified cause." instead of "The runtime did not record a specific cause."
• handle_agent_failure.test.cjs: I kept all tests from both sides and took main's renamed "omits agent-generated diagnostics…" test. I added one combined test: a startup mismatch with a failed Run agent step. It checks the issue title, the mismatch block with its Cause and Fix, failure_categories: model_endpoint_mismatch, and the "Failure Diagnostics" section with the job conclusion and failing step. It also checks that engine_outage and engine_driver_failure are absent and the step log text doesn't appear.
• Add root-cause diagnostics to generic failure issues #67471 as it landed on main uses Actions metadata only. It doesn't download step logs (no 50-line / 8,000-char tail), so the combined test checks the failing-step section and that no log text leaks.
• Tests: handle_agent_failure 503/503 passed. The impacted JS tests (6 files, 1,210 tests) and Go tests run by make agent-report-progress passed.
• One check still fails: the custom Go linter that main just added ([linter-miner] Add unchecked-deferred-close linter #67481, unchecked-deferred-close). It also flags pkg/cli/audit_report.go on main, so I left it, per scope. It also flags this PR's pkg/cli/model_routing_session.go:128 (defer file.Close()); I didn't touch that either.

[gh-aw-bot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes.

Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI.

Sous-chef head: d8b60d9
Sous-chef work:
Sous-chef state: 5f4530984cad27f2ea0fb1978c45d95caa05aeca578b35e86b3b961bcd7f5742

Generated by 👨‍🍳 PR Sous Chef · pi · haiku45 · 3.07 AIC · ⌖ 10.6 AIC · ⊞ 1K · ◷
_{Comment /souschef to run again}

[Copilot]

@copilot address the following outstanding work in one pass:

1. Update this branch with the latest main using make merge-main, resolving any conflicts and preserving the intended changes....

Merged latest main with make merge-main in 2f31276, preserving both this PR’s endpoint-mismatch evidence and main’s sub-agent model-unavailable handling. Build, recompile, formatting, and impacted tests passed; the final gate still reports existing unchecked-deferred-close findings also present in origin/main. No CI was triggered.

[SivaKesava1]

Verified e0861dd (after merging main, including #67471) in our sandbox (AWF v0.28.50, Copilot CLI 1.0.90, org billing):

• Forced mismatch (gpt-5.6-luna with engine.env.COPILOT_PROVIDER_WIRE_API: completions), run 38105198559: fails at startup, no retries. The failure issue is classified model_endpoint_mismatch and shows "Cause: Model endpoint mismatch: configured model 'gpt-5.6-luna'; COPILOT_PROVIDER_WIRE_API=completions (override, /chat/completions); supported endpoints: [/responses, ws:/responses]." and "Fix: Pin a compatible model, remove the COPILOT_PROVIDER_WIRE_API override, or upgrade gh-aw.", plus Add root-cause diagnostics to generic failure issues #67471's failing-step section. No engine-outage text. usage/aw_session.jsonl and gh aw audit show the same record.
• Routed Luna main with a cross-family claude-haiku-4.5 sub-agent (CLI mode), run 38105203891: all jobs green; startup warning names the agent file and links [aw] PR Code Quality Reviewer had a request rejected #67460 and BYOK: sub-agents always use the session's wire API, so a sub-agent on a model from the other family fails with a 400 copilot-cli#5103; the wire-API log says source=routing; audit reports the failed sub-agent attempts; no failure issue is created for this successful run.
• Earlier rounds: model: auto resolves with the wire API from AWF /reflect.

All checks pass, review threads are resolved, and the branch merges cleanly.