Repository navigation
fix(copilot): remove unavailable native web-tool support - #66310
Conversation
Remove native web search and CLI fetch permissions, validate resolved workflow configuration, and preserve SDK custom proxy-aware fetch. Update regression coverage and migration documentation for #65043. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
🧠 Matt Pocock Skills Reviewer has completed the skills-based review. ✅
|
|
✅ Design Decision Gate 🏗️ completed the design decision gate check. See the comment below for the result and any generated ADR draft.
|
|
✅ PR Code Quality Reviewer completed the code quality review. No blocking changed-line issues found in PR #66310; submitted a non-blocking review instead of creating comments.
|
|
✅ Ponytail Reviewer completed successfully! Lean already. Ship.
|
|
✅ Test Quality Sentinel completed test quality analysis. Test Quality Sentinel skipped because pre-fetch PR data was unavailable: unable to fetch test file diff
|
🏗️ Design Decision Gate: ADR RequiredThis PR triggered ADR enforcement and no Architecture Decision Record was found (not in the PR body, not on the branch, not in linked issue #65043). Why enforcement applies
Architectural decision inferred from the diff
A draft ADR has been committed to this branch:
Next action for the authorReview the draft ADR, correct anything that misstates the intent (especially the alternatives and the breaking-change framing /
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
Verdict
I did not find any high-confidence, changed-line bugs worth blocking this PR. The compile-time validation, CLI argument changes, SDK tool-contract enforcement, and imported-engine coverage all line up with the intended behavior change.
🔎 Code quality review by PR Code Quality Reviewer · copilot · gpt54 · 59.7 AIC · ⌖ 5.4 AIC · ⊞ 19.6K
Comment /review to run again
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
The JSON importer still emits rejected Copilot web-search configuration, and several documentation references remain invalid.
Review effort: Balanced
Findings: 1
Open (2)
What changed in this PR
Rejects Copilot native web tools that are unavailable in offline BYOK mode while preserving SDK-mode custom web fetch.
Changes:
- Adds compile-time validation for Copilot web tools.
- Removes obsolete native permissions, version gating, and built-in MCP handling.
- Updates tests, documentation, fixtures, and release notes.
| File | Description |
|---|---|
.changeset/patch-reject-copilot-offline-web-tools.md |
Adds patch release note. |
actions/setup/js/copilot_sdk_tool_config.cjs |
Rejects SDK native web search. |
actions/setup/js/copilot_sdk_tool_config.test.cjs |
Tests SDK rejection. |
docs/src/content/docs/engines/copilot.md |
Documents Copilot limitations. |
docs/src/content/docs/reference/engines.md |
Updates capability matrix. |
docs/src/content/docs/reference/glossary.md |
Corrects web-search definition. |
docs/src/content/docs/reference/tools.md |
Documents rejected configurations and SDK fetch. |
docs/src/content/docs/reference/web-search.md |
Directs Copilot users to MCP search. |
pkg/constants/version_constants.go |
Removes obsolete search minimum version. |
pkg/workflow/agent_validation.go |
Adds Copilot web-tool validation. |
pkg/workflow/compiler_orchestrator_tools.go |
Removes old network-only validation. |
pkg/workflow/compiler_orchestrator_workflow.go |
Validates resolved file workflows. |
pkg/workflow/compiler_string_api.go |
Validates resolved string workflows. |
pkg/workflow/copilot_engine.go |
Marks native search unsupported. |
pkg/workflow/copilot_engine_execution.go |
Always disables built-in MCPs. |
pkg/workflow/copilot_engine_execution_test.go |
Tests omitted native permissions. |
pkg/workflow/copilot_engine_installation.go |
Removes search version gating. |
pkg/workflow/copilot_engine_sdk_tools.go |
Keeps custom fetch and disables search. |
pkg/workflow/copilot_engine_sdk_tools_test.go |
Updates SDK capability expectations. |
pkg/workflow/copilot_engine_tool_arguments_test.go |
Tests CLI and SDK permissions. |
pkg/workflow/copilot_engine_tools.go |
Limits web-fetch permission to SDK mode. |
pkg/workflow/copilot_installer_test.go |
Verifies no search minimum version. |
pkg/workflow/copilot_web_tools_validation_test.go |
Adds validation and import regression tests. |
pkg/workflow/docker_firewall_pin_compile_test.go |
Removes invalid Copilot fetch fixture. |
pkg/workflow/engine_firewall_support.go |
Removes obsolete firewall warnings. |
pkg/workflow/engine_firewall_support_test.go |
Tests unconditional rejection. |
pkg/workflow/fetch_integration_test.go |
Excludes rejected Copilot CLI fetch. |
pkg/workflow/firewall_workflow_test.go |
Removes invalid fetch declarations. |
pkg/workflow/importable_tools_test.go |
Uses SDK mode for imported fetch. |
pkg/workflow/search_integration_test.go |
Verifies compilation failure. |
pkg/workflow/testdata/TestWasmGolden_CompileFixtures/smoke-copilot.golden |
Regenerates Copilot golden output. |
pkg/workflow/testdata/wasm_golden/fixtures/smoke-copilot.md |
Removes unsupported fetch tool. |
| for _, tool := range []string{"web-fetch", "web-search"} { | ||
| if !isCopilotToolValueEnabled(tools, tool) || (tool == "web-fetch" && engineConfig != nil && engineConfig.CopilotSDK) { | ||
| continue |
There was a problem hiding this comment.
Fixed in fa52ada: JSON imports now omit Copilot’s unavailable native web_search tool and emit guidance, with compiler round-trip coverage for the default engine.
| Web search is disabled by default for **Codex**, **Claude**, and **Copilot**. Claude and Copilot enable it only when `web-search:` is declared. Codex enables a shared native search/browsing tool when either `web-search:` or `web-fetch:` is declared; the two capabilities cannot be disabled independently. With neither tool, Codex runs with `-c web_search="disabled"`. Use `network.hosted-web` to restrict Codex's hosted retrieval. | ||
| Web search is disabled by default for **Codex** and **Claude**. Claude enables it only when `web-search:` is declared. Codex enables a shared native search/browsing tool when either `web-search:` or `web-fetch:` is declared; the two capabilities cannot be disabled independently. With neither tool, Codex runs with `-c web_search="disabled"`. Use `network.hosted-web` to restrict Codex's hosted retrieval. | ||
|
|
||
| **Copilot** runs in offline BYOK mode, which disables its native web tools. Compilation rejects `web-search:` and, in CLI mode, `web-fetch:`. Use an MCP server for search or fetch. With `engine.copilot-sdk: true`, `web-fetch:` remains supported by a custom proxy-aware tool that follows `network.allowed`; this does not enable native web search. |
There was a problem hiding this comment.
Fixed in fa52ada: the gallery fetch example now declares Codex, the earlier ADR is marked superseded by ADR-66310, and the network guide documents unconditional Copilot compile errors.
There was a problem hiding this comment.
Skills-Based Review 🧠
Applied /diagnosing-bugs — no actionable issues found; approving.
📋 Analysis
Root cause addressed, not just symptom
The PR removes both the advertisement of unavailable Copilot native web_fetch/web_search tools (capabilities, --allow-tool flags, SDK builtin registration) and the version-gating workaround (CopilotWebSearchMinVersion, copilotSupportsWebSearch) that previously tried to paper over the mismatch. This matches the stated root cause: Copilot behind AWF with COPILOT_OFFLINE=true never exposes these tools regardless of CLI version, so version-gating was the wrong fix. Verified no dangling references remain to the removed helpers (copilotNeedsBuiltinMCPs, copilotSupportsWebSearch, CopilotWebSearchMinVersion).
Regression coverage
copilot_web_tools_validation_test.go is a strong addition: a full matrix over strict × sdk × tool × value plus an imported-engine-config variant verifying SDK settings from imports: are honored before validation runs. Existing tests (engine_firewall_support_test.go, search_integration_test.go, copilot_installer_test.go, copilot_engine_execution_test.go) were updated in place to assert the new rejection behavior rather than being deleted, so prior scenarios stay guarded.
Error paths are explicit
Both the Go compiler (validateCopilotWebToolsSupport) and the JS SDK config (validateToolPermissionParity, buildCopilotSDKSessionToolConfig) now fail loudly with an explanatory message and actionable suggestion (MCP server, engine: codex, or copilot-sdk: true) instead of silently warning or no-op'ing — closing the original gap where permissions were granted for tools that didn't exist at runtime.
Verification performed
go build ./...andgo vet ./pkg/workflow/...— clean.- Targeted
go testrun across the new/updated test names — all pass. - Confirmed golden fixture (
smoke-copilot.golden) and fixture source (smoke-copilot.md) were updated consistently (removedweb-fetch:declaration, no--allow-tool web_fetch/web_searchin compiled output). - Docs (
engines.md,copilot.md,tools.md,glossary.md,web-search.md) consistently describe the new behavior and the SDK custom-fetch escape hatch.
Positive highlights
- ✅ Validation runs after import resolution (
compiler_orchestrator_workflow.go/compiler_string_api.go), correctly honoring SDK settings pulled in viaimports:. - ✅ Clean removal of now-dead
checkToolsNetworkSupport/reportUnfirewalledComponentmachinery that the new blanket rejection supersedes. - ✅ Changeset file included for release notes.
No blocking issues identified.
🧠 Reviewed using Matt Pocock's skills by Matt Pocock Skills Reviewer · copilot · sonnet50 · 84.5 AIC · ⌖ 15.5 AIC · ⊞ 10.1K
Comment /matt to run again
There was a problem hiding this comment.
Review Summary
Reviewed this backend/compiler change (no UI surface, so Impeccable UI modes don't apply — applied a correctness/reliability-focused read instead per the fallback guidance for bug_fix changes).
What changed: Rejects Copilot's unavailable native web-search/CLI-mode web-fetch at compile time (offline BYOK mode disables them), removes dead version-gating code (copilotSupportsWebSearch, CopilotWebSearchMinVersion, copilotNeedsBuiltinMCPs), and preserves the SDK's proxy-aware custom web_fetch.
Findings:
- Validation logic (
validateCopilotWebToolsSupport) correctly resolves engine config post-import so SDK settings from imported files are honored (verified viaTestCopilotWebToolsImportedEngineConfig). - Error messages are clear, differentiate the MCP-server suggestion for
web-searchfrom the SDK-mode suggestion forweb-fetch, and include the root cause (COPILOT_OFFLINE=true). - No orphaned references to removed functions/constants (
copilotSupportsWebSearch,CopilotWebSearchMinVersion,copilotNeedsBuiltinMCPs) remain anywhere in non-test code. --disable-builtin-mcpsis now unconditional, consistent with native web tools always being unavailable.- Test coverage is strong: new
copilot_web_tools_validation_test.gomatrix-tests strict/non-strict × SDK/non-SDK × fetch/search × value shapes, plus updated unit tests for execution args, SDK tool config, and installer version gating. - Docs (
engines.md,tools.md,web-search.md,glossary.md) were updated consistently with the new behavior. - Build (
go build ./...) and targeted tests pass. Fullgo test ./pkg/workflow/...has 4 pre-existing failures (TestVerifyGitCredentials*,TestCloudHypervisorSetupBundleScriptExecutesAgainstFixtures,TestBuildDynamicEnclaveExpiryScript...) reproducible on the base commit without this PR's changes — unrelated to this change (sandbox/time-based environmental flakiness).
No blocking issues found. No inline comments posted — nothing rises above routine engineering judgment calls.
🧵 Reviewed using Impeccable skills by Impeccable Skills Reviewer · copilot · sonnet50 · 75.7 AIC · ⌖ 13.3 AIC · ⊞ 8.1K
|
@copilot address the following outstanding work in one pass:
Push the necessary fixes, reply to each listed review thread and resolve it when addressed. Ignore feedback already answered or resolved. Use the pr-finisher skill and stop when only human review or CI remains; do not trigger CI. Sous-chef head: ffa7dfb
|
…tools Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Co-authored-by: gh-aw-bot <259018956+gh-aw-bot@users.noreply.github.com>
Copilot runs behind AWF with
COPILOT_OFFLINE=true, which disables native web tools. Previously, gh-aw advertised and granted permissions for tools that were not available at runtime. A real Copilot CLI 1.0.91 probe exposed 17 tools but neitherweb_fetchnorweb_search, despite explicitly allowing both.Changes
tools.web-searchand CLI-modetools.web-fetchduring compilation, regardless of strict mode or network restrictions. Validate merged tools against the resolved engine configuration so imported SDK settings work consistently in file and string compilation.web_fetch. An actual SDK 1.0.16 / CLI 1.0.91 probe verified that this replacement remains visible and executes through a proxy in offline mode.Validation
Passed: focused Go unit and integration tests, 113 SDK JavaScript tests, standard Go lint, workflow drift check, and the regenerated Copilot smoke golden fixture. The golden regeneration also includes existing compiler-output drift.
The full repository gate did not pass: pre-existing custom-lint findings and macOS baseline failures remain, and the pinned
vitest@5.0.3dependency was unavailable from the configured npm feeds. JavaScript tests used isolated Vitest 5.0.2 tooling without changing repository dependency manifests or lockfiles.Fixes #65043