fix(read): keep the local fetch tier direct — bypass system proxy env - #88
Merged
Merged
Conversation
…aves-machine opt-in stays exclusive to --use-proxy
Owner
|
@BiBoyang Merged, shipped in 3.38.0. Verified the mechanism rather than taking it on description: with The way you found it is the useful part: a SOCKS-only proxy made the leak loud, where a working HTTP proxy would have kept it silent. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
read's fetch cascade documents the local tier as the privacy-preserving one: "Default (no --use-proxy): local extractor only. URL is never sent to a [third party]". Butfetch_local.pyusesurllib.request.urlopen, which honorshttp_proxy/https_proxyenv by default — so on any machine with a system proxy configured, the "local" tier silently sends the target host through that proxy without the--use-proxyopt-in. The privacy contract is broken exactly where it claims to hold.(We found this the other way around: on a machine whose proxy is SOCKS-only, urllib mishandles the scheme and the local tier always failed with "Remote end closed connection without response", falling through to defuddle. curl worked fine on the same host. The failure mode made the leak visible; on a correctly configured HTTP proxy the leak would be silent.)
Fix
Build the local-tier opener with an empty
ProxyHandler, so the local tier is a direct connection by contract. If a host genuinely requires a proxy for outbound access, the local tier now fails honestly and the cascade still works via the explicit--use-proxyopt-in — which is the documented privacy semantics.Adds a unit test pinning the behavior: with
http_proxy/https_proxyset,fetch_htmlmust build its opener with an empty proxy map.python3 -m pytest tests/python/test_fetchers.pypasses (8 tests, incl. the new one);python3 scripts/verify_skills.pypasses.