Skip to content

fix(read): keep the local fetch tier direct — bypass system proxy env - #88

Merged
tw93 merged 2 commits into
tw93:mainfrom
BiBoyang:fix/read-local-tier-proxy
Sep 19, 2026
Merged

tw93 merged 2 commits into
tw93:mainfrom
BiBoyang:fix/read-local-tier-proxy

Conversation

@BiBoyang

Copy link
Copy Markdown
Contributor

Summary

read's fetch cascade documents the local tier as the privacy-preserving one: "Default (no --use-proxy): local extractor only. URL is never sent to a [third party]". But fetch_local.py uses urllib.request.urlopen, which honors http_proxy/https_proxy env by default — so on any machine with a system proxy configured, the "local" tier silently sends the target host through that proxy without the --use-proxy opt-in. The privacy contract is broken exactly where it claims to hold.

(We found this the other way around: on a machine whose proxy is SOCKS-only, urllib mishandles the scheme and the local tier always failed with "Remote end closed connection without response", falling through to defuddle. curl worked fine on the same host. The failure mode made the leak visible; on a correctly configured HTTP proxy the leak would be silent.)

Fix

Build the local-tier opener with an empty ProxyHandler, so the local tier is a direct connection by contract. If a host genuinely requires a proxy for outbound access, the local tier now fails honestly and the cascade still works via the explicit --use-proxy opt-in — which is the documented privacy semantics.

Adds a unit test pinning the behavior: with http_proxy/https_proxy set, fetch_html must build its opener with an empty proxy map.

python3 -m pytest tests/python/test_fetchers.py passes (8 tests, incl. the new one); python3 scripts/verify_skills.py passes.

@tw93
tw93 merged commit 66a48e6 into tw93:main Sep 19, 2026
2 checks passed
@tw93

tw93 commented Sep 19, 2026

Copy link
Copy Markdown
Owner

@BiBoyang Merged, shipped in 3.38.0.

Verified the mechanism rather than taking it on description: with http_proxy set, the default opener carries a ProxyHandler populated from the environment, so the local tier really was sending the target host through a system proxy while documenting itself as the tier that never leaves the machine. I also red-ran your test by reverting the fix, and it fails, so it pins the behavior rather than passing vacuously.

The way you found it is the useful part: a SOCKS-only proxy made the leak loud, where a working HTTP proxy would have kept it silent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants