Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -217,3 +217,27 @@ docs/source/_autosummary/
*~
*.swp
*.swo

# Local working scratch (debug docs, logs, scan reports). Excluded from
# git so security scanners and contributors don't see false positives
# from cached scan outputs left in the working tree.
tmp/

# Claude Code local config (per-machine; never in repo).
.claude/

# Local virtualenvs that aren't `.venv` (the canonical name). Older
# branches sometimes leave `venv/`, `venv_py313_test/`, etc.; these
# accumulate stale dependency metadata that pollutes vulnerability
# scanners with already-superseded CVEs.
venv/
venv_*/

# Security scan output artifacts (gitleaks/grype/semgrep/checkov/
# hadolint). Sometimes committed accidentally; always regeneratable.
*-report.json
*-stderr.txt

# uv-managed venv lockfile. Project uses poetry.lock as the canonical
# lockfile; committing both creates ambiguity for contributors.
uv.lock
24 changes: 24 additions & 0 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Gitleaks config: extend the default ruleset with an allowlist of
# paths that contain known false positives. Same scope as
# .gitignore / .semgrepignore additions in the IPv6 work (#212).

[extend]
# Inherit gitleaks' bundled default ruleset.
useDefault = true

[allowlist]
description = "Paths excluded from gitleaks scans"
paths = [
# Claude Code per-machine config. May contain short-lived AWS STS
# session tokens for tool integrations (Opsera S3 upload URLs etc.);
# never in repo, expires fast, not a real credential leak.
'''^\.claude/.*''',
# Local working scratch. Regenerated scan reports echo example
# AWS-key-shaped strings back at the scanner (recursive false
# positive).
'''^tmp/.*''',
# Stale local virtualenvs.
'''^\.venv/.*''',
'''^venv/.*''',
'''^venv_.*/.*''',
]
15 changes: 15 additions & 0 deletions .semgrepignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
# Scan-output artifacts contain example AWS-key-shaped strings that
# semgrep would otherwise flag as secrets — recursive false positive.
tmp/
*-report.json
*-stderr.txt

# Claude Code local config (per-machine; not in repo). May contain
# placeholder credentials in example settings.
.claude/

# Stale local virtualenvs accumulate already-superseded dependency
# metadata that's not in pyproject.toml.
.venv/
venv/
venv_*/
78 changes: 78 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,84 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

## [Unreleased]

### Added
- **Full IPv6 support across the stack** (#201). IPv6 is now first-class
alongside IPv4 across detection, validation, anonymity comparison,
SOCKS5 proxying, and `[v6]:port` provider parsing.
- `Resolver.host_is_ip()` accepts both IPv4 and IPv6 literals
(loopback `::1`, documentation prefix `2001:db8::1`, IPv4-mapped
`::ffff:192.0.2.1`, link-local with zone IDs `fe80::1%eth0`).
- `Resolver.get_real_ext_ip()` returns RFC 5952 canonical form and
accepts IPv6 responses from upstream IP-detection services. Adds
`https://api64.ipify.org/` to the default endpoints so proxybroker
works on IPv6-only networks.
- `proxybroker/utils.py`: new `canonicalize_ip(s) -> str | None`
helper exposes the RFC 5952 canonical form via stdlib `ipaddress`.
`get_all_ip()` rewritten to extract IPv4 substrings (legacy
contract) plus IPv6 candidates validated through `ipaddress`,
returning canonical form so equivalent encodings collapse to a
single set element.
- `IPv6BracketedPortPattern` and the new `find_proxy_pairs(text)`
helper extract `[v6]:port` proxy pairs from text (including
link-local zone IDs like `[fe80::1%eth0]:8080`). Wired into the
public `Provider.find_proxies` (kept out of the lower-level
`_find_proxies` raw-regex helper so subclasses that pipe its
output through `b64decode`/custom decoders aren't fed
already-normalized `(host, port)` tuples) and into
`Broker._load`'s file/raw string parsing so every provider feed
can surface IPv6 proxies. Bracketed IPv6 spans are masked from
the IPv4 line regex in both paths to prevent IPv4-mapped IPv6
addresses (`[::ffff:1.2.3.4]:8080`) from spawning a phantom IPv4
entry.
- `_get_anonymity_lvl` and `Judge.check` now use canonical-form set
membership instead of raw substring matching, so v6 leaks are
correctly classified regardless of how the judge formatted the
address (uppercase, expanded with leading zeros, with or without
`::` compression).
- `Proxy(host=...)` accepts IPv6 literals; `as_text()`, `__repr__()`
and `Proxy.log` bracket IPv6 hosts per RFC 3986 § 3.2.2 so the
output is unambiguous.
- `Socks5Ngtr` emits `ATYP=0x04` + 16-byte address for IPv6
destinations (`ATYP=0x01` + 4-byte for IPv4 unchanged), unblocking
SOCKS5 connections to IPv6 origins. Reply ATYP read from the
*response* (not the request) per RFC 1928 § 6 — dual-stack
proxies may bind a different family. SOCKS4 stays IPv4-only by
spec (the SOCKS4 protocol predates IPv6 and only defines a 4-byte
address field; ATYP=0x04 is a SOCKS5/RFC-1928-only construct).
- **Happy Eyeballs DNS** (RFC 8305 § 3) in `Resolver.resolve()`.
When the caller doesn't pin `qtype` or `family`, A and AAAA
queries fire in parallel; the first non-empty answer wins; the
slower task is cancelled. Callers that explicitly pass
`qtype="A"` or `family=socket.AF_INET[/INET6]` get the
single-family path, preserving the legacy A-only contract.
- **Modern type hints** on `canonicalize_ip`, `find_proxy_pairs`,
`_format_host_port` (PEP 604 union syntax). Better IDE/mypy
ergonomics for downstream consumers.

### Changed
- `find_proxy_pairs(text)` now canonicalises both IPv4 AND IPv6
entries (#201). IPv4 canonical form equals identity, so legacy
v4-only feeds see no behavior change — the contract is just
consistent now: every returned `(ip, port)` has a canonical IP.
- `Socks4Ngtr.negotiate(ip=v6, ...)` raises `BadResponseError(
"SOCKS4 protocol does not support IPv6 destinations")` instead
of a cryptic `OSError` from `inet_aton` (#201). Logs point users
at SOCKS5 for IPv6.
- `Resolver.host_is_ip` legacy behavior preserved for IPv4 with leading
zeros (e.g. `127.0.0.001`); accepted by the wrapper even though
CPython 3.9.5+ rejects them under CVE-2021-29921. Documented in the
docstring; some provider feeds in the wild still emit that form.
- `judge.py:Judge.check` and `checker.py:_get_anonymity_lvl` now log
the boolean visibility result (rather than the substring expression)
for symmetry with how the v6 set-membership comparison is done.

### Removed
- **`IPv6Pattern` 700-character regex** (#201) and its
`# nosemgrep: regex_dos` suppression. The pattern carried many
capture groups so `re.findall` returned tuples (not strings) and
IPv6 entries silently never matched. Replaced by the narrow
`_IPV6_CANDIDATE_PATTERN` tokenizer + stdlib `ipaddress` validator.

## [2.0.0b2] - 2026-05-08

🎯 **Custom Provider System & Quality Pass**
Expand Down
18 changes: 18 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,24 @@ PROVIDERS if providers is None else providers
### Protocol Priority
Deterministic order: SOCKS5 → SOCKS4 → CONNECT:80 → CONNECT:25 → HTTPS → HTTP

### IPv6 Support
First-class alongside IPv4 across the stack (PR for #201). Key invariants:
- **Canonical form is RFC 5952** (`str(ipaddress.ip_address(...))`):
lowercase, leading-zeros stripped, longest zero-run as `::`. Use
`proxybroker.utils.canonicalize_ip(s) -> str | None` at every
boundary where IPs cross from text into comparison.
- **IPv4 contracts are preserved**. `IPPattern.findall` still handles
the legacy substring extraction (e.g. `127.0.0.1` from `127.0.0.1:80`).
Canonical form for IPv4 is identity, so set membership behavior is
unchanged for the IPv4 path.
- **IPv6 in URI authority MUST be bracketed** (RFC 3986). `Proxy.as_text`,
`Proxy.__repr__`, and proxy-list output all bracket v6 hosts.
- **SOCKS5 `ATYP=0x04`** for IPv6 destinations; SOCKS4 stays IPv4-only
by spec (no v6 ATYP in RFC 1928).
- **Provider feeds**: `[v6]:port` is parsed via `IPv6BracketedPortPattern`
+ stdlib validation. Helper `find_proxy_pairs(text)` returns
`[(ip_canonical, port), ...]` for both v4 and v6.

### Signal Handler Cleanup
`Broker.stop()` properly removes signal handlers to prevent memory leaks

Expand Down
24 changes: 22 additions & 2 deletions proxybroker/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,12 @@
from .proxy import Proxy
from .resolver import Resolver
from .server import Server
from .utils import IPPortPatternLine, log
from .utils import (
IPPortPatternLine,
IPv6BracketedPortPattern,
canonicalize_ip,
log,
)

# Pause between grabbing cycles; in seconds.
GRAB_PAUSE = 180
Expand Down Expand Up @@ -374,7 +379,22 @@ async def _load(self, data, check=True):
if isinstance(data, io.TextIOWrapper):
data = data.read()
if isinstance(data, str):
data = IPPortPatternLine.findall(data)
# Extract bracketed v6 entries first, then mask their spans
# in the input before running the v4 line regex. Without
# masking, an IPv4-mapped v6 entry like `[::ffff:1.2.3.4]:8080`
# would also produce a phantom v4 entry `1.2.3.4:8080` from
# the embedded literal. RFC 6874 zone IDs in brackets are
# accepted by the regex; validation is via canonicalize_ip.
v6_pairs = []
for raw_v6, port in IPv6BracketedPortPattern.findall(data):
canonical = canonicalize_ip(raw_v6)
if canonical is not None:
v6_pairs.append((canonical, port))
v4_input = IPv6BracketedPortPattern.sub(
lambda m: " " * len(m.group(0)), data
)
v4_pairs = IPPortPatternLine.findall(v4_input)
data = v4_pairs + v6_pairs
proxies = set(data)
for proxy in proxies:
await self._handle(proxy, check=check)
Expand Down
16 changes: 14 additions & 2 deletions proxybroker/checker.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,14 @@
from .judge import Judge, get_judges
from .negotiators import NGTRS
from .resolver import Resolver
from .utils import get_all_ip, get_headers, get_status_code, log, parse_headers
from .utils import (
canonicalize_ip,
get_all_ip,
get_headers,
get_status_code,
log,
parse_headers,
)


class Checker:
Expand Down Expand Up @@ -316,12 +323,17 @@ def _check_test_response(proxy, headers, content, rv):
def _get_anonymity_lvl(real_ext_ip, proxy, judge, content):
content = content.lower()
foundIP = get_all_ip(content)
# Defense in depth: canonicalise the real IP even if the caller already
# passed canonical form. Comparison must be canonical-vs-canonical so
# IPv6 textual encodings (case, leading zeros, compression) compare
# equal. Falls back to the raw value only if canonicalisation fails.
real_canonical = canonicalize_ip(real_ext_ip) or real_ext_ip

via = (content.count("via") > judge.marks["via"]) or (
content.count("proxy") > judge.marks["proxy"]
)

if real_ext_ip in foundIP:
if real_canonical in foundIP:
lvl = "Transparent"
elif via:
lvl = "Anonymous"
Expand Down
16 changes: 12 additions & 4 deletions proxybroker/judge.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

from .errors import ResolveError
from .resolver import Resolver
from .utils import get_headers, log
from .utils import canonicalize_ip, get_all_ip, get_headers, log


class Judge:
Expand Down Expand Up @@ -107,8 +107,16 @@ async def check(self, real_ext_ip):
return

page = page.lower()

if resp.status == 200 and real_ext_ip in page and rv in page:
# Use canonical-form set membership instead of substring `in page`
# so we correctly recognise IPv6 echoes regardless of how the
# judge formatted them (uppercase, expanded, compressed). For
# IPv4, canonical form equals the raw form, so behavior is
# preserved for the legacy v4 path.
page_ips = get_all_ip(page)
real_canonical = canonicalize_ip(real_ext_ip) or real_ext_ip
real_ip_visible = real_canonical in page_ips

if resp.status == 200 and real_ip_visible and rv in page:
self.marks["via"] = page.count("via")
self.marks["proxy"] = page.count("proxy")
self.is_working = True
Expand All @@ -118,7 +126,7 @@ async def check(self, real_ext_ip):
else:
log.debug(
f"{self} is failed. HTTP status code: {resp.status}; "
f"Real IP on page: {real_ext_ip in page}; Version: {rv in page}; "
f"Real IP on page: {real_ip_visible}; Version: {rv in page}; "
f"Response: {page}"
)

Expand Down
Loading
Loading