Client sends getConfiguration over hardcoded plaintext HTTP to the raw endpoint - consumer ISP security products (CUJO/Xfinity/Spectrum) silently break joins, with no possible server-side mitigation

[Rhetorikk]

Summary

During the connect flow, after getEndpoints resolves, the client POSTs getConfiguration to the raw endpoint IP over plaintext http://, unconditionally, with no HTTPS attempt and no retry. Consumer ISP security middleboxes — CUJO AI-powered products such as Xfinity Advanced Security and Spectrum Security Shield, which ship enabled by default on ISP-rented gateways — treat a plaintext HTTP POST to a never-before-seen bare IP on a non-standard port as hostile and silently drop or tamper with it. The join dies at that exact step.

The result: a meaningful fraction of players (we measured ~5–10% of join attempts on our server) cannot connect to any server whose IP is new to their household's gateway — and there is no server-side mitigation, even with the full documented proxy setup deployed.

The code

code/components/citizen-legacy-net-resources/src/ResourceNetBindings.cpp:

auto curServerUrl = fmt::sprintf("https://%s/", m_netLibrary->GetCurrentPeer().ToString());

// #TODO: remove this once server version with `18d5259f60dd203b5705130491ddda4e95665171` becomes mandatory
auto curServerUrlNonTls = fmt::sprintf("http://%s/", m_netLibrary->GetCurrentPeer().ToString());
...
httpClient->DoPostRequest(fmt::sprintf("%sclient", curServerUrlNonTls), ...)

• getConfiguration always uses curServerUrlNonTls — plaintext, hardcoded.
• On failure the flow hard-stops: GlobalError("Obtaining configuration from server failed.%s"), no https retry.
• The #TODO shows this is a known transitional state gated on the h2 flow-control fix (18d5259f) becoming mandatory server-side. That server fix shipped long ago; the client-side plaintext path remains.

Why server operators cannot work around it

We deployed the complete documented mitigation stack (Proxy Setup):

• sv_forceIndirectListing true + sv_listingHostOverride → connect endpoint on our TLS domain behind Cloudflare. Steps 2–5 (info.json, initConnect + deferrals, getEndpoints) all ride TLS and succeed for affected players (confirmed via reverse-proxy access logs: HTTP 200 on initConnect and getEndpoints, then silence).
• fileserver_add → resource downloads on the TLS CDN.
• The one remaining plaintext request — getConfiguration to the raw endpoint — is precisely where affected clients die, and:
  • It cannot be proxied to TLS: the URL is built client-side from the raw peer address.
  • It cannot be TLS-terminated at the port: a TLS-only listener on the game TCP port breaks all clients, because every client insists on plaintext for this request (we verified this before nearly deploying the docs' "Local TLS Proxy" — see note below).

Evidence / reproduction

Environment: server artifact 31689 (Linux), fresh dedicated-server IP + newly-registered asset domain (a normal migration scenario).

1. Affected household: Xfinity (Comcast) customer with ISP-rented gateway, Advanced Security at default (on). 10/10 join attempts over ~45 minutes reproduce identically:
   • Reverse-proxy log: POST /client (initConnect) → 200 (~5.5 KB), POST /client (getEndpoints) → 200 (27 B, endpoint list), both via Cloudflare TLS.
   • Then zero further traffic from that household to the server — the plaintext getConfiguration POST to the raw IP never arrives. Server-side, the connection never reaches playerJoining.
2. Same player connects normally via VPN or mobile hotspot (path avoids the gateway's inspection).
3. Server IP is clean on Spamhaus/SpamCop/SORBS/Barracuda/GSB; the asset domain was explicitly confirmed "not blocked" by Xfinity's own review portal — consistent with the block being the plaintext-to-raw-IP heuristic, not reputation feeds.
4. A second variant observed on other households: the ISP product MITMs the plaintext HTTP exchange and the client surfaces it closed the connection while deferring / TCP sequence desync (client ACKs bytes the server never sent — visible in packet captures).

These products are default-on for tens of millions of North American households and their deployment is growing; any server that migrates to a new IP inherits this failure class for weeks.

Suggested fix

Ship the #TODO: prefer curServerUrl (https) for getConfiguration, falling back to plaintext only if the TLS attempt fails (or gate on a server capability flag returned by initConnect/getEndpoints). The server side already multiplexes TLS/plaintext on the game TCP port, so no server-operator action would be required. Clients don't validate endpoint certificates, so the server's built-in self-signed cert suffices — the goal is not authentication but making the payload opaque to on-path middleboxes.

Note for the docs team

Until the client change ships, the Proxy Setup page's "Local TLS proxy" section is dangerous as written: a TLS-only listener on the game TCP port breaks getConfiguration for every client (plaintext POST into a TLS socket → Obtaining configuration from server failed. for the whole player base). A warning that the game TCP port must continue to accept plaintext would save operators from a full outage.