[miniflare] Support connect() on remote VPC Network bindings in local dev#14712
Conversation
🦋 Changeset detectedLatest commit: 2cc4927 The changes in this PR will be included in the next version bump. This PR includes changesets to release 8 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Codeowners approval required for this PR:
Show detailed file reviewers
|
d3b9589 to
bc96fcb
Compare
|
Thanks for putting this together. As it is a non-trivial change to our remote-bindings system I am going to need a bit of time to review it carefully. I am also talking to the Hyperdrive team to see if it would be useful to them too. |
|
We've just moved a lot of code around so this is now conflicted. Sorry. |
bc96fcb to
59aa143
Compare
|
Rebased onto latest Notes on the rebase:
Thanks for the quick answers on the design questions, @petebacondarwin — happy to adjust anything else that comes up in review. Rest easy — the merge conflicts were personally resolved by this raccoon. 🦝 |
|
Codeowners approval required for this PR:
Show detailed file reviewers |
|
the approach here makes sense to me. I suspect this will work for Hyperdrive |
… proxy client Remote bindings were wired locally as a proxy worker that only relayed fetch/JSRPC, so `binding.connect(address)` against a raw-TCP binding (e.g. a VPC network) failed with "Incoming CONNECT on a worker not supported". Add an inbound `connect` handler to the proxy client worker: it recovers the target address from `socket.opened.localAddress` (the verbatim `connect()` authority, per workerd#6059), opens a dedicated WebSocket to the remote proxy server, and relays bytes in both directions via a new `pipeSocketOverWebSocket` helper (with `truncateCloseReason` for the 123-byte close-reason limit). The relay tears both directions down together so a parked read or a missing close event can't leak the socket. The inbound connect handler is experimental-gated in workerd, so opt the proxy client worker into the `experimental` compat flag only for raw-TCP bindings via a new `rawTcp` option on `remoteProxyClientWorker`; VPC networks pass it. The existing HTTP/JSRPC proxy call sites are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orker The remote-bindings proxy server now detects the WebSocket connect upgrade (carrying the target in `MF-Connect-Address`), opens the real binding socket via `fetcher.connect(address)`, and relays bytes bidirectionally over the WebSocket. This is the edge half of the Miniflare `connect()` tunnel. The relay helper is a byte-for-byte mirror of the one in miniflare's remote-bindings-utils; this template is bundled standalone for the edge, so the logic is duplicated here rather than imported. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 13 tests for the `connect()` tunnel: end-to-end byte relay and address propagation plus a close/error matrix run against the real bundled ProxyServerWorker (in a second "edge" Miniflare instance), unit coverage of `pipeSocketOverWebSocket` teardown/error paths, and opt-in assertions that the `experimental` flag is set only when `rawTcp` is requested. Add a `@relay-under-test` vitest alias so the worker-side relay helper can be unit-tested without pulling worker-typed source into the node-side tsconfig (which excludes `src/workers/**`). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
59aa143 to
919708f
Compare
|
We dug into the Hyperdrive
So it looks like a natural follow-up PR (mostly plugin wiring + support-matrix/config work on top of this relay) rather than something to fold in here. Happy to take a stab at that after this lands, if useful. |
@cloudflare/autoconfig
@cloudflare/config
create-cloudflare
@cloudflare/deploy-helpers
@cloudflare/kv-asset-handler
miniflare
@cloudflare/pages-shared
@cloudflare/unenv-preset
@cloudflare/vite-plugin
@cloudflare/vitest-pool-workers
@cloudflare/workers-auth
@cloudflare/workers-editor-shared
@cloudflare/workers-utils
wrangler
commit: |
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BuKCFc94BFDpbwag1UfPbB
oxlint (typescript-eslint/no-non-null-assertion) forbids `!`; return a 400 for a connect upgrade missing the MF-Connect-Address header instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BuKCFc94BFDpbwag1UfPbB
Mirror the write-path pattern in both relay copies so a non-object rejection can't throw while building the 1011 close reason and mask the original error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BuKCFc94BFDpbwag1UfPbB
workers-devprod
left a comment
There was a problem hiding this comment.
Codeowners reviews satisfied

Fixes #14710.
What
Adds raw-TCP
connect()support for remote VPC Network bindings in local development. Previously,env.VPC_NETWORK.connect("host:port")worked when deployed but failed underwrangler dev/getPlatformProxy()withIncoming CONNECT on a worker not supported, because the local remote-bindings proxy only relayed HTTP and JSRPC.Raw TCP is now tunnelled end-to-end through the existing remote-bindings proxy:
packages/miniflare/src/workers/shared/remote-proxy-client.worker.ts) gains an inboundconnect(socket)handler. It recovers the caller's target address, opens a WebSocket to the remote proxy server, and relays bytes in both directions.ProxyServerWorker(packages/remote-bindings/templates/remoteBindings/ProxyServerWorker.ts) gains a WebSocket tunnel endpoint that callsenv[binding].connect(address)(routing through the real VPC tunnel) and relays bytes back.connecthandler only for raw-TCP bindings (VPC networks today).Why
The full root-cause analysis, source trace, and feasibility spikes are in #14710, so only a summary is repeated here. In short: remote-capable bindings are wired locally as a proxy worker that implements
fetch()/capnweb JSRPC only.env.BINDING.connect(addr)is a native workerd path that delivers an inbound CONNECT to that worker's entrypoint, which workerd rejects unless the worker carries theconnect_pass_throughorexperimentalcompat flag. Neither the proxy client nor the edge proxy server had a socket relay path, so the call died locally before ever reaching the remote proxy session.Design decisions
connecthandler (workerd#6059): the proxy client worker implementsexport default { connect(socket) }. This handler is experimental-gated in workerd, so it is enabled via a per-workerexperimentalcompat flag rather than relying on Miniflare's global--experimentalCLI flag (which is not sufficient — the check is per-worker).socket.opened.localAddress: the authority passed toconnect("host:port")arrives verbatim as the inbound socket'slocalAddresson the service-binding path, matching workerd'sconnect-handler-test.jscontract. No extra address-negotiation protocol is needed; the address is forwarded to the edge in anMF-Connect-Addressheader.connect(): each tunnel opens its own WebSocket to the proxy server rather than multiplexing over the existing capnweb JSRPC session. This keeps the relay independent of the JSRPC framing and simple to reason about for teardown.rawTcpopt-in (existing call sites unaffected):remoteProxyClientWorker()takes a newoptions?: { rawTcp?: boolean }parameter; only VPC networks pass{ rawTcp: true }. All existing HTTP/JSRPC proxy call sites keep their current behaviour and do not enable theexperimentalflag.Design questions — resolved (confirmed by maintainer in #14710)
Both questions raised in #14710 have been answered by @petebacondarwin, and the implementation already matches the preferred direction:
experimentalcompat flag on Miniflare's internal proxy-client worker (scoped to raw-TCP bindings) is acceptable, since it is only used locally and Miniflare controls the workerd version in use (comment). Implemented as such.connect()is preferred to avoid collisions (comment). Implemented as such.Known limitations
Documented in the relay's code comments as well:
socket.readableis drained as fast as it produces; inbound WebSocket messages are written to the socket in order via a serialised write chain.Testing
New test file
packages/miniflare/test/plugins/shared/remote-bindings-connect.spec.ts— 13 tests total:ProxyServerWorker(run in a second "edge" Miniflare instance) and asserts theconnect()address is forwarded verbatim.pipeSocketOverWebSocketunit (7): clean WS close cancels the parked socket read and resolves; socket EOF closes the WS and settles the WS-wait direction; a remote1011close rejects and cancels the opposite direction; a WSerrorevent rejects, cancels, and closes the writer; a socket read error closes with1011and rejects; a socket write failure tears down with1011and rejects; the1011close reason is truncated to <=123 UTF-8 bytes on a char boundary.rawTcpopt-in unit (2): theexperimentalflag is off by default and enabled only whenrawTcpis set.Real-environment E2E (manual): verified against a real Cloudflare Tunnel with a private-network MySQL database.
env.VPC_NETWORK.connect(host:3306)under local dev returned a genuine MySQL server greeting and executed a Drizzle query end-to-end, matching deployed behaviour. (No tunnel IDs, database hosts, or account identifiers are included here.)Build, typecheck,
oxfmt, andoxlintare green.PR template checklist (to fill in the actual PR)
Fixes #14710.
remote-bindings-connect.spec.ts, plus manual real-tunnel E2E as described above.connect()over VPC Networks); no new user-facing API or flag is introduced.