Skip to content

[miniflare] Support connect() on remote VPC Network bindings in local dev#14712

Merged
petebacondarwin merged 7 commits into
cloudflare:mainfrom
mack-erel:vpc-network-connect-tunnel
Jul 24, 2026
Merged

[miniflare] Support connect() on remote VPC Network bindings in local dev#14712
petebacondarwin merged 7 commits into
cloudflare:mainfrom
mack-erel:vpc-network-connect-tunnel

Conversation

@mack-erel

@mack-erel mack-erel commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Fixes #14710.

What

Adds raw-TCP connect() support for remote VPC Network bindings in local development. Previously, env.VPC_NETWORK.connect("host:port") worked when deployed but failed under wrangler dev / getPlatformProxy() with Incoming CONNECT on a worker not supported, because the local remote-bindings proxy only relayed HTTP and JSRPC.

Raw TCP is now tunnelled end-to-end through the existing remote-bindings proxy:

  • Miniflare proxy client (packages/miniflare/src/workers/shared/remote-proxy-client.worker.ts) gains an inbound connect(socket) handler. It recovers the caller's target address, opens a WebSocket to the remote proxy server, and relays bytes in both directions.
  • Wrangler ProxyServerWorker (packages/remote-bindings/templates/remoteBindings/ProxyServerWorker.ts) gains a WebSocket tunnel endpoint that calls env[binding].connect(address) (routing through the real VPC tunnel) and relays bytes back.
  • The proxy client worker opts into workerd's inbound connect handler only for raw-TCP bindings (VPC networks today).

Why

The full root-cause analysis, source trace, and feasibility spikes are in #14710, so only a summary is repeated here. In short: remote-capable bindings are wired locally as a proxy worker that implements fetch()/capnweb JSRPC only. env.BINDING.connect(addr) is a native workerd path that delivers an inbound CONNECT to that worker's entrypoint, which workerd rejects unless the worker carries the connect_pass_through or experimental compat flag. Neither the proxy client nor the edge proxy server had a socket relay path, so the call died locally before ever reaching the remote proxy session.

Design decisions

  • Inbound connect handler (workerd#6059): the proxy client worker implements export default { connect(socket) }. This handler is experimental-gated in workerd, so it is enabled via a per-worker experimental compat flag rather than relying on Miniflare's global --experimental CLI flag (which is not sufficient — the check is per-worker).
  • Address recovery via socket.opened.localAddress: the authority passed to connect("host:port") arrives verbatim as the inbound socket's localAddress on the service-binding path, matching workerd's connect-handler-test.js contract. No extra address-negotiation protocol is needed; the address is forwarded to the edge in an MF-Connect-Address header.
  • A dedicated WebSocket per connect(): each tunnel opens its own WebSocket to the proxy server rather than multiplexing over the existing capnweb JSRPC session. This keeps the relay independent of the JSRPC framing and simple to reason about for teardown.
  • rawTcp opt-in (existing call sites unaffected): remoteProxyClientWorker() takes a new options?: { rawTcp?: boolean } parameter; only VPC networks pass { rawTcp: true }. All existing HTTP/JSRPC proxy call sites keep their current behaviour and do not enable the experimental flag.

Design questions — resolved (confirmed by maintainer in #14710)

Both questions raised in #14710 have been answered by @petebacondarwin, and the implementation already matches the preferred direction:

  1. Experimental gating. Enabling the experimental compat flag on Miniflare's internal proxy-client worker (scoped to raw-TCP bindings) is acceptable, since it is only used locally and Miniflare controls the workerd version in use (comment). Implemented as such.
  2. WebSocket topology. A dedicated WebSocket per connect() is preferred to avoid collisions (comment). Implemented as such.

Known limitations

Documented in the relay's code comments as well:

  • No TCP half-close. When either direction ends, the tunnel is torn down in full (the WebSocket is closed and the socket's writable side is closed) rather than closing a single direction. Protocols relying on one-way half-close while continuing to read the other direction are not supported.
  • No backpressure. WebSockets expose no backpressure signal, so socket.readable is drained as fast as it produces; inbound WebSocket messages are written to the socket in order via a serialised write chain.

Testing

New test file packages/miniflare/test/plugins/shared/remote-bindings-connect.spec.ts13 tests total:

  • End-to-end tunnel (1): tunnels bytes end-to-end through the real, bundled ProxyServerWorker (run in a second "edge" Miniflare instance) and asserts the connect() address is forwarded verbatim.
  • Close/error matrix, end-to-end (3): a target connect failure terminates the tunnel without hanging; a target half-close with the user still open tears the relay down; a user-initiated close propagates to the target pipe.
  • pipeSocketOverWebSocket unit (7): clean WS close cancels the parked socket read and resolves; socket EOF closes the WS and settles the WS-wait direction; a remote 1011 close rejects and cancels the opposite direction; a WS error event rejects, cancels, and closes the writer; a socket read error closes with 1011 and rejects; a socket write failure tears down with 1011 and rejects; the 1011 close reason is truncated to <=123 UTF-8 bytes on a char boundary.
  • rawTcp opt-in unit (2): the experimental flag is off by default and enabled only when rawTcp is set.

Real-environment E2E (manual): verified against a real Cloudflare Tunnel with a private-network MySQL database. env.VPC_NETWORK.connect(host:3306) under local dev returned a genuine MySQL server greeting and executed a Drizzle query end-to-end, matching deployed behaviour. (No tunnel IDs, database hosts, or account identifiers are included here.)

Build, typecheck, oxfmt, and oxlint are green.


PR template checklist (to fill in the actual PR)

Fixes #14710.

  • Tests
    • Tests included/updated — 13 new tests in remote-bindings-connect.spec.ts, plus manual real-tunnel E2E as described above.
  • Public documentation
    • Documentation not necessary because: this fixes a local-dev parity gap for an already-GA feature (TCP connect() over VPC Networks); no new user-facing API or flag is introduced.

A raccoon waving hello


Open in Devin Review

@changeset-bot

changeset-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2cc4927

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages
Name Type
miniflare Minor
wrangler Minor
@cloudflare/deploy-helpers Patch
@cloudflare/pages-shared Patch
@cloudflare/remote-bindings Patch
@cloudflare/runtime-types Patch
@cloudflare/vite-plugin Patch
@cloudflare/vitest-pool-workers Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@workers-devprod
workers-devprod requested review from a team and ascorbic and removed request for a team July 16, 2026 04:52
@workers-devprod

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • @cloudflare/wrangler
Show detailed file reviewers
  • .changeset/vpc-networks-connect-tunnel.md: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/shared/constants.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/plugins/vpc-networks/index.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/shared/remote-bindings-utils.ts: [@cloudflare/wrangler]
  • packages/miniflare/src/workers/shared/remote-proxy-client.worker.ts: [@cloudflare/wrangler]
  • packages/miniflare/test/plugins/shared/remote-bindings-connect.spec.ts: [@cloudflare/wrangler]
  • packages/miniflare/vitest.config.mts: [@cloudflare/wrangler]
  • packages/wrangler/templates/remoteBindings/ProxyServerWorker.ts: [@cloudflare/wrangler]

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no potential bugs to report.

View in Devin Review to see 1 additional finding.

Open in Devin Review

devin-ai-integration[bot]

This comment was marked as resolved.

@mack-erel
mack-erel force-pushed the vpc-network-connect-tunnel branch from d3b9589 to bc96fcb Compare July 21, 2026 03:05
@petebacondarwin
petebacondarwin requested review from petebacondarwin and removed request for ascorbic July 21, 2026 14:17
@petebacondarwin

Copy link
Copy Markdown
Contributor

Thanks for putting this together. As it is a non-trivial change to our remote-bindings system I am going to need a bit of time to review it carefully. I am also talking to the Hyperdrive team to see if it would be useful to them too.

@petebacondarwin

Copy link
Copy Markdown
Contributor

We've just moved a lot of code around so this is now conflicted. Sorry.

@petebacondarwin
petebacondarwin marked this pull request as draft July 21, 2026 14:59
@mack-erel
mack-erel force-pushed the vpc-network-connect-tunnel branch from bc96fcb to 59aa143 Compare July 22, 2026 00:20
@mack-erel

mack-erel commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main — the conflict from the remote-bindings package extraction (#14720) is resolved and the PR is mergeable again.

Notes on the rebase:

  • The WebSocket tunnel endpoint followed ProxyServerWorker.ts to its new home in @cloudflare/remote-bindings; the relay logic is unchanged (one index-access guard added for that package's stricter noUncheckedIndexedAccess tsconfig, mirrored in the miniflare copy to keep the two implementations identical).
  • All green locally after the rebase: turbo build across miniflare/wrangler/remote-bindings, the 13 tunnel tests in remote-bindings-connect.spec.ts, and typechecks for all touched packages.
  • PR description updated: the two design questions are now marked as resolved per your answers in 🐛 BUG: connect() on remote VPC Network bindings fails in local dev with "Incoming CONNECT on a worker not supported" #14710 (per-worker experimental flag kept; dedicated WebSocket per connect() — already how it was implemented).

Thanks for the quick answers on the design questions, @petebacondarwin — happy to adjust anything else that comes up in review.


Conflict-resolving raccoon on duty

Rest easy — the merge conflicts were personally resolved by this raccoon. 🦝

@mack-erel
mack-erel marked this pull request as ready for review July 22, 2026 00:43
@workers-devprod

workers-devprod commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Codeowners approval required for this PR:

  • ✅ @cloudflare/wrangler
Show detailed file reviewers

Comment thread packages/miniflare/src/plugins/vpc-networks/index.ts
@xortive

xortive commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

the approach here makes sense to me. I suspect this will work for Hyperdrive remote: true as well, if we don't have support for that already.

mack-erel and others added 4 commits July 23, 2026 09:36
… proxy client

Remote bindings were wired locally as a proxy worker that only relayed
fetch/JSRPC, so `binding.connect(address)` against a raw-TCP binding (e.g. a
VPC network) failed with "Incoming CONNECT on a worker not supported".

Add an inbound `connect` handler to the proxy client worker: it recovers the
target address from `socket.opened.localAddress` (the verbatim `connect()`
authority, per workerd#6059), opens a dedicated WebSocket to the remote proxy
server, and relays bytes in both directions via a new `pipeSocketOverWebSocket`
helper (with `truncateCloseReason` for the 123-byte close-reason limit). The
relay tears both directions down together so a parked read or a missing close
event can't leak the socket.

The inbound connect handler is experimental-gated in workerd, so opt the proxy
client worker into the `experimental` compat flag only for raw-TCP bindings via
a new `rawTcp` option on `remoteProxyClientWorker`; VPC networks pass it. The
existing HTTP/JSRPC proxy call sites are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…orker

The remote-bindings proxy server now detects the WebSocket connect upgrade
(carrying the target in `MF-Connect-Address`), opens the real binding socket
via `fetcher.connect(address)`, and relays bytes bidirectionally over the
WebSocket. This is the edge half of the Miniflare `connect()` tunnel.

The relay helper is a byte-for-byte mirror of the one in miniflare's
remote-bindings-utils; this template is bundled standalone for the edge, so the
logic is duplicated here rather than imported.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add 13 tests for the `connect()` tunnel: end-to-end byte relay and address
propagation plus a close/error matrix run against the real bundled
ProxyServerWorker (in a second "edge" Miniflare instance), unit coverage of
`pipeSocketOverWebSocket` teardown/error paths, and opt-in assertions that the
`experimental` flag is set only when `rawTcp` is requested.

Add a `@relay-under-test` vitest alias so the worker-side relay helper can be
unit-tested without pulling worker-typed source into the node-side tsconfig
(which excludes `src/workers/**`).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mack-erel
mack-erel force-pushed the vpc-network-connect-tunnel branch from 59aa143 to 919708f Compare July 23, 2026 00:38
@mack-erel

Copy link
Copy Markdown
Contributor Author

We dug into the Hyperdrive remote: true question. The relay primitive (pipeSocketOverWebSocket + the rawTcp opt-in) is reusable as-is, but it's not a drop-in for Hyperdrive, for three reasons:

  1. The Hyperdrive binding isn't a Fetcher — drivers connect via the magic <hex>.hyperdrive.local hostname and workerd's connect override, so the CONNECT that reaches the proxy worker would carry a synthetic hostname rather than a real target (probably fine, given VPC Services also ignore the address, but unverified for Hyperdrive's edge side).
  2. Miniflare's hyperdrive plugin is currently local-only (driven entirely by localConnectionString) — there's no remoteProxyConnectionString wiring, and remote: true is rejected at the config layer.
  3. Remote preview sessions have no provisioning story for Hyperdrive configs by id yet.

So it looks like a natural follow-up PR (mostly plugin wiring + support-matrix/config work on top of this relay) rather than something to fold in here. Happy to take a stab at that after this lands, if useful.

@pkg-pr-new

pkg-pr-new Bot commented Jul 24, 2026

Copy link
Copy Markdown
@cloudflare/autoconfig

npm i https://pkg.pr.new/@cloudflare/autoconfig@14712

@cloudflare/config

npm i https://pkg.pr.new/@cloudflare/config@14712

create-cloudflare

npm i https://pkg.pr.new/create-cloudflare@14712

@cloudflare/deploy-helpers

npm i https://pkg.pr.new/@cloudflare/deploy-helpers@14712

@cloudflare/kv-asset-handler

npm i https://pkg.pr.new/@cloudflare/kv-asset-handler@14712

miniflare

npm i https://pkg.pr.new/miniflare@14712

@cloudflare/pages-shared

npm i https://pkg.pr.new/@cloudflare/pages-shared@14712

@cloudflare/unenv-preset

npm i https://pkg.pr.new/@cloudflare/unenv-preset@14712

@cloudflare/vite-plugin

npm i https://pkg.pr.new/@cloudflare/vite-plugin@14712

@cloudflare/vitest-pool-workers

npm i https://pkg.pr.new/@cloudflare/vitest-pool-workers@14712

@cloudflare/workers-auth

npm i https://pkg.pr.new/@cloudflare/workers-auth@14712

@cloudflare/workers-editor-shared

npm i https://pkg.pr.new/@cloudflare/workers-editor-shared@14712

@cloudflare/workers-utils

npm i https://pkg.pr.new/@cloudflare/workers-utils@14712

wrangler

npm i https://pkg.pr.new/wrangler@14712

commit: 2cc4927

oxlint (typescript-eslint/no-non-null-assertion) forbids `!`; return a
400 for a connect upgrade missing the MF-Connect-Address header instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BuKCFc94BFDpbwag1UfPbB
devin-ai-integration[bot]

This comment was marked as resolved.

Mirror the write-path pattern in both relay copies so a non-object
rejection can't throw while building the 1011 close reason and mask
the original error.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BuKCFc94BFDpbwag1UfPbB
@petebacondarwin
petebacondarwin enabled auto-merge (squash) July 24, 2026 14:00

@workers-devprod workers-devprod left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Codeowners reviews satisfied

@github-project-automation github-project-automation Bot moved this from Untriaged to Approved in workers-sdk Jul 24, 2026
@petebacondarwin
petebacondarwin merged commit 6e0bf6e into cloudflare:main Jul 24, 2026
61 of 62 checks passed
@github-project-automation github-project-automation Bot moved this from Approved to Done in workers-sdk Jul 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

🐛 BUG: connect() on remote VPC Network bindings fails in local dev with "Incoming CONNECT on a worker not supported"

5 participants