Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .changeset/vpc-networks-connect-tunnel.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
---
"miniflare": minor
"wrangler": minor
---

Support `connect()` on remote VPC Network and VPC Service bindings in local development

Remote VPC Network and VPC Service bindings previously only supported HTTP and JSRPC, so calling `binding.connect(address)` against a private TCP service (for example a database) failed in local dev with `Incoming CONNECT on a worker not supported`. Raw TCP connections through remote VPC Network and VPC Service bindings now work in local development.

This feature is experimental. Existing HTTP and JSRPC usage of remote VPC Network and VPC Service bindings is unaffected, and no new configuration is required.
15 changes: 14 additions & 1 deletion packages/miniflare/src/plugins/shared/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -90,9 +90,22 @@ export function objectEntryWorker(
// into a per-binding service. The only static, non-props-able binding is the
// loopback service (used to surface diagnostics back to the Miniflare host,
// e.g. a Cloudflare Access block detected on the remote proxy response).
export function remoteProxyClientWorker(script?: () => string) {
//
// `options.rawTcp` opts the service into raw TCP `connect()` tunnelling (see the
// `experimental` compatibility flag below). That is a property of the service
// itself, not of any individual binding, so it stays valid under the shared,
// props-based service model.
export function remoteProxyClientWorker(
script?: () => string,
options?: { rawTcp?: boolean }
) {
return {
compatibilityDate: "2025-01-01",
// Raw TCP bindings (e.g. VPC networks) tunnel `binding.connect()` traffic
// through this worker's inbound `connect` handler, which requires the
// `experimental` compatibility flag (workerd#6059). Other bindings only
// proxy HTTP/JSRPC and must not opt in.
...(options?.rawTcp ? { compatibilityFlags: ["experimental"] } : {}),
modules: [
{
name: "index.worker.js",
Expand Down
7 changes: 6 additions & 1 deletion packages/miniflare/src/plugins/vpc-networks/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,12 @@ export const VPC_NETWORKS_PLUGIN: Plugin<typeof VpcNetworksOptionsSchema> = {
return [
{
name: VPC_NETWORKS_REMOTE_SERVICE_NAME,
worker: remoteProxyClientWorker(),
// VPC networks expose raw TCP via `binding.connect()`, tunnelled
// through the proxy client's inbound `connect` handler. The shared
// `vpc-networks:remote` service is dedicated to VPC networks, so
// opting it into raw TCP leaves every other binding's service
// untouched.
worker: remoteProxyClientWorker(undefined, { rawTcp: true }),
Comment thread
petebacondarwin marked this conversation as resolved.
},
];
},
Expand Down
7 changes: 6 additions & 1 deletion packages/miniflare/src/plugins/vpc-services/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,12 @@ export const VPC_SERVICES_PLUGIN: Plugin<typeof VpcServicesOptionsSchema> = {
return [
{
name: VPC_SERVICES_REMOTE_SERVICE_NAME,
worker: remoteProxyClientWorker(),
// VPC services also expose raw TCP via `binding.connect()`, tunnelled
// through the proxy client's inbound `connect` handler. The shared
// `vpc-services:remote` service is dedicated to VPC services, so
// opting it into raw TCP leaves every other binding's service
// untouched.
worker: remoteProxyClientWorker(undefined, { rawTcp: true }),
},
];
},
Expand Down
173 changes: 173 additions & 0 deletions packages/miniflare/src/workers/shared/remote-bindings-utils.ts
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,179 @@ export function makeFetch(
};
}

/**
* Clamp a WebSocket close reason to the protocol's 123-byte limit.
*
* `WebSocket.close(code, reason)` requires the reason to be at most 123 UTF-8
* bytes. Slicing by JavaScript string length (UTF-16 code units) can both
* overshoot the byte budget and split a multi-byte character, so truncate on a
* UTF-8 byte boundary instead.
*/
function truncateCloseReason(reason: string): string {
const bytes = new TextEncoder().encode(reason);
if (bytes.length <= 123) {
return reason;
}
// Back off to a UTF-8 character boundary: step left over any trailing
// continuation bytes (10xxxxxx) so we never cut a multi-byte sequence.
let end = 123;
while (end > 0 && ((bytes[end] ?? 0) & 0b1100_0000) === 0b1000_0000) {
end--;
}
return new TextDecoder().decode(bytes.subarray(0, end));
}

/**
* Relay raw TCP bytes between a `connect()` socket and a WebSocket, in both
* directions, propagating close and error state.
*
* Used to tunnel `binding.connect()` traffic through the remote-bindings proxy:
* the local proxy client pipes the caller's inbound socket over a WebSocket to
* the remote proxy server, which pipes it into the real binding's socket.
*
* The returned promise resolves once both directions have closed cleanly, and
* rejects if either side errors. Rejecting lets the caller's `connect()` handler
* surface the failure on its socket.
*
* Both directions are torn down together: when one side finishes — a WebSocket
* close/error, or the socket reaching EOF / erroring — the opposite direction is
* actively cancelled. The socket read is woken with `reader.cancel()` and the
* WebSocket wait is settled directly, so neither a parked `reader.read()` nor a
* never-arriving close event can leak the relay (and the underlying socket) for
* the lifetime of the process.
*
* Note: WebSockets have no backpressure signal, so `socket.readable` is drained
* as fast as it produces. Incoming WebSocket messages are written in order via a
* serialised write chain. There is no TCP half-close: when either direction ends
* the tunnel is torn down in full — the WebSocket is closed and the socket's
* writable side is closed — rather than closing a single direction.
*/
export async function pipeSocketOverWebSocket(
socket: Socket,
ws: WebSocket
): Promise<void> {
const writer = socket.writable.getWriter();
const reader = socket.readable.getReader();

let wsClosed = false;
function closeWebSocket(code: number, reason?: string) {
if (wsClosed) {
return;
}
wsClosed = true;
try {
ws.close(
code,
reason === undefined ? undefined : truncateCloseReason(reason)
);
} catch {
// Already closing/closed.
}
}

// ws -> socket writes are serialised through this chain to preserve byte
// order. The writable side is closed exactly once (guarded by `writerClosed`)
// so both teardown paths below can invoke it idempotently.
let writeChain = Promise.resolve();
let writerClosed = false;
function closeWriter(): Promise<void> {
if (writerClosed) {
return Promise.resolve();
}
writerClosed = true;
return writeChain.then(() => writer.close());
}

// `fromWebSocket` settles from the ws close/error events, or is settled by the
// socket -> ws direction once it has closed the ws itself (in which case no
// inbound close event will arrive to settle it).
let resolveFromWs!: () => void;
let rejectFromWs!: (reason: unknown) => void;
const fromWebSocket = new Promise<void>((resolve, reject) => {
resolveFromWs = resolve;
rejectFromWs = reject;
});

// WebSocket -> socket. Message events aren't awaited by the runtime, so writes
// are serialised through the promise chain to preserve byte order.
ws.addEventListener("message", (event) => {
const chunk =
typeof event.data === "string"
? new TextEncoder().encode(event.data)
: new Uint8Array(event.data);
writeChain = writeChain
.then(() => writer.write(chunk))
.catch((error) => {
// A socket write failed: tear the tunnel down rather than leaving the
// rejection unhandled. Close the ws (1011), cancel the opposite read
// direction, and reject so the caller sees the failure.
closeWebSocket(
1011,
(error as Error)?.message ?? "socket write failed"
);
reader.cancel().catch(() => {});
rejectFromWs(error);
});
});
ws.addEventListener("close", (event) => {
wsClosed = true;
// Actively cancel the opposite direction so a parked `reader.read()` can't
// keep the socket -> ws relay (and this whole pipe) alive forever.
reader.cancel().catch(() => {});
// Close code 1011 signals the remote end errored.
if (event.code === 1011) {
rejectFromWs(
new Error(event.reason || "Remote tunnel closed with an error")
);
return;
}
// Flush any queued writes, then close the writable side so the caller sees
// EOF.
closeWriter().then(resolveFromWs, rejectFromWs);
});
ws.addEventListener("error", () => {
wsClosed = true;
reader.cancel().catch(() => {});
rejectFromWs(new Error("Tunnel WebSocket errored"));
});

// socket -> WebSocket. On EOF close cleanly (1000); on read error close with
// 1011 and reject so the caller sees the failure.
const toWebSocket = (async () => {
try {
for (;;) {
const { value, done } = await reader.read();
if (done) {
break;
}
// Re-check after the parked read: the ws may have closed while we were
// waiting. If so, terminate quietly instead of erroring on `send`.
if (wsClosed) {
break;
}
ws.send(
value.buffer.slice(
value.byteOffset,
value.byteOffset + value.byteLength
)
);
}
closeWebSocket(1000);
} catch (error) {
closeWebSocket(1011, (error as Error)?.message ?? "socket read failed");
throw error;
} finally {
reader.releaseLock();
// We closed (or observed the close of) the ws on this side, so no inbound
// close event will arrive to settle `fromWebSocket`. Close the writable
// side and settle it here; idempotent with the ws `close` handler above.
closeWriter().then(resolveFromWs, rejectFromWs);
}
})();

await Promise.all([toWebSocket, fromWebSocket]);
}

/**
* Create a remote proxy stub that proxies to a remote binding via capnweb.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import { SharedBindings } from "./constants";
import {
makeFetch,
makeRemoteProxyStub,
pipeSocketOverWebSocket,
throwRemoteRequired,
} from "./remote-bindings-utils";
import type {
Expand All @@ -25,6 +26,46 @@ export default class Client extends WorkerEntrypoint<
)(request);
}

// Handles `binding.connect(address)` for raw TCP bindings (e.g. VPC networks).
// Only reachable when the worker is configured with the `experimental`
// compatibility flag, which enables inbound `connect` handlers (workerd#6059).
async connect(socket: Socket): Promise<void> {
const { remoteProxyConnectionString, binding, cfTraceId } = this.ctx.props;
if (!remoteProxyConnectionString) {
throwRemoteRequired(binding);
}

// The address passed to `binding.connect("host:port")` arrives verbatim as
// the inbound socket's `localAddress` on the service-binding path. See
// https://github.com/cloudflare/workerd/pull/6059.
const { localAddress } = await socket.opened;
if (!localAddress) {
throw new Error(
`Binding ${binding} received a connection without a target address`
);
}

const headers = new Headers({
Upgrade: "websocket",
"MF-Binding": binding,
"MF-Connect-Address": localAddress,
});
if (cfTraceId) {
headers.set("cf-trace-id", cfTraceId);
}

const response = await fetch(remoteProxyConnectionString, { headers });
const ws = response.webSocket;
if (!ws) {
throw new Error(
`Binding ${binding} failed to open a tunnel to ${localAddress} (status ${response.status})`
);
}
ws.accept();

await pipeSocketOverWebSocket(socket, ws);
}

constructor(
ctx: ExecutionContext<RemoteBindingProps>,
env: RemoteBindingEnv
Expand Down
Loading
Loading