[pull] main from LibreChat-AI:main

.env.example

@@ -24,6 +24,14 @@ SANDBOX_RUN_CPU_TIME=10000
 SANDBOX_RUN_TIMEOUT=15000
 SANDBOX_OUTPUT_MAX_SIZE=65536
 
+# Remote stateful code bridge (Code API deployment)
+# CODEAPI_SANDBOX_BACKEND=remote-bridge
+# CODEAPI_EXECUTION_PROFILE=stateful
+# CODEAPI_RUNTIME_SESSION_MODE=affinity
+# CODEAPI_BRIDGE_WORKER_ID=my-vm
+# CODEAPI_BRIDGE_TOKEN=replace-with-a-strong-random-secret
+# CODEAPI_BRIDGE_AUTH_MODE=paired
+
 # Service Configuration
 PYTHON_CONCURRENCY=5
 OTHER_CONCURRENCY=15

.github/workflows/ci.yml

@@ -34,6 +34,9 @@ jobs:
       - name: Sandbox-runner liveness checks
         run: tests/sandbox_runner_healthcheck.sh
 
+      - name: Bridge pairing rollout safety
+        run: tests/bridge_pairing_rollout.sh
+
       - name: Validate sandbox Dockerfiles
         run: |
           docker buildx build --check -f api/Dockerfile .

.gitignore

@@ -1,5 +1,6 @@
 data/
 node_modules
+packages/*/dist/
 .env
 .git
 .npmrc

README.md

@@ -15,6 +15,8 @@ Code Interpreter (internally `codeapi`, the prefix used by its env vars, images,
 - **Package Delivery** - Bakes Python, Node, and Bun into the default microVM
   block-root image; a package-init PVC mode remains available for direct NsJail
   development
+- **Remote Code Bridge** - Lets an operator-owned VM connect outbound and serve
+  as a fenced, stateful sandbox through the `@librechat/code` worker
 
 ## Architecture
 
@@ -35,7 +37,9 @@ Set `CODEAPI_EXECUTION_PROFILE` consistently on an API deployment and its
 workers. The default profile keeps the existing `python-queue` and
 `other-queue`; the stateful profile uses `stateful-python-queue` and
 `stateful-other-queue`. This allows both deployments to share Redis without
-cross-consuming jobs.
+cross-consuming jobs. The `remote-bridge` backend additionally uses
+`remote-bridge-python-queue` and `remote-bridge-other-queue`, fencing attached
+worker jobs from Lambda consumers during rolling deployments.
 
 An existing Lambda MicroVM deployment upgraded from a pre-profile release may
 leave `CODEAPI_EXECUTION_PROFILE` unset for its first binary rollout. An
@@ -65,6 +69,18 @@ Two modes are supported:
 - **NsJail mode** (`kvmEnabled: false`): Direct NsJail sandboxing with Linux namespaces and cgroups
 - **MicroVM mode** (`kvmEnabled: true`): libkrun microVM with its own kernel, NsJail runs inside the guest
 
+## Remote stateful environments
+
+The `remote-bridge` backend keeps the Code API as the policy and queue boundary
+while moving execution to a sandbox on an operator-selected VM. The worker only
+makes outbound authenticated requests, so the VM does not need a public ingress
+port. Assignments carry a deadline, a single-active-worker lock, a monotonically
+increasing generation, and a one-time lease token to fence stale workers.
+
+See [Remote Code Bridge](docs/remote-bridge/README.md) for deployment and threat
+model details. The worker protocol and CLI live in the provider-neutral
+[`@librechat/code`](packages/code/README.md) package.
+
 ## Security disclaimer
 
 This service exists to run arbitrary, untrusted code — treat every

api/Dockerfile

@@ -29,8 +29,10 @@ RUN git clone -b master --single-branch https://github.com/google/nsjail.git . \
 RUN make -j$(nproc)
 
 COPY api/src/spec-guard.c /tmp/spec-guard.c
+COPY docker/rootfs-setup.c /tmp/rootfs-setup.c
 RUN gcc -O2 -static -o /usr/local/bin/spec-guard /tmp/spec-guard.c \
-    && chmod 0111 /usr/local/bin/spec-guard
+    && gcc -O2 -static -o /usr/local/bin/sandbox-rootfs-setup /tmp/rootfs-setup.c \
+    && chmod 0111 /usr/local/bin/spec-guard /usr/local/bin/sandbox-rootfs-setup
 
 # ============================================================================
 # Stage 1b: Build language runtime packages (only consumed by sandbox-runner-baked)
@@ -212,6 +214,7 @@ RUN dnf install -y --setopt=install_weak_deps=False \
     && dnf clean all
 
 COPY --from=launcher-builder /launcher/target/release/sandbox-launcher /usr/local/bin/launcher
+COPY --from=nsjail-builder /usr/local/bin/sandbox-rootfs-setup /sandbox-rootfs-setup
 
 COPY launcher/entrypoint.sh /usr/local/bin/launcher-entrypoint.sh
 COPY docker/start-direct-sandbox.sh /usr/local/bin/start-direct-sandbox.sh

api/src/entrypoint.sh

@@ -171,28 +171,33 @@ fi
 	    chmod 777 "$SMOKE_DIR"
 	fi
 SMOKE_LOG=$(mktemp)
+SMOKE_STDERR=$(mktemp)
 
 NSJAIL_CGROUP_ARGS=()
 if [ "$SANDBOX_USE_CGROUPV2" = "true" ]; then
     NSJAIL_CGROUP_ARGS=(--use_cgroupv2)
 fi
 
-if timeout 10 /usr/sbin/nsjail --config "${NSJAIL_CONFIG:-/sandbox_api/config/sandbox.cfg}" \
+if timeout 10 "${NSJAIL_PATH:-/usr/sbin/nsjail}" --config "${NSJAIL_CONFIG:-/sandbox_api/config/sandbox.cfg}" \
     "${NSJAIL_CGROUP_ARGS[@]}" --log "$SMOKE_LOG" \
     --user "65534:${SMOKE_OUTSIDE_UID}:1" --group "65534:${SMOKE_OUTSIDE_GID}:1" \
     -s /usr/bin:/bin -s /usr/lib:/lib -s /usr/lib64:/lib64 \
     -B "$SMOKE_DIR:/mnt/data" \
-    -- /bin/sh -c 'printf "%s\n" sandbox_ok > /mnt/data/smoke.txt && test "$(cat /mnt/data/smoke.txt)" = sandbox_ok' > /dev/null 2>&1; then
+    -- /bin/sh -c 'printf "%s\n" sandbox_ok > /mnt/data/smoke.txt && test "$(cat /mnt/data/smoke.txt)" = sandbox_ok' > /dev/null 2>"$SMOKE_STDERR"; then
     echo "NsJail smoke test passed"
 else
     echo "FATAL: NsJail smoke test failed — sandbox cannot start"
     echo "NsJail log output:"
     cat "$SMOKE_LOG" 2>/dev/null || true
+    echo "NsJail stderr:"
+    cat "$SMOKE_STDERR" 2>/dev/null || true
     rm -f "$SMOKE_LOG"
+    rm -f "$SMOKE_STDERR"
     rm -rf "$SMOKE_DIR"
     exit 1
 fi
 rm -f "$SMOKE_LOG"
+rm -f "$SMOKE_STDERR"
 rm -rf "$SMOKE_DIR"
 
 echo "Starting sandbox API server..."

docker/Dockerfile.worker-sandbox

@@ -42,8 +42,10 @@ RUN git clone -b master --single-branch https://github.com/google/nsjail.git . \
 RUN make -j$(nproc)
 
 COPY api/src/spec-guard.c /tmp/spec-guard.c
+COPY docker/rootfs-setup.c /tmp/rootfs-setup.c
 RUN gcc -O2 -static -o /usr/local/bin/spec-guard /tmp/spec-guard.c \
-    && chmod 0111 /usr/local/bin/spec-guard
+    && gcc -O2 -static -o /usr/local/bin/sandbox-rootfs-setup /tmp/rootfs-setup.c \
+    && chmod 0111 /usr/local/bin/spec-guard /usr/local/bin/sandbox-rootfs-setup
 
 # ============================================================================
 # Stage 1b: Build language runtime packages for the baked KVM root disk
@@ -83,6 +85,7 @@ WORKDIR /app
 COPY service/package.json service/bun.lock ./
 RUN bun install --frozen-lockfile
 COPY service/src ./src
+COPY packages/code/src /packages/code/src
 COPY shared /shared
 COPY service/tsconfig.json ./
 RUN bun build ./src/worker-server.ts --minify --outdir .build --target bun --external '@opentelemetry/*'
@@ -231,6 +234,7 @@ ENV PATH="/root/.bun/bin:${PATH}"
 
 # --- Launcher (runs on host, boots microVM) ---
 COPY --from=launcher-builder /launcher/target/release/sandbox-launcher /usr/local/bin/launcher
+COPY --from=nsjail-builder /usr/local/bin/sandbox-rootfs-setup /sandbox-rootfs-setup
 
 # --- Launcher entrypoint (DNS resolution + socat relay before VM boot) ---
 COPY launcher/entrypoint.sh /usr/local/bin/launcher-entrypoint.sh

docker/rootfs-setup.c

@@ -0,0 +1,92 @@
+#include <errno.h>
+#include <limits.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/mount.h>
+#include <sys/stat.h>
+#include <unistd.h>
+
+static int bind_mount(const char *source, const char *target, int read_only) {
+  if (mount(source, target, NULL, MS_BIND | MS_REC, NULL) != 0) {
+    fprintf(stderr, "bind %s -> %s failed: %s\n", source, target, strerror(errno));
+    return -1;
+  }
+
+  if (read_only &&
+      mount(NULL, target, NULL, MS_BIND | MS_REMOUNT | MS_RDONLY, NULL) != 0) {
+    fprintf(stderr, "read-only remount of %s failed: %s\n", target, strerror(errno));
+    return -1;
+  }
+
+  return 0;
+}
+
+static int bind_rootfs_path(const char *rootfs, const char *path) {
+  char source[PATH_MAX];
+  int written = snprintf(source, sizeof(source), "%s%s", rootfs, path);
+  if (written < 0 || (size_t)written >= sizeof(source)) {
+    fprintf(stderr, "rootfs path is too long: %s%s\n", rootfs, path);
+    return -1;
+  }
+
+  return bind_mount(source, path, 1);
+}
+
+int main(int argc, char **argv) {
+  if (argc < 2) {
+    fprintf(stderr, "usage: sandbox-rootfs-setup ROOTFS [COMMAND ...]\n");
+    return 2;
+  }
+
+  const char *rootfs = argv[1];
+  if (rootfs[0] != '/') {
+    fprintf(stderr, "rootfs must be an absolute path\n");
+    return 2;
+  }
+
+  if (mount(NULL, "/", NULL, MS_REC | MS_PRIVATE, NULL) != 0) {
+    fprintf(stderr, "making the mount namespace private failed: %s\n", strerror(errno));
+    return 1;
+  }
+
+  if ((mkdir("/sandbox_api", 0755) != 0 && errno != EEXIST) ||
+      (mkdir("/pkgs", 0755) != 0 && errno != EEXIST)) {
+    fprintf(stderr, "creating rootfs mount targets failed: %s\n", strerror(errno));
+    return 1;
+  }
+
+  /*
+   * Keep this process statically linked: the final /usr mount replaces
+   * the Fedora launcher's dynamic userspace with the Debian sandbox rootfs.
+   * A shell cannot safely perform this sequence because its next command may
+   * try to load a host binary against guest libraries (or vice versa).
+   */
+  const char *paths[] = {"/sandbox_api", "/pkgs"};
+  for (size_t i = 0; i < sizeof(paths) / sizeof(paths[0]); i++) {
+    if (bind_rootfs_path(rootfs, paths[i]) != 0) {
+      return 1;
+    }
+  }
+
+  if (access("/host-packages", F_OK) == 0 &&
+      bind_mount("/host-packages", "/pkgs", 0) != 0) {
+    fprintf(stderr, "warning: sandbox will run without host packages\n");
+  }
+
+  /* Bind all guest userspace last, then immediately enter it. */
+  if (bind_rootfs_path(rootfs, "/usr") != 0) {
+    return 1;
+  }
+
+  setenv("PATH", "/root/.bun/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", 1);
+  setenv("LD_LIBRARY_PATH", "/usr/lib/aarch64-linux-gnu:/usr/lib/x86_64-linux-gnu", 1);
+
+  setenv("NSJAIL_PATH", "/usr/sbin/nsjail", 1);
+
+  char *default_argv[] = {"/sandbox_api/entrypoint.sh", NULL};
+  char **command_argv = argc > 2 ? &argv[2] : default_argv;
+  execv(command_argv[0], command_argv);
+  fprintf(stderr, "starting sandbox entrypoint failed: %s\n", strerror(errno));
+  return 1;
+}

docker/start-direct-sandbox.sh

@@ -44,35 +44,4 @@ else
 fi
 
 export SANDBOX_ROOTFS="$ROOTFS"
-
-exec unshare --mount bash -c '
-    ROOTFS="${SANDBOX_ROOTFS:-/sandbox-rootfs}"
-
-    mount -o bind,ro "$ROOTFS/usr/sbin"     /usr/sbin    || { echo "FATAL: cannot bind /usr/sbin"; exit 1; }
-    mount -o bind,ro "$ROOTFS/usr/lib"      /usr/lib     || { echo "FATAL: cannot bind /usr/lib"; exit 1; }
-
-    if [ -d "$ROOTFS/usr/lib64" ] && ! [ -L "$ROOTFS/usr/lib64" ]; then
-        mount -o bind,ro "$ROOTFS/usr/lib64" /usr/lib64 2>/dev/null || \
-            echo "[sandbox] WARNING: could not bind /usr/lib64 - sandboxed binaries may fail to exec"
-    fi
-
-    mount -o bind,ro "$ROOTFS/usr/local"    /usr/local   || { echo "FATAL: cannot bind /usr/local"; exit 1; }
-    mount -o bind,ro "$ROOTFS/sandbox_api"  /sandbox_api || { echo "FATAL: cannot bind /sandbox_api"; exit 1; }
-    mount -o bind,ro "$ROOTFS/pkgs"       /pkgs      || { echo "FATAL: cannot bind /pkgs"; exit 1; }
-
-    if [ -d /host-packages ]; then
-        mount --bind /host-packages /pkgs 2>/dev/null || \
-            echo "WARNING: could not bind /host-packages - sandbox will run without packages"
-    fi
-
-    mount -o bind,ro "$ROOTFS/usr/bin" /usr/bin || { echo "FATAL: cannot bind /usr/bin"; exit 1; }
-
-    multiarch_libdir=$(find /usr/lib -maxdepth 1 -type d -name "*-linux-gnu" -print -quit)
-    if [ -n "$multiarch_libdir" ]; then
-        export LD_LIBRARY_PATH="$multiarch_libdir${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
-    fi
-
-    export PATH="/root/.bun/bin:$PATH"
-
-    exec /sandbox_api/entrypoint.sh
-'
+exec unshare --mount /sandbox-rootfs-setup "$ROOTFS"

docs/adr/001-stateful-code-environments.md

@@ -0,0 +1,86 @@
+# ADR 001: Stateful code environments use an outbound Code API bridge
+
+- Status: Accepted for alpha
+- Date: 2026-08-30
+
+## Context
+
+LibreChat needs coding agents to reuse a workspace across conversation turns
+while allowing the environment owner to choose the VM. Internet-facing
+LibreChat instances cannot safely require inbound access to that VM, forward
+end-user tokens to it, or treat an MCP connection as a sandbox boundary.
+
+The first alpha demonstrated a stable runtime-session ID, a single fenced
+worker lease, and workspace persistence across turns. Its static shared worker
+token was sufficient to prove execution flow but is not an acceptable hardened
+enrollment mechanism.
+
+## Decision
+
+The product concept is a **stateful code environment**. Code API remains its
+broker and policy boundary, and `remote-bridge` is a Code API sandbox backend.
+The `@librechat/code` worker connects outbound from the chosen VM and forwards
+assignments only to a loopback or private sandbox endpoint.
+
+Hardened workers enroll through a one-time pairing code:
+
+1. An administrator creates a code scoped to the configured worker ID.
+2. The CLI generates an Ed25519 keypair locally and redeems the code with only
+   its public key.
+3. Code API returns a fifteen-minute credential bound to that public key.
+4. Every worker request signs the method, path, body digest, timestamp, nonce,
+   and credential.
+5. Code API rejects stale timestamps and replayed nonces and supports rotation
+   and immediate revocation.
+
+Static bearer authentication remains a non-hardened compatibility mode.
+
+## Ownership and state
+
+The alpha environment is deployment/operator owned and configured with one
+worker ID. A future LibreChat control plane may persist deployment-, tenant-,
+or user-owned environment records and issue the same pairing operation through
+RBAC-protected APIs without changing the worker execution protocol.
+
+Workspace state belongs to the stable runtime session, not to a transient
+assignment lease. For `remote-bridge`, that state currently survives turns on
+the same worker and backing disk. It is not yet checkpointed or portable across
+worker replacement; the UI and operator documentation must not imply otherwise.
+
+## Security invariants
+
+- The VM requires no inbound internet listener.
+- Code API, not the worker, authenticates LibreChat users and normalizes work.
+- A stolen short-lived credential is insufficient without the worker private
+  key; a stolen private key is insufficient after credential expiry or
+  revocation.
+- Pairing codes and credentials are stored by digest where lookup permits.
+- One configured worker has at most one active fenced assignment.
+- Sandbox isolation and default-deny egress remain mandatory; pairing secures
+  the transport identity but does not make the host a sandbox.
+- A compromised worker can lie about advertised capabilities. Capability
+  labels and policy digests are audit signals until enforcement is coupled to
+  an attested sandbox or trusted host policy.
+
+## Consequences
+
+- `@librechat/code` owns the provider-neutral protocol, identity handling, and
+  worker CLI; Code API owns enrollment, scheduling, and execution policy.
+- LibreChat owns environment persistence, ownership, RBAC, and user experience.
+- The Agents SDK keeps only its adapter until a second concrete consumer proves
+  which coding-tool abstractions are genuinely provider neutral.
+- MCP may expose environment operations later, but it is not the worker
+  transport or isolation boundary.
+- Multi-worker directories, checkpoint/restore, owner-scoped quotas, and
+  enforced network capability profiles remain follow-up decisions.
+
+## Alternatives rejected
+
+- **Inbound SSH/HTTP to the VM:** expands attack surface and complicates NAT and
+  firewall operation.
+- **MCP as the worker protocol:** conflates tool discovery with leases,
+  cancellation, fencing, and sandbox policy.
+- **Put the runtime in the Agents SDK:** couples provider-neutral execution to
+  one agent integration and makes non-agent consumers depend on agent internals.
+- **Long-lived shared bearer token:** easy to bootstrap, but replayable and not
+  bound to a worker-held key.