feat(browser-runtime): implement the Steel browser runtime

CHANGELOG.md

@@ -8,6 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/).
 
 ## [Unreleased]
 ### Added
+- Added [Steel](https://github.com/steel-dev/steel-browser) as a managed remote browser runtime, covering both Steel Cloud (`STEEL_API_KEY`) and self-hosted deployments (`STEEL_BASE_URL`). Selecting `--browser-runtime steel` previously raised a not-implemented error.
 - Added `scripts/export_openeval.py`, an additive script exporting a batch's `rescore-summary.json` as an [EvalPort](https://github.com/adhabnr-ux/evalport) `ResultSet` Thanks to [@adhabnr-ux](https://github.com/adhabnr-ux).
 - Added a `--browser-runtime kernel` mode to the Harbor adapter that runs each task against one Kernel cloud browser, exposing only a credential-free CDP bridge to the agent, and finalizes the replay and deletes the browser during verification.
 

README.md

@@ -337,7 +337,7 @@ Full registry: [`src/clawbench/runtime/harnesses/harnesses.yaml`](src/clawbench/
 
 | I want to… | Where |
 | --- | --- |
-| Use a managed remote browser instead of a local container | [`docs/browser-runtimes.md`](docs/browser-runtimes.md) — Kernel and Browserbase setup, options, and recordings |
+| Use a managed remote browser instead of a local container | [`docs/browser-runtimes.md`](docs/browser-runtimes.md) — Kernel, Browserbase, and Steel setup, options, and recordings |
 | Run V2 through the Harbor framework (and run it fast) | [`docs/harbor.md`](docs/harbor.md) — conversion, judge wiring, concurrency, troubleshooting |
 | See every CLI command and flag | [`docs/cli.md`](docs/cli.md) |
 

docs/browser-runtimes.md

@@ -2,7 +2,7 @@
 
 By default ClawBench launches Chromium inside its own container. You can point it at a managed remote browser instead — useful when the host cannot run containers comfortably, or when you want the provider to handle scaling and session replay.
 
-`--browser-runtime` accepts `local` (default), `kernel`, `browserbase`, `remote-cdp`, and `steel`. **`steel` is reserved and not implemented yet** — selecting it raises an error.
+`--browser-runtime` accepts `local` (default), `kernel`, `browserbase`, `steel`, and `remote-cdp`.
 
 ## Local container (default)
 
@@ -68,6 +68,56 @@ uv run clawbench-batch --models your-model --all-cases \
 
 **Concurrency.** `--max-concurrent` defaults to **1** with Browserbase (2 for local runs) because parallel sessions consume provider quota. Raise it only as far as your plan's concurrent-session limit allows.
 
+## Steel
+
+Works against both [Steel Cloud](https://steel.dev) and a self-hosted
+[steel-browser](https://github.com/steel-dev/steel-browser); they expose the same
+`/v1/sessions` API.
+
+For Steel Cloud, put the key in `.env.local`:
+
+```dotenv
+STEEL_API_KEY=ste-...
+```
+
+For a self-hosted deployment, point ClawBench at it instead. No key is required
+unless your deployment enforces one:
+
+```dotenv
+STEEL_BASE_URL=http://localhost:3000
+```
+
+Then select the runtime on a single or batch run:
+
+```bash
+uv run clawbench-run test-cases/v1/<case> your-model \
+  --browser-runtime steel
+
+uv run clawbench-batch --models your-model --all-cases \
+  --browser-runtime steel
+```
+
+Steel runs reuse the same CDP action capture, screenshots, HTTP logging, and
+request interception as local runs, so scoring is unchanged. Steel serves its
+session replay from the session viewer rather than as a downloadable file, so a
+Steel run has no local `recording.mp4`; the viewer URL is stored as
+`browser_runtime.recording_url` in `run-meta.json`.
+
+Provider options are passed as JSON. Supported fields are `blockAds`,
+`solveCaptcha`, `useProxy`, `proxyUrl`, `region`, `userAgent`, `stealthConfig`,
+`sessionContext`, and `extensionIds`:
+
+```bash
+uv run clawbench-batch --models your-model --all-cases \
+  --browser-runtime steel \
+  --browser-runtime-options '{"blockAds":true,"solveCaptcha":true}'
+```
+
+`dimensions` and `timeout` are set by ClawBench (1920x1080, and the task time
+limit plus 120s of headroom) and are rejected as options. Batch concurrency
+defaults to **1**; raise `--max-concurrent` only as far as your Steel plan or
+self-hosted capacity allows.
+
 ## Attaching to a browser you already run
 
 ```bash

docs/cli.md

@@ -42,7 +42,7 @@ clawbench-run <case-dir> --human      # human reference run
 | `--output-dir <path>` | `<project>/test-output` | Where run directories are written |
 | `--no-build` | off | Skip building the container image (assumes it exists) |
 | `--no-upload` | off | Skip HuggingFace upload even if `HF_TOKEN` is configured |
-| `--browser-runtime <name>` | `local` | `local`, `kernel`, `browserbase`, `remote-cdp` — see [`browser-runtimes.md`](browser-runtimes.md) |
+| `--browser-runtime <name>` | `local` | `local`, `kernel`, `browserbase`, `steel`, `remote-cdp` — see [`browser-runtimes.md`](browser-runtimes.md) |
 | `--browser-cdp-url <url>` | — | CDP endpoint for `--browser-runtime remote-cdp` |
 | `--browser-runtime-options <json>` | — | Provider-specific options, e.g. `'{"region":"us-west-2"}'` |
 

src/clawbench/runner/batch.py

@@ -44,7 +44,7 @@
     "claw-eval": "test-cases/claw-eval",
 }
 DEFAULT_CASES_SUITE = "v2"
-MANAGED_BROWSER_RUNTIMES = frozenset({"browserbase", "kernel"})
+MANAGED_BROWSER_RUNTIMES = frozenset({"browserbase", "kernel", "steel"})
 
 
 def load_models_yaml() -> dict:

src/clawbench/runner/run_support/browser_runtime/providers.py

@@ -31,6 +31,18 @@
     "stealth",
     "tags",
 }
+_STEEL_API_URL = "https://api.steel.dev"
+_STEEL_ALLOWED_OPTIONS = {
+    "blockAds",
+    "extensionIds",
+    "proxyUrl",
+    "region",
+    "sessionContext",
+    "solveCaptcha",
+    "stealthConfig",
+    "useProxy",
+    "userAgent",
+}
 DEFAULT_BROWSER_CDP_URL = os.environ.get(
     "CLAWBENCH_BROWSER_CDP_URL",
     "http://127.0.0.1:9222",
@@ -156,6 +168,12 @@ def __init__(self, message: str, *, status: int | None = None) -> None:
         self.status = status
 
 
+class _SteelApiError(RuntimeError):
+    def __init__(self, message: str, *, status: int | None = None) -> None:
+        super().__init__(message)
+        self.status = status
+
+
 def _parse_options(raw: str | None) -> dict[str, Any]:
     if not raw:
         return {}
@@ -173,6 +191,19 @@ def _env_value(env: dict[str, str], key: str) -> str | None:
     return value if value else None
 
 
+def _scrub_secret(text: str, secret: str | None) -> str:
+    """Remove an API key from an error string, including URL-encoded forms."""
+    if not secret:
+        return text
+    for form in (
+        secret,
+        urllib.parse.quote(secret, safe=""),
+        urllib.parse.quote_plus(secret, safe=""),
+    ):
+        text = text.replace(form, "[REDACTED]")
+    return text
+
+
 def _pick_free_port() -> int:
     with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
         s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
@@ -232,22 +263,184 @@ def cleanup(self, session: BrowserSession) -> None:
 
 
 class SteelBrowserRuntimeProvider:
+    """Steel sessions, either Steel Cloud or a self-hosted steel-browser.
+
+    Both speak the same ``/v1/sessions`` API; the only difference is that
+    Steel Cloud authenticates with ``steel-api-key`` while a self-hosted
+    deployment (``STEEL_BASE_URL``) usually takes no key at all.
+    """
+
     name = "steel"
-    default_recording_mode = "disabled"
+    default_recording_mode = "provider"
 
-    def __init__(self, *, options: dict[str, Any]) -> None:
+    def __init__(
+        self,
+        *,
+        api_key: str | None,
+        options: dict[str, Any],
+        api_url: str = _STEEL_API_URL,
+    ) -> None:
+        unknown = sorted(set(options) - _STEEL_ALLOWED_OPTIONS)
+        if unknown:
+            allowed = ", ".join(sorted(_STEEL_ALLOWED_OPTIONS))
+            raise BrowserRuntimeError(
+                "steel runtime options contain unsupported field(s): "
+                f"{', '.join(unknown)}; allowed fields: {allowed}"
+            )
+        for key in ("blockAds", "solveCaptcha", "useProxy"):
+            value = options.get(key)
+            if value is not None and not isinstance(value, bool):
+                raise BrowserRuntimeError(f"steel {key} option must be a boolean")
+        for key in ("proxyUrl", "region", "userAgent"):
+            value = options.get(key)
+            if value is not None and not isinstance(value, str):
+                raise BrowserRuntimeError(f"steel {key} option must be a string")
+        for key in ("sessionContext", "stealthConfig"):
+            value = options.get(key)
+            if value is not None and not isinstance(value, dict):
+                raise BrowserRuntimeError(f"steel {key} option must be a JSON object")
+        extension_ids = options.get("extensionIds")
+        if extension_ids is not None and not isinstance(extension_ids, list):
+            raise BrowserRuntimeError("steel extensionIds option must be a JSON array")
+        self.api_key = api_key
         self.options = options
+        self.api_url = api_url.rstrip("/")
+
+    @property
+    def _is_cloud(self) -> bool:
+        return self.api_url == _STEEL_API_URL
+
+    def _request(
+        self,
+        method: str,
+        path: str,
+        payload: dict[str, Any] | None = None,
+    ) -> dict[str, Any]:
+        data = (
+            json.dumps(payload, separators=(",", ":")).encode()
+            if payload is not None
+            else None
+        )
+        headers = {"Content-Type": "application/json"}
+        if self.api_key:
+            headers["steel-api-key"] = self.api_key
+        request = urllib.request.Request(
+            f"{self.api_url}{path}",
+            data=data,
+            method=method,
+            headers=headers,
+        )
+        try:
+            with urllib.request.urlopen(request, timeout=15) as response:
+                raw = response.read()
+        except urllib.error.HTTPError as e:
+            if e.code in {401, 403}:
+                message = "Steel authentication failed"
+            elif e.code in {402, 429}:
+                message = "Steel quota or concurrency limit was exceeded"
+            else:
+                message = f"Steel API returned HTTP {e.code}"
+            raise _SteelApiError(message, status=e.code) from None
+        except (urllib.error.URLError, TimeoutError, OSError) as e:
+            reason = _scrub_secret(str(getattr(e, "reason", e)), self.api_key)
+            raise _SteelApiError(f"Steel API request failed: {reason}") from None
+
+        try:
+            result = json.loads(raw)
+        except (json.JSONDecodeError, UnicodeDecodeError):
+            raise _SteelApiError("Steel API returned malformed JSON") from None
+        if not isinstance(result, dict):
+            raise _SteelApiError("Steel API returned non-object JSON")
+        return result
+
+    def _release(self, session_id: str) -> str:
+        try:
+            self._request("POST", f"/v1/sessions/{session_id}/release")
+        except _SteelApiError as e:
+            if e.status in {404, 409}:
+                return "already_closed"
+            raise
+        return "released"
 
     def start(self, task: dict[str, Any], time_limit_s: int) -> BrowserSession:
-        raise BrowserRuntimeError(
-            "steel browser runtime is reserved but not implemented yet"
+        if not self.api_key and self._is_cloud:
+            raise BrowserRuntimeError(
+                "steel browser runtime requires STEEL_API_KEY (or STEEL_BASE_URL "
+                "pointing at a self-hosted steel-browser)"
+            )
+
+        # Steel expresses session timeouts in milliseconds; ClawBench works in
+        # seconds and adds the same 120s of headroom the other managed runtimes
+        # use for startup and teardown.
+        timeout_ms = min(86_400_000, max(60_000, (time_limit_s + 120) * 1000))
+        payload = {
+            **self.options,
+            "dimensions": {"width": 1920, "height": 1080},
+            "timeout": timeout_ms,
+        }
+        try:
+            result = self._request("POST", "/v1/sessions", payload)
+        except _SteelApiError as e:
+            raise BrowserRuntimeError(str(e)) from None
+
+        session_id = result.get("id")
+        cdp_url = result.get("websocketUrl")
+        if not isinstance(session_id, str) or not session_id:
+            raise BrowserRuntimeError(
+                "Steel session response did not include a valid id"
+            )
+        if not isinstance(cdp_url, str) or not cdp_url.startswith(("ws://", "wss://")):
+            try:
+                self._release(session_id)
+            except _SteelApiError:
+                pass
+            raise BrowserRuntimeError(
+                "Steel session response did not include a valid websocketUrl"
+            )
+
+        viewer_url = result.get("sessionViewerUrl")
+        if not isinstance(viewer_url, str) or not viewer_url.startswith(
+            ("http://", "https://")
+        ):
+            viewer_url = None
+        debug_url = result.get("debugUrl")
+        if not isinstance(debug_url, str) or not debug_url.startswith(
+            ("http://", "https://")
+        ):
+            debug_url = None
+
+        return BrowserSession(
+            provider=self.name,
+            mode="remote",
+            session_id=session_id,
+            cdp_url=cdp_url,
+            viewer_url=viewer_url,
+            debug_url=debug_url,
+            # Steel keeps the rrweb session replay behind the same viewer URL
+            # instead of handing back a downloadable file, so a Steel run has
+            # no local recording.mp4.
+            recording_url=viewer_url,
+            metadata={
+                "deployment": "cloud" if self._is_cloud else "self-hosted",
+                "region": result.get("region"),
+                "timeout_ms": timeout_ms,
+                "proxy_source": result.get("proxySource"),
+                "solve_captcha": result.get("solveCaptcha"),
+            },
+            recording_mode="provider",
         )
 
     def finalize(self, session: BrowserSession, output_dir: Path) -> None:
         pass
 
     def cleanup(self, session: BrowserSession) -> None:
-        session.cleanup_status = "not_required"
+        if not session.session_id:
+            session.cleanup_status = "not_required"
+            return
+        try:
+            session.cleanup_status = self._release(session.session_id)
+        except _SteelApiError as e:
+            raise BrowserRuntimeError(str(e)) from None
 
 
 class BrowserbaseRuntimeProvider:
@@ -324,14 +517,7 @@ def _request(
                 message = f"Browserbase API returned HTTP {e.code}"
             raise _BrowserbaseApiError(message, status=e.code) from None
         except (urllib.error.URLError, TimeoutError, OSError) as e:
-            reason = str(getattr(e, "reason", e))
-            for secret in (
-                self.api_key,
-                urllib.parse.quote(self.api_key, safe=""),
-                urllib.parse.quote_plus(self.api_key, safe=""),
-            ):
-                if secret:
-                    reason = reason.replace(secret, "[REDACTED]")
+            reason = _scrub_secret(str(getattr(e, "reason", e)), self.api_key)
             raise _BrowserbaseApiError(
                 f"Browserbase API request failed: {reason}"
             ) from None
@@ -511,14 +697,7 @@ def _request_response(
                 message = f"Kernel API returned HTTP {e.code}"
             raise _KernelApiError(message, status=e.code) from None
         except (urllib.error.URLError, TimeoutError, OSError) as e:
-            reason = str(getattr(e, "reason", e))
-            for secret in (
-                self.api_key,
-                urllib.parse.quote(self.api_key, safe=""),
-                urllib.parse.quote_plus(self.api_key, safe=""),
-            ):
-                if secret:
-                    reason = reason.replace(secret, "[REDACTED]")
+            reason = _scrub_secret(str(getattr(e, "reason", e)), self.api_key)
             raise _KernelApiError(f"Kernel API request failed: {reason}") from None
 
     def _request(
@@ -783,7 +962,20 @@ def make_browser_runtime_provider(
             },
         )
     if runtime == "steel":
-        return SteelBrowserRuntimeProvider(options=options)
+        api_url = _env_value(env, "STEEL_BASE_URL") or _STEEL_API_URL
+        api_key = _env_value(env, "STEEL_API_KEY")
+        # A self-hosted steel-browser normally runs unauthenticated, so the key
+        # is only mandatory when talking to Steel Cloud.
+        if not api_key and api_url.rstrip("/") == _STEEL_API_URL:
+            raise BrowserRuntimeError(
+                "steel browser runtime requires STEEL_API_KEY (or STEEL_BASE_URL "
+                "pointing at a self-hosted steel-browser)"
+            )
+        return SteelBrowserRuntimeProvider(
+            api_key=api_key,
+            options=options,
+            api_url=api_url,
+        )
     if runtime == "browserbase":
         api_key = _env_value(env, "BROWSERBASE_API_KEY")
         if not api_key: