Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions .claude/skills/add-sandbox-provider/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ In [package.json](../../../package.json) add `"@computesdk/<name>": "^<version>"
`dependencies`, **in alphabetical order**. Then sync the lockfile:

```bash
npm install
pnpm install
```

Confirm the diff only added the new package — `git diff --stat package-lock.json`
Expand All @@ -43,7 +43,7 @@ that's fine as long as the committed lockfile only gained the new entry).

## 3. Document the env var(s)

In [env.example](../../../env.example) append a section (staged providers live near the
In [benchmarks/.env.example](../../../benchmarks/.env.example) append a section (staged providers live near the
bottom, by `BEAM`):

```
Expand All @@ -53,7 +53,7 @@ bottom, by `BEAM`):

## 4. Stage the provider entry (commented)

In [src/sandbox/providers.ts](../../../src/sandbox/providers.ts):
In [benchmarks/sandbox/providers.ts](../../../benchmarks/sandbox/providers.ts):

- Add a commented import near the other imports: `// import { <name> } from '@computesdk/<name>';`
- Add a commented entry in the `providers[]` array (direct-mode section, alphabetical):
Expand Down Expand Up @@ -81,7 +81,7 @@ two spots, both commented like the `beam` lines:

## What you do NOT need to touch

- `GATEWAY_PROVIDERS` in `src/sandbox/generate-svg.ts` — only for automatic-mode
- `GATEWAY_PROVIDERS` in `benchmarks/sandbox/generate-svg.ts` — only for automatic-mode
providers routed through the ComputeSDK gateway (railway/render), not direct-mode SDKs.
- A `bench:<name>` script in package.json is optional; CI invokes
`--provider ${{ matrix.provider }}` directly. Skip it unless asked.
Expand All @@ -91,7 +91,7 @@ two spots, both commented like the `beam` lines:

The runner gracefully **skips** any provider whose `requiredEnvVars` are missing — it
returns a `skipped: true` result rather than failing (see
[src/sandbox/benchmark.ts](../../../src/sandbox/benchmark.ts), the `missingVars` check).
[benchmarks/sandbox/benchmark.ts](../../../benchmarks/sandbox/benchmark.ts), the `missingVars` check).
So activating without the secret wouldn't break CI; staging is a convention to keep the
provider list reflecting what's actually being benchmarked until the secret is added.

Expand All @@ -104,5 +104,5 @@ repo settings first.
## Final check

Run `git diff` and confirm the change set is exactly: package.json, package-lock.json,
env.example, src/sandbox/providers.ts, .github/workflows/sandbox-benchmarks.yml — and
benchmarks/.env.example, benchmarks/sandbox/providers.ts, .github/workflows/sandbox-benchmarks.yml — and
that every provider-specific line is commented out (staged).
17 changes: 9 additions & 8 deletions .github/workflows/ai-gateway-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ name: AI Gateway Benchmark
on:
pull_request:
paths:
- 'src/ai-gateway/**'
- 'src/run.ts'
- 'src/merge-results.ts'
- 'benchmarks/ai-gateway/**'
- 'benchmarks/src/run.ts'
- 'benchmarks/src/merge-results.ts'
- 'package.json'
- '.github/workflows/ai-gateway-benchmarks.yml'
schedule:
Expand Down Expand Up @@ -53,16 +53,17 @@ jobs:
with:
# Full history so the rebase-on-push retry below has a merge base.
fetch-depth: 0
- uses: pnpm/action-setup@v4

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2: Third-party pnpm/action-setup action is not pinned to a commit SHA

Uses a mutable version tag for a third-party GitHub Action, allowing supply chain attacks if the tag is compromised.

Pin the action to a full commit SHA and verify it against the official repository.

AI prompt
Check if this security scanner issue is valid. If so, understand the root cause and fix it. If appropriate, update or add tests. Keep the change focused and preserve intended behavior.

<file name=".github/workflows/ai-gateway-benchmarks.yml">
<violation number="1" location=".github/workflows/ai-gateway-benchmarks.yml:56">
<priority>P2</priority>
<title>Third-party pnpm/action-setup action is not pinned to a commit SHA</title>
<evidence>Uses `pnpm/action-setup@v4` with a mutable version tag instead of a commit SHA in `.github/workflows/ai-gateway-benchmarks.yml:56`.</evidence>
<recommendation>Pin the action to a full commit SHA (e.g., `pnpm/action-setup@&lt;sha&gt;`) and verify it against the official pnpm/action-setup repository.</recommendation>
</violation>
</file>

- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Run AI gateway benchmark
env:
Expand All @@ -79,9 +80,9 @@ jobs:
if [ -n "${{ github.event.inputs.provider }}" ]; then
PROVIDER_FLAG="--provider ${{ github.event.inputs.provider }}"
fi
npm run bench:ai-gateway -- $PROVIDER_FLAG \
pnpm run bench:ai-gateway -- $PROVIDER_FLAG \
--iterations ${{ (github.event_name == 'pull_request' && '2') || github.event.inputs.iterations || '10' }}
- run: npm run generate-ai-gateway-svg
- run: pnpm run generate-ai-gateway-svg
- name: Upload results and SVG as artifacts
if: always()
uses: actions/upload-artifact@v4
Expand Down
30 changes: 16 additions & 14 deletions .github/workflows/browser-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ on:
push:
branches: [master]
paths:
- 'src/browser/**'
- 'src/util/**'
- 'src/run.ts'
- 'src/merge-results.ts'
- 'benchmarks/browser/**'
- 'benchmarks/src/util/**'
- 'benchmarks/src/run.ts'
- 'benchmarks/src/merge-results.ts'
- 'package.json'
schedule:
- cron: '0 0 * * 5' # Weekly on Friday at midnight UTC
Expand Down Expand Up @@ -45,16 +45,17 @@ jobs:
# - anchorbrowser
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Clear stale results from checkout
run: rm -rf results/browser/
Expand All @@ -71,7 +72,7 @@ jobs:
TILION_BASE_URL: ${{ secrets.TILION_BASE_URL }}
# ANCHORBROWSER_API_KEY: ${{ secrets.ANCHORBROWSER_API_KEY }}
run: |
npm run bench -- \
pnpm run bench -- \
--mode browser \
--provider ${{ matrix.provider }} \
--iterations ${{ github.event_name == 'push' && '10' || github.event.inputs.iterations || '100' }}
Expand All @@ -96,32 +97,33 @@ jobs:
# A shallow (depth-1) clone can make `git rebase origin/<branch>`
# fail when the remote has advanced past the shallow boundary.
fetch-depth: 0
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Download all artifacts
uses: actions/download-artifact@v4
with:
path: artifacts/
pattern: browser-results-*
- name: Merge results
run: npx tsx src/merge-results.ts --input artifacts --mode browser
run: npx tsx benchmarks/src/merge-results.ts --input artifacts --mode browser
- name: Ingest results to platform
if: github.event_name != 'push'
continue-on-error: true
env:
INGEST_URL: ${{ secrets.INGEST_URL }}
INGEST_SECRET: ${{ secrets.INGEST_SECRET }}
run: npx tsx src/ingest.ts --type browser
- run: npm run generate-browser-svg
run: npx tsx benchmarks/src/ingest.ts --type browser
- run: pnpm run generate-browser-svg
- name: Upload SVG as artifact
if: github.event_name == 'push'
uses: actions/upload-artifact@v4
Expand Down
28 changes: 15 additions & 13 deletions .github/workflows/browser-throughput-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@ on:
push:
branches: [master]
paths:
- 'src/browser/**'
- 'src/util/**'
- 'src/run.ts'
- 'src/merge-results.ts'
- 'benchmarks/browser/**'
- 'benchmarks/src/util/**'
- 'benchmarks/src/run.ts'
- 'benchmarks/src/merge-results.ts'
- 'package.json'
schedule:
- cron: '0 3 * * 5' # Weekly on Friday at 03:00 UTC (offset from main browser benchmark)
Expand Down Expand Up @@ -96,16 +96,17 @@ jobs:
- tilion
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Clear stale results from checkout
run: rm -rf results/browser-throughput/
Expand All @@ -122,7 +123,7 @@ jobs:
TILION_API_KEY: ${{ secrets.TILION_API_KEY }}
TILION_BASE_URL: ${{ secrets.TILION_BASE_URL }}
run: |
npm run bench -- \
pnpm run bench -- \
--mode browser-throughput \
--provider ${{ matrix.provider }} \
--iterations ${{ github.event_name == 'push' && '3' || github.event.inputs.iterations || '100' }}
Expand All @@ -147,25 +148,26 @@ jobs:
# A shallow (depth-1) clone can make `git rebase origin/<branch>`
# fail when the remote has advanced past the shallow boundary.
fetch-depth: 0
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Download all artifacts
uses: actions/download-artifact@v4
with:
path: artifacts/
pattern: browser-throughput-results-*
- name: Merge results
run: npx tsx src/merge-results.ts --input artifacts --mode browser-throughput
- run: npm run generate-browser-throughput-svg
run: npx tsx benchmarks/src/merge-results.ts --input artifacts --mode browser-throughput
- run: pnpm run generate-browser-throughput-svg
- name: Upload SVG as artifact
if: github.event_name == 'push'
uses: actions/upload-artifact@v4
Expand Down
7 changes: 4 additions & 3 deletions .github/workflows/pricing-svg.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
- run: npm ci
- run: npm run generate-pricing-svg
cache: 'pnpm'
- run: pnpm install --frozen-lockfile
- run: pnpm run generate-pricing-svg
- name: Commit and push
run: |
git config user.name "github-actions[bot]"
Expand Down
36 changes: 19 additions & 17 deletions .github/workflows/sandbox-dax-benchmarks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,11 @@ on:
push:
branches: [master]
paths:
- 'src/sandbox/dax.ts'
- 'src/sandbox/providers.ts'
- 'src/run.ts'
- 'src/merge-results.ts'
- 'scripts/dax-benchmark.sh'
- 'benchmarks/sandbox/dax.ts'
- 'benchmarks/sandbox/providers.ts'
- 'benchmarks/src/run.ts'
- 'benchmarks/src/merge-results.ts'
- 'benchmarks/scripts/dax-benchmark.sh'
- 'package.json'
- '.github/workflows/sandbox-dax-benchmarks.yml'
schedule:
Expand Down Expand Up @@ -71,32 +71,33 @@ jobs:
- vercel
steps:
- uses: actions/checkout@v4
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Build E2B template with 8 vCPU / 16 GiB
if: matrix.provider == 'e2b'
env:
E2B_API_KEY: ${{ secrets.E2B_API_KEY }}
run: npx tsx scripts/build-e2b-template.ts
run: npx tsx benchmarks/scripts/build-e2b-template.ts
- name: Build Superserve template with 8 vCPU / 16 GiB
if: matrix.provider == 'superserve'
env:
SUPERSERVE_API_KEY: ${{ secrets.SUPERSERVE_API_KEY }}
run: npx tsx scripts/build-superserve-template.ts
run: npx tsx benchmarks/scripts/build-superserve-template.ts
- name: Find Northflank plan for 8 vCPU / 16 GiB
if: matrix.provider == 'northflank'
env:
NORTHFLANK_TOKEN: ${{ secrets.NORTHFLANK_TOKEN }}
run: npx tsx scripts/find-northflank-plan.ts
run: npx tsx benchmarks/scripts/find-northflank-plan.ts
- name: Clear stale results from checkout
run: rm -rf results/
- name: Run dax benchmark
Expand Down Expand Up @@ -138,7 +139,7 @@ jobs:
VERCEL_TEAM_ID: ${{ secrets.VERCEL_TEAM_ID }}
VERCEL_PROJECT_ID: ${{ secrets.VERCEL_PROJECT_ID }}
run: |
npm run bench -- \
pnpm run bench -- \
--provider ${{ matrix.provider }} \
--mode sandbox-dax \
--iterations ${{ github.event.inputs.iterations || '1' }}
Expand All @@ -160,31 +161,32 @@ jobs:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: pnpm/action-setup@v4
- uses: actions/setup-node@v4
with:
node-version: 24
cache: 'npm'
cache: 'pnpm'
- name: Install dependencies
run: |
if [ "${{ github.event_name }}" = "schedule" ]; then
npm update
pnpm update
else
npm ci
pnpm install --frozen-lockfile
fi
- name: Download all artifacts
uses: actions/download-artifact@v4
with:
path: artifacts/
pattern: results-*
- name: Merge results
run: npx tsx src/merge-results.ts --input artifacts
run: npx tsx benchmarks/src/merge-results.ts --input artifacts
- name: Ingest results to platform
if: github.event_name != 'push' && github.event.inputs.dry_run != 'true'
continue-on-error: true
env:
INGEST_URL: ${{ secrets.INGEST_URL }}
INGEST_SECRET: ${{ secrets.INGEST_SECRET }}
run: npx tsx src/ingest.ts --type sandbox
run: npx tsx benchmarks/src/ingest.ts --type sandbox
- name: Post results to merged PR
if: github.event_name == 'push'
continue-on-error: true
Expand Down
Loading
Loading