Merge branch 'main' into cedric/migrate-braintrust-langchain-into-main-pkg

Author: viadezo1er

.agents/skills/sdk-benchmarking/SKILL.md

@@ -0,0 +1,101 @@
+---
+name: sdk-benchmarking
+description: Run, compare, and extend Braintrust Python SDK pyperf benchmarks. Use when touching hot-path code in `py/src/braintrust/` such as serialization, deep-copy, span creation, or logging; when adding or updating files under `py/benchmarks/`; or when you need baseline/branch performance measurements with `cd py && make bench` and `make bench-compare`.
+---
+
+# SDK Benchmarking
+
+Use this skill for benchmark work in the Braintrust Python SDK repository.
+
+Benchmark support already exists in `py/benchmarks/`. Use the current repo workflow, not commit archaeology, once you have identified the relevant benchmark surface.
+
+## Read First
+
+Always read:
+
+- `AGENTS.md`
+- `CONTRIBUTING.md`
+- `py/Makefile`
+- `py/benchmarks/__main__.py`
+- `py/benchmarks/_utils.py`
+- `py/benchmarks/benches/__init__.py`
+
+Read when relevant:
+
+- `py/benchmarks/benches/bench_bt_json.py` for the module pattern
+- `py/benchmarks/fixtures.py` for shared payload builders
+- `py/setup.py` when benchmarking the optional `orjson` fast path
+- `references/benchmark-patterns.md` in this skill for command and module templates
+
+## Workflow
+
+1. Identify the hot path or API surface that changed.
+2. Find the nearest existing benchmark module under `py/benchmarks/benches/`.
+3. Run the narrowest useful benchmark first.
+4. Add or update a `bench_*.py` module only if the current suite does not cover the changed path.
+5. Reuse or extend `py/benchmarks/fixtures.py` for realistic shared payloads instead of inlining bulky test data.
+6. Save before/after results and compare them when the task is about regression detection or improvement claims.
+
+## Commands
+
+Run benchmarks from `py/`:
+
+```bash
+cd py
+make bench
+make bench BENCH_ARGS="--fast"
+make bench BENCH_ARGS="-o /tmp/before.json"
+make bench BENCH_ARGS="-o /tmp/after.json"
+make bench-compare BENCH_BASE=/tmp/before.json BENCH_NEW=/tmp/after.json
+python -m benchmarks.benches.bench_bt_json
+```
+
+Use `python -m benchmarks --help` for extra `pyperf` flags.
+
+If the benchmark should measure the optional `orjson` path, install the performance extra first:
+
+```bash
+cd py
+python -m uv pip install -e '.[performance]'
+```
+
+## Adding Benchmarks
+
+Put new modules in `py/benchmarks/benches/` and name them `bench_<name>.py`.
+
+Every benchmark module must:
+
+- expose `main(runner: pyperf.Runner | None = None) -> None`
+- create its own `pyperf.Runner()` only when `runner` is `None`
+- call `disable_pyperf_psutil()` before creating that runner
+- register benchmarks with stable, descriptive names via `runner.bench_func(...)`
+- remain executable directly with `python -m benchmarks.benches.bench_<name>`
+
+Do not add manual registration. `python -m benchmarks` auto-discovers every `bench_*.py` module in `py/benchmarks/benches/`.
+
+## Fixtures
+
+Keep reusable payload builders and synthetic objects in `py/benchmarks/fixtures.py`.
+
+Prefer fixture helpers when:
+
+- several benchmark cases share similar payloads
+- the inputs are large enough to distract from the benchmark itself
+- you need variants such as small, medium, large, circular, or non-string-key cases
+
+Keep fixture builders deterministic and focused on representative data shapes.
+
+## Validation
+
+- Run the narrowest affected benchmark first.
+- Use `BENCH_ARGS="--fast"` for quick local sanity checks while iterating.
+- Save JSON outputs and use `make bench-compare` for baseline versus branch comparisons.
+- If you changed code paths that also have correctness tests, run the smallest relevant test target in addition to the benchmark.
+
+## Pitfalls
+
+- Measuring import/setup overhead instead of the hot function under test.
+- Inlining ad hoc payload construction in each benchmark instead of reusing fixtures.
+- Forgetting the standalone `main()` pattern, which breaks auto-discovery and direct execution symmetry.
+- Claiming performance changes from a single unsaved local run instead of comparing saved results.
+- Benchmarking the `orjson` fast path without explicitly installing `.[performance]`.

.agents/skills/sdk-benchmarking/agents/openai.yaml

@@ -0,0 +1,4 @@
+interface:
+  display_name: "SDK Benchmarking"
+  short_description: "Run and extend Braintrust SDK benchmarks"
+  default_prompt: "Use $sdk-benchmarking to run, compare, or add Braintrust Python SDK benchmarks."

.agents/skills/sdk-benchmarking/references/benchmark-patterns.md

@@ -0,0 +1,93 @@
+# Benchmark Patterns
+
+Use this reference when adding or updating SDK benchmarks.
+
+## Command Cheatsheet
+
+```bash
+cd py
+
+# Run everything
+make bench
+
+# Faster local iteration
+make bench BENCH_ARGS="--fast"
+
+# Save results for comparison
+make bench BENCH_ARGS="-o /tmp/before.json"
+make bench BENCH_ARGS="-o /tmp/after.json"
+make bench-compare BENCH_BASE=/tmp/before.json BENCH_NEW=/tmp/after.json
+
+# Run one module directly
+python -m benchmarks.benches.bench_bt_json
+
+# Inspect all forwarded pyperf flags
+python -m benchmarks --help
+```
+
+## Module Skeleton
+
+```python
+import pathlib
+import sys
+
+import pyperf
+
+
+if __package__ in (None, ""):
+    sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[2]))
+
+from benchmarks._utils import disable_pyperf_psutil
+
+
+def target(value):
+    return value
+
+
+def main(runner: pyperf.Runner | None = None) -> None:
+    if runner is None:
+        disable_pyperf_psutil()
+        runner = pyperf.Runner()
+
+    runner.bench_func("example.target[case-name]", target, "value")
+
+
+if __name__ == "__main__":
+    main()
+```
+
+Follow the existing `py/benchmarks/benches/bench_bt_json.py` pattern when importing repo code. The `sys.path` adjustment keeps direct module execution working from inside `py/`.
+
+## Fixture Guidance
+
+Put reusable builders in `py/benchmarks/fixtures.py` when:
+
+- several benchmark cases need the same payload shape
+- the payload should model realistic nested SDK inputs
+- the benchmark should cover edge cases such as circular references or non-string keys
+
+Current fixture patterns already cover:
+
+- small, medium, and large nested payloads
+- circular structures
+- non-string dictionary keys
+- dataclass-like and pydantic-like values
+
+Extend those helpers before creating one-off payload factories in a new benchmark module.
+
+## Comparison Workflow
+
+For branch-to-branch comparisons:
+
+```bash
+cd py
+git checkout main
+make bench BENCH_ARGS="-o /tmp/main.json"
+
+git checkout my-branch
+make bench BENCH_ARGS="-o /tmp/branch.json"
+
+make bench-compare BENCH_BASE=/tmp/main.json BENCH_NEW=/tmp/branch.json
+```
+
+Use `--rigorous` only when you need lower-noise final numbers; use `--fast` while iterating.