Proposal: Python adapter converting GuardResponse/RedactResponse to a portable OpenEval ResultSet

[adhabnr-ux]

What EvalPort is: an open interchange spec + SDK (evalport-sdk on PyPI) for portable LLM evaluation results — a shared schema so results from one safety/eval tool can be read, aggregated, and diffed alongside results from unrelated tools, without bespoke glue per tool.

Why the Superagent Python SDK fits: I read sdk/python/src/safety_agent/client.py and types.py. SafetyClient.guard() already returns a clean, structured verdict object:

class GuardResponse:
    classification: Literal["pass", "block"]
    reasoning: str
    violation_types: list[str]
    cwe_codes: list[str]
    usage: TokenUsage

(built from the JSON-schema in schemas.py's GUARD_RESPONSE_FORMAT, with _aggregate_guard_results() handling chunked inputs). redact() similarly returns RedactResponse(redacted, findings, usage), and scan() returns ScanResponse(result, usage: ScanUsage). These are exactly the shape an evaluation result record wants: an id per tested input, a pass/block verdict, and structured violation metadata.

A small standalone adapter (safety-agent-openeval-adapter, zero footprint on safety-agent itself) could turn a batch of guard() calls into a portable result set:

from safety_agent import create_client
from safety_agent_openeval_adapter import results_to_openeval
from openeval.validate import validate_result_set

client = create_client()
results = [await client.guard(input=p) for p in prompts]
# each: GuardResponse(classification="block", reasoning="...", violation_types=[...], cwe_codes=[...], usage=...)

result_set = results_to_openeval(results, probe_ids=prompt_ids, suite_id="superagent-guard-eval")
validate_result_set(result_set)

This would let a red-team run against client.guard() (or client.redact()) sit next to results from any other framework's adapter in one portable format — useful for anyone benchmarking Superagent's guard model against other guardrail tools with a shared scoring pipeline.

Spec + SDK: https://github.com/adhabnr-ux/evalport

Happy to build and submit this myself as a standalone Python package with tests against the real GuardResponse/RedactResponse shapes — it can live entirely in EvalPort's own adapters/ directory with no changes required to this repo, unless you'd rather see it linked from sdk/python/README.md.

Posting as an independent contributor, not affiliated with Superagent — just read through the Python SDK client and thought the result objects were a good fit.

[homanp]

Sounds like a good idea! Please submit a PR

[adhabnr-ux]

Will do — I'll build it as a standalone adapter and open a PR. Thanks for the quick yes.

— Sahi