Hi AssetOpsBench team,
I read through docs/guideline/ground_truth_design_guideline.md — the ground-truth structure is one of the more carefully thought-out agent-eval formats I've seen: planning_steps (agent + high-level instruction), execution_steps (name/action/agent/arguments/outputs, each with a deterministic flag per field), and execution_links forming a DAG between steps, with the explicit "soft ground truth" idea that deterministic tasks get exact answers while non-deterministic tasks get acceptance criteria instead. That's a genuinely reusable design, not just a benchmark-specific format.
That made me think it's a strong candidate for a portable interchange layer, separate from the AssetOpsBench-specific schema — e.g. so a scenario/trajectory built for AssetOpsBench could be described in a framework-agnostic way for someone comparing it against a differently-shaped agent-eval benchmark, or so the 460+ scenario results (six-dimensional LLM-judge criteria, per the README's leaderboard section) could be archived/diffed independent of your internal format.
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open spec — JSON Schemas under spec/schemas/ (suite.json, testcase.json, resultset.json, grader.json) plus Python/TS SDKs — built for exactly this kind of "framework A's eval data ↔ portable format ↔ framework B" mapping, not another benchmark runner.
A minimal mapping, grounded in your actual field names, would look roughly like:
def ground_truth_to_testcase(scenario: dict) -> evalport.TestCase:
return evalport.TestCase(
id=scenario["id"],
input=scenario["text"], # the utterance
expected_output={
"planning_steps": scenario["planning_steps"],
"execution_steps": scenario["execution_steps"],
"execution_links": scenario["execution_links"],
},
metadata={"category": scenario.get("category"), "type": scenario.get("type")},
)This is a "does this seem useful" check before I spend time on it, not a request to change anything in AssetOpsBench itself — happy to build and maintain the adapter as a separate small package if there's interest, or drop it if not. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — and congrats on the KDD 2026 acceptance and the IJCAI competition, that's a lot of momentum for a benchmark repo.
Hi AssetOpsBench team,
I read through
docs/guideline/ground_truth_design_guideline.md— the ground-truth structure is one of the more carefully thought-out agent-eval formats I've seen:planning_steps(agent + high-level instruction),execution_steps(name/action/agent/arguments/outputs, each with adeterministicflag per field), andexecution_linksforming a DAG between steps, with the explicit "soft ground truth" idea that deterministic tasks get exact answers while non-deterministic tasks get acceptance criteria instead. That's a genuinely reusable design, not just a benchmark-specific format.That made me think it's a strong candidate for a portable interchange layer, separate from the AssetOpsBench-specific schema — e.g. so a scenario/trajectory built for AssetOpsBench could be described in a framework-agnostic way for someone comparing it against a differently-shaped agent-eval benchmark, or so the 460+ scenario results (six-dimensional LLM-judge criteria, per the README's leaderboard section) could be archived/diffed independent of your internal format.
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), a small open spec — JSON Schemas under
spec/schemas/(suite.json,testcase.json,resultset.json,grader.json) plus Python/TS SDKs — built for exactly this kind of "framework A's eval data ↔ portable format ↔ framework B" mapping, not another benchmark runner.A minimal mapping, grounded in your actual field names, would look roughly like:
This is a "does this seem useful" check before I spend time on it, not a request to change anything in AssetOpsBench itself — happy to build and maintain the adapter as a separate small package if there's interest, or drop it if not. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
No pressure — and congrats on the KDD 2026 acceptance and the IJCAI competition, that's a lot of momentum for a benchmark repo.