diff --git a/docs/en/benchmarks/automation_bench.md b/docs/en/benchmarks/automation_bench.md new file mode 100644 index 000000000..51436a179 --- /dev/null +++ b/docs/en/benchmarks/automation_bench.md @@ -0,0 +1,144 @@ +# AutomationBench + + +## Overview + +AutomationBench evaluates agents on realistic business workflows across sales, marketing, operations, support, +finance, and HR. EvalScope runs the public tasks, simulated SaaS services, and assertion-based scoring provided by +Zapier's official Python package. + +## Task Description + +- **Task Type**: Stateful business workflows across 47 simulated SaaS tools. +- **Public Dataset**: 600 tasks, with 100 tasks in each of `sales`, `marketing`, `operations`, `support`, `finance`, + and `hr`. +- **Simple Baseline**: The optional `simple` subset contains 200 foundational single- and two-step tasks. Its metrics + use the `simple_` prefix and are reported separately from the public benchmark score. +- **Scoring**: `partial_credit` is the fraction of scored assertions satisfied. `pass_rate` is the mean official + `task_completed_correctly` signal, which is 1 only when every scored assertion passes. + +## Evaluation Notes + +- Prepare a Python 3.13+ environment and install the pinned dependencies before evaluation: + `python -m pip install "verifiers==0.1.12.dev2" "automation-bench @ git+https://github.com/zapier/AutomationBench.git@a321764ace3cfbe42289e6a13abef2f0f4f56fad"`. +- By default, EvalScope evaluates all six public domains (600 tasks). Use `subset_list` to select domains or `limit` + for a smaller run. +- Use an API-backed EvalScope model with `openai_api`, `openai_responses_api`, or `anthropic_api` evaluation type. +- No Docker runtime, external dataset download, or real SaaS credentials are required. Model API credentials are still + required. +- The released tasks are public. Zapier's official leaderboard uses a separate held-out private set, so local public + scores are directional rather than leaderboard-identical. + + +## Properties + +| Property | Value | +|----------|-------| +| **Benchmark Name** | `automation_bench` | +| **Dataset ID** | [AutomationBench](https://github.com/zapier/AutomationBench) | +| **Paper** | [Paper](https://arxiv.org/abs/2604.18934) | +| **Tags** | `Agent`, `FunctionCalling`, `MultiTurn` | +| **Metrics** | `pass_rate`, `partial_credit`, `error_rate` | +| **Default Shots** | 0-shot | +| **Evaluation Split** | `test` | + + +## Data Statistics + +| Metric | Value | +|--------|-------| +| Total Samples | 600 | +| Prompt Length (Mean) | 983.57 chars | +| Prompt Length (Min/Max) | 586 / 1785 chars | + +**Per-Subset Statistics:** + +| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max | +|--------|---------|-------------|------------|------------| +| `sales` | 100 | 832.89 | 586 | 1245 | +| `marketing` | 100 | 953.02 | 612 | 1275 | +| `operations` | 100 | 1259.54 | 787 | 1785 | +| `support` | 100 | 937.56 | 711 | 1577 | +| `finance` | 100 | 990.51 | 709 | 1259 | +| `hr` | 100 | 927.91 | 689 | 1276 | + +## Sample Example + +**Subset**: `sales` + +```json +{ + "input": [ + { + "id": "97a20b04", + "content": "You are a workflow automation agent. Execute the requested tasks using the available tools. Do not ask clarifying questions - use the information provided and make reasonable assumptions when needed. You have a budget of ~50 tool-using turns ... [TRUNCATED 47 chars] ... searches. When summarizing your work in messages or records, list only items you acted on. Do not name, enumerate, or explain items you skipped, excluded, or rejected — handle exclusions silently in the action, not narratively in the output.", + "source": "input", + "role": "system" + }, + { + "id": "4d75f740", + "content": "We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Be sure to follow the latest routing guidelines. Confirm the account tier from the 'Account Hierarchy' spreadshe ... [TRUNCATED 160 chars] ... support-escalation@example.com, executive-team@example.com, sales-team@example.com, smb-team@example.com, vp-sales@example.com\n\nUse Gmail for all email sends. Include the names of affected entities and the relevant amounts in your message(s).", + "source": "input", + "role": "user" + } + ], + "target": "", + "id": 0, + "group_id": 0, + "subset_key": "sales", + "metadata": { + "record_key": "sales:0", + "domain": "sales", + "task": "sales.multi_hop_lookup", + "official_example_id": 501 + } +} +``` + +*Note: Some content was truncated for display.* + +## Prompt Template + +*No prompt template defined.* + +## Extra Parameters + +| Parameter | Type | Default | Description | +|-----------|------|---------|-------------| +| `toolset` | `str` | `api` | Official tool style exposed to the agent. Choices: ['api', 'zapier', 'limited_zapier'] | + +## Usage + +### Using CLI + +```bash +evalscope eval \ + --model YOUR_MODEL \ + --api-url OPENAI_API_COMPAT_URL \ + --api-key EMPTY_TOKEN \ + --datasets automation_bench \ + --limit 10 # Remove this line for formal evaluation +``` + +### Using Python + +```python +from evalscope import run_task +from evalscope.config import TaskConfig + +task_cfg = TaskConfig( + model='YOUR_MODEL', + api_url='OPENAI_API_COMPAT_URL', + api_key='EMPTY_TOKEN', + datasets=['automation_bench'], + dataset_args={ + 'automation_bench': { + # subset_list: ['sales', 'marketing', 'operations'] # optional, evaluate specific subsets + # extra_params: {} # uses default extra parameters + } + }, + limit=10, # Remove this line for formal evaluation +) + +run_task(task_cfg=task_cfg) +``` diff --git a/docs/en/get_started/supported_dataset/agent.md b/docs/en/get_started/supported_dataset/agent.md index cb92fbbc4..ec35a0bcd 100644 --- a/docs/en/get_started/supported_dataset/agent.md +++ b/docs/en/get_started/supported_dataset/agent.md @@ -4,6 +4,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d | Benchmark Name | Pretty Name | Task Categories | |------------|----------|----------| +| `automation_bench` | [AutomationBench](../../benchmarks/automation_bench.md) | `Agent`, `FunctionCalling`, `MultiTurn` | | `bfcl_v3` | [BFCL-v3](../../benchmarks/bfcl_v3.md) | `Agent`, `FunctionCalling` | | `bfcl_v4` | [BFCL-v4](../../benchmarks/bfcl_v4.md) | `Agent`, `FunctionCalling` | | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` | @@ -37,6 +38,7 @@ Below is the list of supported AGENT benchmarks. Click on a benchmark name for d :hidden: :maxdepth: 1 +../../benchmarks/automation_bench.md ../../benchmarks/bfcl_v3.md ../../benchmarks/bfcl_v4.md ../../benchmarks/browsecomp.md diff --git a/docs/zh/benchmarks/automation_bench.md b/docs/zh/benchmarks/automation_bench.md new file mode 100644 index 000000000..3307c0b67 --- /dev/null +++ b/docs/zh/benchmarks/automation_bench.md @@ -0,0 +1,134 @@ +# AutomationBench + + +## 概述 + +AutomationBench 在销售、营销、运营、支持、财务和人力资源等领域的实际业务工作流中评估智能体。EvalScope 使用 Zapier 官方 Python 包提供的公开任务、模拟的 SaaS 服务以及基于断言的评分机制进行评测。 + +## 任务描述 + +- **任务类型**：涵盖 47 个模拟 SaaS 工具的状态化业务工作流。 +- **公开数据集**：共 600 个任务，分别来自 `sales`（销售）、`marketing`（营销）、`operations`（运营）、`support`（支持）、`finance`（财务）和 `hr`（人力资源）六个领域，每个领域各 100 个任务。 +- **简单基线**：可选的 `simple` 子集包含 200 个基础性的单步或两步任务。其指标使用 `simple_` 前缀，并与公开基准测试分数分开报告。 +- **评分方式**：`partial_credit` 表示满足的评分断言所占比例；`pass_rate` 是官方 `task_completed_correctly` 信号的平均值，仅当所有评分断言全部通过时该值才为 1。 + +## 评测说明 + +- 评测前请准备 Python 3.13+ 环境并安装指定依赖： + `python -m pip install "verifiers==0.1.12.dev2" "automation-bench @ git+https://github.com/zapier/AutomationBench.git@a321764ace3cfbe42289e6a13abef2f0f4f56fad"`。 +- 默认情况下，EvalScope 会评测全部六个公开领域（共 600 个任务）。可通过 `subset_list` 参数选择特定领域，或使用 `limit` 参数进行小规模运行。 +- 需使用基于 API 的 EvalScope 模型，并指定 `openai_api`、`openai_responses_api` 或 `anthropic_api` 作为评测类型。 +- 无需 Docker 运行环境、外部数据集下载或真实的 SaaS 凭据，但仍需提供模型 API 凭据。 +- 发布的任务是公开的。Zapier 官方排行榜使用的是独立的私有保留集，因此本地公开分数仅具参考意义，无法与排行榜完全一致。 + +## 属性 + +| 属性 | 值 | +|----------|-------| +| **基准测试名称** | `automation_bench` | +| **数据集ID** | [AutomationBench](https://github.com/zapier/AutomationBench) | +| **论文** | [Paper](https://arxiv.org/abs/2604.18934) | +| **标签** | `Agent`, `FunctionCalling`, `MultiTurn` | +| **指标** | `pass_rate`, `partial_credit`, `error_rate` | +| **默认示例数** | 0-shot | +| **评测划分** | `test` | + +## 数据统计 + +| 指标 | 值 | +|--------|-------| +| 总样本数 | 600 | +| 提示词长度（平均） | 983.57 字符 | +| 提示词长度（最小/最大） | 586 / 1785 字符 | + +**各子集统计数据：** + +| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 | +|--------|---------|-------------|------------|------------| +| `sales` | 100 | 832.89 | 586 | 1245 | +| `marketing` | 100 | 953.02 | 612 | 1275 | +| `operations` | 100 | 1259.54 | 787 | 1785 | +| `support` | 100 | 937.56 | 711 | 1577 | +| `finance` | 100 | 990.51 | 709 | 1259 | +| `hr` | 100 | 927.91 | 689 | 1276 | + +## 样例示例 + +**子集**: `sales` + +```json +{ + "input": [ + { + "id": "97a20b04", + "content": "You are a workflow automation agent. Execute the requested tasks using the available tools. Do not ask clarifying questions - use the information provided and make reasonable assumptions when needed. You have a budget of ~50 tool-using turns ... [TRUNCATED 47 chars] ... searches. When summarizing your work in messages or records, list only items you acted on. Do not name, enumerate, or explain items you skipped, excluded, or rejected — handle exclusions silently in the action, not narratively in the output.", + "source": "input", + "role": "system" + }, + { + "id": "4d75f740", + "content": "We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Be sure to follow the latest routing guidelines. Confirm the account tier from the 'Account Hierarchy' spreadshe ... [TRUNCATED 160 chars] ... support-escalation@example.com, executive-team@example.com, sales-team@example.com, smb-team@example.com, vp-sales@example.com\n\nUse Gmail for all email sends. Include the names of affected entities and the relevant amounts in your message(s).", + "source": "input", + "role": "user" + } + ], + "target": "", + "id": 0, + "group_id": 0, + "subset_key": "sales", + "metadata": { + "record_key": "sales:0", + "domain": "sales", + "task": "sales.multi_hop_lookup", + "official_example_id": 501 + } +} +``` + +*注：部分内容因展示需要已被截断。* + +## 提示模板 + +*未定义提示模板。* + +## 额外参数 + +| 参数 | 类型 | 默认值 | 描述 | +|-----------|------|---------|-------------| +| `toolset` | `str` | `api` | 提供给智能体的官方工具风格。选项：['api', 'zapier', 'limited_zapier'] | + +## 使用方法 + +### 使用 CLI + +```bash +evalscope eval \ + --model YOUR_MODEL \ + --api-url OPENAI_API_COMPAT_URL \ + --api-key EMPTY_TOKEN \ + --datasets automation_bench \ + --limit 10 # 正式评测时请删除此行 +``` + +### 使用 Python + +```python +from evalscope import run_task +from evalscope.config import TaskConfig + +task_cfg = TaskConfig( + model='YOUR_MODEL', + api_url='OPENAI_API_COMPAT_URL', + api_key='EMPTY_TOKEN', + datasets=['automation_bench'], + dataset_args={ + 'automation_bench': { + # subset_list: ['sales', 'marketing', 'operations'] # 可选，用于评测特定子集 + # extra_params: {} # 使用默认额外参数 + } + }, + limit=10, # 正式评测时请删除此行 +) + +run_task(task_cfg=task_cfg) +``` diff --git a/docs/zh/get_started/supported_dataset/agent.md b/docs/zh/get_started/supported_dataset/agent.md index 70c5c4dd1..4aaaab622 100644 --- a/docs/zh/get_started/supported_dataset/agent.md +++ b/docs/zh/get_started/supported_dataset/agent.md @@ -4,6 +4,7 @@ | 数据集名称 | 标准名称 | 任务类别 | |------------|----------|----------| +| `automation_bench` | [AutomationBench](../../benchmarks/automation_bench.md) | `Agent`, `FunctionCalling`, `MultiTurn` | | `bfcl_v3` | [BFCL-v3](../../benchmarks/bfcl_v3.md) | `Agent`, `FunctionCalling` | | `bfcl_v4` | [BFCL-v4](../../benchmarks/bfcl_v4.md) | `Agent`, `FunctionCalling` | | `browsecomp` | [BrowseComp](../../benchmarks/browsecomp.md) | `Agent`, `Knowledge`, `QA` | @@ -37,6 +38,7 @@ :hidden: :maxdepth: 1 +../../benchmarks/automation_bench.md ../../benchmarks/bfcl_v3.md ../../benchmarks/bfcl_v4.md ../../benchmarks/browsecomp.md diff --git a/evalscope/benchmarks/_meta/automation_bench.json b/evalscope/benchmarks/_meta/automation_bench.json new file mode 100644 index 000000000..61ccc270b --- /dev/null +++ b/evalscope/benchmarks/_meta/automation_bench.json @@ -0,0 +1,152 @@ +{ + "meta": { + "pretty_name": "AutomationBench", + "dataset_id": "https://github.com/zapier/AutomationBench", + "paper_url": "https://arxiv.org/abs/2604.18934", + "tags": [ + "Agent", + "FunctionCalling", + "MultiTurn" + ], + "metrics": [ + "pass_rate", + "partial_credit", + "error_rate" + ], + "few_shot_num": 0, + "eval_split": "test", + "train_split": "", + "subset_list": [ + "sales", + "marketing", + "operations", + "support", + "finance", + "hr" + ], + "description": "\n## Overview\n\nAutomationBench evaluates agents on realistic business workflows across sales, marketing, operations, support,\nfinance, and HR. EvalScope runs the public tasks, simulated SaaS services, and assertion-based scoring provided by\nZapier's official Python package.\n\n## Task Description\n\n- **Task Type**: Stateful business workflows across 47 simulated SaaS tools.\n- **Public Dataset**: 600 tasks, with 100 tasks in each of `sales`, `marketing`, `operations`, `support`, `finance`,\n and `hr`.\n- **Simple Baseline**: The optional `simple` subset contains 200 foundational single- and two-step tasks. Its metrics\n use the `simple_` prefix and are reported separately from the public benchmark score.\n- **Scoring**: `partial_credit` is the fraction of scored assertions satisfied. `pass_rate` is the mean official\n `task_completed_correctly` signal, which is 1 only when every scored assertion passes.\n\n## Evaluation Notes\n\n- Prepare a Python 3.13+ environment and install the pinned dependencies before evaluation:\n `python -m pip install \"verifiers==0.1.12.dev2\" \"automation-bench @ git+https://github.com/zapier/AutomationBench.git@a321764ace3cfbe42289e6a13abef2f0f4f56fad\"`.\n- By default, EvalScope evaluates all six public domains (600 tasks). Use `subset_list` to select domains or `limit`\n for a smaller run.\n- Use an API-backed EvalScope model with `openai_api`, `openai_responses_api`, or `anthropic_api` evaluation type.\n- No Docker runtime, external dataset download, or real SaaS credentials are required. Model API credentials are still\n required.\n- The released tasks are public. Zapier's official leaderboard uses a separate held-out private set, so local public\n scores are directional rather than leaderboard-identical.\n", + "prompt_template": "", + "system_prompt": "", + "few_shot_prompt_template": "", + "aggregation": "mean", + "extra_params": { + "toolset": { + "type": "str", + "description": "Official tool style exposed to the agent.", + "value": "api", + "choices": [ + "api", + "zapier", + "limited_zapier" + ] + } + }, + "sandbox_config": {}, + "category": "agent" + }, + "statistics": { + "total_samples": 600, + "subset_stats": [ + { + "name": "sales", + "sample_count": 100, + "prompt_length_mean": 832.89, + "prompt_length_min": 586, + "prompt_length_max": 1245, + "prompt_length_std": 147.16, + "target_length_mean": null + }, + { + "name": "marketing", + "sample_count": 100, + "prompt_length_mean": 953.02, + "prompt_length_min": 612, + "prompt_length_max": 1275, + "prompt_length_std": 146.53, + "target_length_mean": null + }, + { + "name": "operations", + "sample_count": 100, + "prompt_length_mean": 1259.54, + "prompt_length_min": 787, + "prompt_length_max": 1785, + "prompt_length_std": 198.5, + "target_length_mean": null + }, + { + "name": "support", + "sample_count": 100, + "prompt_length_mean": 937.56, + "prompt_length_min": 711, + "prompt_length_max": 1577, + "prompt_length_std": 143.69, + "target_length_mean": null + }, + { + "name": "finance", + "sample_count": 100, + "prompt_length_mean": 990.51, + "prompt_length_min": 709, + "prompt_length_max": 1259, + "prompt_length_std": 114.98, + "target_length_mean": null + }, + { + "name": "hr", + "sample_count": 100, + "prompt_length_mean": 927.91, + "prompt_length_min": 689, + "prompt_length_max": 1276, + "prompt_length_std": 123.31, + "target_length_mean": null + } + ], + "prompt_length": { + "mean": 983.57, + "min": 586, + "max": 1785, + "std": 198.22 + }, + "target_length_mean": null, + "computed_at": "2026-07-22T16:02:16.107002" + }, + "sample_example": { + "data": { + "input": [ + { + "id": "97a20b04", + "content": "You are a workflow automation agent. Execute the requested tasks using the available tools. Do not ask clarifying questions - use the information provided and make reasonable assumptions when needed. You have a budget of ~50 tool-using turns ... [TRUNCATED 47 chars] ... searches. When summarizing your work in messages or records, list only items you acted on. Do not name, enumerate, or explain items you skipped, excluded, or rejected — handle exclusions silently in the action, not narratively in the output.", + "source": "input", + "role": "system" + }, + { + "id": "4d75f740", + "content": "We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Be sure to follow the latest routing guidelines. Confirm the account tier from the 'Account Hierarchy' spreadshe ... [TRUNCATED 160 chars] ... support-escalation@example.com, executive-team@example.com, sales-team@example.com, smb-team@example.com, vp-sales@example.com\n\nUse Gmail for all email sends. Include the names of affected entities and the relevant amounts in your message(s).", + "source": "input", + "role": "user" + } + ], + "target": "", + "id": 0, + "group_id": 0, + "subset_key": "sales", + "metadata": { + "record_key": "sales:0", + "domain": "sales", + "task": "sales.multi_hop_lookup", + "official_example_id": 501 + } + }, + "subset": "sales", + "truncated": true + }, + "readme": { + "en": "# AutomationBench\n\n\n## Overview\n\nAutomationBench evaluates agents on realistic business workflows across sales, marketing, operations, support,\nfinance, and HR. EvalScope runs the public tasks, simulated SaaS services, and assertion-based scoring provided by\nZapier's official Python package.\n\n## Task Description\n\n- **Task Type**: Stateful business workflows across 47 simulated SaaS tools.\n- **Public Dataset**: 600 tasks, with 100 tasks in each of `sales`, `marketing`, `operations`, `support`, `finance`,\n and `hr`.\n- **Simple Baseline**: The optional `simple` subset contains 200 foundational single- and two-step tasks. Its metrics\n use the `simple_` prefix and are reported separately from the public benchmark score.\n- **Scoring**: `partial_credit` is the fraction of scored assertions satisfied. `pass_rate` is the mean official\n `task_completed_correctly` signal, which is 1 only when every scored assertion passes.\n\n## Evaluation Notes\n\n- Prepare a Python 3.13+ environment and install the pinned dependencies before evaluation:\n `python -m pip install \"verifiers==0.1.12.dev2\" \"automation-bench @ git+https://github.com/zapier/AutomationBench.git@a321764ace3cfbe42289e6a13abef2f0f4f56fad\"`.\n- By default, EvalScope evaluates all six public domains (600 tasks). Use `subset_list` to select domains or `limit`\n for a smaller run.\n- Use an API-backed EvalScope model with `openai_api`, `openai_responses_api`, or `anthropic_api` evaluation type.\n- No Docker runtime, external dataset download, or real SaaS credentials are required. Model API credentials are still\n required.\n- The released tasks are public. Zapier's official leaderboard uses a separate held-out private set, so local public\n scores are directional rather than leaderboard-identical.\n\n\n## Properties\n\n| Property | Value |\n|----------|-------|\n| **Benchmark Name** | `automation_bench` |\n| **Dataset ID** | [AutomationBench](https://github.com/zapier/AutomationBench) |\n| **Paper** | [Paper](https://arxiv.org/abs/2604.18934) |\n| **Tags** | `Agent`, `FunctionCalling`, `MultiTurn` |\n| **Metrics** | `pass_rate`, `partial_credit`, `error_rate` |\n| **Default Shots** | 0-shot |\n| **Evaluation Split** | `test` |\n\n\n## Data Statistics\n\n| Metric | Value |\n|--------|-------|\n| Total Samples | 600 |\n| Prompt Length (Mean) | 983.57 chars |\n| Prompt Length (Min/Max) | 586 / 1785 chars |\n\n**Per-Subset Statistics:**\n\n| Subset | Samples | Prompt Mean | Prompt Min | Prompt Max |\n|--------|---------|-------------|------------|------------|\n| `sales` | 100 | 832.89 | 586 | 1245 |\n| `marketing` | 100 | 953.02 | 612 | 1275 |\n| `operations` | 100 | 1259.54 | 787 | 1785 |\n| `support` | 100 | 937.56 | 711 | 1577 |\n| `finance` | 100 | 990.51 | 709 | 1259 |\n| `hr` | 100 | 927.91 | 689 | 1276 |\n\n## Sample Example\n\n**Subset**: `sales`\n\n```json\n{\n \"input\": [\n {\n \"id\": \"97a20b04\",\n \"content\": \"You are a workflow automation agent. Execute the requested tasks using the available tools. Do not ask clarifying questions - use the information provided and make reasonable assumptions when needed. You have a budget of ~50 tool-using turns ... [TRUNCATED 47 chars] ... searches. When summarizing your work in messages or records, list only items you acted on. Do not name, enumerate, or explain items you skipped, excluded, or rejected — handle exclusions silently in the action, not narratively in the output.\",\n \"source\": \"input\",\n \"role\": \"system\"\n },\n {\n \"id\": \"4d75f740\",\n \"content\": \"We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Be sure to follow the latest routing guidelines. Confirm the account tier from the 'Account Hierarchy' spreadshe ... [TRUNCATED 160 chars] ... support-escalation@example.com, executive-team@example.com, sales-team@example.com, smb-team@example.com, vp-sales@example.com\\n\\nUse Gmail for all email sends. Include the names of affected entities and the relevant amounts in your message(s).\",\n \"source\": \"input\",\n \"role\": \"user\"\n }\n ],\n \"target\": \"\",\n \"id\": 0,\n \"group_id\": 0,\n \"subset_key\": \"sales\",\n \"metadata\": {\n \"record_key\": \"sales:0\",\n \"domain\": \"sales\",\n \"task\": \"sales.multi_hop_lookup\",\n \"official_example_id\": 501\n }\n}\n```\n\n*Note: Some content was truncated for display.*\n\n## Prompt Template\n\n*No prompt template defined.*\n\n## Extra Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `toolset` | `str` | `api` | Official tool style exposed to the agent. Choices: ['api', 'zapier', 'limited_zapier'] |\n\n## Usage\n\n### Using CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets automation_bench \\\n --limit 10 # Remove this line for formal evaluation\n```\n\n### Using Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['automation_bench'],\n dataset_args={\n 'automation_bench': {\n # subset_list: ['sales', 'marketing', 'operations'] # optional, evaluate specific subsets\n # extra_params: {} # uses default extra parameters\n }\n },\n limit=10, # Remove this line for formal evaluation\n)\n\nrun_task(task_cfg=task_cfg)\n```\n\n\n", + "zh": "# AutomationBench\n\n\n## 概述\n\nAutomationBench 在销售、营销、运营、支持、财务和人力资源等领域的实际业务工作流中评估智能体。EvalScope 使用 Zapier 官方 Python 包提供的公开任务、模拟的 SaaS 服务以及基于断言的评分机制进行评测。\n\n## 任务描述\n\n- **任务类型**：涵盖 47 个模拟 SaaS 工具的状态化业务工作流。\n- **公开数据集**：共 600 个任务，分别来自 `sales`（销售）、`marketing`（营销）、`operations`（运营）、`support`（支持）、`finance`（财务）和 `hr`（人力资源）六个领域，每个领域各 100 个任务。\n- **简单基线**：可选的 `simple` 子集包含 200 个基础性的单步或两步任务。其指标使用 `simple_` 前缀，并与公开基准测试分数分开报告。\n- **评分方式**：`partial_credit` 表示满足的评分断言所占比例；`pass_rate` 是官方 `task_completed_correctly` 信号的平均值，仅当所有评分断言全部通过时该值才为 1。\n\n## 评测说明\n\n- 评测前请准备 Python 3.13+ 环境并安装指定依赖：\n `python -m pip install \"verifiers==0.1.12.dev2\" \"automation-bench @ git+https://github.com/zapier/AutomationBench.git@a321764ace3cfbe42289e6a13abef2f0f4f56fad\"`。\n- 默认情况下，EvalScope 会评测全部六个公开领域（共 600 个任务）。可通过 `subset_list` 参数选择特定领域，或使用 `limit` 参数进行小规模运行。\n- 需使用基于 API 的 EvalScope 模型，并指定 `openai_api`、`openai_responses_api` 或 `anthropic_api` 作为评测类型。\n- 无需 Docker 运行环境、外部数据集下载或真实的 SaaS 凭据，但仍需提供模型 API 凭据。\n- 发布的任务是公开的。Zapier 官方排行榜使用的是独立的私有保留集，因此本地公开分数仅具参考意义，无法与排行榜完全一致。\n\n## 属性\n\n| 属性 | 值 |\n|----------|-------|\n| **基准测试名称** | `automation_bench` |\n| **数据集ID** | [AutomationBench](https://github.com/zapier/AutomationBench) |\n| **论文** | [Paper](https://arxiv.org/abs/2604.18934) |\n| **标签** | `Agent`, `FunctionCalling`, `MultiTurn` |\n| **指标** | `pass_rate`, `partial_credit`, `error_rate` |\n| **默认示例数** | 0-shot |\n| **评测划分** | `test` |\n\n## 数据统计\n\n| 指标 | 值 |\n|--------|-------|\n| 总样本数 | 600 |\n| 提示词长度（平均） | 983.57 字符 |\n| 提示词长度（最小/最大） | 586 / 1785 字符 |\n\n**各子集统计数据：**\n\n| 子集 | 样本数 | 提示词平均长度 | 提示词最小长度 | 提示词最大长度 |\n|--------|---------|-------------|------------|------------|\n| `sales` | 100 | 832.89 | 586 | 1245 |\n| `marketing` | 100 | 953.02 | 612 | 1275 |\n| `operations` | 100 | 1259.54 | 787 | 1785 |\n| `support` | 100 | 937.56 | 711 | 1577 |\n| `finance` | 100 | 990.51 | 709 | 1259 |\n| `hr` | 100 | 927.91 | 689 | 1276 |\n\n## 样例示例\n\n**子集**: `sales`\n\n```json\n{\n \"input\": [\n {\n \"id\": \"97a20b04\",\n \"content\": \"You are a workflow automation agent. Execute the requested tasks using the available tools. Do not ask clarifying questions - use the information provided and make reasonable assumptions when needed. You have a budget of ~50 tool-using turns ... [TRUNCATED 47 chars] ... searches. When summarizing your work in messages or records, list only items you acted on. Do not name, enumerate, or explain items you skipped, excluded, or rejected — handle exclusions silently in the action, not narratively in the output.\",\n \"source\": \"input\",\n \"role\": \"system\"\n },\n {\n \"id\": \"4d75f740\",\n \"content\": \"We just closed the Meridian Corp Platform Deal! Mark it as won and route the win notice to the right team per our routing policy. Be sure to follow the latest routing guidelines. Confirm the account tier from the 'Account Hierarchy' spreadshe ... [TRUNCATED 160 chars] ... support-escalation@example.com, executive-team@example.com, sales-team@example.com, smb-team@example.com, vp-sales@example.com\\n\\nUse Gmail for all email sends. Include the names of affected entities and the relevant amounts in your message(s).\",\n \"source\": \"input\",\n \"role\": \"user\"\n }\n ],\n \"target\": \"\",\n \"id\": 0,\n \"group_id\": 0,\n \"subset_key\": \"sales\",\n \"metadata\": {\n \"record_key\": \"sales:0\",\n \"domain\": \"sales\",\n \"task\": \"sales.multi_hop_lookup\",\n \"official_example_id\": 501\n }\n}\n```\n\n*注：部分内容因展示需要已被截断。*\n\n## 提示模板\n\n*未定义提示模板。*\n\n## 额外参数\n\n| 参数 | 类型 | 默认值 | 描述 |\n|-----------|------|---------|-------------|\n| `toolset` | `str` | `api` | 提供给智能体的官方工具风格。选项：['api', 'zapier', 'limited_zapier'] |\n\n## 使用方法\n\n### 使用 CLI\n\n```bash\nevalscope eval \\\n --model YOUR_MODEL \\\n --api-url OPENAI_API_COMPAT_URL \\\n --api-key EMPTY_TOKEN \\\n --datasets automation_bench \\\n --limit 10 # 正式评测时请删除此行\n```\n\n### 使用 Python\n\n```python\nfrom evalscope import run_task\nfrom evalscope.config import TaskConfig\n\ntask_cfg = TaskConfig(\n model='YOUR_MODEL',\n api_url='OPENAI_API_COMPAT_URL',\n api_key='EMPTY_TOKEN',\n datasets=['automation_bench'],\n dataset_args={\n 'automation_bench': {\n # subset_list: ['sales', 'marketing', 'operations'] # 可选，用于评测特定子集\n # extra_params: {} # 使用默认额外参数\n }\n },\n limit=10, # 正式评测时请删除此行\n)\n\nrun_task(task_cfg=task_cfg)\n```", + "content_hash": "5c0094941e963cbdc9cb2c12f43f9025", + "needs_translation": false + }, + "updated_at": "2026-07-22T16:02:16.242636", + "translation_updated_at": "2026-07-22T16:02:21" +} diff --git a/evalscope/benchmarks/automation_bench/__init__.py b/evalscope/benchmarks/automation_bench/__init__.py new file mode 100644 index 000000000..9fb2cdd2a --- /dev/null +++ b/evalscope/benchmarks/automation_bench/__init__.py @@ -0,0 +1,3 @@ +from .automation_bench_adapter import AutomationBenchAdapter + +__all__ = ['AutomationBenchAdapter'] diff --git a/evalscope/benchmarks/automation_bench/automation_bench_adapter.py b/evalscope/benchmarks/automation_bench/automation_bench_adapter.py new file mode 100644 index 000000000..efdae21fb --- /dev/null +++ b/evalscope/benchmarks/automation_bench/automation_bench_adapter.py @@ -0,0 +1,224 @@ +import json +from typing import Any, Dict, List, Tuple + +from evalscope.api.agent import NativeAgentConfig +from evalscope.api.benchmark import AgentAdapter, BenchmarkMeta +from evalscope.api.dataset import DatasetDict, Sample, build_dataset_dict_from_record_map +from evalscope.api.evaluator import InferenceResult +from evalscope.api.metric import Score +from evalscope.api.model import Model, ModelOutput, ModelUsage +from evalscope.api.registry import register_benchmark +from evalscope.constants import EvalType, Tags +from evalscope.utils.argument_utils import get_secret_value +from .utils import ( + DEFAULT_AUTOMATION_BENCH_PACKAGE, + DEFAULT_AUTOMATION_BENCH_VERIFIERS, + PUBLIC_DOMAINS, + convert_automation_bench_messages, + ensure_automation_bench_runtime, + load_automation_bench_records, + run_automation_bench_task, +) + +_EXTRA_PARAMS = { + 'toolset': { + 'type': 'str', + 'description': 'Official tool style exposed to the agent.', + 'value': 'api', + 'choices': ['api', 'zapier', 'limited_zapier'], + }, +} + +_DESCRIPTION = f""" +## Overview + +AutomationBench evaluates agents on realistic business workflows across sales, marketing, operations, support, +finance, and HR. EvalScope runs the public tasks, simulated SaaS services, and assertion-based scoring provided by +Zapier's official Python package. + +## Task Description + +- **Task Type**: Stateful business workflows across 47 simulated SaaS tools. +- **Public Dataset**: 600 tasks, with 100 tasks in each of `sales`, `marketing`, `operations`, `support`, `finance`, + and `hr`. +- **Simple Baseline**: The optional `simple` subset contains 200 foundational single- and two-step tasks. Its metrics + use the `simple_` prefix and are reported separately from the public benchmark score. +- **Scoring**: `partial_credit` is the fraction of scored assertions satisfied. `pass_rate` is the mean official + `task_completed_correctly` signal, which is 1 only when every scored assertion passes. + +## Evaluation Notes + +- Prepare a Python 3.13+ environment and install the pinned dependencies before evaluation: + `python -m pip install "{DEFAULT_AUTOMATION_BENCH_VERIFIERS}" "{DEFAULT_AUTOMATION_BENCH_PACKAGE}"`. +- By default, EvalScope evaluates all six public domains (600 tasks). Use `subset_list` to select domains or `limit` + for a smaller run. +- Use an API-backed EvalScope model with `openai_api`, `openai_responses_api`, or `anthropic_api` evaluation type. +- No Docker runtime, external dataset download, or real SaaS credentials are required. Model API credentials are still + required. +- The released tasks are public. Zapier's official leaderboard uses a separate held-out private set, so local public + scores are directional rather than leaderboard-identical. +""" + + +@register_benchmark( + BenchmarkMeta( + name='automation_bench', + pretty_name='AutomationBench', + dataset_id='https://github.com/zapier/AutomationBench', + tags=[Tags.AGENT, Tags.FUNCTION_CALLING, Tags.MULTI_TURN], + description=_DESCRIPTION, + paper_url='https://arxiv.org/abs/2604.18934', + metric_list=['pass_rate', 'partial_credit', 'error_rate'], + aggregation='mean', + eval_split='test', + subset_list=PUBLIC_DOMAINS, + default_subset='sales', + extra_params=_EXTRA_PARAMS, + ) +) +class AutomationBenchAdapter(AgentAdapter): + """EvalScope wrapper around the official AutomationBench Python environment.""" + + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._task_records: Dict[str, Dict[str, Any]] = {} + ensure_automation_bench_runtime() + + def load(self) -> Tuple[DatasetDict, None]: + domains = list(self.subset_list) + official_records = load_automation_bench_records(domains) + records_by_domain: Dict[str, List[Dict[str, Any]]] = {} + self._task_records.clear() + + for domain in domains: + records_by_domain[domain] = [] + for index, record in enumerate(official_records[domain]): + record_key = f'{domain}:{index}' + self._task_records[record_key] = dict(record) + records_by_domain[domain].append({ + 'record_key': record_key, + 'domain': domain, + 'example_id': record.get('example_id'), + 'task': record.get('task', record_key), + 'prompt': record.get('prompt') or [], + 'answer': record.get('answer') or '', + }) + + return build_dataset_dict_from_record_map( + record_map=records_by_domain, + sample_fields=self.record_to_sample, + location=self.dataset_id, + limit=self.limit, + repeats=self.repeats, + shuffle=self.shuffle, + seed=self.seed, + ), None + + def record_to_sample(self, record: Dict[str, Any]) -> Sample: + return Sample( + input=convert_automation_bench_messages(record['prompt']), + target=record.get('answer') or '', + subset_key=record['domain'], + metadata={ + 'record_key': record['record_key'], + 'domain': record['domain'], + 'task': record['task'], + 'official_example_id': record.get('example_id'), + }, + ) + + def _on_inference(self, model: Model, sample: Sample) -> InferenceResult: + record_key = str(sample.metadata.get('record_key') or '') + task_record = self._task_records.get(record_key) + if task_record is None: + raise ValueError(f'AutomationBench task record is unavailable: {record_key!r}.') + + result = run_automation_bench_task( + task_record=task_record, + model_name=model.name, + model_api=model.api, + api=self._resolve_api_protocol(), + toolset=self.extra_params['toolset'], + max_turns=self._resolve_max_turns(), + sampling_args=self._build_sampling_args(model), + extra_headers=dict(get_secret_value(model.config.extra_headers) or {}), + ) + metrics = result.get('metrics') or {} + content = json.dumps( + { + 'task': result.get('task'), + 'partial_credit': metrics.get('partial_credit', 0.0), + 'task_completed_correctly': metrics.get('task_completed_correctly', 0.0), + 'error': result.get('error'), + }, + ensure_ascii=False, + ) + output = ModelOutput.from_content( + model=model.name, + content=content, + error=str(result['error']) if result.get('error') else None, + ) + usage = result.get('usage') or {} + output.usage = ModelUsage( + input_tokens=int(usage.get('input_tokens', 0) or 0), + output_tokens=int(usage.get('output_tokens', 0) or 0), + total_tokens=int(usage.get('input_tokens', 0) or 0) + int(usage.get('output_tokens', 0) or 0), + ) + messages = convert_automation_bench_messages(result.pop('messages', [])) + output.metadata = result + return InferenceResult(output=output, messages=messages or None) + + def match_score( + self, + original_prediction: str, + filtered_prediction: str, + reference: str, + task_state: Any, + ) -> Score: + result = getattr(task_state.output, 'metadata', None) or {} + metrics = result.get('metrics') or {} + error = result.get('error') + prefix = 'simple_' if task_state.metadata.get('domain') == 'simple' else '' + pass_rate = f'{prefix}pass_rate' + return Score( + extracted_prediction=filtered_prediction, + prediction=original_prediction, + value={ + pass_rate: float(metrics.get('task_completed_correctly', 0.0) or 0.0), + f'{prefix}partial_credit': float(metrics.get('partial_credit', 0.0) or 0.0), + f'{prefix}error_rate': float(bool(error)), + }, + main_score_name=pass_rate, + metadata={ + 'task': result.get('task'), + 'error': error, + 'usage': result.get('usage') or {}, + 'debug': result.get('debug') or {}, + 'assertion_results': result.get('assertion_results') or [], + 'end_state': result.get('end_state'), + 'perf': result.get('perf') or {}, + }, + ) + + def _resolve_api_protocol(self) -> str: + eval_type = self._task_config.eval_type if self._task_config is not None else None + if eval_type == EvalType.ANTHROPIC_API: + return 'anthropic' + if eval_type == EvalType.OPENAI_RESPONSES_API: + return 'responses' + return 'chat_completions' + + def _resolve_max_turns(self) -> int: + agent_config = self._task_config.agent_config if self._task_config is not None else None + if isinstance(agent_config, NativeAgentConfig) and 'max_steps' in agent_config.model_fields_set: + return agent_config.max_steps + return 50 + + @staticmethod + def _build_sampling_args(model: Model) -> Dict[str, Any]: + fields = ('temperature', 'top_p', 'max_tokens', 'reasoning_effort', 'extra_body') + return { + field: getattr(model.config, field) + for field in fields + if getattr(model.config, field, None) is not None + } diff --git a/evalscope/benchmarks/automation_bench/utils.py b/evalscope/benchmarks/automation_bench/utils.py new file mode 100644 index 000000000..faacef5e1 --- /dev/null +++ b/evalscope/benchmarks/automation_bench/utils.py @@ -0,0 +1,241 @@ +import importlib +import json +import sys +from typing import Any, Callable, Dict, Iterable, List + +from evalscope.api.messages import ChatMessage, dict_to_chat_message +from evalscope.utils.function_utils import AsyncioLoopRunner + +DEFAULT_AUTOMATION_BENCH_COMMIT = 'a321764ace3cfbe42289e6a13abef2f0f4f56fad' +DEFAULT_AUTOMATION_BENCH_PACKAGE = ( + f'automation-bench @ git+https://github.com/zapier/AutomationBench.git@{DEFAULT_AUTOMATION_BENCH_COMMIT}' +) +DEFAULT_AUTOMATION_BENCH_VERIFIERS = 'verifiers==0.1.12.dev2' +PUBLIC_DOMAINS = ['sales', 'marketing', 'operations', 'support', 'finance', 'hr'] + + +def ensure_automation_bench_runtime() -> None: + """Ensure the pinned official AutomationBench package is available.""" + if sys.version_info < (3, 13): + raise RuntimeError( + 'AutomationBench requires Python 3.13+. Create and activate a Python 3.13 conda environment, ' + 'then run EvalScope from that environment.' + ) + + try: + importlib.import_module('automationbench.runner') + except ImportError as error: + raise ImportError( + 'AutomationBench is not installed in the active Python environment. Install the pinned official ' + f'package with `python -m pip install "{DEFAULT_AUTOMATION_BENCH_VERIFIERS}" ' + f'"{DEFAULT_AUTOMATION_BENCH_PACKAGE}"`, then rerun EvalScope.' + ) from error + + +def load_automation_bench_records(domains: Iterable[str]) -> Dict[str, List[Dict[str, Any]]]: + """Load task records from the official domain datasets.""" + from automationbench.domains import get_domain_dataset + + return {domain: [dict(row) for row in get_domain_dataset(domain)] for domain in domains} + + +def run_automation_bench_task( + task_record: Dict[str, Any], + model_name: str, + model_api: Any, + api: str, + toolset: str, + max_turns: int, + sampling_args: Dict[str, Any], + extra_headers: Dict[str, str], +) -> Dict[str, Any]: + """Evaluate one task with the official environment on EvalScope's model client.""" + return AsyncioLoopRunner.run( + _run_automation_bench_task( + task_record=task_record, + model_name=model_name, + model_api=model_api, + api=api, + toolset=toolset, + max_turns=max_turns, + sampling_args=sampling_args, + extra_headers=extra_headers, + ) + ) + + +async def _run_automation_bench_task( + task_record: Dict[str, Any], + model_name: str, + model_api: Any, + api: str, + toolset: str, + max_turns: int, + sampling_args: Dict[str, Any], + extra_headers: Dict[str, str], +) -> Dict[str, Any]: + from automationbench.clients import ( + OpenAIResponsesClient, + RetryingOpenAIChatCompletionsClient, + StreamingAnthropicClient, + ) + from automationbench.rubric import create_rubric + from automationbench.runner import AutomationBenchEnv + from datasets import Dataset + from verifiers.decorators import discover_decorated + + try: + native_client = model_api.async_client + except AttributeError as error: + raise TypeError( + 'AutomationBench requires an API model with an async client. Use openai_api, openai_responses_api, ' + 'or anthropic_api evaluation type.' + ) from error + if extra_headers: + native_client = native_client.with_options(default_headers=extra_headers) + + if api == 'anthropic': + client = StreamingAnthropicClient(native_client) + elif api == 'responses': + client = OpenAIResponsesClient(native_client) + else: + client = RetryingOpenAIChatCompletionsClient(native_client) + + env = _create_automation_bench_env( + environment_cls=AutomationBenchEnv, + discover_decorated=discover_decorated, + dataset=Dataset.from_list([task_record]), + rubric=create_rubric(), + max_turns=max_turns, + toolset=toolset, + search_top_k=20, + ) + + results = await env.evaluate( + client=client, + model=model_name, + sampling_args=sampling_args or None, + num_examples=1, + rollouts_per_example=1, + max_concurrent=1, + state_columns=['_usage', '_debug', '_assertion_results', '_end_state', '_perf'], + ) + outputs = results.get('outputs') if isinstance(results, dict) else None + if not isinstance(outputs, list) or len(outputs) != 1: + raise RuntimeError( + f'AutomationBench returned {len(outputs) if isinstance(outputs, list) else 0} outputs for one task.' + ) + return _normalize_result(outputs[0]) + + +def _create_automation_bench_env( + environment_cls: Any, + discover_decorated: Callable[[Any, str], Any], + **kwargs: Any, +) -> Any: + """Construct the official environment without Verifiers' process-level hooks.""" + + class EvalScopeAutomationBenchEnv(environment_cls): + + def __post_init__(self) -> None: + self._stop_conditions = discover_decorated(self, 'stop') + self._cleanup_handlers = discover_decorated(self, 'cleanup') + self._teardown_handlers = discover_decorated(self, 'teardown') + + return EvalScopeAutomationBenchEnv(**kwargs) + + +def _normalize_result(raw_output: Dict[str, Any]) -> Dict[str, Any]: + raw = dict(raw_output) + metrics = dict(raw.get('metrics') or {}) + debug = dict(raw.get('_debug') or {}) + prompt = [_serialize_message(message) for message in (raw.get('prompt') or [])] + completion = [_serialize_message(message) for message in (raw.get('completion') or [])] + return { + 'task': raw.get('task', 'unknown'), + 'metrics': { + 'partial_credit': float(metrics.get('partial_credit', 0.0) or 0.0), + 'task_completed_correctly': float(metrics.get('task_completed_correctly', 0.0) or 0.0), + }, + 'messages': prompt + completion, + 'usage': raw.get('_usage') or {}, + 'debug': debug, + 'assertion_results': raw.get('_assertion_results') or [], + 'end_state': raw.get('_end_state'), + 'perf': raw.get('_perf') or {}, + 'error': raw.get('error'), + } + + +def convert_automation_bench_messages(messages: Iterable[Any]) -> List[ChatMessage]: + """Convert official Verifiers messages into EvalScope chat messages.""" + converted: List[ChatMessage] = [] + for raw_message in messages: + data = _serialize_message(raw_message) + role = data.get('role') + if role not in ('system', 'user', 'assistant', 'tool'): + continue + + data['content'] = _message_content(data.get('content')) + if role == 'assistant': + data['tool_calls'] = _normalize_tool_calls(data.get('tool_calls')) or None + reasoning = data.pop('reasoning_content', None) + if reasoning: + data['reasoning'] = reasoning + + try: + message = dict_to_chat_message(data) + except ValueError: + if role != 'assistant' or not data.get('tool_calls'): + raise + data['metadata'] = {**(data.get('metadata') or {}), 'unparsed_tool_calls': data.pop('tool_calls')} + message = dict_to_chat_message(data) + message.source = 'input' if role in ('system', 'user') else 'generate' + converted.append(message) + return converted + + +def _normalize_tool_calls(tool_calls: Any) -> List[Any]: + if not tool_calls: + return [] + raw_calls = tool_calls if isinstance(tool_calls, list) else [tool_calls] + normalized = [] + for raw_call in raw_calls: + if isinstance(raw_call, str): + try: + call = json.loads(raw_call) + except json.JSONDecodeError: + normalized.append(raw_call) + continue + else: + call = _serialize_message(raw_call) + if isinstance(call, dict) and not isinstance(call.get('function'), dict): + call['function'] = { + 'name': call.pop('name', ''), + 'arguments': call.pop('arguments', '{}'), + } + normalized.append(call) + return normalized + + +def _serialize_message(message: Any) -> Dict[str, Any]: + if isinstance(message, dict): + return dict(message) + return message.model_dump(mode='json') + + +def _message_content(content: Any) -> str: + if content is None: + return '' + if isinstance(content, str): + return content + if isinstance(content, list): + parts = [] + for item in content: + if isinstance(item, str): + parts.append(item) + continue + data = _serialize_message(item) + parts.append(str(data.get('text') or data.get('content') or data)) + return '\n'.join(parts) + return str(content) diff --git a/tests/benchmark/test_agent.py b/tests/benchmark/test_agent.py index a849abbf2..c09008864 100644 --- a/tests/benchmark/test_agent.py +++ b/tests/benchmark/test_agent.py @@ -17,6 +17,11 @@ from evalscope.api.agent.mcp import MCPServerConfigStdio from evalscope.api.metric import SampleScore, Score from evalscope.api.registry import get_benchmark +from evalscope.benchmarks.automation_bench.utils import ( + _create_automation_bench_env, + _normalize_result, + ensure_automation_bench_runtime, +) from evalscope.benchmarks.claw_eval.claw_eval_adapter import ClawEvalAdapter from evalscope.benchmarks.claw_eval.utils import ( DEFAULT_CLAW_EVAL_SANDBOX_IMAGE, @@ -492,6 +497,262 @@ def fake_image_preparer(repo_root_arg): self.assertNotIn('selected_task_ids', metadata) self.assertNotIn('selected_records', metadata) + def test_automation_bench(self): + """Test AutomationBench's official task wrapper with a mocked in-process runner.""" + run_calls = [] + + def fake_record_loader(domains): + self.assertEqual(domains, ['sales']) + return { + 'sales': [{ + 'example_id': 501, + 'task': 'sales.update_contact', + 'prompt': [ + { + 'role': 'system', + 'content': 'Use the available business APIs.' + }, + { + 'role': 'user', + 'content': 'Update the contact phone number.' + }, + ], + 'answer': '', + 'info': '{}', + }] + } + + def fake_task_runner(**kwargs): + run_calls.append(kwargs) + return { + 'task': 'sales.update_contact', + 'reward': 0.5, + 'metrics': { + 'partial_credit': 0.5, + 'task_completed_correctly': 0.0, + }, + 'messages': [ + { + 'role': 'system', + 'content': 'Use the available business APIs.' + }, + { + 'role': 'user', + 'content': 'Update the contact phone number.' + }, + { + 'role': 'assistant', + 'content': None, + 'tool_calls': [ + '{"id":"call-1","name":"salesforce_update_contact",' + '"arguments":"{\\"contact_id\\":\\"123\\"}"}' + ], + }, + { + 'role': 'tool', + 'tool_call_id': 'call-1', + 'content': '{"ok": true}' + }, + { + 'role': 'assistant', + 'content': 'Done.' + }, + ], + 'usage': { + 'input_tokens': 20, + 'output_tokens': 5 + }, + 'debug': {}, + 'assertion_results': [{ + 'type': 'contact_phone_equals', + 'passed': True, + 'excluded': False + }], + 'end_state': { + 'salesforce': {} + }, + 'perf': { + 'tool_calls': 1 + }, + 'error': None, + } + + dataset_args = { + 'subset_list': ['sales'], + 'extra_params': { + 'toolset': 'api', + }, + } + with patch('evalscope.benchmarks.automation_bench.automation_bench_adapter.ensure_automation_bench_runtime'), \ + patch('evalscope.benchmarks.automation_bench.automation_bench_adapter.load_automation_bench_records', + side_effect=fake_record_loader), \ + patch('evalscope.benchmarks.automation_bench.automation_bench_adapter.run_automation_bench_task', + side_effect=fake_task_runner): + self._run_dataset_test( + 'automation_bench', + dataset_args, + use_mock=True, + limit=1, + repeats=2, + eval_batch_size=1, + no_timestamp=True, + work_dir='outputs/test_agent_automation_bench', + api_url='http://localhost:8000/v1', + api_key='local-key', + agent_config=NativeAgentConfig(max_steps=7), + generation_config={ + 'temperature': 0.2, + 'reasoning_effort': 'low', + 'extra_body': { + 'enable_thinking': True + }, + 'extra_headers': { + 'X-Test': 'value' + }, + }, + ) + + self.assertEqual(len(run_calls), 2) + self.assertTrue(all(call['model_name'] == 'qwen3-max' for call in run_calls)) + self.assertTrue(all(call['api'] == 'chat_completions' for call in run_calls)) + self.assertTrue(all(call['toolset'] == 'api' for call in run_calls)) + self.assertTrue(all(call['max_turns'] == 7 for call in run_calls)) + self.assertTrue(all(call['extra_headers'] == {'X-Test': 'value'} for call in run_calls)) + self.assertTrue( + all( + call['sampling_args'] == { + 'temperature': 0.2, + 'reasoning_effort': 'low', + 'extra_body': { + 'enable_thinking': True + }, + } for call in run_calls + ) + ) + + review_files = list(Path('outputs/test_agent_automation_bench').glob('reviews/*/automation_bench_sales.jsonl')) + self.assertEqual(len(review_files), 1) + reviews = [json.loads(line) for line in review_files[0].read_text().splitlines() if line] + self.assertEqual(len(reviews), 2) + review = reviews[0] + self.assertEqual(review['sample_score']['score']['main_score_name'], 'pass_rate') + self.assertEqual(review['sample_score']['score']['value']['pass_rate'], 0.0) + self.assertEqual(review['sample_score']['score']['value']['partial_credit'], 0.5) + self.assertEqual(review['sample_score']['sample_metadata']['domain'], 'sales') + self.assertNotIn('automation_bench_result', review['sample_score']['sample_metadata']) + prediction_file = next(Path('outputs/test_agent_automation_bench').glob('predictions/*/automation_bench_sales.jsonl')) + prediction = json.loads(prediction_file.read_text().splitlines()[-1]) + self.assertNotIn('automation_bench_result', prediction['metadata']) + self.assertNotIn('messages', prediction['model_output']['metadata']) + assistant = next(message for message in prediction['messages'] if message['role'] == 'assistant') + self.assertEqual(assistant['tool_calls'][0]['function']['name'], 'salesforce_update_contact') + + def test_automation_bench_requires_python_313(self): + """Test AutomationBench rejects unsupported Python interpreters before import.""" + with patch('evalscope.benchmarks.automation_bench.utils.sys.version_info', (3, 12, 0)): + with self.assertRaisesRegex(RuntimeError, 'Python 3.13'): + ensure_automation_bench_runtime() + + def test_automation_bench_missing_package_has_install_hint(self): + """Test a missing official package reports the pinned installation command.""" + with patch('evalscope.benchmarks.automation_bench.utils.sys.version_info', (3, 13, 0)), \ + patch('evalscope.benchmarks.automation_bench.utils.importlib.import_module', + side_effect=ImportError('missing')): + with self.assertRaisesRegex(ImportError, 'python -m pip install'): + ensure_automation_bench_runtime() + + def test_automation_bench_simple_aggregate_is_separate(self): + """Test the simple baseline does not contribute to the public pass-rate metric.""" + with patch('evalscope.benchmarks.automation_bench.automation_bench_adapter.ensure_automation_bench_runtime'): + adapter = get_benchmark( + 'automation_bench', + TaskConfig( + datasets=['automation_bench'], + dataset_args={'automation_bench': { + 'subset_list': ['simple'] + }}, + ), + ) + + def simple_score(completed, partial_credit, error=None): + task_state = Mock( + metadata={'domain': 'simple'}, + output=Mock(metadata={ + 'metrics': { + 'task_completed_correctly': completed, + 'partial_credit': partial_credit, + }, + 'error': error, + }), + ) + return adapter.match_score('', '', '', task_state) + + scores = [ + SampleScore( + score=simple_score(1.0, 1.0), + sample_id=0, + sample_metadata={'domain': 'simple'}, + ), + SampleScore( + score=simple_score(0.0, 0.5, 'failed'), + sample_id=1, + sample_metadata={'domain': 'simple'}, + ), + ] + + aggregated = {score.metric_name: score.score for score in adapter.aggregate_scores(scores)} + self.assertEqual(aggregated['simple_pass_rate'], 0.5) + self.assertEqual(aggregated['simple_partial_credit'], 0.75) + self.assertEqual(aggregated['simple_error_rate'], 0.5) + self.assertNotIn('pass_rate', aggregated) + + def test_automation_bench_environment_init_has_no_process_hooks(self): + """Test the EvalScope environment discovers handlers without registering process hooks.""" + + class FakeOfficialEnvironment: + + def __init__(self, **kwargs): + self.kwargs = kwargs + self.__post_init__() + + handlers = { + 'stop': ['stop'], + 'cleanup': ['cleanup'], + 'teardown': ['teardown'], + } + discover_decorated = Mock(side_effect=lambda _, kind: handlers[kind]) + with patch('signal.signal') as signal_register, patch('atexit.register') as atexit_register: + env = _create_automation_bench_env( + environment_cls=FakeOfficialEnvironment, + discover_decorated=discover_decorated, + dataset='dataset', + ) + + self.assertEqual(env._stop_conditions, ['stop']) + self.assertEqual(env._cleanup_handlers, ['cleanup']) + self.assertEqual(env._teardown_handlers, ['teardown']) + self.assertEqual(env.kwargs, {'dataset': 'dataset'}) + signal_register.assert_not_called() + atexit_register.assert_not_called() + + def test_automation_bench_normalization_uses_official_result_fields(self): + """Test reward and debug diagnostics do not override official metrics or error state.""" + result = _normalize_result({ + 'task': 'sales.update_contact', + 'reward': 1.0, + 'metrics': { + 'partial_credit': 0.25, + }, + '_debug': { + 'errors': ['diagnostic only'] + }, + }) + + self.assertEqual(result['metrics']['partial_credit'], 0.25) + self.assertEqual(result['metrics']['task_completed_correctly'], 0.0) + self.assertIsNone(result['error']) + self.assertEqual(result['debug']['errors'], ['diagnostic only']) + def test_claw_eval_init_checks_runtime(self): """Test Claw-Eval fails early when the official package is unavailable.""" with patch.object(ClawEvalAdapter, '_check_runtime') as check: