Add AutomationBench benchmark

[Yunnglin]

Summary

• add an AgentAdapter for Zapier's official AutomationBench Python environment
• load all 600 public business-workflow tasks and keep the optional 200-task simple baseline separate
• reuse EvalScope API clients and task/generation configuration while preserving official tools and assertion scoring
• add focused runtime, scoring, message-conversion, and worker-thread compatibility tests
• generate benchmark metadata and English/Chinese documentation

Runtime

AutomationBench requires Python 3.13+. The adapter reports the pinned installation command when the official package is unavailable; it does not create environments or install dependencies at runtime. Evaluation runs in process and does not require Docker or real SaaS credentials.

Validation

• LC_ALL=C LANG=C conda run -n eval make lint
• five focused tests in tests/benchmark/test_agent.py (5 passed)
• LC_ALL=C LANG=C conda run -n eval make docs (English and Chinese Sphinx builds succeeded)
• live Python 3.13 smoke evaluation against the official environment (simple task: pass rate 1.0)

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[Yunnglin]

/review

[qoderai]

👋 Review Summary

Nice work wiring AutomationBench into EvalScope’s agent benchmarking pipeline, including the adapter, runtime guards, and focused tests around Python 3.13 and the simple baseline. The integration aligns well with the existing benchmark architecture and keeps the official environment in-process.

🛡️ Key Risks & Issues

• AutomationBench environment initialization currently suppresses signal.signal and atexit.register globally while worker-thread environments are constructed. Because these functions are process-wide, any other thread registering signal handlers or atexit callbacks in that window will silently have its registrations dropped. This is a subtle concurrency risk that can lead to missing cleanup or signal handling in unrelated code.
• Result normalization currently infers partial_credit and task_completed_correctly from reward and defaults, and treats any non-empty error (including items surfaced from debug.errors) as a full error_rate=1.0. If AutomationBench evolves its metrics or uses debug errors for diagnostics, EvalScope’s metrics could diverge from the intended semantics or over-count errors.

🧪 Verification Advice

• Exercise AutomationBench against real upstream outputs beyond the mocked tests, especially cases where metrics are partially missing, reward is present, or errors are reported only in debug structures, and confirm that pass_rate, partial_credit, and error_rate match Zapier’s reference implementation.
• Run evaluations under workloads that import other libraries or register atexit hooks during execution to ensure that the environment init suppression does not interfere with unrelated code; if any registration is unexpectedly missing, consider narrowing the monkeypatch scope.

💡 Thoughts & Suggestions

• Consider replacing the process-wide monkeypatch of signal/atexit with a more targeted patch of the Verifiers environment or helper that installs hooks, so AutomationBench worker setup does not affect other threads.
• Tighten _normalize_result and match_score to rely primarily on AutomationBench’s explicit metrics and completion flags, treating reward and debug errors as diagnostic metadata. Small unit tests around these paths would give a lot of confidence if the upstream environment changes.
• Overall, the adapter and tests are in good shape; a bit more coverage around the runner glue and normalization logic will make this integration very robust over time.

---

🤖 Generated by Qoder • View workflow run

fixed