fix: per-subset float limit and Anthropic tool call id sanitization

[Yunnglin]

Summary

Fixes two P1 bugs:

1. Bug: 浮点 --limit 在 reformat_subset 数据集(如 MMLU-Pro)上被错误地按第一个 subset 的大小套用到所有 subset #1525 — On datasets with multiple subsets built via DatasetDict.from_dataset (e.g. MMLU-Pro), a float --limit was resolved into an integer using the first subset's size, then that integer was reused for all subsets (the loop mutated the limit variable in place). With --limit 0.1 on MMLU-Pro, every subset got exactly 41 samples (574 total) instead of ~10% of each subset (~1203 total).
2. anthropic_api不支持工具调用类型的数据集评测（General-FunctionCalling） #1523 — general_fc (and any dataset with pre-recorded tool-call history) failed with 400 errors when evaluated via --eval-type anthropic_api. The conversion layer passed dataset-provided tool call ids (e.g. functions.search:0) straight through to the Anthropic API, violating its ^[a-zA-Z0-9_-]+$ id constraint; the same id also repeats across turns in the data, breaking tool_use/tool_result pairing.

Closes #1525
Closes #1523

Changes

• evalscope/api/dataset/dataset.py: resolve float limits into a per-subset local subset_limit instead of mutating limit on the first iteration.
• evalscope/models/utils/anthropic.py: new _sanitize_tool_call_ids() pre-pass in anthropic_chat_messages() — replaces illegal characters with _, deduplicates ids across turns with a numeric suffix, and applies the same per-turn mapping to subsequent tool messages so tool_use/tool_result pairing is preserved. Already-legal unique ids (e.g. model-generated toolu_xxx) are kept unchanged, and messages are copied via model_copy so original samples are never mutated.
• New regression tests: tests/api/test_dataset_dict_limit.py (3 cases), tests/models/test_anthropic_tool_id_sanitize.py (4 cases, no network).

Testing

• Unit: 7 new tests pass; tests/models/test_anthropic_prompt_cache.py regression passes; tests/cli/test_all.py::TestRun::test_ci_lite passes; make lint clean.
• E2E Bug: 浮点 --limit 在 reformat_subset 数据集(如 MMLU-Pro)上被错误地按第一个 subset 的大小套用到所有 subset #1525 (local vLLM, Qwen2.5-0.5B-Instruct): --datasets mmlu_pro --limit 0.01 now yields per-subset Num of 4/13/11/9/11/7/8/12/7/4/8/9/7/3 (113 total ≈ 12032 x 1%), instead of a constant count reused from the first subset.
• E2E anthropic_api不支持工具调用类型的数据集评测（General-FunctionCalling） #1523 (local vLLM /v1/messages with --enable-auto-tool-choice --tool-call-parser hermes): 100 general_fc samples via --eval-type anthropic_api completed with zero 400 errors; 68 of them carried tool-message history with illegal ids (functions.img_gen:0, functions.search:1, ...), so the sanitization path was genuinely exercised. Offline check: all 2000 general_fc conversations convert to legal, per-conversation-unique, correctly paired ids (2136 tool_use/tool_result pairs).

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[qoderai]

👋 Review Summary

This PR addresses two concrete P1 issues: per-subset float limit handling in DatasetDict.from_dataset and Anthropic tool-call id sanitization for datasets with pre-recorded tool histories. The changes are focused, well-scoped, and supported by targeted regression tests.

🛡️ Key Risks & Issues

• Dataset limits: The new subset_limit logic in DatasetDict.from_dataset correctly applies float and int limits per subset without mutating shared state, and the tests cover the previously broken scenario where the first subset’s size was reused. The semantics for float limits (flooring via int(len * limit)) and for extreme values (negative or very large limits) remain implicit but are inherited behavior, not new risk.
• Anthropic tool-call ids: _sanitize_tool_call_ids enforces Anthropic’s id pattern, keeps already-legal ids unchanged, and preserves tool_use/tool_result pairing across turns by using a per-turn id map and a conversation-wide used_ids set. The main subtle edge case is when multiple ToolCall entries in a single assistant turn share the same original id: the map stores only the last sanitized id for that key, so subsequent tool-result messages referencing the original id will all point at the final tool_use id, leaving earlier tool_use blocks unreferenced. This is unlikely with well-formed datasets (general_fc currently reuses ids across turns, not within a turn), but it is worth being aware of if you ever see duplicate tool-call ids inside one assistant message.

🧪 Verification Advice

• Run a multi-subset benchmark (such as one built from DatasetDict) with both float and int --limit values and inspect per-subset sample counts to confirm each subset gets its own fraction or count rather than a shared value.
• With Anthropic enabled and a tool-using dataset like general_fc, run a small eval via --eval-type anthropic_api and confirm there are no 400 errors related to tool ids and that tool_use/tool_result blocks in the Anthropic messages carry legal, unique ids that pair as expected.
• Execute the new test modules together with existing Anthropic prompt cache and bridge tests to ensure prompt caching and tool id behavior remain consistent end-to-end.

💡 Thoughts & Suggestions

• If you want the "unique across the conversation" guarantee to hold even for malformed histories, consider explicitly handling or logging duplicate tool-call ids within a single assistant turn so any ambiguity in pairing is easier to diagnose. Otherwise, the current implementation is a solid, pragmatic fix for the real-world failures described in the PR and should be safe to ship.

---

🤖 Generated by Qoder • View workflow run