feat(perf): long-context prefix injection via prefix_file/prefix_role

[Yunnglin]

feat(perf): long-context prefix injection via prefix_file / prefix_role

Closes #1524

What

Benchmarking 128K/256K-scale long contexts with real text was previously impossible: random gives fixed lengths but high-entropy meaningless tokens (prefix-cache hit rate / MTP acceptance rate unrealistic), while real instruction datasets are mostly 4K-8K tokens, so a 128K target_input_len filters everything out.

This PR adds prefix injection to the perf dataset pipeline: point prefix_file at a long text corpus, and the framework slices it to exactly target_input_len − prompt_len tokens, prepending it to each short prompt as a system (default) or user message — every request hits the target length while keeping real-language entropy.

evalscope perf \
  --dataset openqa \
  --tokenizer-path Qwen/Qwen2.5-0.5B-Instruct \
  --dataset-args '{"target_input_len": 8192, "prefix_file": "long_text.txt", "prefix_role": "system"}' \
  ...

Key changes

Feature (evalscope/perf/plugin/datasets/)

• dataset_args.py: TextLengthArgs gains prefix_file / prefix_role with cross-field validation (prefix_file requires target_input_len; unknown keys still fail fast).
• base.py: load + encode the corpus once at construction (auto-tile with a warning when shorter than the budget), compute the per-prompt token budget, inject into messages; plain-text concatenation fallback for /v1/completions (no chat template) with a warning.
• utils.py: extract converge_to_token_len() as the shared decode→re-encode convergence skeleton; gen_prompt_decode_to_target_len (random dataset) and the new fit_prefix_to_budget are now thin wrappers that only differ in their fill strategy (random vocab sampling vs. sequential corpus continuation). Removes the previously duplicated retry loop.
• Enabled for openqa / longalpaca / line_by_line / single-turn ShareGPT (net −16 lines in the four plugins).

Tests (tests/perf/test_prefix_injection.py)

• Prefix budget fitting, corpus tiling, schema validation.
• Tokenizer round-trip drift convergence via Merge/Split fake tokenizers — also adds the first direct coverage for the previously mocked-out random-dataset convergence loop.

Docs (zh + en, second commit)

• stress_test/examples.md: new "Long-context Benchmarking (Prefix Injection)" section; restructured 14 flat H2 sections into 7 workflow-ordered groups; fixed a bash line-continuation-comment bug.
• stress_test/parameters.md: restructured by workflow (input construction / load patterns / replay); new "Long-context Prefix Injection" reference.
• stress_test/multi_turn.md: folded orphan sections (--duration soft exit, chat-template FAQ) into owning sections; moved swe_smith-only params under its dataset entry; synced the en table with the zh [int, int] range syntax.
• Fixed 6 pre-existing Sphinx xref warnings (stale anchors into clip_benchmark / mteb / agent/native); make docs now builds zh+en with zero warnings.

Verification

• make lint passes; tests/perf/test_prefix_injection.py + test_request_generation.py: 34 passed.
• Refactor equivalence: real Qwen2.5 tokenizer × 20 fixed-seed cases — output hash identical before/after extracting the shared convergence loop.
• Real vLLM (Qwen2.5-0.5B-Instruct) end-to-end runs:
  • system-role 8192: 20/20 success, server-reported Input Tokens 8205.0 avg = p50 = p99 = max (zero variance; 8192 + 13 chat-template tokens), TTFT ~69 ms
  • user-role 8221, random-dataset hot path 2176 (8/8, no drift warnings), /v1/completions fallback ±1 token (concatenation-boundary token merge, documented in the parameter reference)
  • Prefix-cache effect observable: second identical run TTFT 81.6 → 51.2 ms
• make docs: zh + en Sphinx builds succeed with 0 warnings.

[gemini-code-assist]

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

[qoderai]

👋 Review Summary

Nice work on adding real-text long-context prefix injection and centralizing the token-length convergence logic. The design is thoughtfully wired through the perf datasets and backed by solid unit and integration tests with fake tokenizers.

🛡️ Key Risks & Issues

• Long-context prefix combined with input_len_mode='drop' ends up effectively disabling the prefix: only prompts whose bare-token length already equals target_input_len are kept, which in turn gives a zero prefix budget, so no prefix is ever injected even when prefix_file is set. This makes that configuration misleading for users who expect genuine long-context behavior.
• For multi-turn ShareGPT datasets, the prefix budget is measured only against the last user turn, while earlier conversation history remains untouched. That means the total input length (history + prefix + last user turn) can significantly exceed target_input_len and potentially the backend’s context window. This is more of a semantic/operational risk than a bug, but it would be good to clarify or enforce bounds so perf runs don’t accidentally push requests beyond supported limits.
• The new converge_to_token_len helper does repeated decode/re-encode loops over long token sequences for both random prompts and prefixes. At large target_input_len values and high request counts, this can become a noticeable CPU cost on the driver side, and the drift-warning path in fit_prefix_to_budget may produce a lot of log output if a real tokenizer fails to converge, which could impact perf runs.

🧪 Verification Advice

• Add a targeted test that exercises prefix_file together with input_len_mode='drop' and target_input_len, making the current behavior explicit and providing a guard rail if you decide to change semantics (e.g., by rejecting this combination or warning at construction).
• Extend prefix-injection integration tests beyond line_by_line/share_gpt_zh to include openqa and longalpaca, using small local fixtures and a fake tokenizer. This will help ensure prepare_messages and apply_prefix_to_messages behave as expected across all wired datasets.
• Consider a lightweight perf/CLI smoke test that uses prefix_file through run_perf_benchmark (with a fake tokenizer and tiny dataset) to validate the end-to-end wiring from CLI dataset_args to the plugins.

💡 Thoughts & Suggestions

• It may be worth tightening or documenting the semantics around prefix_file + input_len_mode='drop' so users don’t accidentally configure a long prefix that never applies. A schema-level validator or constructor warning would make intent clearer.
• For multi-turn datasets like ShareGPT, consider whether target_input_len should represent the total request length (including history) or just the final turn. If the former, introducing an optional cap on overall message tokens or documenting the current behavior explicitly would help avoid context overruns.
• If prefix-based long-context perf runs become common at very high token counts, you might want to monitor and, if necessary, optimize converge_to_token_len (e.g., by reducing retries when drift is small or caching converged slices for repeated budgets) to keep the driver overhead under control.

---

🤖 Generated by Qoder • View workflow run

[Copilot]

Pull request overview

Adds long-context prefix injection to the perf dataset pipeline so real-text prompts can be reliably expanded to 128K/256K-scale fixed input lengths via --dataset-args (prefix_file, prefix_role), alongside a small refactor to share the decode→re-encode convergence loop and extensive doc updates for the stress-test guide.

Changes:

• Add prefix_file / prefix_role to perf dataset args with validation and implement prefix injection in the dataset base pipeline (incl. tiling + /v1/completions fallback).
• Refactor the “decode→re-encode until token length converges” loop into a shared helper and reuse it for both random prompt generation and prefix fitting.
• Add end-to-end tests for prefix injection and update/restructure zh+en stress-test documentation accordingly.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/perf/test_prefix_injection.py	New end-to-end tests covering prefix budgeting, tiling, drift convergence, and validation.
evalscope/perf/plugin/datasets/utils.py	Extract shared convergence helper and add prefix-budget fitting utility.
evalscope/perf/plugin/datasets/base.py	Load/tokenize prefix corpus once and inject prefix into prompts/messages based on role/template mode.
evalscope/perf/plugin/datasets/dataset_args.py	Add prefix_file / prefix_role to TextLengthArgs with cross-field validation.
evalscope/perf/plugin/datasets/line_by_line.py	Route string prompts through prepare_messages() to enable prefix injection.
evalscope/perf/plugin/datasets/openqa.py	Route prompts through prepare_messages() to enable prefix injection.
evalscope/perf/plugin/datasets/longalpaca.py	Route prompts through prepare_messages() to enable prefix injection.
evalscope/perf/plugin/datasets/share_gpt.py	Apply prefix injection to multi-message conversations (budgeted against last user turn).
docs/zh/user_guides/stress_test/quick_start.md	Update anchors and wording to match restructured stress-test docs.
docs/zh/user_guides/stress_test/parameters.md	Restructure parameter reference; document long-context prefix injection and related workflow sections.
docs/zh/user_guides/stress_test/multi_turn.md	Restructure multi-turn guide; fold orphan sections and align headings/anchors.
docs/zh/user_guides/stress_test/examples.md	Add prefix-injection examples and reorganize examples into workflow order.
docs/zh/third_party/deepsearchqa.md	Fix/adjust anchor reference to agent MCP guide section.
docs/zh/get_started/parameters.md	Fix/adjust anchor reference for Agent evaluation section.
docs/zh/advanced_guides/custom_dataset/embedding.md	Fix/adjust anchor reference into MTEB backend docs.
docs/zh/advanced_guides/custom_dataset/clip.md	Fix/adjust anchor reference into CLIP backend docs.
docs/en/user_guides/stress_test/quick_start.md	Update anchors and wording to match restructured stress-test docs.
docs/en/user_guides/stress_test/parameters.md	Restructure parameter reference; document long-context prefix injection and related workflow sections.
docs/en/user_guides/stress_test/multi_turn.md	Restructure multi-turn guide; fold orphan sections and align headings/anchors.
docs/en/user_guides/stress_test/examples.md	Add prefix-injection examples and reorganize examples into workflow order.
docs/en/advanced_guides/custom_dataset/embedding.md	Fix/adjust anchor reference into MTEB backend docs.
docs/en/advanced_guides/custom_dataset/clip.md	Fix/adjust anchor reference into CLIP backend docs.