Skip to content

docs(speed-bench): add generated benchmark summary#413

Open
dutifulbob wants to merge 11 commits into
antirez:mainfrom
dutifulbob:docs/generated-speed-summary
Open

docs(speed-bench): add generated benchmark summary#413
dutifulbob wants to merge 11 commits into
antirez:mainfrom
dutifulbob:docs/generated-speed-summary

Conversation

@dutifulbob

Copy link
Copy Markdown

Opened on behalf of Onur Solmaz (osolmaz).

Summary

Speed benchmark CSVs are easy to add but hard to compare at a glance.
This change adds a generated summary table to speed-bench/README.md.
It also adds a small Python updater so the table can be rebuilt whenever CSV files change.

What Changed

The speed bench folder now has one command that rebuilds the summary from the checked-in CSV files.
The generated README block has explicit begin and end markers so future updates replace only that section.

  • Added speed-bench/update_summary.py using only the Python standard library.
  • Appended a generated benchmark summary table to speed-bench/README.md.
  • The table includes best generation, generation at 32k context, average generation, best prefill, prefill at 32k context, and average prefill.
  • CSV files with UTF-8 BOMs are handled with utf-8-sig.

Testing

I tested the updater directly and checked that the generated Markdown is clean.
This is a documentation/script-only change; I did not run model inference benchmarks.

  • python3 speed-bench/update_summary.py
  • PYTHONDONTWRITEBYTECODE=1 python3 -m py_compile speed-bench/update_summary.py
  • git diff --check -- speed-bench/README.md speed-bench/update_summary.py

Machine/backend/model quant: not applicable for this docs-only change.
Notable failures: none.

Risks

The risk is low because this does not touch inference code or benchmark CSV data.
The main limitation is that the summary only reflects CSV files in speed-bench/, not one-off numbers in the top-level README.

@osolmaz

osolmaz commented Jun 15, 2026

Copy link
Copy Markdown

Feel free to refactor/reimplement

I thought it would be cool to have a single place to see all the perf results

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants