Benchkit is a shell-first benchmarking framework for the CX Framework. It supports build and run workflows for multiple applications and systems, result collection, profiler data handling, estimation workflows, and result portal integration.
Repository migration details are documented in docs/repository-history.md.
- Run benchmarks across multiple codes and systems with a shared workflow.
- Support both cross-build and native execution environments.
- Keep site-specific configuration separate from benchmark logic.
- Collect result data, profiler outputs, and estimation inputs in a consistent format.
- Integrate with the CX result portal and related estimation workflows.
- Provide a practical base for performance analysis, estimation, and feedback.
- Contributing guide: CONTRIBUTING.md
- Add a new application: docs/guides/add-app.md
- Add a new system: docs/guides/add-site.md
- Add estimation support: docs/guides/add-estimation.md
- CI execution control: docs/ci.md
- Security policy: SECURITY.md
- Profiler support guide: docs/guides/profiler-support.md
- Profiler level reference: docs/guides/profiler-level-reference.md
Core specifications:
- CX framework: top-level concept and terminology.
- CX platform: system-level responsibilities and component boundaries.
- Benchkit specification: Benchkit responsibilities, interfaces, and future extension points.
Estimation and storage specifications:
- Estimation: common rules for accepting, running, storing, and presenting estimation functions.
- Estimate JSON: data-format requirements for stored estimation results.
- Estimation input acquisition: handoff rules for inputs required by estimation packages.
- Estimation package: responsibilities and structure of estimation packages.
- Estimation package metadata: metadata fields used to describe package identity, inputs, and fallback behavior.
- Estimation package shell API: shell-level interface for calling package implementations.
- Re-estimation: rules for re-running estimation from stored benchmark and estimation records.
- Result storage design: storage-design memo for result and estimate artifacts.
Operational specifications:
- Audit log specification: result-server audit log events and handling rules.
The detailed developer-oriented reference has moved to docs:
This includes:
- project structure
- result portal architecture
- CI pipeline structure
- configuration files
- CI execution control
- system-specific execution environments
- runtime requirements
result_serverrequires Python 3.12 or later.
This project is licensed under the BSD 3-Clause License. See LICENSE.