We should be able to compare results across different hardware/machine types similar to how Clickbench does it. This will need atleast the following changes, but may need more. These changes will probably span across both sketch-bench and sketch-leaderboard repos.
- Input data from sketch-bench should include a unique identifier for the kind of machine it was run on (e.g. GCP
c4d-standard-2), and a unique identifier that denotes the cloud/cluster environment (AWS vs GCP vs Azure)
- sketch-leaderboard tabs should allow users to filter based on one or more machine types or cluster types
- Experiment infrastructure in sketch-bench to run experiments on each of the different cloud providers
- Hand-maintained machine catalog (new reference file, F2-shaped): machine-id → provider, vCPU/RAM, on-demand $/hr
- $-cost priority/constraint in Decision Support
- Shared React Context for machine filter state (multi-select) — first cross-tab shared state in the app; every tab (Table, Accuracy-vs-Cost, Throughput, Decision Support, Visualization) reads from it
- Cost-efficiency (metric ÷ catalog $/hr) added as another selectable cost-axis on the existing Accuracy-vs-Cost tab
We should be able to compare results across different hardware/machine types similar to how Clickbench does it. This will need atleast the following changes, but may need more. These changes will probably span across both sketch-bench and sketch-leaderboard repos.
c4d-standard-2), and a unique identifier that denotes the cloud/cluster environment (AWS vs GCP vs Azure)