Skip to content

Experimental Tag for benchmarks #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nv-rborkar wants to merge 5 commits into
base: master
Choose a base branch
from
Open

Experimental Tag for benchmarks #171

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fe7a203
Add experimental tag to a benchmark,
nv-rborkar Apr 2, 2024
f32e1f3
Update submission_rules.adoc
nv-rborkar Apr 2, 2024
c81772b
Experimental tag definition
nv-rborkar Apr 2, 2024
39cda73
Change name to Agile
nv-rborkar Apr 11, 2024
0f35893
Update submission_rules.adoc
nv-rborkar May 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion submission_rules.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s
Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps:

. *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round.
. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case.
. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _agile_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. Minimum life of an _agile_ benchmark is 1 year. The suite will have at most two benchmarks with _agile_ tag in any round.
. *Sync Domains* across working groups (e.g. Inference, Training, and HPC)
. *Identify PIC* (person in charge) to drive this domain addition across all working groups
. For each domain addition, do the two following steps, possibly in parallel:
Expand Down