From fe7a203538eed228a5bfd5c8600277009bc90814 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 2 Apr 2024 12:57:55 -0700 Subject: [PATCH 1/5] Add experimental tag to a benchmark, --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 81c6fb6..1ecf1cf 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps: . *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round. -. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. +. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. The suite will have at most one benchmark with _experimental_ tag in any round. . *Sync Domains* across working groups (e.g. Inference, Training, and HPC) . *Identify PIC* (person in charge) to drive this domain addition across all working groups . For each domain addition, do the two following steps, possibly in parallel: From f32e1f322ae76f0bf3bab4a94c9bdc195aeaf797 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 2 Apr 2024 13:30:46 -0700 Subject: [PATCH 2/5] Update submission_rules.adoc --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 1ecf1cf..f5b6d66 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps: . *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round. -. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. The suite will have at most one benchmark with _experimental_ tag in any round. +. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. . *Sync Domains* across working groups (e.g. Inference, Training, and HPC) . *Identify PIC* (person in charge) to drive this domain addition across all working groups . For each domain addition, do the two following steps, possibly in parallel: From c81772b699f1eaa228646ada56cddebefee4d7f2 Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Tue, 2 Apr 2024 13:55:25 -0700 Subject: [PATCH 3/5] Experimental tag definition --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index f5b6d66..9b13ef5 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps: . *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round. -. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. +. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. The suite will have at most two benchmarks with _experimental_ tag in any round. . *Sync Domains* across working groups (e.g. Inference, Training, and HPC) . *Identify PIC* (person in charge) to drive this domain addition across all working groups . For each domain addition, do the two following steps, possibly in parallel: From 39cda731ce2adf78331f3a3169d2a2163ed9fa4f Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Thu, 11 Apr 2024 11:11:37 -0700 Subject: [PATCH 4/5] Change name to Agile Also added minimum lifetime of agile to be 1 year based on discussion in WG. --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index 9b13ef5..eba46a3 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps: . *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round. -. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _experimental_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. The suite will have at most two benchmarks with _experimental_ tag in any round. +. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _agile_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 4 rounds. Minimum life of an _agile_ benchmark is 2 rounds. The suite will have at most two benchmarks with _agile_ tag in any round. . *Sync Domains* across working groups (e.g. Inference, Training, and HPC) . *Identify PIC* (person in charge) to drive this domain addition across all working groups . For each domain addition, do the two following steps, possibly in parallel: From 0f35893bb4f3d6966392be37e820089aa7a3971f Mon Sep 17 00:00:00 2001 From: Ritika Borkar <54604832+nv-rborkar@users.noreply.github.com> Date: Thu, 30 May 2024 09:49:24 -0700 Subject: [PATCH 5/5] Update submission_rules.adoc --- submission_rules.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/submission_rules.adoc b/submission_rules.adoc index eba46a3..24c43df 100644 --- a/submission_rules.adoc +++ b/submission_rules.adoc @@ -217,7 +217,7 @@ The exact review period schedule needs to be agreed upon 4 weeks in advance of s Each Working Group decides what benchmarks they want in each round. This is a pipelined process, with the following steps: . *Carrying Capacity Decision* - Each working group decides how many benchmarks they can handle for this round. -. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _agile_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 4 rounds. Minimum life of an _agile_ benchmark is 2 rounds. The suite will have at most two benchmarks with _agile_ tag in any round. +. *Domain Identification* - Working groups review proposals for domain adds/removals from members. The working group will attempt to come to majority consensus in 1 or 2 meetings. If consensus cannot be had, this will go to a vote according to the MLCommons voting rules. Working groups may add up to 2 benchmarks max per round, but will strive for 1 or 0 as the typical case. Optionally, working group can tag a benchmark as _agile_ implying the benchmark is likely to be updated earlier than the otherwise expected lifetime of 2 years. Minimum life of an _agile_ benchmark is 1 year. The suite will have at most two benchmarks with _agile_ tag in any round. . *Sync Domains* across working groups (e.g. Inference, Training, and HPC) . *Identify PIC* (person in charge) to drive this domain addition across all working groups . For each domain addition, do the two following steps, possibly in parallel: