Skip to content

Add first-class Gang scheduling - #10

Open
dejanzele wants to merge 1 commit into
mainfrom
gang-scheduling
Open

Add first-class Gang scheduling#10
dejanzele wants to merge 1 commit into
mainfrom
gang-scheduling

Conversation

@dejanzele

@dejanzele dejanzele commented Jul 13, 2026

Copy link
Copy Markdown
Member

Adds armada_flyte.Gang, a first-class way to declare that a set of Flyte tasks must be scheduled all-or-nothing on Armada.

You add members with Gang.add(task, ...) (or Gang.map(task, items, ...) for the common homogeneous case) and await gang.run(). The gang id and the cardinality are derived from the members added, so they can never drift from the fan-out the way a hand-set value could. Gang.run() submits every member together, and Armada places all of them or none.

Under the hood the count is only known at the driver's fan-out, and Flyte hands the connector one task at a time with no group size in its metadata. So Gang carries the gang id, cardinality, and uniformity label to the connector through per-member container environment variables under an ARMADAFLYTE_GANG_ prefix. The connector translates those into Armada's gang annotations and strips them from the submitted pod, so Armada's own ARMADA_GANG_* runtime injection stays authoritative for the application. Gang.add rejects a non-Armada task, since such a member would run as an ordinary pod and deadlock the rest of the gang.

This is a breaking change to ArmadaConfig, so the package version moves from 0.1.0 to 0.2.0.

Removed from ArmadaConfig:

  • gang_id
  • gang_cardinality
  • gang_node_uniformity_label

Express gangs with Gang instead. Passing one of these as a keyword now raises TypeError.

This branch is independent and can merge in any order relative to the others.

Signed-off-by: Dejan Zele Pejchev <pejcev.dejan@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant