Select devices per-container in the ActorTemplate

[eliranw]

Problem

Follow-up to #502, which lands GPU passthrough at the worker level. A pool requests N GPUs and every actor container gets all of them. The ActorTemplate containers have no say in the device layout, so device assignment is coupled to the worker.

Per Benjamin Elder (@BenTheElder) (review), device assignment should move into the ActorTemplate so it is decoupled from the worker and can control which containers in an actor receive the devices.

Use cases

Actors are pod-shaped (multiple containers, one gVisor sandbox, shared netns), so per-container device control unlocks:

• Accelerator container plus CPU-only sidecar. The model or trainer container claims the accelerator (nvidia.com/gpu, amd.com/gpu, an FPGA); the logging, proxy, or metrics sidecar next to it claims none. Today every sidecar is handed the device nodes.
• Security isolation. An untrusted or user-supplied sidecar is denied device access as defense in depth. Today it silently receives whatever the pod holds (/dev/nvidia*, an SR-IOV VF), widening the attack surface that feat: CDI-based NVIDIA GPU passthrough into gVisor actor containers #502 otherwise works to keep narrow.
• Devices split across containers. On a multi-device node, worker containers each claim a distinct GPU for data-parallel or shard-per-container work over the shared netns.

Proposed API

Declare devices per-container using the standard Kubernetes resources shape, the same corev1.ResourceRequirements type the pool already uses. The pool reserves capacity; each container claims from it.

kind: WorkerPool
spec:
  sandboxClass: gvisor
  template:
    resources:
      limits:
        nvidia.com/gpu: "2"        # pool reserves 2 GPUs on the node (unchanged; this is what schedules onto a GPU node)
    containers:
      - name: trainer
        image: pytorch@sha256:...
        resources:
          limits:
            nvidia.com/gpu: "2"    # this container claims both
      - name: sidecar-logger
        image: fluentbit@sha256:...
        # no gpu request, none injected (today it would wrongly get both)

One new field on Container:

// Resources for this container, including device-plugin resources such as
// nvidia.com/gpu. Only nvidia.com/gpu is supported today.
// +optional
Resources *corev1.ResourceRequirements `json:"resources,omitempty"`

This reuses an existing type instead of inventing a devices selector, and:

• is idiomatic: it matches vanilla Kubernetes per-container GPU requests;
• is device-generic for free: extended resources already cover RDMA, FPGA, and others.

Scope

• Add resources to ActorTemplate.Container.
• CEL validation:
  • sum of container nvidia.com/gpu must not exceed the pool reservation;
  • reject device classes the runtime cannot yet inject. Only nvidia.com/gpu is injectable today; the field type stays vendor-neutral and the gate lifts per class;
  • extend the existing sandboxClass == 'gvisor' guard to the container level.
• Plumb the per-container request through WorkerAssignment to ateom.
• In ateom, partition the pod's assigned GPUs across the containers that request them, the way the device plugin does in real Kubernetes, instead of the unconditional per-container maybeInjectGPU. The partition must be deterministic, ordered by container.
• Update the API guide.

Risks

• Device identity is not stable across resume. CDI names are pod-local, so on restore the actor may land on different physical devices with new UUIDs and PCI IDs. Injection already re-runs on the restore path, so device paths are re-resolved. Remaining care:
  • an app that cached a device UUID or PCI ID before snapshot sees a different one after;
  • the container-to-device partition must be deterministic so a container keeps its ordinal across resume.
• Pool reservation stays the scheduling knob. Keep the pool's resources.limits.nvidia.com/gpu as the reservation and validate that the containers' sum does not exceed it, rather than deriving the pool total from containers. Smaller change, single scheduling knob.

Out of scope

• DRA. Dynamic Resource Allocation (resource.k8s.io, GA in 1.34) is the home for physical index pinning, topology co-allocation, and device sharing such as MIG and time-slicing. If those are needed, revisit through a DRA ResourceClaim rather than extending counts.
• Non-GPU device classes. The surface allows them; the implementation lands with the first such device.
• Snapshot and restore of live device state, such as a live CUDA context. Tracked separately by the cuda-checkpoint follow-up.

References

• GPU support for gVisor actors #627: GPU support for gVisor actors, the parent this follows up.
• Actor lifecycle hooks #450: actor lifecycle hooks. Adjacent container-spec surface; device setup and teardown may want to interact with it.

[BenTheElder]

Actors are pod-shaped (multiple containers, one gVisor sandbox, shared netns), so per-container device control unlocks:

It also unlocks a possible future where we are not 1:1 with actor : pod.

Declare devices per-container using the standard Kubernetes resources shape, the same corev1.ResourceRequirements type the pool already uses. The pool reserves capacity; each container claims from it.

I'm wondering how much requests vs limits makes sense and if we should allow requests at this stage. Like what would it mean if you requested 2GI for uVM and then on restore got only 1?

cc Julian Gutierrez Oschmann (@juli4n)