Skip to content

bug: forge-dpu-agent crash-loops when a DPU interface has ifindex >= 256 (parsed as u8) #4872

Description

@everlearner

Version

v2.1.0-rc.1

Describe the bug.

After a NICo instance update that added a VF interface, forge-dpu-agent on the DPU went into a crash loop — main_loop error exit, restarting every ~90 seconds with a new PID each time (due to ifindex declared as u8 and linux interface ID went past 255).

The instance update never completed. It stayed in Configuring for over 6 hours, new interfaces stuck in Pending, the superseded one in Deleting. Core sat in Assigned/NetworkConfigUpdate { WaitingForConfigSynced } reporting network observation count does not match DPU count, dpu_count=1 network_observation_count=0 — the DPU never reported network status because the agent kept dying first.

Minimum reproducible example

On a DPU where any interface has ifindex >= 256, issue an instance update that adds a VF:

{"secondaryVpcIds":["<vpc-id>"],
 "interfaces":[{"vpcPrefixId":"<pf-prefix-id>","isPhysical":true,"ipAddress":"<pf-ip>"},
               {"vpcId":"<vpc-id>","ipFamilies":["IPv4"],"isPhysical":false,"virtualFunctionId":0}]}

nicocli instance update --data '<above>' <instance-id>

The agent starts crash-looping within ~90 seconds and the instance never leaves Configuring. The same payload worked fine on a DPU whose interface indexes are all below 256.

Note the indexes are sparse — this DPU has only 101 interfaces, but they run to 268 with large gaps (60, 61, 126, 205, 206, 207, …). Creating and deleting SFs/VFs burns indexes without reusing them, so any long-lived DPU gets there regardless of how many interfaces exist at once.

Relevant log output

Forge-dpu-agent logs on DPU:
Aug 11 23:44:29 7-243-180-19. forge-dpu-agent[1115137]: level=INFO component=nico-dpu-agent msg="HBN container ID is new to us, updating its neighbor learning config" container_id=4031a343757d683635ce27e45eff6c20fc7f50a07d682855eca73764005cf35a previous_container_id=None location="crates/agent/src/hbn.rs:206"
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]: Error: main_loop error exit
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]: Caused by:
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]:     invalid value: integer `256`, expected u8 at line 1 column 15


On DPU:
$ ip -o link show
256: pf0vf51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\    link/ether 26:97:d8:57:c8:31 brd ff:ff:ff:ff:ff:ff\    altname enp3s0f0nc1pf0vf51
257: pf0vf52: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\    link/ether 22:1b:b3:c5:ea:34 brd ff:ff:ff:ff:ff:ff\    altname enp3s0f0nc1pf0vf52
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
258: pf0vf53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\    link/ether a2:06:15:71:54:8a brd ff:ff:ff:ff:ff:ff\    altname enp3s0f0nc1pf0vf53
2: oob_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master mgmt state UP mode DEFAULT group default qlen 1000\    link/ether e0:9d:73:7b:02:e2 brd ff:ff:ff:ff:ff:ff\    altname enamlnxbf17i0
259: pf0vf54: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\    link/ether 5e:f6:eb:97:bf:d3 brd ff:ff:ff:ff:ff:ff\    altname enp3s0f0nc1pf0vf54

Core side, repeating for hours:
state="Assigned/NetworkConfigUpdate { network_config_update_state: WaitingForConfigSynced }"
msg="network observation count does not match DPU count" dpu_count=1 network_observation_count=0
msg="checking dpu" dpu_machine_id=....
msg=missing

Other/Misc.

Workaround: rebooting the DPU clears it — indexes restart low, the agent stops crashing, config synced and the VF came up.

Code of Conduct

  • I agree to follow NVIDIA Infra Controller's Code of Conduct
  • I have searched the open bugs and have found no duplicates for this bug report

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugA defect in existing software (deprecated - use issue type, but it's needed for reporting now)

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions