Version
v2.1.0-rc.1
Describe the bug.
After a NICo instance update that added a VF interface, forge-dpu-agent on the DPU went into a crash loop — main_loop error exit, restarting every ~90 seconds with a new PID each time (due to ifindex declared as u8 and linux interface ID went past 255).
The instance update never completed. It stayed in Configuring for over 6 hours, new interfaces stuck in Pending, the superseded one in Deleting. Core sat in Assigned/NetworkConfigUpdate { WaitingForConfigSynced } reporting network observation count does not match DPU count, dpu_count=1 network_observation_count=0 — the DPU never reported network status because the agent kept dying first.
Minimum reproducible example
On a DPU where any interface has ifindex >= 256, issue an instance update that adds a VF:
{"secondaryVpcIds":["<vpc-id>"],
"interfaces":[{"vpcPrefixId":"<pf-prefix-id>","isPhysical":true,"ipAddress":"<pf-ip>"},
{"vpcId":"<vpc-id>","ipFamilies":["IPv4"],"isPhysical":false,"virtualFunctionId":0}]}
nicocli instance update --data '<above>' <instance-id>
The agent starts crash-looping within ~90 seconds and the instance never leaves Configuring. The same payload worked fine on a DPU whose interface indexes are all below 256.
Note the indexes are sparse — this DPU has only 101 interfaces, but they run to 268 with large gaps (60, 61, 126, 205, 206, 207, …). Creating and deleting SFs/VFs burns indexes without reusing them, so any long-lived DPU gets there regardless of how many interfaces exist at once.
Relevant log output
Forge-dpu-agent logs on DPU:
Aug 11 23:44:29 7-243-180-19. forge-dpu-agent[1115137]: level=INFO component=nico-dpu-agent msg="HBN container ID is new to us, updating its neighbor learning config" container_id=4031a343757d683635ce27e45eff6c20fc7f50a07d682855eca73764005cf35a previous_container_id=None location="crates/agent/src/hbn.rs:206"
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]: Error: main_loop error exit
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]: Caused by:
Aug 11 23:44:45 7-243-180-19. forge-dpu-agent[1115137]: invalid value: integer `256`, expected u8 at line 1 column 15
On DPU:
$ ip -o link show
256: pf0vf51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\ link/ether 26:97:d8:57:c8:31 brd ff:ff:ff:ff:ff:ff\ altname enp3s0f0nc1pf0vf51
257: pf0vf52: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\ link/ether 22:1b:b3:c5:ea:34 brd ff:ff:ff:ff:ff:ff\ altname enp3s0f0nc1pf0vf52
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000\ link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
258: pf0vf53: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\ link/ether a2:06:15:71:54:8a brd ff:ff:ff:ff:ff:ff\ altname enp3s0f0nc1pf0vf53
2: oob_net0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master mgmt state UP mode DEFAULT group default qlen 1000\ link/ether e0:9d:73:7b:02:e2 brd ff:ff:ff:ff:ff:ff\ altname enamlnxbf17i0
259: pf0vf54: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 128\ link/ether 5e:f6:eb:97:bf:d3 brd ff:ff:ff:ff:ff:ff\ altname enp3s0f0nc1pf0vf54
Core side, repeating for hours:
state="Assigned/NetworkConfigUpdate { network_config_update_state: WaitingForConfigSynced }"
msg="network observation count does not match DPU count" dpu_count=1 network_observation_count=0
msg="checking dpu" dpu_machine_id=....
msg=missing
Other/Misc.
Workaround: rebooting the DPU clears it — indexes restart low, the agent stops crashing, config synced and the VF came up.
Code of Conduct
Version
v2.1.0-rc.1
Describe the bug.
After a NICo instance update that added a VF interface, forge-dpu-agent on the DPU went into a crash loop — main_loop error exit, restarting every ~90 seconds with a new PID each time (due to
ifindexdeclared asu8and linux interface ID went past 255).The instance update never completed. It stayed in Configuring for over 6 hours, new interfaces stuck in Pending, the superseded one in Deleting. Core sat in Assigned/NetworkConfigUpdate { WaitingForConfigSynced } reporting network observation count does not match DPU count, dpu_count=1 network_observation_count=0 — the DPU never reported network status because the agent kept dying first.
Minimum reproducible example
Relevant log output
Other/Misc.
Workaround: rebooting the DPU clears it — indexes restart low, the agent stops crashing, config synced and the VF came up.
Code of Conduct