[CaseStudy] MAX_GRO_SKBS

[zhongjiechen]

GRO (Generic Receive Offload) is a mechanism in the kernel network stack, located at the driver layer, which aggregates multiple packets from the network into a large packet and then passes it to the network stack for processing. GRO can significantly reduce CPU overhead and improve long flow throughput.

MAX_GRO_SBS is used to limit the maximum number of different flows that the GRO layer can process.

The goal of MAX_GRO_SKBS is to balance throughput and latency. The default value is 8. For example, if MAX_GRO_SKBS=8, the GRO layer is currently processing 6 different flows. When a packet from the 7th flow arrives, the GRO layer buffers it and attempts to aggregate it. When a packet from the 8th flow arrives, the GRO layer sends all packets from the oldest flow to the upper network stack. This works somewhat like LRU.

A larger value of MAX_GRO_SKBS will probably improve the throughput as it gives more opportunity to merge packets, while at the cost of latency inflation. A smaller value of MAX_GRO_SKBS will reduce the latency but sacrifice the throughput.

[zhongjiechen]

Co-locate many mouse flows (64B) with an elephant flow (512KB), and vary the number of mouse flows.
Compare 8 (default value) with 16.

We use two CloudLab C6620 machines connected to each other. In order to allow all flows to be processed on one CPU, FlowDirect is used to aggregate all flows into one network card queue, e.g.,:

sudo ethtool -U eno12409np1 flow-type tcp4 dst-ip 192.168.25.1 action 1

This command will redirect all flows destined to 192.168.25.1 to NIC hardware queue 1 (and thus all flows are processed on the same CPU core).

Launch mouse flows:

iperf3 -s -p 5201
iperf3 -c 192.168.25.1 -P 16/32/64 -l 64 -p 5201

Launch elephant flows:

iperf3 -s -p 5202
iperf3 -c 192.168.25.1 -P 1 -l 512k -p 5202

[zhongjiechen]

The performance improvement is based on the following fact:
Mixing a large number of mouse flows with elephant flows reduces the kernel network stack's opportunity to perform GRO, which significantly degrades throughput for elephant flows.

Thus, increasing MAX_GRO_SKBS helps the kernel network stack to perform GRO for elephant flows. But the mouse flow will be delayed, so we can see a slight performance drop in mouse flows.

When the number of mouse flows is 16, even if MAX_GRO_SKBS=8, there is still a great chance to aggregate the elephant flow, so the difference is not big.
As the number of mouse flows increases to 32, MAX_GRO_SKBS=8 can no longer provide enough aggregation opportunities for the elephant flow.
As the number of mouse flows increases to 64, even MAX_GRO_SKBS=16 can no longer provide enough aggregation opportunities for the elephant flow.

[zhongjiechen]

We argue that if there is a mixture of flows, it is better to set a larger MAX_GRO_SKBS so that the elephant flow has more opportunities to benefit from GRO but does not degrade the performance of other flows too much.