[Bug] --io-uring slower than TX_RING when testing on dummy interface

[fklassen]

In branch 4.6.0 I tried running performance tests on dummy network and found --io-uring actually reduced performanc.

I speculate that io_uring has been implemented with parallel paths, which probably is causing extra context switching. I also speculate that that would cause reordering of packets, which is not desirable. Possibly moving to a single stream would fix both concerns

To Reproduce
Steps to reproduce the behavior:

sudo ip link add dummy0 type dummy
sudo ip link set dummy0 up
sudo tcpreplay -i dummy0 -l 1000 --io-uring --topspeed test/test.pcap 2> /dev/null
Actual: 179000 packets (69000000 bytes) sent in 0.472379 seconds
Rated: 146069152.1 Bps, 1168.55 Mbps, 378933.01 pps
Flows: 47 flows, 99.49 fps, 176000 unique flow packets, 2000 unique non-flow packets
Statistics for network device: dummy0
	Successful packets:        179000
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0

Expected behavior

I would expect --io-uring to be faster, instead its slower. Here is the same machine without --io-uring

sudo tcpreplay -i dummy0 -l 1000 --topspeed test/test.pcap 2> /dev/null
Actual: 179000 packets (69000000 bytes) sent in 0.135882 seconds
Rated: 507793526.7 Bps, 4062.34 Mbps, 1317319.43 pps
Flows: 47 flows, 345.88 fps, 176000 unique flow packets, 2000 unique non-flow packets
Statistics for network device: dummy0
	Successful packets:        179000
	Failed packets:            0
	Truncated packets:         0
	Retried packets (ENOBUFS): 0
	Retried packets (EAGAIN):  0

System (please complete the following information):

• Linux Debian 13 (trixie)
• Tcpreplay Version 4.6.0-beta3

[fklassen]

Triage

Root cause confirmed by code inspection. The reported numbers did not reproduce on my test box — they inverted — but there is a real defect underneath, described below.

Root cause: one syscall per packet

sendpacket_send_io_uring() (src/common/sendpacket.c:2165) calls io_uring_submit() once per packet (line 2208). The ring is created with flags = 0 (line 2087), so there is no IORING_SETUP_SQPOLL — every io_uring_submit() is therefore a real io_uring_enter(2).

The io_uring path consequently issues exactly as many syscalls per packet as a plain send(), while adding on top:

• a memcpy of every packet into a slot buffer (line 2195) — the plain send() path transmits straight out of the pcap buffer
• an io_uring_peek_cqe() drain loop on every packet (line 2178)
• SQE prep, CQE processing, and free-slot stack bookkeeping per packet

io_uring's whole advantage is amortizing syscalls across a batch. Submitting per packet discards that and keeps only the overhead. The 256-slot buffer pool (sendpacket.h:192) provides depth for pipelining that the code never actually exploits.

By contrast the AF_XDP backend does batch properly: src/send_packets.c:1425-1442 accumulates sp->batch_size descriptors and sendpacket.c:601 submits them in one go. io_uring received no equivalent — it hangs off the generic per-packet sendpacket() call at src/send_packets.c:610.

Reported numbers did not reproduce

Built at e79fba27, same OS and kernel as the report (Debian 13, 6.12.96), same dummy0 setup and command lines:

	this issue	my box
default	1,317,319 pps	214,200 pps
--io-uring	378,933 pps	756,138 pps

Here --io-uring is 3.5x faster, not slower. Two observations:

1. The baseline is not libpcap. With HAVE_TX_RING and HAVE_PF_PACKET both set, the default path is txring_put() (sendpacket.c:439) — the mmap'd TX ring. That backend has a quirk of its own: its helper thread usleep(100)s whenever the ring drains (src/common/txring.c:69), so on a CPU-constrained host the producer can starve the consumer into 100µs sleeps. That likely explains why my default figure is 6x below the one reported here, and it makes this VM a poor environment for the comparison.

Could you confirm whether HAVE_TX_RING is set in your build? If it is off, your baseline was a plain send() at 1.3 Mpps, which makes the io_uring result a clean 3.5x loss entirely consistent with the root cause above. The issue title should be updated either way — the comparison is against TX_RING or raw PF_PACKET, not libpcap.

2. The "extra context switching" hypothesis is not confirmed. Sampling the thread list during a live --io-uring run shows zero iou-* io-wq kernel workers and a process that stays single-threaded. On this kernel the AF_PACKET sends complete inline on the nowait path, so nothing is being punted to worker threads.

The reordering concern is real, but not yet triggered

Ordering is currently preserved, by accident: each io_uring_enter() carries exactly one SQE and completes inline, so packets cannot overtake one another. But there is no IOSQE_IO_LINK and no IORING_SETUP_SINGLE_ISSUER anywhere in the path. As soon as the performance bug is fixed by batching submissions, any SQE punted to io-wq under backpressure can be overtaken by later ones — meaning reordering would appear exactly when the NIC is busy, which is precisely when a replay tool must not reorder.

This needs to be designed into the batching fix rather than bolted on afterwards.

Suggested direction

Batch submissions (queue N SQEs, one io_uring_submit() per batch, flushing on drain, EOF, or a timer), and/or enable IORING_SETUP_SQPOLL to remove the submit syscall entirely. Explicit ordering control has to land together with the batching change, not after it.

This also needs a defensible benchmark environment. A dummy interface on a 6-CPU VM measures scheduler behaviour more than TX path efficiency — the TX_RING usleep(100) result above shows how badly that can distort the comparison.

Not a release blocker in my view, since --io-uring is opt-in and off by default.

Two smaller issues in the same code, unrelated to this report

• a short send (0 <= res < uring_lens[slot]) is counted as a full success in sendpacket_uring_process_cqe()
• the EAGAIN/ENOBUFS resubmit at line 2136 has no backoff

[fklassen]

Correction: it does reproduce — my earlier baseline was wrong

Reporter confirms HAVE_TX_RING was not set in their build, so the baseline in the original report is a plain PF_PACKET send() (sendpacket.c:441), not the mmap'd TX ring.

My previous comment compared --io-uring against TX_RING, which was the wrong baseline and produced the inverted result. Rebuilding with HAVE_TX_RING compiled out (verified: zero txring symbols in the binary) and re-measuring on the same dummy0 setup, 5 runs each, interleaved:

backend	runs (pps)	mean
plain send()	1174078, 852178, 993396, 1164508, 930818	1,022,995
--io-uring	865215, 977154, 705119, 866195, 1001505	883,037
TX_RING	487505, 293414, 230738, 191714, 449022	330,478

--io-uring is slower than plain send(), so the report reproduces. My plain-send() baseline (1.02 Mpps) also lines up reasonably with the 1.32 Mpps in the original report.

Caveat on magnitude: the gap I measure is ~14% on these means, or ~24% on a quieter 3-run set earlier, versus the ~71% in the original report. The per-run spread here is wide enough that the ranges partially overlap, so this VM is establishing direction reliably but not magnitude. The root cause in my previous comment is unchanged and is a code-level fact rather than a measurement: one io_uring_enter(2) per packet, zero syscall savings, plus a memcpy and a CQE drain loop per packet on top.

Separate finding: TX_RING is 3x slower than plain send() here

TX_RING averages 330 kpps against plain send()'s 1.02 Mpps, with a 2.5x spread across runs (191k–487k). That is the usleep(100) in the helper thread at src/common/txring.c:69 — whenever the ring drains, the consumer sleeps 100µs while the producer waits. On a CPU-constrained host the two never settle into a rhythm.

This is arguably worse than the io_uring problem, since TX_RING is the default path when autodetected. I'd suggest a separate issue for it.

Also worth checking: why was TX_RING off in that build?

The probe at configure.ac:1615 is pure autodetect — there is no --disable-tx-ring. The known failure mode (the <netpacket/packet.h> / <linux/if_packet.h> collision from #1043/#1044) was fixed in 82c6ff9, which is an ancestor of v4.6.0-beta3, and on my Debian 13 / 6.12.96 box both autotools and CMake detect HAVE_TX_RING=yes.

So a beta3 build on Debian 13 detecting TX_RING as no is unexpected and may be its own bug. One candidate: a stale generated src/config.h left in the source tree shadowing the fresh one in an out-of-tree build directory. I hit exactly that while triaging — a stale in-tree src/config.h broke my out-of-tree CMake build with a struct bpf_program redefinition at src/defines.h:61. Worth confirming which config.h that build actually compiled against.

[fklassen]

Addressed in 01936ab on 4.6.0-beta3, with an important correction to the premise below.

Two real defects found and fixed in the io_uring path

sendpacket_send_io_uring() called io_uring_submit() once per packet, so --io-uring cost exactly one io_uring_enter syscall per packet — the same syscall count as the plain send() path it is meant to beat — plus a memcpy into a slot buffer and CQE bookkeeping on top. Submissions are now batched 64 deep: 17900 packets cost 280 io_uring_enter calls instead of 17900, for ~11% less CPU per packet (1246 → 1105 ns/pkt, min-of-18 interleaved runs).

Batching alone would have broken every paced replay — a partly filled batch sat in the submission queue for the length of the nap, and a --pps=10 run delivered nothing for 1.8s then dumped all 20 packets at once. A new sendpacket_flush() is called before the replay loop naps and at close.

Your reordering hypothesis was half right. There was no io-wq punting (ps -L on a running replay shows a single thread), so no parallel kernel workers or extra context switching. But ordering genuinely was not guaranteed: unlinked SQEs let a send that cannot complete inline be retried out of band while packets behind it go out first, and the EAGAIN/ENOBUFS retry path resubmitted a packet behind newer ones. Each batch is now chained with IOSQE_IO_HARDLINK so the kernel issues it strictly in order — a hard link rather than a plain IOSQE_IO_LINK, since a plain link is severed on failure and one ENOBUFS would cancel the rest of the batch with -ECANCELED.

Also registered the socket with the ring (IOSQE_FIXED_FILE) to skip a file-table lookup per packet.

The baseline in this report was not libpcap — it was TX_RING

This build has HAVE_TX_RING, so the no-flag path goes through txring_put(), an mmap'd ring, not send(). No flag selects it; it is chosen automatically whenever configure/cmake detects it.

And that baseline was over-reporting — see #1078, filed and fixed off the back of this. TX_RING was stranding frames in the ring and counting them as successfully sent: at -l 1 it reported 179 packets sent and transmitted zero. Some of the throughput advantage measured here was packets that never left the host.

With both backends fixed and honest, on dummy0, -l 1000, all 179000 packets verified delivered against the interface counter:

backend	rate
TX_RING	1.07 Mpps
io_uring	0.86 Mpps

So the direction of this report holds up: TX_RING really is faster, because an mmap ring has no per-packet syscall at all, whereas io_uring still does one send per packet — batched, but still a send. The gap is ~1.25x, not the 3.5x measured here.

I could not reproduce the 3.5x, even against the unfixed code — io_uring measured at or slightly ahead of TX_RING before any of these changes. The numbers in the report look like desktop noise: unpinned runs of an identical invocation on this box ranged from 0.13s to 1.55s, which spans the entire reported gap. Anyone re-measuring should pin with taskset, compare CPU time rather than wall clock, and check delivery against /sys/class/net/<dev>/statistics/tx_packets.

[fklassen]

Closing — both defects this turned up are fixed on 4.6.0-beta3:

• 01936ab — io_uring: batched submissions (17900 io_uring_enter calls → 280, ~11% less CPU per packet), IOSQE_IO_HARDLINK ordering, sendpacket_flush() for paced replays, registered socket fd.
• a4a4bfe — TX_RING: in-order ring fill ([Bug] TX_RING silently drops undrained packets at close and counts them as sent #1078). The baseline this was measured against was stranding frames and counting them as sent — at -l 1 it reported 179 packets sent and transmitted zero.

On the remaining gap. With both backends honest, TX_RING does 1.07 Mpps to io_uring's 0.86 Mpps on dummy0, all 179000 packets verified delivered against the interface counter. That gap is architectural, not a bug: an mmap'd TX ring costs no syscall per packet at all, while io_uring still issues one send per packet — batched now, but still a send. io_uring is not going to overtake TX_RING on Linux by tuning; its value here is portability of the async model and lossless delivery, not raw pps.

The 3.5x in the original report was never reproducible — io_uring measured at or slightly ahead of TX_RING even before any of these changes, and unpinned runs of an identical invocation on that box ranged 0.13s to 1.55s, which spans the whole reported gap.

Reopen if the batched path still underperforms on real NICs — the dummy interface has no qdisc backpressure, so the EAGAIN/ENOBUFS retry path is essentially untested by this reproducer.