Interface Traffic Out Slightly Slower Than In

AdriftAtlas

HW:
4 Core Intel N5105
4 x Intel 2.5GbE I226-V LAN

SW:
Proxmox 7.3-4
pfSense Plus 22.05

Config:
2 Cores, 4GB RAM assigned
WAN/LAN in PCI Passthrough
Hardware checksum, TCP segmentation, and large receive offload enabled
No traffic shaping enabled

I was previously using an ASUS RT-AX88U. Speedtest.net would come back with 942Mbps down and 942 up max. With pfSense I am getting 910 Mbps down and 942Mbps up. CPU usage stays below 30%.

Testing against LAN with iPerf3 and watching the pfSense traffic graph I see that the LAN interface sustains 945Mbps out and 975Mbps in. This would explain the slight asymmetry.

I have tried the following:

Disabling all hardware offload (all three boxes were checked)
Disabled Flow Control for igc using system tunables
Set net.isr.dispatch=deferred instead of net.isr.dispatch=direct
Used VirtIO interfaces with offload disabled instead of PCI Passthrough

Not quite sure why this is happening? Is the NIC itself to blame?

stephenw10

Do you still see that running pfSense bare metal?

AdriftAtlas

@stephenw10
I did not try it on bare metal and the firewall is already running so I don't want to rebuild it. I am using PCI Passthrough though so it should be pretty close.

stephenw10

You see any errors on the NICs?

Can you see the full 941Mbps between internal interfaces?

AdriftAtlas

@stephenw10

sysctl dev.igc shows no errors, not even checksum errors.

I'm not sure what you mean by between internal interfaces. I am running the iperf3 server on pfsense and the client is running on windows 11.

I'll try to do a packet capture later and see if anything stands out.

stephenw10

Ah, running one side on pfSense itself will almost always be slower.

Try testing between two internal hosts on different interfaces to exclude the WAN.