Low routing performance new build, N3160, i211-AT

pfBasic

I can iperf between each client with gigabit speeds, so I feel that I can rule out the clients and the switch as being the issue.

As for the router, I have tried both 2.3 and 2.4 and have gotten similar results on each. I am at a loss at what to troubleshoot next. I have included the system activity while running an iperf test in which the router was only returning 350 Mb/s to the client, in the hope some sees something that is off.

You're just using the iperf package in the wrong way. If you click the question mark icon on the pfsense page while using the package it will take you to the wiki page for the package.

https://doc.pfsense.org/index.php/Iperf_package

Usage
iperf running on pfSense is NOT a suitable way of testing firewall throughput, as there is a significant difference between performance of traffic initiated or terminated on the firewall and traffic traversing the firewall. There are many suitable uses for iperf running on pfSense, but testing the throughput capabilities of the firewall is not one of them.

pfBasic

That being said, my box gets gigabit throughput any way i run iperf. From it, to it, across it. I only saw ~5% on top while it was running.

I don't think the problem is your hardware though. It doesn't take much to just route at gigabit speeds and that's a pretty modern processor.

Capture.JPG_thumb

pfBasic

Are you using a sideloaded version of iperf or the iperf package?

iphonelover

I used both, the perf package, which is version 2, and installed iperf3 from the shell.

So I just finished testing this setup-

client1 running iperf3 -s connected to router on the wan port
client2 running iperf3 -c connected to router on the lan port

I didn't have my routes set up correctly, so I couldn't make the full connection, but I was able to iperf from the router lan interface to client1 at full gig speed of about 940 Mbps.

While not 100% conclusive it does lead me to believe that I can route at full gig speed.

Also I found that the Intel speed step was disabled in the BIOS, so I enabled that, and checked the PowerD settings in Pfsense, and cranked everything up to 11 and I can perf at 600 Mbps, where as before it was 350 Mbps.

pfBasic

yeah that hardware should have no issues routing full gig without any tweaking.

tirsojrp

Do not run iperf on the pfsense box.

VioletDragon

This can be fixed by enabling PowerD and set it to Hiadaptive and applying the following tweaks for network stack. once applied reboot and test.

Disable hardware checksum offload

Disable hardware TCP segmentation offload

Disable hardware large receive offload

kern.ipc.nmbclusters="1000000"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240
hw.igb.rx_process_limit="-1"
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0

quaint-engineer

@VioletDragon said in Low routing performance new build, N3160, i211-AT:

This can be fixed by enabling PowerD and set it to Hiadaptive and applying the following tweaks for network stack. once applied reboot and test.

Disable hardware checksum offload

Disable hardware TCP segmentation offload

Disable hardware large receive offload

kern.ipc.nmbclusters="1000000"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240
hw.igb.rx_process_limit="-1"
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0

Thanks! This got me from 144Mbits/sec -> 204Mbits/sec -> 940Mbits/sec on my Odyssey X86J4105800.

VioletDragon

That is great. Glad it helped :D

BuiltOnSelfSuccess

@violetdragon

I already had:
PowerD enabled and set to Hiadaptive ,
Disable hardware large receive offload
Disable hardware checksum offload

So I went and made your suggested changes:
Disable hardware TCP segmentation offload
and I added all of the network stack tweaks that you suggested but I haven't seen an improvement in speed, are there ay other tweaks that you can suggest or different values for the tweaks that you'll already suggested that I can try please?

Thanks in advance.

BuiltOnSelfSuccess

I found another post (https://hardforum.com/threads/pfsense-2-5-new-configurations-for-igb-performance.2008210/#post-1044942639) and made some additional changes (only to the tunables as I'm not confident and knowledgeable enough to modify the loader.conf) which seemed to improve throughput, which one of these did the trick I have no idea but on my 200/20 speed I was previously getting around 90/12 I'm now achieving around 150/16 but I know that my device has previously been capable of 180/19 so I'll keep playing until I can improve the speed further but I'd be very grateful for any additional ideas/changes/tweaks etc...