Low routing performance new build, N3160, i211-AT

[iphonelover]

I purchased a Jetway device (http://www.jetwaycomputer.com/JBC313U591.html) that has a N3160 CPU (has aes-ni support) and two intel i211-AT nics onboard, plopped in 8gb ram and 120GB pci-e ssd.  Thought this was going to be a great build that would last me the next couple years.

I have gotten the parts and have built and been testing it.  I can't seem to get it to iperf more than 600 Mb/s. Based on remarks I have seen on the forum I was under the impression that this should be able to route gigabit speeds. For my testing I have two different clients, both have iperf2 and iperf3, and the router.

I can iperf between each client with gigabit speeds, so I feel that I can rule out the clients and the switch as being the issue.

As for the router, I have tried both 2.3 and 2.4 and have gotten similar results on each.  I am at a loss at what to troubleshoot next.  I have included the system activity while running an iperf test in which the router was only returning 350 Mb/s to the client, in the hope some sees something that is off.

Next I am probably going to put my clients on either side of NAT and see if it will truly route at gigabit speeds, and it is just an iperf or some other issue.


[w0w]

Did you try tunes for IGB?
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
Something else like
kern.ipc.nmbclusters="1000000"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240
hw.igb.rx_process_limit="-1"
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0

[heper]

measure thoughput, not from or to pfSense

[iphonelover]

I tried adjusting the tunables, one at a time, I would make the setting change in the UI, save and then reboot and retest.  Nothing seemed to make any difference, positive or negative.  I guess, I am just going to have to get all of my devices inline to do a true test of the routing.

I will keep plugging away, but I have never seen where perf was not able to show what the server/box/router was capable of before, so I am guessing once I get this test done, I will see the same results of only routing in the 300-400 Mbps range, instead of the 900ish I was expecting.

[pfBasic]

@iphonelover:

I can iperf between each client with gigabit speeds, so I feel that I can rule out the clients and the switch as being the issue.

As for the router, I have tried both 2.3 and 2.4 and have gotten similar results on each.  I am at a loss at what to troubleshoot next.  I have included the system activity while running an iperf test in which the router was only returning 350 Mb/s to the client, in the hope some sees something that is off.

You're just using the iperf package in the wrong way. If you click the question mark icon on the pfsense page while using the package it will take you to the wiki page for the package.

https://doc.pfsense.org/index.php/Iperf_package

Usage
iperf running on pfSense is NOT a suitable way of testing firewall throughput, as there is a significant difference between performance of traffic initiated or terminated on the firewall and traffic traversing the firewall. There are many suitable uses for iperf running on pfSense, but testing the throughput capabilities of the firewall is not one of them.

[pfBasic]

That being said, my box gets gigabit throughput any way i run iperf. From it, to it, across it. I only saw ~5% on top while it was running.

I don't think the problem is your hardware though. It doesn't take much to just route at gigabit speeds and that's a pretty modern processor.

[pfBasic]

Are you using a sideloaded version of iperf or the iperf package?

[iphonelover]

I used both, the perf package, which is version 2, and installed iperf3 from the shell.

So I just finished testing this setup-

client1 running iperf3 -s connected to router on the wan port
client2 running iperf3 -c connected to router on the lan port

I didn't have my routes set up correctly, so I couldn't make the full connection, but I was able to iperf from the router lan interface to client1 at full gig speed of about 940 Mbps.

While not 100% conclusive it does lead me to believe that I can route at full gig speed.

Also I found that the Intel speed step was disabled in the BIOS, so I enabled that, and checked the PowerD settings in Pfsense, and cranked everything up to 11 and I can perf at 600 Mbps, where as before it was 350 Mbps.

[pfBasic]

yeah that hardware should have no issues routing full gig without any tweaking.

[tirsojrp]

Do not run iperf on the pfsense box.

[VioletDragon]

This can be fixed by enabling PowerD and set it to Hiadaptive and applying the following tweaks for network stack. once applied reboot and test.

Disable hardware checksum offload

Disable hardware TCP segmentation offload

Disable hardware large receive offload

kern.ipc.nmbclusters="1000000"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240
hw.igb.rx_process_limit="-1"
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0

[quaint-engineer]

@VioletDragon said in Low routing performance new build, N3160, i211-AT:

This can be fixed by enabling PowerD and set it to Hiadaptive and applying the following tweaks for network stack. once applied reboot and test.

Disable hardware checksum offload

Disable hardware TCP segmentation offload

Disable hardware large receive offload

kern.ipc.nmbclusters="1000000"
hw.igb.rxd="4096"
hw.igb.txd="4096"
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100
net.isr.defaultqlimit=4096
net.link.ifqmaxlen=10240
hw.igb.rx_process_limit="-1"
hw.igb.num_queues=2
dev.igb.0.fc=0
dev.igb.1.fc=0

Thanks! This got me from 144Mbits/sec -> 204Mbits/sec -> 940Mbits/sec on my Odyssey X86J4105800.

[VioletDragon]

That is great. Glad it helped :D

[BuiltOnSelfSuccess]

@violetdragon

I already had:
PowerD enabled and set to Hiadaptive ,
Disable hardware large receive offload
Disable hardware checksum offload

So I went and made your suggested changes:
Disable hardware TCP segmentation offload
and I added all of the network stack tweaks that you suggested but I haven't seen an improvement in speed, are there ay other tweaks that you can suggest or different values for the tweaks that you'll already suggested that I can try please?

Thanks in advance.

[BuiltOnSelfSuccess]

I found another post (https://hardforum.com/threads/pfsense-2-5-new-configurations-for-igb-performance.2008210/#post-1044942639) and made some additional changes (only to the tunables as I'm not confident and knowledgeable enough to modify the loader.conf) which seemed to improve throughput, which one of these did the trick I have no idea but on my 200/20 speed I was previously getting around 90/12 I'm now achieving around 150/16 but I know that my device has previously been capable of 180/19 so I'll keep playing until I can improve the speed further but I'd be very grateful for any additional ideas/changes/tweaks etc...