8200 Performance Expectations?
-
I recently received my 8200, fresh out of the box running 23.05.1.
I moved LAN to ix0 (first 10gbit interface) and adding 20 VLANs each with an IP address and associated allow firewall rule.
Out of the box with iperf3 testing between two endpoints (8200 is a router on a stick off a switch):
Endpoints on both VLAN = 9.3gbps
Endpoints on different VLAN (traffic passing through 8200) = 1.1gbpson 2.5g ports I can get slightly higher single process iperf3 tests at 2gbps.
testing with iperf -P8 i can see upwards of 4.8gbps, testing any more processes does not net any faster speeds.
turning off pf (pfctl -d) I can get single iperf speeds of 3.8gbps
moving all cables from the 8200 to a mikrotik CCR2216: 9.3gbps (iperf3 with -P1).
I've been through numerous SFP+ module types from intel to gtek to ubiquiti, all have same results. Also turning off offloads had no effect.
During iperf3 testing, if_io_tqg_# processes is using 100% cpu, a single iperf3 process uses ~20% cpu of the firewall @ 1.1gbps.
Is there something I am missing here?
-
So that's testing between VLANs on the same NIC, ix0?
Is that using a single process when you saw the 1.1Gbps result? Is it the same both ways?
@ma2885 said in 8200 Performance Expectations?:
a single iperf3 process uses ~20% cpu of the firewall @ 1.1gbps.
That's an iperf process running on pfSense or just the total usage whilst testing through it?
How is that spread across the CPU cores? Since there are 8 that could still be one core at 100%.
I would certainly expect to see more than 1.1Gbps between VLANs.
Steve
-
iperf was running through the firewall and achieving those rates.
I do not have an explanation as to why this solved (I feel sort of crazy saying what I am about to say) it but I'll post this in case it helps someone else out...
After trying every DAC/SFP+ module I could get my hands on reinstalling pfsense more than 4 times, I started playing with different OS's on the box, installed vyos - equally slow, installed ubuntu 22.04 - equally slow, installed ubuntu 23.04 = ~5gbps single tcp stream in iperf, 2 iperf streams passing through it, I see 9.4gbps...
I then reinstalled pfsense and restored my earlier backups (same process I followed the previous 4 times) to continue troubleshooting and it too now is getting ~4gbps single stream and 4 streams gets 9.4gbps - this is more in line of what I would expect.
I made no cabling/sfp+ or any other changes than changing the OS on the 8200 during each cycle - I have no clue why ubuntu 23.04 changed behavior but now that it works, I'm not trying to relive the past. For what its worth I just got two more 8200's to deploy this week, and fresh out of the box, the two new ones perform as the original 8200 post installing ubuntu 23.04 and then going back to pfsense does.
-
Hmm, weird! I'd make sure you've done a full power cycle on it at some point. I could just about imagine the newer driver in Unbuntu set something in the NIC and it's still set. I could also imagine it may have updated some NIC firmware, which would be more interesting.
Steve
-
This post is deleted! -
It would be interesting to see the output from:
[23.05.1-RELEASE][root@8200-2.stevew.lan]/root: sysctl -a | grep fw_version dev.qat.0.fw_version: 4.18.0 dev.ix.3.fw_version: eTrack 0x80000889 dev.ix.2.fw_version: eTrack 0x80000889 dev.ix.1.fw_version: eTrack 0x8000084b PHY FW V65535 dev.ix.0.fw_version: eTrack 0x8000084b PHY FW V65535If running some other OS did change anything I'd expect it to show there.
Steve
