Cannot Achieve 10g pfsense bottleneck
-
@pwood999 said in Cannot Achieve 10g pfsense bottleneck:
You should probably separate the problem into stages rather the just assume is a CPU issue.
Can you name the stages you have in mind to divide the problem into?
CPU performance is crucial on a platform where packet forwarding is performed in software. Testing with UDP is a non-sense, cause it has no flow control and you have to calculate the possible throughput by accounting the packet lost. Running pfSense as iPerf server coast also CPU time to process packets received and does not reflect the possible throughput of the platform. Testing between two endpoints connected to Pfsense is the right way to do. It's why it is called throughput ...
Devices in between may affect possible throughput, especially when packet lost occurs there for whatever reason and TCP congestion algorithms kicks in. So it's good to check the interface counter in the path and ensure there are no drops on interfaces in the path for whatever reason.My suggestion is to disable HT / SMT, scale queues down to 4 and there might be another improvement. The Intel SpeedShift may work better on packed level rather than core level.
-
@Averlon what would you suggest for other BIOS power settings? Can pfSense manage power well enough for me to disable BIOS control? From my testing, when I let pfSense manage the power, it did not go over the 2200 limit (or I put some wrong settings in the BIOS and got stuck at 2200 when I let pfsense handle the power management)
-
@Laxarus
And never getting closer with your settings.This is what I am using on a Windows machine
iperf3 -c 192.168.40.40 -P 8 -t 30 -O 3 -w 2M -N -RAs @stephenw10 already said, this is not a CPU frequency issue, this is just maximum reported by pfSense.
@Laxarus said in Cannot Achieve 10g pfsense bottleneck:
I cannot afford testing this hardware in a virtual environment. This is currently in production and there is no backup for it.
Are you using UEFI mode and SATA drive or this is NVME installation?
-
@w0w ı was using legacy but switched to uefi later thinking this will improve things but it did not change anything. Installation is on a m.2 nvme.
-
Mmm, drive speed and boot type really shouldn't make any difference to throughput.
You could be hitting some bus limit perhaps. Some hardware off-loading not playing nicely?
-
@stephenw10 said in Cannot Achieve 10g pfsense bottleneck:
about 2 hours ago
Mmm, drive speed and boot type really shouldn't make any difference to throughput.
This makes a difference if we want to migrate to a Proxmox VM. When a SATA drive is used, you can prepare a new drive with Proxmox and a pfSense VM on another system, then just move it over and reassign the interfaces in Proxmox. Just use a USB Ethernet adapter as the management interface on both PCs. With NVMe it can be more complicated and may require more downtime.
@Laxarus try the iperf command provided in my previous message and post back the results.
-
Right but no difference to the the throughput of the resulting install.
-
@stephenw10 said in Cannot Achieve 10g pfsense bottleneck:
about 2 hours ago
Right but no difference to the the throughput of the resulting install.
Definitely yes.
-
@Averlon The reason I suggest testing each server to & from PfSense was just to verify that part of the E2E path - especially as the 25G link is used for all VLAN's to the Microtik.
Server1 --> Microtik --> PfSense (DS & US)
Server2 --> Microtik --> PfSense (DS & US)It would at least verify the firewall rules on the VLAN's & the VLAN's through the Microtik can pass the full bandwidth.
-
@w0w said in Cannot Achieve 10g pfsense bottleneck:
@Laxarus try the iperf command provided in my previous message and post back the results.
still 5G and occasional 6G
@Averlon said in Cannot Achieve 10g pfsense bottleneck:
My suggestion is to disable HT / SMT, scale queues down to 4 and there might be another improvement. The Intel SpeedShift may work better on packed level rather than core level.
disabled HT but this did not make any difference
-
@Laxarus said in Cannot Achieve 10g pfsense bottleneck:
still 5G and occasional 6G
OK, so how exactly is the Intel XXV710 dual 25G connected to the Ubiquiti switch, and what is the exact switch model, ports, cables, and transceivers you’re using if any?
-
@w0w
Switch: USW-EnterpriseXG-24
Connection: Unifi SFP28 DAC cable (UC-DAC-SFP28)
I disabled the LAGG so there is only a single cable now.
Do you think these cables dont play nice with pfsense?
But I also tested the 10g rj-45 built-in port but still no difference so I've ruled this out.
At this point, I am entertaining the idea of putting all 10G devices in same vlan/switch and stick with L2.
-
@Laxarus said in Cannot Achieve 10g pfsense bottleneck:
Do you think these cables dont play nice with pfsense?
I don’t think so. The more I look at it, the more I think it’s some software glitch — but where exactly is the bottleneck? It looks just like some queues/limiters. This CPU should do 30-40 Gbit with fw filtering and 60 Gbit just for routing. I don’t know — something is broken.
-
Maybe share your PfSense config, with any public IP's, Certs, etc. obfuscated ?
Or just screenshots of the VLAN firewall rules & any Limiter/Shaper queue settings ?
Check this post or an XML Redactor that might be helpful.
link redactor -
@Laxarus said in Cannot Achieve 10g pfsense bottleneck:
disabled HT but this did not make any difference
Did you configure the NIC queues down to 4 as well and tested SpeedShift at Package Level? The hwpstate_intel driver works quite well with Broadwell CPUs and does shown improvements (according to your post) towards 6Gbps on your Skylake CPUs. Compared to your previous posted results, this is an improvement of almost 1Gbps.
How is the throughput if you disable the firewall (pfctl -d) and use pfsense as router only. NAT won't be available once you disable the firewall. You can re-enable by running pfctl -e and it will load your last ruleset. If you don't see any significant difference with firewall disabled, you can be at least sure, it's not the firewall ruleset slowing things down.
What about the interface counter on that Ubiquiti switch, especially the ones for the 25gbps Uplinks - are there any error counter / drops shown?
-
Just for info.
When transferring large files between my TrueNas system and my Windows11 Pro PC, both using NVME SSD. I have transfer speeds above 5Gbit.
Situation is as follows:
- NAS <> 10G-switch <> pfSense <(lagg)> 10G-switch <> PC.
- NAS, pfSense and PC all equipped with ConnectX4 cards used at a speed of 10G.
- using jumbo frames (9000) on the connection
- transferring data between two NVME SSD's
- PC to NAS 5Gbit
- NAS to PC almost 9Gbit
- my fpSense system is build arround a older PC-mainbord having an Intel i5 6600K systeem (kaby lake Q1 2017). 4 core CPU
I am almost sure the PC is the speed limiting factor. The PC performance when transferring small files is 'dramatic'Intel i5 6600K systeem (kaby lake Q1 2017). 4 core
-
@pwood999 said in Cannot Achieve 10g pfsense bottleneck:
Maybe share your PfSense config, with any public IP's, Certs, etc. obfuscated ?
Or just screenshots of the VLAN firewall rules & any Limiter/Shaper queue settings ?
Check this post or an XML Redactor that might be helpful.
link redactorI will check what I can do about sharing the config. I think I saw some github repo for anonymizing the config.
Edit: Yep found it
Github pfsense-redactor@Averlon said in Cannot Achieve 10g pfsense bottleneck:
Did you configure the NIC queues down to 4 as well and tested SpeedShift at Package Level? The hwpstate_intel driver works quite well with Broadwell CPUs and does shown improvements (according to your post) towards 6Gbps on your Skylake CPUs. Compared to your previous posted results, this is an improvement of almost 1Gbps.
Yeah, I did all that. But 6G is not consistent, I am still getting mostly 5G.
I still think some configuration issue on the pfsense side of things. I am considering making a fresh install and testing things out then reloading my config.
@Averlon said in Cannot Achieve 10g pfsense bottleneck:
What about the interface counter on that Ubiquiti switch, especially the ones for the 25gbps Uplinks - are there any error counter / drops shown?
I see no errors.
@louis2 said in Cannot Achieve 10g pfsense bottleneck:
I am almost sure the PC is the speed limiting factor. The PC performance when transferring small files is 'dramatic'Intel i5 6600K systeem (kaby lake Q1 2017). 4 core
not similar to my case since I can achieve 10g on L2 with the same devices I test so I've ruled out the clients as the limiting factor.
I will try to adjust my settings as close to defaults as possible to see if it makes any difference.
-
Hi all
Very interesting topic: I'm experiencing the same issues with similar limitations on 10Gbit/s link.
I'm experimenting since a year with possible settings and test-scenarios. No success so far.One session limited to ~600 Mbit/s.
10 sessions limited to ~5 Gbit/s -
I was able to increase the throughput per session from 600 Mbit/s to 1.2 Gbit/s by adding this config
hw.pci.honor_msi_blacklist=0to /boot/loader.conf
Then a reboot is required.
Source: https://lists.freebsd.org/pipermail/freebsd-bugs/2015-October/064355.html
-
@TomTheOne Are you on VMware?
