Abysmal Performance after pfSense hardware upgrade
-
I'm just going to grab another m.2 and do a clean install of 2.7.2
Only config I'll be making would be assigning a MAC address to the Wan interface as the ISP locks the router MAC to their OTN handoff.
I'll be in the dark again while I get this setup.
-
Hmm, should be identical then. Could be a bad PHY/port?
-
@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:
You will see loading there from either the ix or igc NICs.
What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.
Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).
-
@Gblenn said in Abysmal Performance after pfSense hardware upgrade:
@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:
You will see loading there from either the ix or igc NICs.
What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.
Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).
This is the hardware offloading options I have selected
-
@8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated. -
The altq setting only affects hn NICs.
What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit
qwhile it's running it leaves the output on the console so you can copy/paste it out. -
Clean install
@Gblenn said in Abysmal Performance after pfSense hardware upgrade:
@8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated.I'll give this a shot and see if there is a difference
-
@Gblenn said in Abysmal Performance after pfSense hardware upgrade:
Does HW offloading play a part in this, which I have activated as I'm running Suricata
https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
"11.2.3. OffloadingNetwork cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.
LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.
Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."
On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.
-
Current Settings after reboot to apply
And the speed test results
Enabled SSH
Ran top -HaSP
https://streamable.com/tjs0r1 -
@Gblenn said in Abysmal Performance after pfSense hardware upgrade:
@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:
Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).
My previous system had been on a dual sfp+ x520. I have another dual in my Desktop I've been testing with and FreeNAS box
@SteveITS said in Abysmal Performance after pfSense hardware upgrade:
@Gblenn said in Abysmal Performance after pfSense hardware upgrade:
Does HW offloading play a part in this, which I have activated as I'm running Suricata
https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
"11.2.3. OffloadingNetwork cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.
LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.
Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."
On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.
So is your thought/suggestion to check/disable these feature
The clean install's web UI is substantially more responsive than what I've grown accustomed to.
-
@8ayM said in Abysmal Performance after pfSense hardware upgrade:
So is your thought/suggestion to check/disable these feature
Yes we check the three "offloading" checkboxes. Those need a restart.
-
Well I've gone through my stack of SFP+ Dac cables short of a 1' as I really don't have room to cable the unit, and be within reach for the short data cable.
So I broke out 2 x Gtek 10G SFP+ Modules and attached a OM4 fibre cable between the two
I'm at a loss as the DAC's have worked up until I did this firewall upgrade.
@SteveITS said in Abysmal Performance after pfSense hardware upgrade:
@8ayM said in Abysmal Performance after pfSense hardware upgrade:
So is your thought/suggestion to check/disable these feature
Yes we check the three "offloading" checkboxes. Those need a restart.
I'll revert these changes in a moment and see how that makes a difference
Is this the state that your sugesting?
After that I'll slap in my original mirrored m.2's as originally intended and see how things fare.
You almost can't be a normal person with out spare parts to trouble shoot a lot of this. 99% of the time I'm just a packrat until events like this occur.
Although I would like to get back to DAC's for the lower power draw and heat.
-
The ix NICs in the C3K chipset a missing the lines that allow reading the link status from SFP modules. That's why you see RJ-45-copper modules are not supported.
DAC cables normally work but you often see the link status shown as 'unknown'.
The X520 does not have that issue and I suspect that's what you're seeing here.
-
I wasn't aware I couldn't utilize a SFP+ to RJ45 module
Making the changes as suggested by @SteveITS yielded similar results as before speed wise.
I'm going to try swapping back to my mirrored m.2 and see how that goes. with the sfp+ modules and fiber back to the switching infrastructure. -
For reference:
https://downloadmirror.intel.com/732258/readme.txtIn addition, SFP+ devices based on the Intel(R) Ethernet Connection X552 and Intel(R) Ethernet Connection X553 do not support the following features: * Speed and duplex auto-negotiation. * Wake on LAN * 1000BASE-T SFP ModulesThough in reality we have seen some modules will work.
-
@stephenw10
Explains the Module not supported message I'm getting at the console for using a 10Gtek Ubiquity 10G SFP+ module.Might look at some new DAC's
Anyway restored old config, made the hardware offloading changes as mentioned above, and things are looking better. Also remove traffic shaping as I'm struggling to image hitting that limit short of benchmarking.
-
A 10G DAC cable will usually work in my experience. A DAC connected to 1G at the other end will almost always fail and doesn't allow setting 1G manually.
-
@8ayM said in Abysmal Performance after pfSense hardware upgrade:
@stephenw10
Explains the Module not supported message I'm getting at the console for using a 10Gtek Ubiquity 10G SFP+ module.Might look at some new DAC's
Anyway restored old config, made the hardware offloading changes as mentioned above, and things are looking better. Also remove traffic shaping as I'm struggling to image hitting that limit short of benchmarking.
So it seems the one thing that made the difference was that you turned off HW checksum offload (the first item in the list)??
@SteveITS said in Abysmal Performance after pfSense hardware upgrade:
@8ayM said in Abysmal Performance after pfSense hardware upgrade:
So is your thought/suggestion to check/disable these feature
Yes we check the three "offloading" checkboxes. Those need a restart.
Is there no benefit at all having any of the HW offloading active, even with e.g. X520 NIC? I think I have always had all three turned on, on both my sites (other site has i211 NIC's),
-
@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:
A 10G DAC cable will usually work in my experience. A DAC connected to 1G at the other end will almost always fail and doesn't allow setting 1G manually.
Threw on one of my 10g DAC's again just for giggles
I'm going back to the modules after testing the 1' DAC. When not working on it I can get by with that. Otherwise I'll deal with the coil of fiber for now.
-
@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:
The altq setting only affects hn NICs.
What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit
qwhile it's running it leaves the output on the console so you can copy/paste it out.Yes absolutely, Suricata comes out at the top when I run speedtest...
