Abysmal Performance after pfSense hardware upgrade

8ayM · Mar 15, 2024, 10:44 PM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

A 10G DAC cable will usually work in my experience. A DAC connected to 1G at the other end will almost always fail and doesn't allow setting 1G manually.

Threw on one of my 10g DAC's again just for giggles
login-to-view

I'm going back to the modules after testing the 1' DAC. When not working on it I can get by with that. Otherwise I'll deal with the coil of fiber for now.

Gblenn · Mar 15, 2024, 10:49 PM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

The altq setting only affects hn NICs.

What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit q while it's running it leaves the output on the console so you can copy/paste it out.

Yes absolutely, Suricata comes out at the top when I run speedtest...

login-to-view

SteveITS · Mar 15, 2024, 10:53 PM

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

Is there no benefit at all having any of the HW offloading active, even with e.g. X520 NIC? I think I have always had all three turned on, on both my sites (other site has i211 NIC's),

We run Suricata so disable it. There's potentially a benefit otherwise but if it causes false positives what's the point? E.g. packets have no checksum as they arrive in Suricata IIRC, so trip that rule unless the rule is disabled.

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

Suricata comes out at the top when I run speedtest

It will eat a lot of CPU as it has to process every packet through its ruleset. It can be a noticeable difference on fast Internet connections with a slow CPU. 7G is pretty good though. :) CPU usage also can vary widely depending on what rules are enabled. (e.g. no sense running web server rules without a web server...or really Suricata in general without any servers...most outgoing traffic is encrypted otherwise)

8ayM · Mar 16, 2024, 1:34 AM

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

@8ayM said in Abysmal Performance after pfSense hardware upgrade:

@stephenw10
Explains the Module not supported message I'm getting at the console for using a 10Gtek Ubiquity 10G SFP+ module.

Might look at some new DAC's

Anyway restored old config, made the hardware offloading changes as mentioned above, and things are looking better. Also remove traffic shaping as I'm struggling to image hitting that limit short of benchmarking.
login-to-view

So it seems the one thing that made the difference was that you turned off HW checksum offload (the first item in the list)??

@SteveITS said in Abysmal Performance after pfSense hardware upgrade:

@8ayM said in Abysmal Performance after pfSense hardware upgrade:

So is your thought/suggestion to check/disable these feature

Yes we check the three "offloading" checkboxes. Those need a restart.

Is there no benefit at all having any of the HW offloading active, even with e.g. X520 NIC? I think I have always had all three turned on, on both my sites (other site has i211 NIC's),

I made those changes yes, but the item that appears to have resolved my issue is I stopped using the 10G DAC's and installed SFP+ Modules and OM4 fiber cable.

I tried 2 x Cisco SFP-H10GB-CU3M, and 5 various 10Gtek cables all SFP+

The only DAC that looks to be working is a 1' CAB-10ZGSFP-P0.3M.

So it looks like I might be the proud owner of 7 questionable SPF+ DACs

Results with the 1' / 0.3m DAC currently in use
login-to-view

Gblenn · Mar 16, 2024, 9:24 AM

@8ayM said in Abysmal Performance after pfSense hardware upgrade:

I tried 2 x Cisco SFP-H10GB-CU3M, and 5 various 10Gtek cables all SFP+

The only DAC that looks to be working is a 1' CAB-10ZGSFP-P0.3M.

So it looks like I might be the proud owner of 7 questionable SPF+ DACs

Quite the bummer! And I guess it's clear that it is the X553 that doesn't like the DAC's, and not the switch (which sounds unlikely)?

[EDIT] A bit of googling reveals some problems with linking up X553. Checking out a few of them it seems to boil down to driver updates. And I found a reference to OPNsense working fine with X553. https://vyos.dev/T5619
But I also found this on servethehome.
https://forums.servethehome.com/index.php?threads/10gbit-interface-compatibility-intel-x553-mellanox-connectx-2.21477/
Look at the very bottom where someone has been able to make a parameter change in the kernel module to make it work by setting "allow_unsupported_sfp". Not sure if that would change anything given that you get a link up, it's just the performance that sucks. But I remember having seen a similar thing, but wrt firmware in OEM Intel cards, back when I was looking to upgrade my machine.

I have no idea if this possibility even exists in pfsense, perhaps @SteveITS or @stephenw10 knows?

stephenw10 · Mar 16, 2024, 2:34 PM

You can set 'allow unsupported SFP' but that won't help here. It's already allowing the module it's just unable to read or set the link speed. As far as I know there's nothing we c an do about that.

@Gblenn What CPU are you using that's passing 8Gbps with Suricata enabled?

Gblenn · Mar 16, 2024, 2:43 PM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

You can set 'allow unsupported SFP' but that won't help here. It's already allowing the module it's just unable to read or set the link speed. As far as I know there's nothing we c an do about that.

Yeah, that's what I thought, as the link is actually up. But it seems to differ depending on HW connected, given that at least one DAC is working. And as others have reported, with the right drivers it seems to work.

@Gblenn What CPU are you using that's passing 8Gbps with Suricata enabled?

It's an i5-11400, but I am running suricata in legacy mode. I can't remember exactly, but I believe I got around 3.5Gbps running inline mode.
And I have it virtualized on Proxmox, set to host CPU with 4 cores assigned.

stephenw10 · Mar 16, 2024, 2:48 PM

Ah, yes so significantly more powerful than any C3K CPU.

It is interesting that you see no interrupt load though, I agree. I suspect you would see that with Suricata in in-line mode.

You do see the expected kernel mode iflib task queue processes though. That's where the traffic and pf load usually appears.

Gblenn · Mar 16, 2024, 4:41 PM

@stephenw10 Indeed it is, and with 12 cores I am able to run a few other things as separate VM's without affecting throughput, (NtopNG being one of them).

Are you thinking that if I shift to inline mode for Suricata, I would start seeing interrupt going up? @8ayM doesn't seem to have Suricata activated but perhaps Ntop would have the same effect?

BTW, I changed the HW offloads this morning (none activated now) and although time of day may affect speedtest results, I did manage to get similar speeds just now.

login-to-view
login-to-view

Also tried disabling Suricata but I don't see any difference in performance...

stephenw10 · Mar 16, 2024, 9:30 PM

Mmm, the interrupt loading is interesting. What I expect to see is the task queue group values as you are seeing them.

I have to think it's ntop putting the NIC in promiscuous mode doing something there. I don't see that on a C3K system here:

last pid: 39097;  load averages:  0.67,  0.30,  0.21                                            up 2+08:27:39  21:29:14
340 threads:   6 running, 290 sleeping, 44 waiting
CPU 0:  5.5% user,  0.0% nice, 20.0% system,  0.0% interrupt, 74.5% idle
CPU 1:  2.4% user,  0.0% nice, 10.2% system,  0.0% interrupt, 87.5% idle
CPU 2:  3.1% user,  0.0% nice,  5.5% system,  0.0% interrupt, 91.4% idle
CPU 3:  3.1% user,  0.0% nice,  5.1% system,  0.0% interrupt, 91.8% idle
Mem: 98M Active, 215M Inact, 521M Wired, 3002M Free
ARC: 133M Total, 33M MFU, 93M MRU, 1121K Anon, 976K Header, 5440K Other
     99M Compressed, 244M Uncompressed, 2.47:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B    64K CPU2     2  55.4H  90.08% [idle{idle: cpu2}]
   11 root        187 ki31     0B    64K RUN      3  55.4H  89.88% [idle{idle: cpu3}]
   11 root        187 ki31     0B    64K CPU1     1  55.4H  85.69% [idle{idle: cpu1}]
   11 root        187 ki31     0B    64K CPU0     0  55.3H  76.17% [idle{idle: cpu0}]
    0 root        -60    -     0B  1648K -        2   0:03   4.75% [kernel{if_io_tqg_2}]
    0 root        -60    -     0B  1648K -        1   0:02   3.55% [kernel{if_io_tqg_1}]
    0 root        -60    -     0B  1648K -        3   0:04   2.29% [kernel{if_io_tqg_3}]
10536 root          4    0    84M    33M RUN      3   0:00   2.06% /usr/local/bin/python3.11 /usr/local/bin/speedtest{p
10536 root         56    0    84M    33M usem     1   0:01   1.87% /usr/local/bin/python3.11 /usr/local/bin/speedtest{p

Though it's also clearly not anywhere near the same throughput.

8ayM · Mar 16, 2024, 11:20 PM

@stephenw10

Would you want me to test something on my unit?

I just finished updating to the 5.6.x build so I may have some slightly different results over factory ntopng which is usually behind

8ayM · Mar 17, 2024, 12:07 AM

@8ayM

Latest test running top -HaSP
ntopng Community v.5.6.240304 rev.0 running in background. It had been disabled for most of our testing after it was suggested to do so

https://streamable.com/ir3e6j

stephenw10 · Mar 17, 2024, 3:35 PM

Did you try testing with ntop-ng disabled? Also try with bandwidthd and darkstat disabled.

8ayM · Mar 18, 2024, 4:13 PM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

Did you try testing with ntop-ng disabled? Also try with bandwidthd and darkstat disabled.

I'll try again when I get home

8ayM · Mar 18, 2024, 11:44 PM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

Did you try testing with ntop-ng disabled? Also try with bandwidthd and darkstat disabled.

As requested

Preformed the test disabling one at a time announcing which ones. Then a final test again Turing all back on
https://streamable.com/77ahrq

stephenw10 · Mar 19, 2024, 1:05 AM

Hmm, so still interrupt load with all three disabled? There must be something else set there. You have any custom sysctls set?

8ayM · Mar 19, 2024, 1:13 AM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

Hmm, so still interrupt load with all three disabled? There must be something else set there. You have any custom sysctls set?

Not that I recall, then again this has been an evolution of my early usage of pfSense which is going for about 15 years at this point. Always seemed to complicated to start over, and over time that feeling continued to grow.

Here is my current System Tunables:
login-to-view

stephenw10 · Mar 19, 2024, 8:43 PM

Hmm, nothing unexpected there. You have any custom loader values in /boot/loader.conf.local?

What other packages do you have installed?

8ayM · Mar 19, 2024, 8:43 PM

@stephenw10

login-to-view

[2.7.2-RELEASE][admin@pfSense-Edge01.scs.lan]/boot: vi loader.conf
kern.cam.boot_delay=10000
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbo9="524288"
kern.ipc.nmbjumbop="524288"
opensolaris_load="YES"
zfs_load="YES"
opensolaris_load="YES"
zfs_load="YES"
kern.cam.boot_delay=10000
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbo9="524288"
kern.ipc.nmbjumbop="524288"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
cryptodev_load="YES"
zfs_load="YES"
boot_serial="NO"
autoboot_delay="3"
hw.hn.vf_transparent="0"
hw.hn.use_if_start="1"
net.link.ifqmaxlen="128"
machdep.hwpstate_pkg_ctrl="1"
net.pf.states_hashsize="4194304"

stephenw10 · Mar 19, 2024, 9:50 PM

No loader.conf.local file though?