Abysmal Performance after pfSense hardware upgrade

stephenw10

That video shows ntopng using a lot CPU cycles. The first thing I'd do is restest with it disabled. No single CPU core looks to be pegged at 100% though.

8ayM

Well I just swapped things back to the ix1 interface and the speeds tanked again

stephenw10

Hmm, interesting. Are those using the same NIC chip? What does pciconf -lv show?

Gblenn

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

That video shows ntopng using a lot CPU cycles. The first thing I'd do is restest with it disabled. No single CPU core looks to be pegged at 100% though.

I just ran a few speedtests, and I don't see any of the "intr{swi1: netisr x}" that show up during your testing. And my interrupt stays flat at 0% throughout... what is that?

stephenw10

You will see that if the loading is significant. That's where the loading for moving traffic and filtering appears.

Gblenn

@stephenw10 Ok, so in my case the NIC's are doing the work...? I have Suricata showing up at the top of the list instead...

However, I just found this bit from servethehome who tested this a while back https://www.servethehome.com/the-everything-fanless-home-server-firewall-router-and-nas-appliance-qotom-qnap-teamgroup/2/

I guess that doesn't mean it's using the CPU for the SFP+ ports but perhaps it's a driver issue?

The 2.5GB ports are i225V according to servethehome...

stephenw10

You will see loading there from either the ix or igc NICs.

8ayM

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

Hmm, interesting. Are those using the same NIC chip? What does pciconf -lv show?

Yes should be off the CPU

===

[2.7.2-RELEASE][admin@pfSense-Edge01.scs.lan]/root: pciconf -lv
hostb0@pci0:0:0:0:      class=0x060000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x1980 subvendor=0x8086 subdevice=0x1999
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series System Agent'
    class      = bridge
    subclass   = HOST-PCI
hostb1@pci0:0:4:0:      class=0x060000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19a1 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Error Registers'
    class      = bridge
    subclass   = HOST-PCI
none0@pci0:0:5:0:       class=0x080700 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19a2 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Root Complex Event Collector'
    class      = base peripheral
    subclass   = Root Complex Event Collector
pcib1@pci0:0:6:0:       class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Integrated QAT Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib2@pci0:0:9:0:       class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:10:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a5 subvendor=0x8086 subdevice=0x19a5
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:11:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a6 subvendor=0x8086 subdevice=0x19a6
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:12:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a7 subvendor=0x8086 subdevice=0x19a7
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:0:14:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a8 subvendor=0x8086 subdevice=0x19a8
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:0:15:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19a9 subvendor=0x8086 subdevice=0x19a9
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib8@pci0:0:16:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19aa subvendor=0x8086 subdevice=0x19aa
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:0:17:0:      class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19ab subvendor=0x8086 subdevice=0x19ab
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
none1@pci0:0:18:0:      class=0x088000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19ac subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series SMBus Contoller - Host'
    class      = base peripheral
ahci0@pci0:0:19:0:      class=0x010601 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19b2 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series SATA Controller 0'
    class      = mass storage
    subclass   = SATA
ahci1@pci0:0:20:0:      class=0x010601 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19c2 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series SATA Controller 1'
    class      = mass storage
    subclass   = SATA
xhci0@pci0:0:21:0:      class=0x0c0330 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19d0 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series USB 3.0 xHCI Controller'
    class      = serial bus
    subclass   = USB
pcib11@pci0:0:22:0:     class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19d1 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Integrated LAN Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib12@pci0:0:23:0:     class=0x060400 rev=0x11 hdr=0x01 vendor=0x8086 device=0x19d2 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Integrated LAN Root Port'
    class      = bridge
    subclass   = PCI-PCI
none2@pci0:0:24:0:      class=0x078000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19d3 subvendor=0x8086 subdevice=0x19d3
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series ME HECI 1'
    class      = simple comms
uart2@pci0:0:26:0:      class=0x070002 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19d8 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series HSUART Controller'
    class      = simple comms
    subclass   = UART
isab0@pci0:0:31:0:      class=0x060100 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19dc subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series LPC or eSPI'
    class      = bridge
    subclass   = PCI-ISA
none3@pci0:0:31:2:      class=0x058000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19de subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series Power Management Controller'
    class      = memory
ichsmb0@pci0:0:31:4:    class=0x0c0500 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19df subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series SMBus controller'
    class      = serial bus
    subclass   = SMBus
none4@pci0:0:31:5:      class=0x0c8000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19e0 subvendor=0x8086 subdevice=0x7270
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series SPI Controller'
    class      = serial bus
none5@pci0:1:0:0:       class=0x0b4000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x19e2 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Atom Processor C3000 Series QuickAssist Technology'
    class      = processor
nvme0@pci0:3:0:0:       class=0x010802 rev=0x03 hdr=0x00 vendor=0x8086 device=0x0975 subvendor=0x8086 subdevice=0x8410
    vendor     = 'Intel Corporation'
    class      = mass storage
    subclass   = NVM
igc0@pci0:4:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc1@pci0:5:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc2@pci0:6:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc3@pci0:7:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
igc4@pci0:8:0:0:        class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x15f3 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Controller I225-V'
    class      = network
    subclass   = ethernet
pcib10@pci0:9:0:0:      class=0x060400 rev=0x03 hdr=0x01 vendor=0x1a03 device=0x1150 subvendor=0x1a03 subdevice=0x1150
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'AST1150 PCI-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:10:0:0:    class=0x030000 rev=0x30 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x1a03 subdevice=0x2000
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
ix0@pci0:11:0:0:        class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
ix1@pci0:11:0:1:        class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
ix2@pci0:12:0:0:        class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
ix3@pci0:12:0:1:        class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
[2.7.2-RELEASE][admin@pfSense-Edge01.scs.lan]/root:

===

8ayM

I'm just going to grab another m.2 and do a clean install of 2.7.2

Only config I'll be making would be assigning a MAC address to the Wan interface as the ISP locks the router MAC to their OTN handoff.

I'll be in the dark again while I get this setup.

stephenw10

Hmm, should be identical then. Could be a bad PHY/port?

Gblenn

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

You will see loading there from either the ix or igc NICs.

What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.

Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

8ayM

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

You will see loading there from either the ix or igc NICs.

What could be the reason I'm not seeing any of the intr{swi1: netisr x} when I run the same test? I am getting around 7-8Gbit both directions. For me it's Suricata taking over during the down and upload runs.

Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

This is the hardware offloading options I have selected

Gblenn

@8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated.

stephenw10

The altq setting only affects hn NICs.

What output do you actually see? Suricata will likely be top of the list of you're running it. If you hit q while it's running it leaves the output on the console so you can copy/paste it out.

8ayM

Clean install

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

@8ayM My settings are quite different in that I have not ticked any of the boxes you have marked... So I have all HW offloading enabled but the ALTQ disabled.
Of course I'm on a different system altogether, with different CPU and NIC's. But yours is definitely Intel based and should behave well also with those activated.

I'll give this a shot and see if there is a difference

SteveITS

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

Does HW offloading play a part in this, which I have activated as I'm running Suricata

https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
"11.2.3. Offloading

Network cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.

LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.

Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."

On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.

8ayM

Current Settings after reboot to apply

And the speed test results

Enabled SSH

Ran top -HaSP
https://streamable.com/tjs0r1

8ayM

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

@stephenw10 said in Abysmal Performance after pfSense hardware upgrade:

Does HW offloading play a part in this, which I have activated as I'm running Suricata in legacy mode (Intel X520 NIC).

My previous system had been on a dual sfp+ x520. I have another dual in my Desktop I've been testing with and FreeNAS box

@SteveITS said in Abysmal Performance after pfSense hardware upgrade:

@Gblenn said in Abysmal Performance after pfSense hardware upgrade:

Does HW offloading play a part in this, which I have activated as I'm running Suricata

https://docs.suricata.io/en/suricata-7.0.2/performance/packet-capture.html#offloading
"11.2.3. Offloading

Network cards, drivers and the kernel itself have various techniques to speed up packet handling. Generally these will all have to be disabled.

LRO/GRO lead to merging various smaller packets into big 'super packets'. These will need to be disabled as they break the dsize keyword as well as TCP state tracking.

Checksum offloading can be left enabled on AF_PACKET and PF_RING, but needs to be disabled on PCAP, NETMAP and others."

On pfSense inline mode uses NETMAP, and my notes from long ago said to disable offloading if using legacy due to false positives.

So is your thought/suggestion to check/disable these feature

The clean install's web UI is substantially more responsive than what I've grown accustomed to.

SteveITS

@8ayM said in Abysmal Performance after pfSense hardware upgrade:

So is your thought/suggestion to check/disable these feature

Yes we check the three "offloading" checkboxes. Those need a restart.

8ayM

Well I've gone through my stack of SFP+ Dac cables short of a 1' as I really don't have room to cable the unit, and be within reach for the short data cable.

So I broke out 2 x Gtek 10G SFP+ Modules and attached a OM4 fibre cable between the two

I'm at a loss as the DAC's have worked up until I did this firewall upgrade.

@SteveITS said in Abysmal Performance after pfSense hardware upgrade:

@8ayM said in Abysmal Performance after pfSense hardware upgrade:

So is your thought/suggestion to check/disable these feature

Yes we check the three "offloading" checkboxes. Those need a restart.

I'll revert these changes in a moment and see how that makes a difference

Is this the state that your sugesting?

After that I'll slap in my original mirrored m.2's as originally intended and see how things fare.

You almost can't be a normal person with out spare parts to trouble shoot a lot of this. 99% of the time I'm just a packrat until events like this occur.

Although I would like to get back to DAC's for the lower power draw and heat.