Load updated Intel IX module to get 10Gbps

ogghi · Jan 20, 2024, 1:47 PM

Hi there,

First: 2.7.2-RELEASE (amd64)

in order to achieve 10Gbps I am trying to load the if_ix_updated.ko during boot.

I enabled FreeBSD repo in order to have the kernel module installed from fresh ports.

I have the if_ix_updated.ko under /boot/modules/ and made changes to /boot/loader.conf.local :

net.inet.tcp.tso="0"
if_ix_updated_load="YES"
hw.ix.flow_control="0"
hw.ix.num_queues=40
hw.ix.enable_aim=1
hw.ix.max_interrupt_rate=30000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
machdep.hyperthreading_intr_allowed=1

The kernel module is not getting loaded though.

To test I even added the if_ix_updated_load="YES" line to /boot/loader.conf directly, no change.

In fact I think the settings I made here are not taken in consideration?

dmesg:
ix0: <Intel(R) X552 (SFP+)> mem 0xfbc00000-0xfbdfffff,0xfbe04000-0xfbe07fff irq 11 at device 0.0 on pci5
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: ac:1f:6b:67:04:ec
ix0: eTrack 0x800005b9 PHY FW V65535
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
ix1: <Intel(R) X552 (SFP+)> mem 0xfba00000-0xfbbfffff,0xfbe00000-0xfbe03fff irq 10 at device 0.1 on pci5
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: ac:1f:6b:67:04:ed
ix1: eTrack 0x800005b9 PHY FW V65535
ix1: netmap queues/slots: TX 8/2048, RX 8/2048

Any idea on this?

Thanks and have a nice weekend!

stephenw10 · Jan 21, 2024, 9:35 PM

Why do you think it's not loading? It should at the console at boot before loading the kernel.
Or check the sysctl:

[24.03-DEVELOPMENT][admin@5100-2.stevew.lan]/root: sysctl dev.ix.0.iflib.driver_version
dev.ix.0.iflib.driver_version: 4.0.1-k

That won't exist in the Intel kmod driver.

However that driver doesn't exist in our pkg repo. Where did you get it? It has to match the kernel very closely.

ogghi · Jan 25, 2024, 5:14 AM

Hi @stephenw10

Thanks for the reply here!

I thought it would not be loading as I am confused by version mismatch between the FreshPorts number and the one reported by
sysctl dev.ix.0.iflib.driver_version
dev.ix.0.iflib.driver_version: 4.0.1-k

Whereas on FreshPorts:
net/intel-ix-kmod: update to 3.3.35

Or are those driver versions that don't have anything to do with package version here?

If that is already all fine, I am again out of ideas why we won't get to the 10Gbps (or at least closer to it)...

stephenw10 · Jan 25, 2024, 1:57 PM

The Intel kmod driver doesn't use iflib as I understand it. So it has a completely different version number.

If it's trying to load a module and failing you should see an error at boot on the console. It would only be on the console though because it's before the kernel starts.

What speeds are you seeing? What hardware are you using?

ogghi · Jan 26, 2024, 12:25 PM

@stephenw10 Hi there

I am usually seeing around 4Gbps in both directions.

See here the output of pciconf -lv

none0@pci0:255:11:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f81 subvendor=0x8086 subdevice=0x6f81
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1'
    class      = base peripheral
none1@pci0:255:11:1:	class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f36 subvendor=0x8086 subdevice=0x6f36
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1'
    class      = dasp
    subclass   = performance counters
none2@pci0:255:11:2:	class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f37 subvendor=0x8086 subdevice=0x6f37
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1'
    class      = dasp
    subclass   = performance counters
none3@pci0:255:11:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f76 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug'
    class      = base peripheral
none4@pci0:255:12:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe0 subvendor=0x8086 subdevice=0x6fe0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none5@pci0:255:12:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe1 subvendor=0x8086 subdevice=0x6fe1
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none6@pci0:255:12:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe2 subvendor=0x8086 subdevice=0x6fe2
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none7@pci0:255:12:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe3 subvendor=0x8086 subdevice=0x6fe3
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none8@pci0:255:15:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ff8 subvendor=0x8086 subdevice=0x6ff8
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none9@pci0:255:15:4:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffc subvendor=0x8086 subdevice=0x6fe0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none10@pci0:255:15:5:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffd subvendor=0x8086 subdevice=0x6fe0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none11@pci0:255:15:6:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffe subvendor=0x8086 subdevice=0x6fe0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent'
    class      = base peripheral
none12@pci0:255:16:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1d subvendor=0x8086 subdevice=0x6f1d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent'
    class      = base peripheral
none13@pci0:255:16:1:	class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f34 subvendor=0x8086 subdevice=0x6f34
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent'
    class      = dasp
    subclass   = performance counters
none14@pci0:255:16:5:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1e subvendor=0x8086 subdevice=0x6f1e
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox'
    class      = base peripheral
none15@pci0:255:16:6:	class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f7d subvendor=0x8086 subdevice=0x6f7d
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox'
    class      = dasp
    subclass   = performance counters
none16@pci0:255:16:7:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1f subvendor=0x8086 subdevice=0x6f1f
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox'
    class      = base peripheral
none17@pci0:255:18:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fa0 subvendor=0x8086 subdevice=0x6fa0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0'
    class      = base peripheral
none18@pci0:255:18:1:	class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f30 subvendor=0x8086 subdevice=0x6f30
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0'
    class      = dasp
    subclass   = performance counters
none19@pci0:255:19:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fa8 subvendor=0x8086 subdevice=0x6fa8
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS'
    class      = base peripheral
none20@pci0:255:19:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f71 subvendor=0x8086 subdevice=0x6f71
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS'
    class      = base peripheral
none21@pci0:255:19:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6faa subvendor=0x8086 subdevice=0x6faa
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder'
    class      = base peripheral
none22@pci0:255:19:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fab subvendor=0x8086 subdevice=0x6fab
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder'
    class      = base peripheral
none23@pci0:255:19:4:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fac subvendor=0x8086 subdevice=0x6fac
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder'
    class      = base peripheral
none24@pci0:255:19:5:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fad subvendor=0x8086 subdevice=0x6fad
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder'
    class      = base peripheral
none25@pci0:255:19:6:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fae subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast'
    class      = base peripheral
none26@pci0:255:19:7:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6faf subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast'
    class      = base peripheral
none27@pci0:255:20:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb0 subvendor=0x8086 subdevice=0x6fb0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control'
    class      = base peripheral
none28@pci0:255:20:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb1 subvendor=0x8086 subdevice=0x6fb1
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control'
    class      = base peripheral
none29@pci0:255:20:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb2 subvendor=0x8086 subdevice=0x6fb2
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error'
    class      = base peripheral
none30@pci0:255:20:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb3 subvendor=0x8086 subdevice=0x6fb3
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error'
    class      = base peripheral
none31@pci0:255:20:4:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbc subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface'
    class      = base peripheral
none32@pci0:255:20:5:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbd subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface'
    class      = base peripheral
none33@pci0:255:20:6:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbe subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface'
    class      = base peripheral
none34@pci0:255:20:7:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbf subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface'
    class      = base peripheral
none35@pci0:255:21:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb4 subvendor=0x8086 subdevice=0x6fb4
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control'
    class      = base peripheral
none36@pci0:255:21:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb5 subvendor=0x8086 subdevice=0x6fb5
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control'
    class      = base peripheral
none37@pci0:255:21:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb6 subvendor=0x8086 subdevice=0x6fb6
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error'
    class      = base peripheral
none38@pci0:255:21:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb7 subvendor=0x8086 subdevice=0x6fb7
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error'
    class      = base peripheral
none39@pci0:255:30:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f98 subvendor=0x8086 subdevice=0x6f98
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none40@pci0:255:30:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f99 subvendor=0x8086 subdevice=0x6f99
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none41@pci0:255:30:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f9a subvendor=0x8086 subdevice=0x6f9a
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none42@pci0:255:30:3:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fc0 subvendor=0x8086 subdevice=0x6fc0
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none43@pci0:255:30:4:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f9c subvendor=0x8086 subdevice=0x6f9c
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none44@pci0:255:31:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f88 subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
none45@pci0:255:31:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f8a subvendor=0x0000 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit'
    class      = base peripheral
hostb0@pci0:0:0:0:	class=0x060000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f00 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2'
    class      = bridge
    subclass   = HOST-PCI
pcib2@pci0:0:1:0:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f02 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib3@pci0:0:1:1:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f03 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1'
    class      = bridge
    subclass   = PCI-PCI
pcib4@pci0:0:2:0:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f04 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2'
    class      = bridge
    subclass   = PCI-PCI
pcib5@pci0:0:2:2:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f06 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2'
    class      = bridge
    subclass   = PCI-PCI
pcib6@pci0:0:3:0:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f08 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3'
    class      = bridge
    subclass   = PCI-PCI
pcib7@pci0:0:3:2:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f0a subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3'
    class      = bridge
    subclass   = PCI-PCI
none46@pci0:0:5:0:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f28 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management'
    class      = base peripheral
none47@pci0:0:5:1:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f29 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug'
    class      = base peripheral
none48@pci0:0:5:2:	class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f2a subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors'
    class      = base peripheral
xhci0@pci0:0:20:0:	class=0x0c0330 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c31 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB xHCI'
    class      = serial bus
    subclass   = USB
none49@pci0:0:22:0:	class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3a subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family MEI Controller'
    class      = simple comms
none50@pci0:0:22:1:	class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3b subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family MEI Controller'
    class      = simple comms
ehci0@pci0:0:26:0:	class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c2d subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
pcib8@pci0:0:28:0:	class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c10 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib9@pci0:0:28:1:	class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c12 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib10@pci0:0:28:3:	class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c16 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
pcib12@pci0:0:28:4:	class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c18 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family PCI Express Root Port'
    class      = bridge
    subclass   = PCI-PCI
ehci1@pci0:0:29:0:	class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c26 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family USB EHCI'
    class      = serial bus
    subclass   = USB
isab0@pci0:0:31:0:	class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c54 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'C224 Series Chipset Family Server Standard SKU LPC Controller'
    class      = bridge
    subclass   = PCI-ISA
ahci0@pci0:0:31:2:	class=0x010601 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c02 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]'
    class      = mass storage
    subclass   = SATA
ichsmb0@pci0:0:31:3:	class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c22 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series/C220 Series Chipset Family SMBus Controller'
    class      = serial bus
    subclass   = SMBus
pchtherm0@pci0:0:31:6:	class=0x118000 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c24 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = '8 Series Chipset Family Thermal Management Controller'
    class      = dasp
none51@pci0:3:0:0:	class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f50 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon Processor D Family QuickData Technology Register DMA Channel 0'
    class      = base peripheral
none52@pci0:3:0:1:	class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f51 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon Processor D Family QuickData Technology Register DMA Channel 1'
    class      = base peripheral
none53@pci0:3:0:2:	class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f52 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon Processor D Family QuickData Technology Register DMA Channel 2'
    class      = base peripheral
none54@pci0:3:0:3:	class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f53 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'Intel Corporation'
    device     = 'Xeon Processor D Family QuickData Technology Register DMA Channel 3'
    class      = base peripheral
ix0@pci0:4:0:0:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X552 10 GbE SFP+'
    class      = network
    subclass   = ethernet
ix1@pci0:4:0:1:	class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X552 10 GbE SFP+'
    class      = network
    subclass   = ethernet
igb0@pci0:7:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb1@pci0:8:0:0:	class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533
    vendor     = 'Intel Corporation'
    device     = 'I210 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
pcib11@pci0:9:0:0:	class=0x060400 rev=0x03 hdr=0x01 vendor=0x1a03 device=0x1150 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'AST1150 PCI-to-PCI Bridge'
    class      = bridge
    subclass   = PCI-PCI
vgapci0@pci0:10:0:0:	class=0x030000 rev=0x30 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x15d9 subdevice=0x0921
    vendor     = 'ASPEED Technology, Inc.'
    device     = 'ASPEED Graphics Family'
    class      = display
    subclass   = VGA
igb2@pci0:11:0:0:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb3@pci0:11:0:1:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb4@pci0:11:0:2:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet
igb5@pci0:11:0:3:	class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521
    vendor     = 'Intel Corporation'
    device     = 'I350 Gigabit Network Connection'
    class      = network
    subclass   = ethernet

Also I made sure, link speed on the ix0 and ix1 interfaces is 10Gb

Best
David

stephenw10 · Jan 26, 2024, 1:39 PM

Ok, what CPU is that exactly? You can see the driver is assigning the full 8 queues to the NICs which is good. Check the per-core CPU usage when you are testing the throughput using: top -HaSP

ogghi · Jan 26, 2024, 1:45 PM

@stephenw10 said in Load updated Intel IX module to get 10Gbps:

top -HaSP

Running:

iperf3 -c speedtest.shinternet.ch -p 5200-5209 -P 10 -t 100

last pid: 38688;  load averages:  1.34,  0.59,  0.47                                                                                                                  up 6+00:05:49  14:40:29
500 threads:   10 running, 464 sleeping, 26 waiting
CPU 0:  0.8% user,  0.0% nice, 64.7% system,  0.0% interrupt, 34.5% idle
CPU 1:  0.8% user,  0.0% nice, 20.0% system,  0.0% interrupt, 79.2% idle
CPU 2:  0.8% user,  0.0% nice, 33.3% system,  0.0% interrupt, 65.9% idle
CPU 3:  1.6% user,  0.0% nice, 27.1% system,  0.0% interrupt, 71.4% idle
CPU 4:  0.8% user,  0.0% nice, 32.2% system,  0.0% interrupt, 67.1% idle
CPU 5:  0.8% user,  0.0% nice, 27.8% system,  0.0% interrupt, 71.4% idle
CPU 6:  3.5% user,  0.0% nice,  0.8% system,  0.4% interrupt, 95.3% idle
CPU 7:  0.4% user,  0.0% nice, 36.9% system,  0.4% interrupt, 62.4% idle
Mem: 253M Active, 2162M Inact, 1160M Wired, 4179M Free
ARC: 489M Total, 210M MFU, 268M MRU, 776K Anon, 2855K Header, 7730K Other
     425M Compressed, 1529M Uncompressed, 3.59:1 Ratio
Swap: 1024M Total, 1024M Free

  PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
   11 root        187 ki31     0B   128K CPU6     6 137.8H  95.91% [idle{idle: cpu6}]
   11 root        187 ki31     0B   128K CPU1     1 138.5H  79.08% [idle{idle: cpu1}]
   11 root        187 ki31     0B   128K CPU5     5 137.6H  73.62% [idle{idle: cpu5}]
   11 root        187 ki31     0B   128K CPU4     4 137.5H  72.60% [idle{idle: cpu4}]
   11 root        187 ki31     0B   128K RUN      3 137.5H  71.97% [idle{idle: cpu3}]
   11 root        187 ki31     0B   128K RUN      2 138.2H  69.60% [idle{idle: cpu2}]
   11 root        187 ki31     0B   128K RUN      7 138.0H  64.44% [idle{idle: cpu7}]
    0 root        -60    -     0B  2368K CPU0     0 126:29  62.11% [kernel{if_io_tqg_0}]
   11 root        187 ki31     0B   128K CPU0     0 137.0H  36.71% [idle{idle: cpu0}]
    0 root        -60    -     0B  2368K -        7  85:13  34.52% [kernel{if_io_tqg_7}]
    0 root        -60    -     0B  2368K -        2 102:04  28.68% [kernel{if_io_tqg_2}]
    0 root        -60    -     0B  2368K -        3  94:49  25.96% [kernel{if_io_tqg_3}]
    0 root        -60    -     0B  2368K -        4  89:03  25.91% [kernel{if_io_tqg_4}]
    0 root        -60    -     0B  2368K -        5  98:08  24.47% [kernel{if_io_tqg_5}]
    0 root        -60    -     0B  2368K -        1 151:48  19.72% [kernel{if_io_tqg_1}]
88144 root         24    0    19M  8712K select   5  17.6H   6.36% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn
78123 root         20    0    22M    12M kqread   6   3:32   1.05% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
29667 unbound      20    0  1771M  1655M kqread   2   0:49   0.90% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
 4206 root         20    0    29M    16M select   5  32:22   0.83% /usr/local/sbin/openvpn --config /var/etc/openvpn/server3/config.ovpn{openvpn}
29667 unbound      20    0  1771M  1655M kqread   4   0:47   0.60% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
29667 unbound      20    0  1771M  1655M kqread   3   2:03   0.57% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    0 root        -60    -     0B  2368K -        6  98:06   0.54% [kernel{if_io_tqg_6}]
29667 unbound      20    0  1771M  1655M kqread   5   1:24   0.54% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
   12 root        -60    -     0B   288K WAIT     0   1:44   0.36% [intr{swi1: netisr 7}]
29667 unbound      20    0  1771M  1655M kqread   5   1:59   0.29% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    2 root        -60    -     0B   128K WAIT     0  54:04   0.22% [clock{clock (0)}]
69491 root         20    0    26M    11M select   3  12:09   0.21% /usr/local/sbin/openvpn --config /var/etc/openvpn/server12/config.ovpn{openvpn}
29667 unbound      20    0  1771M  1655M kqread   6   2:02   0.21% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
29667 unbound      20    0  1771M  1655M kqread   2   2:13   0.17% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
   12 root        -60    -     0B   288K WAIT     2   3:11   0.16% [intr{swi1: netisr 2}]
    7 root        -16    -     0B    16K pftm     2   9:05   0.13% [pf purge]
17299 root         20    0    17M  4820K CPU3     3   0:00   0.10% top -HaSP
   12 root        -60    -     0B   288K WAIT     7   4:23   0.09% [intr{swi1: netisr 4}]
64908 root         20    0  5108M   189M uwait    7  29:25   0.08% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf}

Edit: How do I get the CPU model from terminal or so?

stephenw10 · Jan 26, 2024, 1:51 PM

It shows on the dashboard. It's shown in /var/log/dmesg.boot. sysctl hw.model should show it.

Nothing shown as limited there. Are you sure you can 10Gbps to that host?

ogghi · Jan 26, 2024, 2:12 PM

@stephenw10
LOL, easy, but sometimes still hidden from men's eyes...

Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
QAT Crypto: No

When our provider once ran a check (some raw ping flood I guess?) they filled up the 10Gbps.

Also I tried some other iperf3 servers, when they don't report busy, always the same limitation in speed.
Also this is something visible when downloading / uploading from/to a fast server in the net.

stephenw10 · Jan 26, 2024, 2:38 PM

Hmm if you're able to test between two local machines with iperf I would try that first. And if you can confirm they will reach 10Gbps when on the same subnet that eliminates any doubt.

ogghi · Jan 26, 2024, 2:47 PM

@stephenw10 Thanks for getting back to me that quickly!
From a VM on an ESX host to a VM on another ESX host I get ~9Gbps, so looks fine.

From or to one of those VMs to the pFsense (I installed iperf3 to test) I get even less, around 2,7Gbps.

This seems odd? To exclude the network behind the pFsense I also once tried to have a PC with a 10Gb fiber card directly attached to the switch that comes right after the pFsense.
Same results though.

stephenw10 · Jan 26, 2024, 3:01 PM

Running iperf to or from pfSense dircetly will always give a lower result. Both because iperf itself uses significant CPU cycles and because pfSense is optimised as a router and not a server.

Test through pfSense between different interfaces if you can.

ogghi · Jan 26, 2024, 3:12 PM

@stephenw10 our setup here is more simple than others perhaps.
Just ix0 for WAN, ix1 for LAN.
Some VLAN interfaces on LAN (ix1) and a slower backup link on igb5

stephenw10 · Jan 26, 2024, 3:32 PM

Well if you're able to test against a local server on the WAN side that would be a good test.

ogghi · Jan 30, 2024, 8:21 AM

@stephenw10 I should probably enable post notifications...
The only way here would be for me to disconnect the WAN interface from the modem and attach a server to it?

Only other way could be if the provider modem has another 10Gb port and I connect a server there, then give it another public IP we still have. But then traffic would already passing through the modem (which should certainly support 10Gb as the provider uses it for lots of customers)

stephenw10 · Jan 30, 2024, 1:10 PM

Hmm, you could use a switch on the WAN side but it would obviously have to be 10G capable.

That CPU is probably somewhere between the 1537 and 1541 for routing that traffic. I'd expect to see somewhere around 5-6Gbps with pf and NAT enabled.

ogghi · Jan 31, 2024, 5:15 AM

@stephenw10
What do you mean with between 1537 and 1541?
The Intel(R) Xeon(R) CPU D-1518 ?

So you think the limiting factor here could be the CPU?

Would we need to get another Netgate certified appliance here to get the full 10Gb?

stephenw10 · Jan 31, 2024, 1:29 PM

It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.

However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.

ogghi · Jan 31, 2024, 1:29 PM

@stephenw10
That's a tough one then?
How could I figure out what is limiting here?

stephenw10 · Feb 2, 2024, 5:12 AM

Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611

I still find that hard to believe though.