Load updated Intel IX module to get 10Gbps
-
Hi there,
First: 2.7.2-RELEASE (amd64)
in order to achieve 10Gbps I am trying to load the if_ix_updated.ko during boot.
I enabled FreeBSD repo in order to have the kernel module installed from fresh ports.
I have the if_ix_updated.ko under /boot/modules/ and made changes to /boot/loader.conf.local :
net.inet.tcp.tso="0"
if_ix_updated_load="YES"
hw.ix.flow_control="0"
hw.ix.num_queues=40
hw.ix.enable_aim=1
hw.ix.max_interrupt_rate=30000
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbop="524288"
machdep.hyperthreading_intr_allowed=1The kernel module is not getting loaded though.
To test I even added the if_ix_updated_load="YES" line to /boot/loader.conf directly, no change.
In fact I think the settings I made here are not taken in consideration?
dmesg:
ix0: <Intel(R) X552 (SFP+)> mem 0xfbc00000-0xfbdfffff,0xfbe04000-0xfbe07fff irq 11 at device 0.0 on pci5
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: Using 8 RX queues 8 TX queues
ix0: Using MSI-X interrupts with 9 vectors
ix0: allocated for 8 queues
ix0: allocated for 8 rx queues
ix0: Ethernet address: ac:1f:6b:67:04:ec
ix0: eTrack 0x800005b9 PHY FW V65535
ix0: netmap queues/slots: TX 8/2048, RX 8/2048
ix1: <Intel(R) X552 (SFP+)> mem 0xfba00000-0xfbbfffff,0xfbe00000-0xfbe03fff irq 10 at device 0.1 on pci5
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: Using 8 RX queues 8 TX queues
ix1: Using MSI-X interrupts with 9 vectors
ix1: allocated for 8 queues
ix1: allocated for 8 rx queues
ix1: Ethernet address: ac:1f:6b:67:04:ed
ix1: eTrack 0x800005b9 PHY FW V65535
ix1: netmap queues/slots: TX 8/2048, RX 8/2048Any idea on this?
Thanks and have a nice weekend!
-
Why do you think it's not loading? It should at the console at boot before loading the kernel.
Or check the sysctl:[24.03-DEVELOPMENT][admin@5100-2.stevew.lan]/root: sysctl dev.ix.0.iflib.driver_version dev.ix.0.iflib.driver_version: 4.0.1-kThat won't exist in the Intel kmod driver.
However that driver doesn't exist in our pkg repo. Where did you get it? It has to match the kernel very closely.
-
Hi @stephenw10
Thanks for the reply here!
I thought it would not be loading as I am confused by version mismatch between the FreshPorts number and the one reported by
sysctl dev.ix.0.iflib.driver_version
dev.ix.0.iflib.driver_version: 4.0.1-kWhereas on FreshPorts:
net/intel-ix-kmod: update to 3.3.35Or are those driver versions that don't have anything to do with package version here?
If that is already all fine, I am again out of ideas why we won't get to the 10Gbps (or at least closer to it)...
-
The Intel kmod driver doesn't use iflib as I understand it. So it has a completely different version number.
If it's trying to load a module and failing you should see an error at boot on the console. It would only be on the console though because it's before the kernel starts.
What speeds are you seeing? What hardware are you using?
-
@stephenw10 Hi there
I am usually seeing around 4Gbps in both directions.
See here the output of pciconf -lv
none0@pci0:255:11:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f81 subvendor=0x8086 subdevice=0x6f81 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1' class = base peripheral none1@pci0:255:11:1: class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f36 subvendor=0x8086 subdevice=0x6f36 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1' class = dasp subclass = performance counters none2@pci0:255:11:2: class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f37 subvendor=0x8086 subdevice=0x6f37 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link 0/1' class = dasp subclass = performance counters none3@pci0:255:11:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f76 subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R3 QPI Link Debug' class = base peripheral none4@pci0:255:12:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe0 subvendor=0x8086 subdevice=0x6fe0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none5@pci0:255:12:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe1 subvendor=0x8086 subdevice=0x6fe1 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none6@pci0:255:12:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe2 subvendor=0x8086 subdevice=0x6fe2 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none7@pci0:255:12:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fe3 subvendor=0x8086 subdevice=0x6fe3 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none8@pci0:255:15:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ff8 subvendor=0x8086 subdevice=0x6ff8 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none9@pci0:255:15:4: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffc subvendor=0x8086 subdevice=0x6fe0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none10@pci0:255:15:5: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffd subvendor=0x8086 subdevice=0x6fe0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none11@pci0:255:15:6: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6ffe subvendor=0x8086 subdevice=0x6fe0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent' class = base peripheral none12@pci0:255:16:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1d subvendor=0x8086 subdevice=0x6f1d vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent' class = base peripheral none13@pci0:255:16:1: class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f34 subvendor=0x8086 subdevice=0x6f34 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D R2PCIe Agent' class = dasp subclass = performance counters none14@pci0:255:16:5: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1e subvendor=0x8086 subdevice=0x6f1e vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox' class = base peripheral none15@pci0:255:16:6: class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f7d subvendor=0x8086 subdevice=0x6f7d vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox' class = dasp subclass = performance counters none16@pci0:255:16:7: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f1f subvendor=0x8086 subdevice=0x6f1f vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Ubox' class = base peripheral none17@pci0:255:18:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fa0 subvendor=0x8086 subdevice=0x6fa0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0' class = base peripheral none18@pci0:255:18:1: class=0x110100 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f30 subvendor=0x8086 subdevice=0x6f30 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0' class = dasp subclass = performance counters none19@pci0:255:19:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fa8 subvendor=0x8086 subdevice=0x6fa8 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS' class = base peripheral none20@pci0:255:19:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f71 subvendor=0x8086 subdevice=0x6f71 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS' class = base peripheral none21@pci0:255:19:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6faa subvendor=0x8086 subdevice=0x6faa vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder' class = base peripheral none22@pci0:255:19:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fab subvendor=0x8086 subdevice=0x6fab vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder' class = base peripheral none23@pci0:255:19:4: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fac subvendor=0x8086 subdevice=0x6fac vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder' class = base peripheral none24@pci0:255:19:5: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fad subvendor=0x8086 subdevice=0x6fad vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder' class = base peripheral none25@pci0:255:19:6: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fae subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Broadcast' class = base peripheral none26@pci0:255:19:7: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6faf subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast' class = base peripheral none27@pci0:255:20:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb0 subvendor=0x8086 subdevice=0x6fb0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Thermal Control' class = base peripheral none28@pci0:255:20:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb1 subvendor=0x8086 subdevice=0x6fb1 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Thermal Control' class = base peripheral none29@pci0:255:20:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb2 subvendor=0x8086 subdevice=0x6fb2 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 0 Error' class = base peripheral none30@pci0:255:20:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb3 subvendor=0x8086 subdevice=0x6fb3 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 1 Error' class = base peripheral none31@pci0:255:20:4: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbc subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface' class = base peripheral none32@pci0:255:20:5: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbd subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface' class = base peripheral none33@pci0:255:20:6: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbe subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface' class = base peripheral none34@pci0:255:20:7: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fbf subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Channel 0/1 Interface' class = base peripheral none35@pci0:255:21:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb4 subvendor=0x8086 subdevice=0x6fb4 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Thermal Control' class = base peripheral none36@pci0:255:21:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb5 subvendor=0x8086 subdevice=0x6fb5 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Thermal Control' class = base peripheral none37@pci0:255:21:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb6 subvendor=0x8086 subdevice=0x6fb6 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 2 Error' class = base peripheral none38@pci0:255:21:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fb7 subvendor=0x8086 subdevice=0x6fb7 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel 3 Error' class = base peripheral none39@pci0:255:30:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f98 subvendor=0x8086 subdevice=0x6f98 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none40@pci0:255:30:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f99 subvendor=0x8086 subdevice=0x6f99 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none41@pci0:255:30:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f9a subvendor=0x8086 subdevice=0x6f9a vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none42@pci0:255:30:3: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6fc0 subvendor=0x8086 subdevice=0x6fc0 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none43@pci0:255:30:4: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f9c subvendor=0x8086 subdevice=0x6f9c vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none44@pci0:255:31:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f88 subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral none45@pci0:255:31:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f8a subvendor=0x0000 subdevice=0x0000 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Power Control Unit' class = base peripheral hostb0@pci0:0:0:0: class=0x060000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f00 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DMI2' class = bridge subclass = HOST-PCI pcib2@pci0:0:1:0: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f02 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1' class = bridge subclass = PCI-PCI pcib3@pci0:0:1:1: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f03 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 1' class = bridge subclass = PCI-PCI pcib4@pci0:0:2:0: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f04 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2' class = bridge subclass = PCI-PCI pcib5@pci0:0:2:2: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f06 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 2' class = bridge subclass = PCI-PCI pcib6@pci0:0:3:0: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f08 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3' class = bridge subclass = PCI-PCI pcib7@pci0:0:3:2: class=0x060400 rev=0x03 hdr=0x01 vendor=0x8086 device=0x6f0a subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D PCI Express Root Port 3' class = bridge subclass = PCI-PCI none46@pci0:0:5:0: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f28 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management' class = base peripheral none47@pci0:0:5:1: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f29 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO Hot Plug' class = base peripheral none48@pci0:0:5:2: class=0x088000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x6f2a subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D IIO RAS/Control Status/Global Errors' class = base peripheral xhci0@pci0:0:20:0: class=0x0c0330 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c31 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family USB xHCI' class = serial bus subclass = USB none49@pci0:0:22:0: class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3a subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family MEI Controller' class = simple comms none50@pci0:0:22:1: class=0x078000 rev=0x04 hdr=0x00 vendor=0x8086 device=0x8c3b subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family MEI Controller' class = simple comms ehci0@pci0:0:26:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c2d subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family USB EHCI' class = serial bus subclass = USB pcib8@pci0:0:28:0: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c10 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family PCI Express Root Port' class = bridge subclass = PCI-PCI pcib9@pci0:0:28:1: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c12 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family PCI Express Root Port' class = bridge subclass = PCI-PCI pcib10@pci0:0:28:3: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c16 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family PCI Express Root Port' class = bridge subclass = PCI-PCI pcib12@pci0:0:28:4: class=0x060400 rev=0xd5 hdr=0x01 vendor=0x8086 device=0x8c18 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family PCI Express Root Port' class = bridge subclass = PCI-PCI ehci1@pci0:0:29:0: class=0x0c0320 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c26 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family USB EHCI' class = serial bus subclass = USB isab0@pci0:0:31:0: class=0x060100 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c54 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'C224 Series Chipset Family Server Standard SKU LPC Controller' class = bridge subclass = PCI-ISA ahci0@pci0:0:31:2: class=0x010601 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c02 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]' class = mass storage subclass = SATA ichsmb0@pci0:0:31:3: class=0x0c0500 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c22 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series/C220 Series Chipset Family SMBus Controller' class = serial bus subclass = SMBus pchtherm0@pci0:0:31:6: class=0x118000 rev=0x05 hdr=0x00 vendor=0x8086 device=0x8c24 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = '8 Series Chipset Family Thermal Management Controller' class = dasp none51@pci0:3:0:0: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f50 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon Processor D Family QuickData Technology Register DMA Channel 0' class = base peripheral none52@pci0:3:0:1: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f51 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon Processor D Family QuickData Technology Register DMA Channel 1' class = base peripheral none53@pci0:3:0:2: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f52 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon Processor D Family QuickData Technology Register DMA Channel 2' class = base peripheral none54@pci0:3:0:3: class=0x088000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x6f53 subvendor=0x15d9 subdevice=0x0921 vendor = 'Intel Corporation' device = 'Xeon Processor D Family QuickData Technology Register DMA Channel 3' class = base peripheral ix0@pci0:4:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac vendor = 'Intel Corporation' device = 'Ethernet Connection X552 10 GbE SFP+' class = network subclass = ethernet ix1@pci0:4:0:1: class=0x020000 rev=0x00 hdr=0x00 vendor=0x8086 device=0x15ac subvendor=0x15d9 subdevice=0x15ac vendor = 'Intel Corporation' device = 'Ethernet Connection X552 10 GbE SFP+' class = network subclass = ethernet igb0@pci0:7:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet igb1@pci0:8:0:0: class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1533 subvendor=0x15d9 subdevice=0x1533 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet pcib11@pci0:9:0:0: class=0x060400 rev=0x03 hdr=0x01 vendor=0x1a03 device=0x1150 subvendor=0x15d9 subdevice=0x0921 vendor = 'ASPEED Technology, Inc.' device = 'AST1150 PCI-to-PCI Bridge' class = bridge subclass = PCI-PCI vgapci0@pci0:10:0:0: class=0x030000 rev=0x30 hdr=0x00 vendor=0x1a03 device=0x2000 subvendor=0x15d9 subdevice=0x0921 vendor = 'ASPEED Technology, Inc.' device = 'ASPEED Graphics Family' class = display subclass = VGA igb2@pci0:11:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521 vendor = 'Intel Corporation' device = 'I350 Gigabit Network Connection' class = network subclass = ethernet igb3@pci0:11:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521 vendor = 'Intel Corporation' device = 'I350 Gigabit Network Connection' class = network subclass = ethernet igb4@pci0:11:0:2: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521 vendor = 'Intel Corporation' device = 'I350 Gigabit Network Connection' class = network subclass = ethernet igb5@pci0:11:0:3: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x1521 subvendor=0x15d9 subdevice=0x1521 vendor = 'Intel Corporation' device = 'I350 Gigabit Network Connection' class = network subclass = ethernetAlso I made sure, link speed on the ix0 and ix1 interfaces is 10Gb
Best
David -
Ok, what CPU is that exactly? You can see the driver is assigning the full 8 queues to the NICs which is good. Check the per-core CPU usage when you are testing the throughput using:
top -HaSP -
@stephenw10 said in Load updated Intel IX module to get 10Gbps:
top -HaSP
Running:
iperf3 -c speedtest.shinternet.ch -p 5200-5209 -P 10 -t 100last pid: 38688; load averages: 1.34, 0.59, 0.47 up 6+00:05:49 14:40:29 500 threads: 10 running, 464 sleeping, 26 waiting CPU 0: 0.8% user, 0.0% nice, 64.7% system, 0.0% interrupt, 34.5% idle CPU 1: 0.8% user, 0.0% nice, 20.0% system, 0.0% interrupt, 79.2% idle CPU 2: 0.8% user, 0.0% nice, 33.3% system, 0.0% interrupt, 65.9% idle CPU 3: 1.6% user, 0.0% nice, 27.1% system, 0.0% interrupt, 71.4% idle CPU 4: 0.8% user, 0.0% nice, 32.2% system, 0.0% interrupt, 67.1% idle CPU 5: 0.8% user, 0.0% nice, 27.8% system, 0.0% interrupt, 71.4% idle CPU 6: 3.5% user, 0.0% nice, 0.8% system, 0.4% interrupt, 95.3% idle CPU 7: 0.4% user, 0.0% nice, 36.9% system, 0.4% interrupt, 62.4% idle Mem: 253M Active, 2162M Inact, 1160M Wired, 4179M Free ARC: 489M Total, 210M MFU, 268M MRU, 776K Anon, 2855K Header, 7730K Other 425M Compressed, 1529M Uncompressed, 3.59:1 Ratio Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 187 ki31 0B 128K CPU6 6 137.8H 95.91% [idle{idle: cpu6}] 11 root 187 ki31 0B 128K CPU1 1 138.5H 79.08% [idle{idle: cpu1}] 11 root 187 ki31 0B 128K CPU5 5 137.6H 73.62% [idle{idle: cpu5}] 11 root 187 ki31 0B 128K CPU4 4 137.5H 72.60% [idle{idle: cpu4}] 11 root 187 ki31 0B 128K RUN 3 137.5H 71.97% [idle{idle: cpu3}] 11 root 187 ki31 0B 128K RUN 2 138.2H 69.60% [idle{idle: cpu2}] 11 root 187 ki31 0B 128K RUN 7 138.0H 64.44% [idle{idle: cpu7}] 0 root -60 - 0B 2368K CPU0 0 126:29 62.11% [kernel{if_io_tqg_0}] 11 root 187 ki31 0B 128K CPU0 0 137.0H 36.71% [idle{idle: cpu0}] 0 root -60 - 0B 2368K - 7 85:13 34.52% [kernel{if_io_tqg_7}] 0 root -60 - 0B 2368K - 2 102:04 28.68% [kernel{if_io_tqg_2}] 0 root -60 - 0B 2368K - 3 94:49 25.96% [kernel{if_io_tqg_3}] 0 root -60 - 0B 2368K - 4 89:03 25.91% [kernel{if_io_tqg_4}] 0 root -60 - 0B 2368K - 5 98:08 24.47% [kernel{if_io_tqg_5}] 0 root -60 - 0B 2368K - 1 151:48 19.72% [kernel{if_io_tqg_1}] 88144 root 24 0 19M 8712K select 5 17.6H 6.36% /usr/local/sbin/openvpn --config /var/etc/openvpn/server1/config.ovpn 78123 root 20 0 22M 12M kqread 6 3:32 1.05% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 29667 unbound 20 0 1771M 1655M kqread 2 0:49 0.90% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 4206 root 20 0 29M 16M select 5 32:22 0.83% /usr/local/sbin/openvpn --config /var/etc/openvpn/server3/config.ovpn{openvpn} 29667 unbound 20 0 1771M 1655M kqread 4 0:47 0.60% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 29667 unbound 20 0 1771M 1655M kqread 3 2:03 0.57% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 0 root -60 - 0B 2368K - 6 98:06 0.54% [kernel{if_io_tqg_6}] 29667 unbound 20 0 1771M 1655M kqread 5 1:24 0.54% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 12 root -60 - 0B 288K WAIT 0 1:44 0.36% [intr{swi1: netisr 7}] 29667 unbound 20 0 1771M 1655M kqread 5 1:59 0.29% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 2 root -60 - 0B 128K WAIT 0 54:04 0.22% [clock{clock (0)}] 69491 root 20 0 26M 11M select 3 12:09 0.21% /usr/local/sbin/openvpn --config /var/etc/openvpn/server12/config.ovpn{openvpn} 29667 unbound 20 0 1771M 1655M kqread 6 2:02 0.21% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 29667 unbound 20 0 1771M 1655M kqread 2 2:13 0.17% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound} 12 root -60 - 0B 288K WAIT 2 3:11 0.16% [intr{swi1: netisr 2}] 7 root -16 - 0B 16K pftm 2 9:05 0.13% [pf purge] 17299 root 20 0 17M 4820K CPU3 3 0:00 0.10% top -HaSP 12 root -60 - 0B 288K WAIT 7 4:23 0.09% [intr{swi1: netisr 4}] 64908 root 20 0 5108M 189M uwait 7 29:25 0.08% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf{telegraf}Edit: How do I get the CPU model from terminal or so?
-
It shows on the dashboard. It's shown in /var/log/dmesg.boot.
sysctl hw.modelshould show it.Nothing shown as limited there. Are you sure you can 10Gbps to that host?
-
@stephenw10
LOL, easy, but sometimes still hidden from men's eyes...Intel(R) Xeon(R) CPU D-1518 @ 2.20GHz 8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads AES-NI CPU Crypto: Yes (active) QAT Crypto: NoWhen our provider once ran a check (some raw ping flood I guess?) they filled up the 10Gbps.
Also I tried some other iperf3 servers, when they don't report busy, always the same limitation in speed.
Also this is something visible when downloading / uploading from/to a fast server in the net. -
Hmm if you're able to test between two local machines with iperf I would try that first. And if you can confirm they will reach 10Gbps when on the same subnet that eliminates any doubt.
-
@stephenw10 Thanks for getting back to me that quickly!
From a VM on an ESX host to a VM on another ESX host I get ~9Gbps, so looks fine.From or to one of those VMs to the pFsense (I installed iperf3 to test) I get even less, around 2,7Gbps.
This seems odd? To exclude the network behind the pFsense I also once tried to have a PC with a 10Gb fiber card directly attached to the switch that comes right after the pFsense.
Same results though. -
Running iperf to or from pfSense dircetly will always give a lower result. Both because iperf itself uses significant CPU cycles and because pfSense is optimised as a router and not a server.
Test through pfSense between different interfaces if you can.
-
@stephenw10 our setup here is more simple than others perhaps.
Just ix0 for WAN, ix1 for LAN.
Some VLAN interfaces on LAN (ix1) and a slower backup link on igb5 -
Well if you're able to test against a local server on the WAN side that would be a good test.
-
@stephenw10 I should probably enable post notifications...
The only way here would be for me to disconnect the WAN interface from the modem and attach a server to it?Only other way could be if the provider modem has another 10Gb port and I connect a server there, then give it another public IP we still have. But then traffic would already passing through the modem (which should certainly support 10Gb as the provider uses it for lots of customers)
-
Hmm, you could use a switch on the WAN side but it would obviously have to be 10G capable.
That CPU is probably somewhere between the 1537 and 1541 for routing that traffic. I'd expect to see somewhere around 5-6Gbps with pf and NAT enabled.
-
@stephenw10
What do you mean with between 1537 and 1541?
The Intel(R) Xeon(R) CPU D-1518 ?So you think the limiting factor here could be the CPU?
Would we need to get another Netgate certified appliance here to get the full 10Gb?
-
It doesn't need to be a Netgate appliance but you might need some other device to pass 10Gbps.
However your top output above does not show any CPU core at 100%. It appears there is some other limit in play here before it exhausts the CPU.
-
@stephenw10
That's a tough one then?
How could I figure out what is limiting here? -
Well we did see one user report nearly 25Gbps using that CPU with Mellanox NICs: https://forum.netgate.com/post/1119611
I still find that hard to believe though.
