Intel 10Gb ix X552
-
hw.machine: amd64
hw.model: Intel(R) Pentium(R) CPU D1508 @ 2.20GHz
hw.ncpu: 4
hw.machine_arch: amd64iperf3 -P16 -N -c 192.168.1.1
[ 5] 9.00-10.02 sec 92.0 MBytes 761 Mbits/sec 0 1.34 MBytes
[ 7] 9.00-10.02 sec 71.7 MBytes 593 Mbits/sec 0 1023 KBytes
[ 9] 9.00-10.02 sec 69.7 MBytes 576 Mbits/sec 0 981 KBytes
[ 11] 9.00-10.02 sec 51.6 MBytes 427 Mbits/sec 0 610 KBytes
[ 13] 9.00-10.02 sec 63.4 MBytes 524 Mbits/sec 4 464 KBytes
[ 15] 9.00-10.02 sec 48.1 MBytes 398 Mbits/sec 11 337 KBytes
[ 17] 9.00-10.02 sec 71.3 MBytes 589 Mbits/sec 0 924 KBytes
[ 19] 9.00-10.02 sec 67.9 MBytes 561 Mbits/sec 0 855 KBytes
[ 21] 9.00-10.02 sec 21.4 MBytes 177 Mbits/sec 0 65.1 KBytes
[ 23] 9.00-10.02 sec 57.9 MBytes 479 Mbits/sec 4 405 KBytes
[ 25] 9.00-10.02 sec 22.1 MBytes 183 Mbits/sec 0 65.1 KBytes
[ 27] 9.00-10.02 sec 49.1 MBytes 406 Mbits/sec 4 293 KBytes
[ 29] 9.00-10.02 sec 48.0 MBytes 397 Mbits/sec 6 283 KBytes
[ 31] 9.00-10.02 sec 89.6 MBytes 741 Mbits/sec 0 1.43 MBytes
[ 33] 9.00-10.02 sec 70.6 MBytes 584 Mbits/sec 0 947 KBytes
[ 35] 9.00-10.02 sec 67.7 MBytes 560 Mbits/sec 0 971 KBytes
[SUM] 9.00-10.02 sec 962 MBytes 7.96 Gbits/sec 29pfctl disabled on both pfsense devices:
[2.4.0-RC][root@pfSense.localdomain]/root: pfctl -d
pfctl: pf not enabled
[2.4.0-RC][root@pfSense.localdomain]/root: iperf -s
–----------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 63.7 KByte (default)[ 4] local 192.168.1.1 port 5001 connected with 192.168.1.2 port 17134
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 4.44 GBytes 3.81 Gbits/secstill doesn't reach vanilla FreeBSD performance
-
That is about what I might expect from that CPU. It's not clear what the difference is there between that and FreeBSD though. Interesting. What FreeBSD version is it? Same NIC driver version?
Do you have powerd enabled on both/either? You may need that to get Turbo mode.
Steve
-
We tried PfSense 2.3.4,2.3.4-p1 and 2.4
we also tried freebsd 10.3 and freebsd 11 and the results are very similar
we also tried standard ubuntu installation and it seems that only pfSesne is adversely affected.
We can get 9.5Gbps+ on all the other setups.Both pfSesne and freebsd instalaltion were done using official iso files that can be downloaded from respective sites
How do I check the driver versions?
-
turbo mode was disabled in BIOS before any installation happened for stability reasons
-
Given in the boot log or in the sysctls.
I don't have an ix NIC here but for example:
[2.4.0-RC][root@8860.stevew.lan]/root: sysctl dev.igb.0.%desc dev.igb.0.%desc: Intel(R) PRO/1000 Network Connection, Version - 2.5.3-kYou should also compare the values in /boot/loader.conf(.local)
Steve
-
[2.4.0-RC][root@pfSense.localdomain]/root: sysctl dev.ix.0.%desc
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-kI will check the freebsd once I rebuild the machines.
we copied the settings in /boot/loader.conf(.local) to no effect (we used the default freebsd ones on both freebsd and pfsense and we used pfsense ones on pfsense and freebsd)
-
freebsd 10.3 with pf enabled, still operates at full speed
driver:
root@firewall01:~ # sysctl dev.ix.0.%desc
dev.ix.0.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.1.13-k
-
Have you made any configuration changes for the NIC? May I suggest these then a test (loader.conf):
legal.intel_iwi.license_ack=1
legal.intel_ipw.license_ack=1
hw.igb.enable_aim=0
hw.igb.rxd=4096
hw.igb.txd=4096
hw.igb.rx_process_limit=-1
hw.igb.max_interrupt_rate=64000In system tunables. For each port 0, 1, etc:
dev.igb.0.fc 0
dev.igb.0.eee_disabled 1If you have more ports then configure dev.igb.1.fc 0, dev.igb.1.eee_disabled 1
You can also set hw.igb.num_queues. For quad core you set to 2, e.g. hw.igb.num_queues=2 within loader.conf.
These are igb configurations. You will need to find the ix configuration equivalent like hw.igb.max_interrupt_rate / hw.ix.max_interrupt_rate
Some may be here: https://downloadmirror.intel.com/14687/eng/readme.txt
Also, do a test with all offloading disabled with each option, i.e. checksum, LRO, segmentation. Then enabled, only, checksum and test. Some features of the NIC must have all features enabled to work properly. For examaple, checksum would need LRO and segmentation too the description is in the driver notes, IIRC. 10GbE is a different beast to configure at times. BTW, can you post a dmesg?
-
Have you tried polling?
-
/root: dmesg
Copyright
1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.3-RELEASE-p19 #1 76a12c4e6(RELENG_2_3_4): Fri Jul 14 15:02:35 CDT 2017
root@ce23-amd64-builder:/builder/pfsense-234/tmp/obj/builder/pfsense-234/tmp/FreeBSD-src/sys/pfSense amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
CPU: Intel(R) Pentium(R) CPU D1508 @ 2.20GHz (2200.05-MHz K8-class CPU)
Origin="GenuineIntel" Id=0x50663 Family=0x6 Model=0x56 Stepping=3
Features=0xbfebfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x7ffefbff <sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand>AMD Features=0x2c100800 <syscall,nx,page1gb,rdtscp,lm>AMD Features2=0x121 <lahf,abm,prefetch>Structured Extended Features=0x21cbfbb <fsgsbase,tscadj,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,pqm,nfpusg,pqe,rdseed,adx,smap,proctrace>XSAVE Features=0x1 <xsaveopt>VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
TSC: P-state invariant, performance statistics
real memory = 4294967296 (4096 MB)
avail memory = 3957551104 (3774 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <superm smci--mb="">FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
random: <software, yarrow="">initialized
ioapic0 <version 2.0="">irqs 0-23 on motherboard
lapic0: Forcing LINT1 to edge trigger
wlan: mac acl policy registered
netmap: loaded module
kbd1 at kbdmux0
module_register_init: MOD_LOAD (vesa, 0xffffffff81017260, 0) error 19
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <superm smci--mb="">on motherboard
acpi0: Power Button (fixed)
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
atrtc0: <at realtime="" clock="">port 0x70-0x71,0x74-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <at timer="">port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
Event timer "HPET3" frequency 14318180 Hz quality 440
Event timer "HPET4" frequency 14318180 Hz quality 440
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <acpi host-pci="" bridge="">on acpi0
pci255: <acpi pci="" bus="">on pcib0
pcib1: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib1
pcib2: <acpi pci-pci="" bridge="">irq 16 at device 1.0 on pci0
pci1: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">irq 16 at device 2.0 on pci0
pci2: <acpi pci="" bus="">on pcib3
pcib4: <acpi pci-pci="" bridge="">irq 16 at device 2.2 on pci0
ACPI Warning: \134_SB_.PCI0.BR2C._PRT: Return Package has no elements (empty) (20150515/nsprepkg-137)
pci3: <acpi pci="" bus="">on pcib4
pcib4: no PRT entry for 3.0.INTA
pcib4: no PRT entry for 3.0.INTB
ix0: <intel(r) pro="" 10gbe="" pci-express="" network="" driver,="" version="" -="" 3.1.14="">mem 0xfbc00000-0xfbdfffff,0xfbe04000-0xfbe07fff irq 11 at device 0.0 on pci3
ix0: Using MSIX interrupts with 5 vectors
ix0: Ethernet address: 0c:c4:7a:c9:44:b2
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: <intel(r) pro="" 10gbe="" pci-express="" network="" driver,="" version="" -="" 3.1.14="">mem 0xfba00000-0xfbbfffff,0xfbe00000-0xfbe03fff irq 10 at device 0.1 on pci3
ix1: Using MSIX interrupts with 5 vectors
ix1: Ethernet address: 0c:c4:7a:c9:44:b3
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
pcib5: <acpi pci-pci="" bridge="">irq 16 at device 3.0 on pci0
pci4: <acpi pci="" bus="">on pcib5
xhci0: <intel lynx="" point="" usb="" 3.0="" controller="">mem 0xfb300000-0xfb30ffff irq 19 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0: waiting for BIOS to give up control
xhci0: Port routing mask set to 0xffffffff
usbus0 on xhci0
pci0: <simple comms="">at device 22.0 (no driver attached)
pci0: <simple comms="">at device 22.1 (no driver attached)
ehci0: <intel lynx="" point="" usb="" 2.0="" controller="" usb-b="">mem 0xfb314000-0xfb3143ff irq 18 at device 26.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci0
pcib6: <acpi pci-pci="" bridge="">irq 16 at device 28.0 on pci0
pci5: <acpi pci="" bus="">on pcib6
pcib7: <acpi pci-pci="" bridge="">irq 16 at device 28.4 on pci0
pci6: <acpi pci="" bus="">on pcib7
pcib8: <acpi pci-pci="" bridge="">at device 0.0 on pci6
pci7: <acpi pci="" bus="">on pcib8
vgapci0: <vga-compatible display="">port 0xe000-0xe07f mem 0xfa000000-0xfaffffff,0xfb000000-0xfb01ffff irq 18 at device 0.0 on pci7
vgapci0: Boot video device
ehci1: <intel lynx="" point="" usb="" 2.0="" controller="" usb-a="">mem 0xfb313000-0xfb3133ff irq 18 at device 29.0 on pci0
usbus2: EHCI version 1.0
usbus2 on ehci1
isab0: <pci-isa bridge="">at device 31.0 on pci0
isa0: <isa bus="">on isab0
ahci0: <intel lynx="" point="" ahci="" sata="" controller="">port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf020-0xf03f mem 0xfb312000-0xfb3127ff irq 16 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <ahci channel="">at channel 0 on ahci0
ahcich1: <ahci channel="">at channel 1 on ahci0
ahcich2: <ahci channel="">at channel 2 on ahci0
ahcich3: <ahci channel="">at channel 3 on ahci0
ahcich4: <ahci channel="">at channel 4 on ahci0
ahcich5: <ahci channel="">at channel 5 on ahci0
ahciem0: <ahci enclosure="" management="" bridge="">on ahci0
acpi_button0: <power button="">on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
orm0: <isa option="" rom="">at iomem 0xc0000-0xc7fff on isa0
sc0: <system console="">at flags 0x100 on isa0
sc0: CGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3d0-0x3db iomem 0xb8000-0xbffff on isa0
ppc0: cannot reserve I/O port range
coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
est0: <enhanced speedstep="" frequency="" control="">on cpu0
coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
est1: <enhanced speedstep="" frequency="" control="">on cpu1
coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
est2: <enhanced speedstep="" frequency="" control="">on cpu2
coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3
est3: <enhanced speedstep="" frequency="" control="">on cpu3
Timecounters tick every 1.000 msec
random: unblocking device.
usbus0: 5.0Gbps Super Speed USB v3.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 480Mbps High Speed USB v2.0
ugen0.1: <0x8086> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
ugen2.1: <intel>at usbus2
uhub1: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus2
ugen1.1: <intel>at usbus1
uhub2: <intel 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub0: 21 ports with 21 removable, self powered
ugen0.2: <vendor 0x05e3="">at usbus0
uhub3: <vendor 1="" 9="" 0x05e3="" usb2.0="" hub,="" class="" 0,="" rev="" 2.00="" 32.98,="" addr="">on usbus0
uhub3: MTT enabled
ugen2.2: <vendor 0x8087="">at usbus2
uhub4: <vendor 2="" 9="" 0x8087="" product="" 0x8000,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr="">on usbus2
ugen1.2: <vendor 0x8087="">at usbus1
uhub5: <vendor 2="" 9="" 0x8087="" product="" 0x8008,="" class="" 0,="" rev="" 2.00="" 0.05,="" addr="">on usbus1
uhub4: 4 ports with 4 removable, self powered
uhub5: 4 ports with 4 removable, self powered
uhub3: 4 ports with 4 removable, self powered
ugen0.3: <vendor 0x0557="">at usbus0
uhub6: <vendor 2="" 9="" 0x0557="" product="" 0x7000,="" class="" 0,="" rev="" 2.00="" 0.00,="" addr="">on usbus0
uhub6: 4 ports with 3 removable, self powered
ugen0.4: <vendor 0x0557="">at usbus0
ukbd0: <vendor 0="" 3="" 0x0557="" product="" 0x2419,="" class="" 0,="" rev="" 1.10="" 1.00,="" addr="">on usbus0
kbd0 at ukbd0
ses0 at ahciem0 bus 0 scbus6 target 0 lun 0
ses0: <ahci sgpio="" enclosure="" 1.00="" 0001="">SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <ts64gssd370s p1225ca="">ACS-2 ATA SATA 3.x device
ada0: Serial Number D646741877
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 61057MB (125045424 512 byte sectors)
ada0: Previously was known as ad4
lapic1: Forcing LINT1 to edge trigger
SMP: AP CPU #1 Launched!
lapic2: Forcing LINT1 to edge trigger
SMP: AP CPU #2 Launched!
lapic3: Forcing LINT1 to edge trigger
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1100023234 Hz quality 1000
Trying to mount root from ufs:/dev/ufsid/59b7cd4862406270 [rw]…
pflog0: promiscuous mode enabled
ix0: link state changed to UP
ix1: link state changed to UP
ix0: link state changed to DOWN
ix0: link state changed to UP</ts64gssd370s></ahci></vendor></vendor></vendor></vendor></vendor></vendor></vendor></vendor></vendor></vendor></intel></intel></intel></intel></enhanced></cpu></enhanced></cpu></enhanced></cpu></enhanced></cpu></generic></system></isa></power></ahci></ahci></ahci></ahci></ahci></ahci></ahci></intel></isa></pci-isa></intel></vga-compatible></acpi></acpi></acpi></acpi></acpi></acpi></intel></simple></simple></intel></acpi></acpi></intel(r)></intel(r)></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></acpi></high></at></at></acpi></acpi></acpi></acpi></superm></software></version></software,></superm></xsaveopt></fsgsbase,tscadj,bmi1,hle,avx2,smep,bmi2,erms,invpcid,rtm,pqm,nfpusg,pqe,rdseed,adx,smap,proctrace></lahf,abm,prefetch></syscall,nx,page1gb,rdtscp,lm></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,sdbg,fma,cx16,xtpr,pdcm,pcid,dca,sse4.1,sse4.2,x2apic,movbe,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
Intel(R) Pentium(R) CPU D1508 @ 2.20GHz
Do you disable hyper-threading when the system is installed/using pfSense? It is enabled in the dmesg.
-
Hyper-threading is enabled but it is also enabled on freebsd where the driver works ok.
Upon further testing we discovered that it is receiving end that is slowing down
i.e sending data from pfsene to freebsd works at 9.4-9.5 Gbit
sending data from freebsd to pfsense works at 2.8 Gbitsending data between to freebsd boxes either way works at 9.5 Gbit
I also tried device polling to no effect.
-
I'm not 100% I have the screenshots matched up correctly but it looks like iperf itself is using far more CPU in pfSense than it is in FreeBSD. And that the interrupt load from the driver queues is far higher in pfSense. Would you agree?
That can be typical of a CPU running at a lower frequency. Did you check the sysctls to make sure the CPUs are running at the same speed in both cases?
The loading caused by pf appears as the interrupt load on the NIC queues. If it was not actually disabled that's where it would appear. That would also match the one direction is OK finding as pfSense allows out all traffic by default.
It's hard to believe that CPU would push close to 10Gbps with pf enabled in FreeBSD if it was actually filtering anything. Did you actually see any drop in speed or increase in CPU load when you tested that?
Interesting issue though.
Steve
-
Hi, I'm working on that problem with Belgarath.
sysctl show the same for both systems.
-
pfSense is not pushing 10GBps even with pf disabled, so that i why I am confused.
pfctl -d has no impact on the throughput on the pfsense. -
Hmm, pfctl -d will disable pf only until any change is made in the gui or the ruleset is reloaded which can be triggered by a number of things. However if will report it's already disabled if you run it again (and it is still disabled). You might try disabling it in the GUI in System > Advanced > Firewall/NAT to be sure.
It's interesting that neither box is running at 2201MHz which would be Turbo mode. But they are certainly comparable though with both set the same.
Steve
-
Hardware on both is identical, we disabled turbo mode in bios on both.
we made sure pf was disabled through the duration of the test but will check again with gui suggestion just to be sure. -
Same results when natting i disabled from GUI
-
I would recommend not using pfSense as a client or server for iperf as it does not reflect actual routing performance AT ALL. pfSense is configured to be good at routing not hosting. Both use the network but in completely different ways.
-
Try with both a client and server other than pfSense, but on different sides of the box. You'll probably have to open a firewall port for that.
