Netmap not supported for Intel X553 driver in pfSense 2.5.0
-
Hello,
I have the following board: https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F
I use Netmap(Suricata) on pfSense 2.5.0:
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4
The driver for FreeBSD as far as I know is this:
https://downloadcenter.intel.com/do...10-Gigabit-Network-Connections-Under-FreeBSD-
or from source net/intel-ix-kmod
pciconf -lvc output:
ix3@pci0:8:0:1: class=0x020000 card=0x00008086 chip=0x15e58086 rev=0x11 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection X553 1GbE' class = network subclass = ethernet cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 64 messages, enabled Table in map 0x20[0x0], PBA in map 0x20[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO link x1(x1) speed 2.5(2.5) ASPM L0s/L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected ecap 0003[140] = Serial 1 0100c9ffff000000 ecap 000e[150] = ARI 1 ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 0 VFs configured out of 64 supported First VF RID Offset 0x0080, VF RID Stride 0x0002 VF Device ID 0x15c5 Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 ecap 000d[1b0] = ACS 1
After compiling NETMAP will start only in Emulation mode, hence I will get a very high speed penalty, of only ~150 Mbs/s in comparison with Linux or FreeBSD 11.3 where it worked.
ix3: link state changed to UP 252.220553 [1130] generic_netmap_attach Emulated adapter for ix3 created (prev was NULL) 252.220605 [1035] generic_netmap_dtor Emulated netmap adapter for ix3 destroyed ix3: permanently promiscuous mode enabled 252.245959 [1130] generic_netmap_attach Emulated adapter for ix3 created (prev was NULL) 252.498624 [ 320] generic_netmap_register Emulated adapter for ix3 activated
After some discussions on FreeBSD forums here: https://forums.freebsd.org/threads/intel-x553-driver-support-for-freebsd-12-1.75588/
I understood that NETMAP is disabled on FreeBSD 12.1, as someone kindly provided this line:
MAX_NETMAP_OSVERSION= 1199999 # Doesn't build w/NETMAP on 12
Also I requested help from Intel also here:
https://forums.intel.com/s/question/0D50P00004g25BHSAY/intel-x553-driver-support-for-freebsd-121Basically they've told me to not use FreeBSD 12.1 :)
I contacted also SuperMicro:
What I like to point out is, on pfSense 2.4.5 or 2.4.4 Netmap was running on Native mode, so it's not a hardware issue.
Also please note that ixgbe driver changes names depending on devices names as quoted in my discussion with FreeBSD guys:
Log Message: Update to the Intel ixgbe driver: - The driver loadables will now match the device names, something that has been requested for some time. - Rather than a modules/ixgbe there is now modules/ix and modules/ixv
I didnt want to post a question before I tried to gather all the information I could. But it seems my last chance is with you.
Please go through my discussions with Intel and FreeBSD if you have time, and if you can, help me in order to solve this.
Thank you
-
why are you using pfsense 2.5.0 and not 2.4.5 then? it's not ready for production and it migrated to freebsd 12.1 only a couple of weeks ago
also I don't think anyone here can help you with this, it's something that intel/freebsd should do upstream -
@kiokoman said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
why are you using pfsense 2.5.0 and not 2.4.5 then? it's not ready for production and it migrated to freebsd 12.1 only a couple of weeks ago
also I don't think anyone here can help you with this, it's something that intel/freebsd should do upstreamI wanted to test 2.5.0, it's pretty solid besides the Netmap issue. Maybe pfSense 2.5.0 is not ready for production but what about FreeBSD 12.1, it isn't ready also? A great part of pfSense is still FreeBSD, and it seems nobody knows about this. If nobody will test, how we will find this issues?
Don't get me wrong, I'm not trying to point fingers, just to find a solution if possible, and if not just to report stuff, and if there is no solution, sure I will revert back to a previous version, but why shouldn't we benefit somehow from this in advance?
-
I got the following response from Intel:
The issue now is if I go with the driver included I got very low speeds ~150 Mbs/s, and I can't tune the interface because some parameters are missing:
sysctl hw.ix on 12.0-RELEASE: --- hw.ix.enable_rss: 1 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 3 hw.ix.max_interrupt_rate: 31250 --- sysctl hw.ix on 11.3-RELEASE: --- hw.ix.enable_rss: 1 hw.ix.enable_legacy_tx: 0 hw.ix.enable_fdir: 0 hw.ix.unsupported_sfp: 0 hw.ix.rxd: 2048 hw.ix.txd: 2048 hw.ix.num_queues: 8 hw.ix.enable_msix: 1 hw.ix.advertise_speed: 0 hw.ix.flow_control: 0 hw.ix.tx_process_limit: -1 hw.ix.rx_process_limit: -1 hw.ix.max_interrupt_rate: 31250 hw.ix.enable_aim: 1
Also I found this issue with the same chipset:
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=239704How can I enable or fine tune the interface with the new iflib driver?
This is what I got from sysctl dev.ix.3:
dev.ix.3.iflib.driver_version: 4.0.1-k dev.ix.3.%parent: pci7 dev.ix.3.%pnpinfo: vendor=0x8086 device=0x15e5 subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.ix.3.%location: slot=0 function=1 dbsf=pci0:8:0:1 handle=\_SB_.PCI0.VRP1.LAN3 dev.ix.3.%driver: ix dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
How can I enable the missing parameters or fine tune the interface with the new iflib driver?
-
did you try https://www.freshports.org/net/intel-ix-kmod/
?already compiled available here https://drive.google.com/drive/folders/1fM-Jlmf8BY21kIEGueSxFWmrISZqcDj3
if_ix_updated.ko
it is built for freebsd 12.1 / pfsense 2.5.0 latest snapshot
copy to /boot/modules
create /boot/loader.conf.local with
if_ix_updated_load="YES"
reboot[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: kldstat Id Refs Address Size Name 1 30 0xffffffff80200000 38cefb0 kernel 2 1 0xffffffff83ad0000 58f30 if_ix_updated.ko 3 1 0xffffffff83d1a000 ff0 cpuctl.ko 4 1 0xffffffff83d1b000 2698 intpm.ko 5 1 0xffffffff83d1e000 b40 smbus.ko 6 1 0xffffffff83d1f000 8c90 aesni.ko 7 1 0xffffffff83d28000 10e48 dummynet.ko 8 1 0xffffffff83d39000 27d8 vmmemctl.ko 9 1 0xffffffff83d3c000 2e78 vmblock.ko
[2.5.0-DEVELOPMENT][root@pfSense.kiokoman.home]/root: cat /boot/loader.conf.local if_ix_updated_load="YES"
-
@kiokoman
I tried already, but I tried with your module also:[2.5.0-DEVELOPMENT][root@Entaro.Blueshift]/root: kldstat Id Refs Address Size Name 1 17 0xffffffff80200000 38cefc0 kernel 2 1 0xffffffff83acf000 eed8 aesni.ko 3 1 0xffffffff83adf000 58f30 if_ix_updated.ko 4 1 0xffffffff84011000 ff0 cpuctl.ko 5 1 0xffffffff84012000 37e8 cryptodev.ko 6 1 0xffffffff84016000 b28 coretemp.ko
The idea is if I load if_ix_kmod in /boot/modules/ or if I compile from Intel site and copy if_ix.ko to /boot/kernel, Netmap will not run in Native mode, so I achieve nothing. This process works well with FreeBSD 11.2, 11.3, but not above. The driver cannot be compiled with Netmap support, at least that I have understood from Eric's mail.
From Suricata log:
31/5/2020 -- 16:28:38 - <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - Couldn't query netmap for ix3, error Operation not supported 31/5/2020 -- 16:28:38 - <Info> -- Going to use 1 thread(s) 31/5/2020 -- 16:28:38 - <Error> -- [ERRCODE: SC_ERR_NETMAP_CREATE(263)] - opening devname netmap:ix3/R failed: Operation not supported
I started netmap with dev.netmap.admode:1 in loader.conf.local which forces Netmap to start in Native mode, or otherwise fail.
-
did you try snort ?
-
@kiokoman I've installed it just now, the speed is actually worse ~ 98 Mbs/s. I don't see any startup entries or logs like Suricata have. There is one alert.log which is a txt format, and two logs in pcap format that I have read with tcpdump, but I cannot find nothing related to Netmap.
Maybe if you are more familiar with Snort:[2.5.0-DEVELOPMENT][root@Entaro.Blueshift]/var/log/snort/snort_ix322137: ls -al total 488 drwxr-xr-x 3 root wheel 512 May 31 18:15 . drwxr-xr-x 3 root wheel 512 May 31 18:13 .. -rw-r--r-- 1 root wheel 101523 May 31 18:21 alert -rw-r--r-- 1 root wheel 0 May 31 18:13 app-stats.log drwxr-xr-x 2 root wheel 512 May 31 18:13 barnyard2 -rw------- 1 root wheel 100 May 31 18:15 snort.log.1590938061 -rw------- 1 root wheel 377805 May 31 18:21 snort.log.1590938143
-
Netmap has turned out to be a big disappointment to me. I had very high hopes in the beginning when I first included netmap support in Suricata, and later in Snort on pfSense-2.5, that Inline IPS Mode with netmap would be a fantastic feature in both packages. However, the reality has been that the various NIC drivers seem to haphazardly support netmap operation, and the internal coding and exposed API of the netmap kernel device itself has changed several times over the last couple of years or so. Based on that, no wonder the NIC drivers have a hard time keeping up. So netmap operation is failing to live up to its promise.
Snort probably has the slower performance under emulation mode than Suricata due to the fact the Snort implementation is a bit older and uses the API version that only exposed a single host ring. The latest netmap API exposes multiple host rings if the NIC driver supports them.
-
The ideea is as I'm testing it right now, it will be impossible to run Suricata or Snort anymore, with this ifllib framework. On Linux works ok, I can reach from 620 to 960 Mbps so it's not a hardware issue.
Also it worked at full speed on FreeBSD 11.2, I think maybe it's not only Netmap but this iflib framework that Intel talks about.
I used Snort only to respond to @kiokoman for testing, but I'm not able to tell from logs in which mode NETMAP started like in Suricata. Do you know how?
Also there is somehow the possibility that NETMAP will start to emulated mode regardless of dev.netmap.admode:1 setting with iflib framework? I'm thinking maybe this iflib framework hides or doesn't report correctly the starting mode?
On FreeBSD 11.2 for example I achieved full speed only after I compiled my own driver, so I had:
- NETMAP native mode : 600 - 960 Mbs/s - with Intel compiled driver
- NETMAP emulated mode: 150 Mb/s - with included FreeBSD driver in pfSense installation
On FreeBSD 12.1 I have:
- Netmap native mode(at least that's what the system reports) : 150 Mbs/s - with included FreeBSD driver in pfSense installation
- Netmap emulated mode: 150 Mbs/s - with Intel driver compiled from Intel site, or from FreeBSD source: intel-ix-kmod
I don't know what to ask anymore. Only one question, what will we do with Suricata or Snort when pfSense 2.5.0 will be production ready?
-
Be careful with the term "pfSense driver" as that is not really accurate. There are no "pfSense drivers" at all. They simply use what is included by default from upstream FreeBSD according to the FreeBSD version they are using.
Now on to the topic at hand --
I am not a netmap device expert by any means. I have also read a limited amount about the iflib framework. Still have not fully digested what little bit I did read about it, but I understand it to be yet another new gizmo for abstracting stuff. That seems to be the favorite pastime of developers these days -- abstracting things in layer upon layer of additional software to supposedly make it all easier to use. Well, I'm not 100% convinced, but then nobody asked me anyway ... .
When it comes to trendy hardware support (and that would include things like netmap, NIC drivers and so forth), Linux is better than FreeBSD. So living with less than optimum hardware support is just one of the things we must accept when using a FreeBSD platform.
As for what happens with Snort and Suricata in pfSense-2.5, well, Legacy Mode still works and should continue working. Of course it is not ideal when it comes to IPS. On the other hand, the utility of an IDS/IPS is steadily diminishing as more and more traffic becomes end-to-end encrypted. The rules have less and less traffic to actually inspect in any meaningful way.
-
By pfSense driver I've meant the default driver, and to point out I did nothing to change it. You are correct it comes from FreeBSD not from pfSense. I will change the term.
For the rest, what is to be said. I like your diplomacy
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I understood that NETMAP is disabled on FreeBSD 12.1, as someone kindly provided this line:
That is not true ... in FreeBSD 12.0/12.1, Netmap had been moved to IFLib.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
How can I enable or fine tune the interface with the new iflib driver?
You cannot. The problem is currently FreeBSD 12.0 and 12.1 is still using the old Intel Pro1000 driver instead of Intel 25 driver released in January 2020. The solution is either wait for FreeBSD to incorporate the new driver into FreeBSD 12.1 or encourages pfSense to incorporate it in pfSense 2.5 final release.
I really wished it was easy for us to install driver updates than having to wait for it to be compiled by FreeBSD ... that's why I am encouraging pfSense to do it since the NIC is the essence of a firewall.
-
- Why we must wait for upstream, the driver and the framework it's not open source? Or maybe not the Intel driver.
- It requires recompilation of the kernel?
- There is no new compatible driver with iflib from Intel?
I noticed that you talked with Luigi. I sent an email to him and he introduced me to two of his friends: Vincenzo Maffione and Giuseppe Lettieri. Should I go further with asking them for a solution, or your investigation with Luigi points to the conclusion that the Netmap implementation is fine?
What about the parameters that we used to tune for getting the right buffers, queues, etc. My understanding is that all of those have different names now, and we have to tune them through iflib, before Intel driver is even loaded. All the tutorials are now obsolete?
I don't think it's ok for us to play a waiting game here, we may see the final version omitting this issue, due to lack of upstream implementation
I also started a topic on FreeBSD forums here https://forums.freebsd.org/threads/intel-x553-driver-support-for-freebsd-12-1.75588/ maybe you can drop a line there also, someone may see it.
I will also sent this issue to FreeBSD e-mail group freebsd-net@freebsd.org if you didn't do this already
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
It requires recompilation of the kernel?
Yes, I wish I knew how to do it ... it can take up to 18Hrs to compile. The compatible driver is the Pro1000 which I am using now.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
he introduced me to two of his friends: Vincenzo Maffione and Giuseppe Lettieri
I had communicated with Vincenzo ... cool dude, went out-of-his way to explain the situation.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I don't think it's ok for us to play a waiting game here, we may see the final version omitting this issue, due to lack of upstream implementation
That why I believe it is pfSense responsibility to make sure releases have the latest drivers for NIC and stop leaving it up to FreeBSD ... without the NIC, the firewall is useless. Unfortunately, addressing the issue on FreeBSD forum is a moot point because they think or believe it's pfSense responsibility. They don't encourage discussion on pfSense. Pretty soon pfSense 2.5RC will be out; so, we need folks making noise.
-
@NollipfSense Thank for your input, but I'm not following on the following:
@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
You cannot. The problem is currently FreeBSD 12.0 and 12.1 is still using the old Intel Pro1000 driver instead of Intel 25 driver released in January 2020.
How do you know what driver version FreeBSD 12 is using? All I can see is:
dev.ix.3.iflib.driver_version: 4.0.1-k dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
By 25 version of the driver I think you are referring to this ?
https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-PackBut if go for PRO 1000 specifically you will find that the latest version for FreeBSD is 2.5.14 or 7.7.8 depending the card
For my chipset X553 it uses PROXGB driver:
and the last version is from last year:
https://downloadcenter.intel.com/download/14688/Intel-Network-Adapters-Driver-for-PCIe-10-Gigabit-Network-Connections-Under-FreeBSD-?wapkw=intel%20x550%20networkSo I think it should have been included by now, but I can't tell because the system reports 4.0.1-k.
What is your status now, are you using NETMAP in emulated mode, not at all and you are waiting for this to get fixed?
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
How do you know what driver version FreeBSD 12 is using?
I have been trying to remember the command since you had sent me a message.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 grep
usage: dmesg [-ac] [-M core [-N system]]
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 | grep
usage: dmesg [-ac] [-M core [-N system]]
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root:I know I had used: dmesg
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
What is your status now, are you using NETMAP in emulated mode, not at all and you are waiting for this to get fixed?
Yes, I am using Netmap; however, it in default mode and not in-emulated mode. My NIC is the Intel i350, and I am using Netmap on WAN - Suricata as well as on LAN - Snort, both in-line mode. That's why I had switched to pfSense 2.5.
-
@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
How do you know what driver version FreeBSD 12 is using?
I have been trying to remember the command since you had sent me a message.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 grep
usage: dmesg [-ac] [-M core [-N system]]
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 | grep
usage: dmesg [-ac] [-M core [-N system]]
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root:I know I had used: dmesg
Maybe I should've told you in advance, I already tried dmesg, but I appreciate your intention to help
If I run : dmesg | grep ix3 I will get:
[2.5.0-DEVELOPMENT][root@Entaro.Blueshift]/root: dmesg | grep ix3 ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7d7f400000-0x7d7f5fffff,0x7d7f800000-0x7d7f803fff at device 0.1 on pci7 ix3: Using 2048 TX descriptors and 2048 RX descriptors ix3: Using 4 RX queues 4 TX queues ix3: Using MSI-X interrupts with 5 vectors ix3: allocated for 4 queues ix3: allocated for 4 rx queues ix3: Ethernet address: ac:1f:6b:45:fa:8b ix3: netmap queues/slots: TX 4/2048, RX 4/2048 ix3: link state changed to UP ix3: link state changed to DOWN ix3: link state changed to UP
So I will not get anything.
The only way I can get something is using sysctl like this sysctl dev.ix.3
The result is:ev.ix.3.iflib.driver_version: 4.0.1-k dev.ix.3.%parent: pci7 dev.ix.3.%pnpinfo: vendor=0x8086 device=0x15e5 subvendor=0x8086 subdevice=0x0000 class=0x020000 dev.ix.3.%location: slot=0 function=1 dbsf=pci0:8:0:1 handle=\_SB_.PCI0.VRP1.LAN3 dev.ix.3.%driver: ix dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver
Yes, I am using Netmap; however, it in default mode and not in-emulated mode. My NIC is the Intel i350, and I am using Netmap on WAN - Suricata as well as on LAN - Snort, both in-line mode. That's why I had switched to pfSense 2.5.
But if you are using Netmap in NATIVE mode, what is your issue then ? Or it got fixed after you updated to FreeBSD 12.1 by default? What speeds do you achieve?
My issue is that I have a very high speed penalty.
With FreeBSD 12.1 default driver I get 150 Mbs/s, and NETMAP starts in NATIVE mode
If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
But if you are using Netmap in NATIVE mode, what is your issue then ?
The current driver when in iflib does not allow traffic graph to show. That's great info to have on the WebGUI at a glance. The new driver should make that happened.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.Did you compile with the new Intel 25 driver? I take it yes ... wow that a hell of a difference.
-
@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
But if you are using Netmap in NATIVE mode, what is your issue then ?
The current driver when in iflib does not allow traffic graph to show. That's great info to have on the WebGUI at a glance. The new driver will make that happened.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.Did you compile with the new Intel 25 driver? I take it yes ... wow that a hell of a difference.
I'm using 25.1 even, but I think you are referring to the driver package rele
On FreeBSD 11.2 it worked this way, compile the driver and override the ko in /boot/kernel/ or copy as a module in /boot/modules. Sure, adding the proper line in loader.conf.local is also needed
On FreeBSD 11.3,12.1 if I compile my own driver I will achieve nothing, because my own compilation, will not include NETMAP native support due to iflib framework, hence it will run in Emulated mode at 150 Mbs/s, and if I go with the FreeBSD 12.1 driver, NETMAP will start in Native mode, but I will achieve the same speed 150 Mbs/s, so something is not right with the driver.I contacted you because in your thread here: https://forum.netgate.com/topic/144979/porting-bge-driver-to-iflib/5
you mentioned you had problems with NETMAP Native support and you attempted to compile some drivers.
In this context we have the same issue I think. Can you elaborate, what was your status with BGE in the end?
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
Can you elaborate, what was your status with BGE in the end?
So, I had bought an Apple Mac Mini server (2011) because I am a Mac user and I like the small form factor; however, it uses Broadcom NIC hence, the BGE. I wanted to port the driver, but it was too much work for me as a newbie to porting. So, the solution was to get a thunderbolt PCI enclosure (Akitio) and placed the Intel i350 in it. That also allows me to upgrade to the 10GBe NIC when I move to an area with fiber.
-
@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
Can you elaborate, what was your status with BGE in the end?
So, I had bought an Apple Mac Mini server (2011) because I am a Mac user and I like the small form factor; however, it uses Broadcom NIC hence, the BGE. I wanted to port the driver, but it was too much work for me as a newbie to porting. So, the solution was to get a thunderbolt PCI enclosure and placed the Intel i350 in it. That also allows me to upgrade to the 10GBe NIC when I move to an area with fiber.
- With i350 Netmap works by default, no tinckering from your side whatsoever?
- I don't mind recompiling the kernel, but your steps from that thread are accurate?
- I am asking you because you said it's a lot of waiting, trial and error,etc, and I don't want to reach step 5 for example, and see it's "a no go", but if Luigi explained it to you, then it must work, right?
- Did you got the chance to do a speed test on Fiber?
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
With i350 Netmap works by default, no tinckering from your side whatsoever?
Yes, no problem.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I don't mind recompiling the kernel, but your steps from that thread are accurate?
That's the instructions given to me by Vincenzo.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
Did you got the change to do a speed test on Fiber?
Not yet ... I am planning on moving in December to an area with fiber.
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I am asking you because you said it's a lot of waiting, trial and error,etc, and I don't want to reach step 5 for example, and see it's "a no go", but if Luigi explained it to you, then it must work, right?
As I had said earlier, Vincenzo went out-of-his way to explain everything to me and it is correct as far as I know. Also, I was very lucky to find that Thunderbolt 2 PCI enclosure used on eBay for $78.
-
4.0.1-k is the ixgbe driver version used in pfSense 2.5. You can check the source:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/if_ix.c#L50Steve
-
@stephenw10 said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
4.0.1-k is the ixgbe driver version used in pfSense 2.5. You can check the source:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/if_ix.c#L50Steve
I don't see my chipset in the the source list, maybe that's the issue.
I saw that the version is 4.0.1-k, but I don't find this version on Intel site, that was my dilemma. Before you gave me the link to the github, I couldn't track any changes.
What versioning scheme we have for this drivers ? How can I compare this driver version number to the Intel official site?
I mean I know FreeBSD implements alot of bugfixes, and other optimisations, but I though I could see something like this:As an example:
FreeBSD 4.0.1-k driver contains the code from Intel driver version 3.3.10 plus the following optimisations,etc
This way I could clearly understand if the driver is old or not, if it's compatible, etc
Do you know where to look for this?
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
vendor=0x8086 device=0x15e5
It is listed though: https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/ixgbe_type.h#L146
That is the code it's running.Steve
-
@stephenw10 I thought ixbge was Chelsio's ... no?
-
ixgbe = intel 10 GbE
i= intel
x = 10
gbe = GbE
GbE= Gigabit Ethernet -
Yup, Chelsio uses cxgbe for the same reasons.
Steve
-
@stephenw10 said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
vendor=0x8086 device=0x15e5
It is listed though: https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/ixgbe_type.h#L146
That is the code it's running.Steve
Thank you for pointing me to the code. But I'm not trying to do a code audit.
What I want is, to be able to correlate a Intel driver release with FreeBSD included driver.
There are any standard release notes? (besides fixing this and that, in git)For example, for IXGBE driver for Linux I have a clear release notes that states that X553 chipset is supported.
Please check the description here:
https://downloadcenter.intel.com/download/14687/Intel-Network-Adapter-Driver-for-PCIe-Intel-10-Gigabit-Ethernet-Network-Connections-Under-Linux-
For the upstream driver, version 4.0.1-k doesn't tell me anything. How can I correlate that version to anything?
I don't think if I should bother you with this, because it's not a pfSense issue, but I need to understand this in order to be able find the root cause.
Thank you
-
I got a very interesting response from Intel, after I asked if their latest release 3.3.10 supports X553 chipsets.
To give this response as an real life case scenario, how can I track if the code from Intel's driver release 3.3.14, is included or will be included with FreeBSD upstream driver?
Thank you
-
By comparing the Netgate XG7100 box https://store.netgate.com/pfSense/XG-7100-desktop.aspx
with mine
https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F
I see it has the same configuration as in SOC, RAM, and I hope NICS.
Also it is recommended for Suricata and Snort.
When I bought mine, I was looking over the Netgate XG7100 configuration, to find something close, and I think the Supermicro board is at close at it gets.
My question is, why the Supermicro board has issues with Netmap, and I got speeds as low as 150 Mbs/s?
The speeds are the same with Netgate box? Is there any custom Intel driver for it? Should I need to tweak something?
I don't think that Netgate's box transfer speeds are bad as Supermicro's box.
Any opinion on this?Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
By comparing the Netgate XG7100 box https://store.netgate.com/pfSense/XG-7100-desktop.aspx
with mine
https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F
I see it has the same configuration as in SOC, RAM, and I hope NICS.
Also it is recommended for Suricata and Snort.
When I bought mine, I was looking over the Netgate XG7100 configuration, to find something close, and I think the Supermicro board is at close at it gets.
My question is, why the Supermicro board has issues with Netmap, and I got speeds as low as 150 Mbs/s?
The speeds are the same with Netgate box? Is there any custom Intel driver for it? Should I need to tweak something?
I don't think that Netgate's box transfer speeds are bad as Supermicro's box.
Any opinion on this?Thank you
There is a very high probability that if the Netgate box has the same NICs as your box, then it will have the same netmap issues. What the "recommended for Snort and Suricata" statement really means is the box has enough horsepower (RAM and CPU) to handle a robust rule set. So far as I know, there are no detailed speed tests performed using Inline IPS Mode to determine how that mode functions with various hardware.
The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.
-
@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.
All the users that are using pfSense will have nothing but a great respect for you as a person, and for your contribution.
But as it was with iOS and Android, Apple had just one phone released every year, and it was easy to customize and optimize it. The same thing here, I don't think it's Netgate job to customize third party boards, even if they're not Chinese knock offs, but their boxes I would like to believe they are doing some tinkering and optimizations, maybe compiling some drivers from time to time, it's their hardware. If I remember correctly a few years back the Denverton platform was supported only on Netgate's hardware.So what am I looking for is, maybe some steps, to be able to compile an Intel driver with iflib support? On Linux I know how to compile a kernel, it shouldn't be that hard on FreeBSD...
As for pointing fingers...for my discussions with Intel, FreeBSD and some guys from Netmap, I tend to believe that FreeBSD should update the driver versions, with iflib support. It works on Linux, it worked before FreeBSD 11.3.
Unfortunately I cannot find a Bill Meeks on FreeBSD forums.
Any further help, links to where I should ask more questions will be much appreciated. I'm not referring only to you Bill, I'm asking in general.
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.
All the users that are using pfSense will have nothing but a great respect for you as a person, and for your contribution.
But as it was with Apple and Android, Apple had just one phone released every year, and it was easy to customize and optimize it. The same thing here, I don't think it's Netgate job to customize third party boards, even if they're not Chinese knock offs, but their boxes I would like to believe they are doing some tinkering, it's they're hardware.So what am I looking for is maybe some steps, to be able to compile an Intel driver with iflib support? On Linux I know how to compile a kernel, it shouldn't be that hard on FreeBSD...
As for pointing fingers...for my discussions with Intel, FreeBSD and some guys from Netmap, I tend to believe that FreeBSD should update the drivers versions, with iflib support. It works on Linux, it worked before FreeBSD 11.3.
Unfortunately I cannot find a Bill Meeks on FreeBSD forums.
Any further help, links to where I should ask more questions will be much appreciated. I'm not referring to you Bill, I'm asking in general.
Thank you
The pfSense source is available on Github here: https://github.com/pfsense/FreeBSD-src. There you will find the majority of what you need to compile a pfSense FreeBSD kernel. There are branches for both 2.4.5 and 2.5 at that link. Now setting up a builder is no trivial task and requires quite a bit of FreeBSD compiler "foo" to get it all working. You start by building a FreeBSD bare-bones machine and then create a pfSense "builder" on top of that. You will have to make some manual edits to various shell scripts and conf files to get it working. Several users over the years have tried this. A select few have been successful. Most have not. I have never attempted to build a full kernel. All I do is use my builder to create packages via Poudriere. That's a lot easier to set up.
The pfSense team has never shown much interest in hardware driver development, especially when it relates to specific packages. So while they would make sure a given Intel NIC driver worked properly in their custom hardware for normal operations, they would not necessarily invest the time and effort into making it 100% netmap compatible as that would only benefit the tiny percentage of users that might install and use one of the IDS/IPS packages with inline IPS mode. Without Suricata or Snort (with inline mode) installed, nothing in pfSense uses netmap, thus there is no big incentive for them to spend time and effort making some NIC driver work with netmap.
I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed. And as you have noted, the real issue seems to sit with the new iflib stuff in FreeBSD 12.1.
-
@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed.
So even if, in the end I will buy support from Netgate for this particular issue, will it matter? Or if you're not the proper person to respond, I will leave this question pending.
It's very frustrating when something worked and then it will stop because some driver refactoring.
I will continue on FreeBSD forums, and update if I have something useful.
Thank you
-
@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed.
So even if, in the end I will buy support from Netgate for this particular issue, will it matter? Or if you're not the proper person to respond, I will leave this question pending.
It's very frustrating when something worked and then it will stop because some driver refactoring.
I will continue on FreeBSD forums, and update if I have something useful.
Thank you
I have no inside knowledge nor any special influence with the pfSense team. I'm just one of several volunteer package maintainers.
I don't think they just don't care, but rather they have a lot of things "on the stove" commercial wise at the moment with their appliances and TNSR and they don't have a lot of extra time or money to spend working on FreeBSD drivers unless there was a decent profit opportunity. And profit opportunity is hard to come by when you are talking about free open-source software ... .
-
Sure, I understand. I'm glad that pfSense exists for free
-
@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:
they don't have a lot of extra time or money to spend working on FreeBSD drivers unless there was a decent profit opportunity. And profit opportunity is hard to come by when you are talking about free open-source software ... .
Good reality check, info here ... thanks for sharing.
-
I did some 4 speedtests using iperf on both pfSense versions.
Tests were done on pfSense 2.4.5-p1 and pfSense 2.5.0 clean installsiperf server was started on pfSense interface, and the client was a local Linux host, on the same LAN as pfSense
pfSense was installed on bare metal, no virtualization was used.
The results are as follows:
Test Results for pfSense 2.4.5-p1
With In-kernel Driver 3.3.12-k NETMAP disabled
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 289 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 48430 connected with 172.18.0.12 port 5201 write failed: Broken pipe [ ID] Interval Transfer Bandwidth [ 3] 0.0- 8.1 sec 885 MBytes 911 Mbits/sec
With Suricata enabled - Netmap Native mode
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 153 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 47494 connected with 172.18.0.12 port 5201 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 348 MBytes 291 Mbits/sec
With Intel-ix-kmod driver 3.3.14 NETMAP disabled
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 298 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 47070 connected with 172.18.0.12 port 5201 write failed: Broken pipe [ ID] Interval Transfer Bandwidth [ 3] 0.0- 8.0 sec 885 MBytes 923 Mbits/sec
With Suricata enabled and Intel-ix-kmod driver 3.3.14 - Netmap Native mode
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 162 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 47262 connected with 172.18.0.12 port 5201 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 349 MBytes 292 Mbits/sec
Test Results for pfSense 2.5.0 - latest snapshot
With In-kernel Driver 4.0.1-k NETMAP disabled
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 280 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 50368 connected with 172.18.0.12 port 5201 write failed: Broken pipe [ ID] Interval Transfer Bandwidth [ 3] 0.0- 8.5 sec 885 MBytes 876 Mbits/sec
With Suricata enabled - Netmap Native mode
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 187 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 50376 connected with 172.18.0.12 port 5201 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 336 MBytes 282 Mbits/sec
With Intel-ix-kmod driver 3.3.14 NETMAP disabled
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 187 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 50444 connected with 172.18.0.12 port 5201 write failed: Broken pipe [ ID] Interval Transfer Bandwidth [ 3] 0.0- 8.2 sec 885 MBytes 905 Mbits/sec
With Suricata enabled and Intel-ix-kmod driver 3.3.14 - Netmap Native mode
------------------------------------------------------------ Client connecting to 172.18.0.12, TCP port 5201 TCP window size: 153 KByte (default) ------------------------------------------------------------ [ 3] local 172.18.0.10 port 50530 connected with 172.18.0.12 port 5201 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 340 MBytes 285 Mbits/sec
As we can see after Netmap ( or in my case Suricata, that will start Netmap also) is started the speed penalty is immense.