Netmap not supported for Intel X553 driver in pfSense 2.5.0

NRgia

Why we must wait for upstream, the driver and the framework it's not open source? Or maybe not the Intel driver.
It requires recompilation of the kernel?
There is no new compatible driver with iflib from Intel?

I noticed that you talked with Luigi. I sent an email to him and he introduced me to two of his friends: Vincenzo Maffione and Giuseppe Lettieri. Should I go further with asking them for a solution, or your investigation with Luigi points to the conclusion that the Netmap implementation is fine?

What about the parameters that we used to tune for getting the right buffers, queues, etc. My understanding is that all of those have different names now, and we have to tune them through iflib, before Intel driver is even loaded. All the tutorials are now obsolete?

I don't think it's ok for us to play a waiting game here, we may see the final version omitting this issue, due to lack of upstream implementation

I also started a topic on FreeBSD forums here https://forums.freebsd.org/threads/intel-x553-driver-support-for-freebsd-12-1.75588/ maybe you can drop a line there also, someone may see it.

I will also sent this issue to FreeBSD e-mail group freebsd-net@freebsd.org if you didn't do this already

Thank you

NollipfSense

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

It requires recompilation of the kernel?

Yes, I wish I knew how to do it ... it can take up to 18Hrs to compile. The compatible driver is the Pro1000 which I am using now.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

he introduced me to two of his friends: Vincenzo Maffione and Giuseppe Lettieri

I had communicated with Vincenzo ... cool dude, went out-of-his way to explain the situation.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I don't think it's ok for us to play a waiting game here, we may see the final version omitting this issue, due to lack of upstream implementation

That why I believe it is pfSense responsibility to make sure releases have the latest drivers for NIC and stop leaving it up to FreeBSD ... without the NIC, the firewall is useless. Unfortunately, addressing the issue on FreeBSD forum is a moot point because they think or believe it's pfSense responsibility. They don't encourage discussion on pfSense. Pretty soon pfSense 2.5RC will be out; so, we need folks making noise.

NRgia

@NollipfSense Thank for your input, but I'm not following on the following:

@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

You cannot. The problem is currently FreeBSD 12.0 and 12.1 is still using the old Intel Pro1000 driver instead of Intel 25 driver released in January 2020.

How do you know what driver version FreeBSD 12 is using? All I can see is:

dev.ix.3.iflib.driver_version: 4.0.1-k
dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver

By 25 version of the driver I think you are referring to this ?
https://downloadcenter.intel.com/download/22283/Intel-Ethernet-Adapter-Complete-Driver-Pack

But if go for PRO 1000 specifically you will find that the latest version for FreeBSD is 2.5.14 or 7.7.8 depending the card
Intel Pro 1000 version.png

For my chipset X553 it uses PROXGB driver:
Intrel PROXGB.png

and the last version is from last year:
https://downloadcenter.intel.com/download/14688/Intel-Network-Adapters-Driver-for-PCIe-10-Gigabit-Network-Connections-Under-FreeBSD-?wapkw=intel%20x550%20network

So I think it should have been included by now, but I can't tell because the system reports 4.0.1-k.

What is your status now, are you using NETMAP in emulated mode, not at all and you are waiting for this to get fixed?

Thank you

NollipfSense

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

How do you know what driver version FreeBSD 12 is using?

I have been trying to remember the command since you had sent me a message.

[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 grep
usage: dmesg [-ac] [-M core [-N system]]
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 | grep
usage: dmesg [-ac] [-M core [-N system]]
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root:

I know I had used: dmesg

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

What is your status now, are you using NETMAP in emulated mode, not at all and you are waiting for this to get fixed?

Yes, I am using Netmap; however, it in default mode and not in-emulated mode. My NIC is the Intel i350, and I am using Netmap on WAN - Suricata as well as on LAN - Snort, both in-line mode. That's why I had switched to pfSense 2.5.

NRgia

@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

How do you know what driver version FreeBSD 12 is using?

I have been trying to remember the command since you had sent me a message.

[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 grep
usage: dmesg [-ac] [-M core [-N system]]
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: dmesg igb0 | grep
usage: dmesg [-ac] [-M core [-N system]]
Usage: grep [OPTION]... PATTERN [FILE]...
Try `grep --help' for more information.
[2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root:

I know I had used: dmesg

Maybe I should've told you in advance, I already tried dmesg, but I appreciate your intention to help

If I run : dmesg | grep ix3 I will get:

[2.5.0-DEVELOPMENT][root@Entaro.Blueshift]/root: dmesg | grep ix3
ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver> mem 0x7d7f400000-0x7d7f5fffff,0x7d7f800000-0x7d7f803fff at device 0.1 on pci7
ix3: Using 2048 TX descriptors and 2048 RX descriptors
ix3: Using 4 RX queues 4 TX queues
ix3: Using MSI-X interrupts with 5 vectors
ix3: allocated for 4 queues
ix3: allocated for 4 rx queues
ix3: Ethernet address: ac:1f:6b:45:fa:8b
ix3: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: link state changed to UP
ix3: link state changed to DOWN
ix3: link state changed to UP

So I will not get anything.
The only way I can get something is using sysctl like this sysctl dev.ix.3
The result is:

ev.ix.3.iflib.driver_version: 4.0.1-k

dev.ix.3.%parent: pci7

dev.ix.3.%pnpinfo: vendor=0x8086 device=0x15e5 subvendor=0x8086 subdevice=0x0000 class=0x020000

dev.ix.3.%location: slot=0 function=1 dbsf=pci0:8:0:1 handle=\_SB_.PCI0.VRP1.LAN3

dev.ix.3.%driver: ix

dev.ix.3.%desc: Intel(R) PRO/10GbE PCI-Express Network Driver

Yes, I am using Netmap; however, it in default mode and not in-emulated mode. My NIC is the Intel i350, and I am using Netmap on WAN - Suricata as well as on LAN - Snort, both in-line mode. That's why I had switched to pfSense 2.5.

But if you are using Netmap in NATIVE mode, what is your issue then ? Or it got fixed after you updated to FreeBSD 12.1 by default? What speeds do you achieve?

My issue is that I have a very high speed penalty.

With FreeBSD 12.1 default driver I get 150 Mbs/s, and NETMAP starts in NATIVE mode
If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.

Thank you

NollipfSense

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

But if you are using Netmap in NATIVE mode, what is your issue then ?

The current driver when in iflib does not allow traffic graph to show. That's great info to have on the WebGUI at a glance. The new driver should make that happened.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.

Did you compile with the new Intel 25 driver? I take it yes ... wow that a hell of a difference.

NRgia

@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

But if you are using Netmap in NATIVE mode, what is your issue then ?

The current driver when in iflib does not allow traffic graph to show. That's great info to have on the WebGUI at a glance. The new driver will make that happened.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

If I compile my own driver I will get the same speed, but NETMAP will not start in NATIVE mode, only in emulated mode
In comparison with FreeBSD 11.2 where I got between 800-960 Mbs/s it's a huge difference.

Did you compile with the new Intel 25 driver? I take it yes ... wow that a hell of a difference.
I'm using 25.1 even, but I think you are referring to the driver package rele
On FreeBSD 11.2 it worked this way, compile the driver and override the ko in /boot/kernel/ or copy as a module in /boot/modules. Sure, adding the proper line in loader.conf.local is also needed
On FreeBSD 11.3,12.1 if I compile my own driver I will achieve nothing, because my own compilation, will not include NETMAP native support due to iflib framework, hence it will run in Emulated mode at 150 Mbs/s, and if I go with the FreeBSD 12.1 driver, NETMAP will start in Native mode, but I will achieve the same speed 150 Mbs/s, so something is not right with the driver.

I contacted you because in your thread here: https://forum.netgate.com/topic/144979/porting-bge-driver-to-iflib/5

you mentioned you had problems with NETMAP Native support and you attempted to compile some drivers.

In this context we have the same issue I think. Can you elaborate, what was your status with BGE in the end?

Thank you

NollipfSense

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

Can you elaborate, what was your status with BGE in the end?

So, I had bought an Apple Mac Mini server (2011) because I am a Mac user and I like the small form factor; however, it uses Broadcom NIC hence, the BGE. I wanted to port the driver, but it was too much work for me as a newbie to porting. So, the solution was to get a thunderbolt PCI enclosure (Akitio) and placed the Intel i350 in it. That also allows me to upgrade to the 10GBe NIC when I move to an area with fiber.

NRgia

@NollipfSense said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

Can you elaborate, what was your status with BGE in the end?

So, I had bought an Apple Mac Mini server (2011) because I am a Mac user and I like the small form factor; however, it uses Broadcom NIC hence, the BGE. I wanted to port the driver, but it was too much work for me as a newbie to porting. So, the solution was to get a thunderbolt PCI enclosure and placed the Intel i350 in it. That also allows me to upgrade to the 10GBe NIC when I move to an area with fiber.

With i350 Netmap works by default, no tinckering from your side whatsoever?
I don't mind recompiling the kernel, but your steps from that thread are accurate?
I am asking you because you said it's a lot of waiting, trial and error,etc, and I don't want to reach step 5 for example, and see it's "a no go", but if Luigi explained it to you, then it must work, right?
Did you got the chance to do a speed test on Fiber?
Thank you

NollipfSense

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

With i350 Netmap works by default, no tinckering from your side whatsoever?

Yes, no problem.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I don't mind recompiling the kernel, but your steps from that thread are accurate?

That's the instructions given to me by Vincenzo.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

Did you got the change to do a speed test on Fiber?

Not yet ... I am planning on moving in December to an area with fiber.

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I am asking you because you said it's a lot of waiting, trial and error,etc, and I don't want to reach step 5 for example, and see it's "a no go", but if Luigi explained it to you, then it must work, right?

As I had said earlier, Vincenzo went out-of-his way to explain everything to me and it is correct as far as I know. Also, I was very lucky to find that Thunderbolt 2 PCI enclosure used on eBay for $78.

stephenw10

4.0.1-k is the ixgbe driver version used in pfSense 2.5. You can check the source:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/if_ix.c#L50

Steve

NRgia

@stephenw10 said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

4.0.1-k is the ixgbe driver version used in pfSense 2.5. You can check the source:
https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/if_ix.c#L50

Steve

I don't see my chipset in the the source list, maybe that's the issue.
I saw that the version is 4.0.1-k, but I don't find this version on Intel site, that was my dilemma. Before you gave me the link to the github, I couldn't track any changes.
What versioning scheme we have for this drivers ? How can I compare this driver version number to the Intel official site?
I mean I know FreeBSD implements alot of bugfixes, and other optimisations, but I though I could see something like this:

As an example:

FreeBSD 4.0.1-k driver contains the code from Intel driver version 3.3.10 plus the following optimisations,etc

This way I could clearly understand if the driver is old or not, if it's compatible, etc

Do you know where to look for this?

Thank you

stephenw10

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

vendor=0x8086 device=0x15e5

It is listed though: https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/ixgbe_type.h#L146
That is the code it's running.

Steve

NollipfSense

@stephenw10 I thought ixbge was Chelsio's ... no?

kiokoman

ixgbe = intel 10 GbE
i= intel
x = 10
gbe = GbE
GbE= Gigabit Ethernet

stephenw10

Yup, Chelsio uses cxgbe for the same reasons.

Steve

NRgia

@stephenw10 said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

vendor=0x8086 device=0x15e5

It is listed though: https://github.com/pfsense/FreeBSD-src/blob/RELENG_2_5/sys/dev/ixgbe/ixgbe_type.h#L146
That is the code it's running.

Steve

Thank you for pointing me to the code. But I'm not trying to do a code audit.
What I want is, to be able to correlate a Intel driver release with FreeBSD included driver.
There are any standard release notes? (besides fixing this and that, in git)

For example, for IXGBE driver for Linux I have a clear release notes that states that X553 chipset is supported.

Please check the description here:

https://downloadcenter.intel.com/download/14687/Intel-Network-Adapter-Driver-for-PCIe-Intel-10-Gigabit-Ethernet-Network-Connections-Under-Linux-

For the upstream driver, version 4.0.1-k doesn't tell me anything. How can I correlate that version to anything?

I don't think if I should bother you with this, because it's not a pfSense issue, but I need to understand this in order to be able find the root cause.

Thank you

NRgia

I got a very interesting response from Intel, after I asked if their latest release 3.3.10 supports X553 chipsets.

Intel response about the driver.png

To give this response as an real life case scenario, how can I track if the code from Intel's driver release 3.3.14, is included or will be included with FreeBSD upstream driver?

Thank you

NRgia

By comparing the Netgate XG7100 box https://store.netgate.com/pfSense/XG-7100-desktop.aspx

with mine

https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

I see it has the same configuration as in SOC, RAM, and I hope NICS.

Also it is recommended for Suricata and Snort.

When I bought mine, I was looking over the Netgate XG7100 configuration, to find something close, and I think the Supermicro board is at close at it gets.

My question is, why the Supermicro board has issues with Netmap, and I got speeds as low as 150 Mbs/s?

The speeds are the same with Netgate box? Is there any custom Intel driver for it? Should I need to tweak something?

I don't think that Netgate's box transfer speeds are bad as Supermicro's box.
Any opinion on this?

Thank you

bmeeks

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

By comparing the Netgate XG7100 box https://store.netgate.com/pfSense/XG-7100-desktop.aspx

with mine

https://www.supermicro.com/en/products/motherboard/A2SDi-4C-HLN4F

I see it has the same configuration as in SOC, RAM, and I hope NICS.

Also it is recommended for Suricata and Snort.

When I bought mine, I was looking over the Netgate XG7100 configuration, to find something close, and I think the Supermicro board is at close at it gets.

My question is, why the Supermicro board has issues with Netmap, and I got speeds as low as 150 Mbs/s?

The speeds are the same with Netgate box? Is there any custom Intel driver for it? Should I need to tweak something?

I don't think that Netgate's box transfer speeds are bad as Supermicro's box.
Any opinion on this?

Thank you

There is a very high probability that if the Netgate box has the same NICs as your box, then it will have the same netmap issues. What the "recommended for Snort and Suricata" statement really means is the box has enough horsepower (RAM and CPU) to handle a robust rule set. So far as I know, there are no detailed speed tests performed using Inline IPS Mode to determine how that mode functions with various hardware.

The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.