Porting BGE Driver to IFLIB...

NollipfSense

Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/

These are the following steps:

1.	Install FreeBSD (11 or 12) on a virtual machine.
2.	Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed.
3.	Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 
4.	Go to the Netmap tool:  $ cd /netmap/apps/pkt-gen
5.	Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx
6.	Run Gmake, which converts the driver and launch cc to compile it: $ gmake
7.	copy the file and place in your pfSense production box: /usr/local/bin - $ sudo cp pkt-gen /usr/local/bin

Your NIC driver would now be supported by Netmap so you can use Suricata and soon Snort in inline mode. Please note in the above step 7, you will be copying the file from your FreeBSD VM, so you may need a jump drive to transport. Also, please note that I did not need to do this as I took another option as stated here: https://forum.netgate.com/topic/144928/thunderbolt-pcie-enclosere-and-intel-nic

NollipfSense

@NollipfSense said in Porting BGE Driver to IFLIB...:

That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/

Correction per developer:

I'm sorry, but I think there has been a misunderstanding.
There are no "tools to enable to port any driver to iflib". The end result of the steps listed there is just that you have the pkt-gen test program built and installed.
You can read about pkt-gen here https://github.com/freebsd/freebsd/blob/master/tools/tools/netmap/pkt-gen.8
In any case, you don't really need to worry about iflib and native netmap support. Netmap will work on any NIC because it has a feature called "emulated netmap adapter. This means that you can use Suricata/Bro with netmap on any NIC.

NRgia

@NollipfSense said in Porting BGE Driver to IFLIB...:

Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/

These are the following steps:

Install FreeBSD (11 or 12) on a virtual machine.

Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed.

Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap

Go to the Netmap tool: $ cd /netmap/apps/pkt-gen

Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx

Run Gmake, which converts the driver and launch cc to compile it: $ gmake

Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:
gmake error.png

Do you know what's missing?
Thanks

bmeeks

@NRgia said in Porting BGE Driver to IFLIB...:

@NollipfSense said in Porting BGE Driver to IFLIB...:
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/

These are the following steps:
1.	Install FreeBSD (11 or 12) on a virtual machine.
2.	Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed.
3.	Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 
4.	Go to the Netmap tool:  $ cd /netmap/apps/pkt-gen
5.	Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx
6.	Run Gmake, which converts the driver and launch cc to compile it: $ gmake
Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:

Do you know what's missing?
Thanks

The instructions you are following make no sense. There is no "conversion of a driver to iflib" possible by simply compiling and running pkt-gen. The pkt-gen program is just a demonstation utility (or think sample application) provided for netmap developers to test with. It just generates a packet stream for transmission using netmap. The interface parameter you provide on the command line is just to tell the utility which interface to send the packet stream out on.

To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.

If you want a sample app that more closely mimics the inline mode of the IDS/IPS packages, you want to use the bridge sample application found in the same directory tree as pkt-gen. It bridges a traffic stream between two interfaces, or an interface and the host stack, using netmap.

The particular error you are hitting during the compilation of pkt-gen is likely due to a problem with include files and macro definitions in your FreeBSD environment. Specifically it is complaining about the syntax, but I suspect the compiler is getting confused by some missing macro definitions or else missing include files. But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.

NollipfSense

I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.

NRgia

@NollipfSense said in Porting BGE Driver to IFLIB...:

I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.

You cannot just install drivers on pfSense,
you cannot do: "make install" - there is no make
Instead you compile a new driver from Intel site and then load the ko module.

The issue here is, even after the latest driver is compiled, there will be no Native Netmap support. The driver must be compiled with iflib support.

We've discussed about this on my thread here:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0

My understanding was that following the steps you explained will get me a driver with iflib support.

Compiling a new driver will get me with no Native Netmap Support

NRgia

@bmeeks said in Porting BGE Driver to IFLIB...:

To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.

If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?

But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.

I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap.

bmeeks

@NRgia said in Porting BGE Driver to IFLIB...:

@bmeeks said in Porting BGE Driver to IFLIB...:

To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.

If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?

But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.

I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap.

To fix the Intel driver will require modifying the Intel driver's C source code. You could probably find examples by looking at the FreeBSD NIC driver source code commits on Github and finding some that say "porting driver to iflib" or something similar. Trying to make the changes yourself, unless you are a seasoned C programmer familiar with FreeBSD hardware driver interfaces, will be a challenge.

I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.

bmeeks

Here is an example from the FreeBSD source code tree of the ixgbe adapter being converted to use the iflib framework: https://github.com/freebsd/freebsd/commit/4fd3548cada3b9bc2f491612804009142dea73eb. Notice that 20 source code files were changed. The link shows the diff output highlighting the changes.

NRgia

@bmeeks said in Porting BGE Driver to IFLIB...:

I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.

They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you

bmeeks

@NRgia said in Porting BGE Driver to IFLIB...:

@bmeeks said in Porting BGE Driver to IFLIB...:

I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.

They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you

Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.

bmeeks

@NRgia said in Porting BGE Driver to IFLIB...:

@bmeeks said in Porting BGE Driver to IFLIB...:

I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.

They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you

I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.

The real incentive for getting open-source hardware drivers fixed lies with the manufacturers since they need wider compatibility of their product in order to sell more of it. So the more operating systems that have support for their hardware, the better for them profit-wise. Thus the incentive to fix the netmap problem would logically rest more with Intel than FreeBSD itself.

NRgia

@bmeeks said in Porting BGE Driver to IFLIB...:

I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.

I spoke with him already:
Intel response.png

Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0

Any other replies asking for a new driver or so were met with silence.

I also contacted the FreeBSD intel-ix-kmod maintainer:
intel-ix-kmod response.png

Believe me I tried :)

NRgia

@bmeeks said in Porting BGE Driver to IFLIB...:

@NRgia said in Porting BGE Driver to IFLIB...:

@bmeeks said in Porting BGE Driver to IFLIB...:

I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.

They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you

Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.

I can see, I don't know C, only Java, and even if I knew, I must understand first what to change. But it will be time consuming, not a quick fix.

bmeeks

@NRgia said in Porting BGE Driver to IFLIB...:

@bmeeks said in Porting BGE Driver to IFLIB...:

I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.

I spoke with him already:

Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0

Any other replies asking for a new driver or so were met with silence.

I also contacted the FreeBSD intel-ix-kmod maintainer:

Believe me I tried :)

Hmm...curious since I saw where Eric specifically modified some of the ixgbe driver source files to port them to iflib. Of course that was in 2017.

NollipfSense

What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.

bmeeks

@NollipfSense said in Porting BGE Driver to IFLIB...:

What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.

@NRgia and I have been in an extensive private chat today and may be on the way to a solution. He is trying a custom compile of a new driver from the lastest ports tree.

NRgia

I have managed to compile the new driver from the new version of intel-ix-kmod that Serghey Kozlvov updated for me.

I have good news and bad news:

The good one: Netmap works in Native mode, and it seems it doesn't use iflib
The bad one: The Speed didn't improve.

Guided by @bmeeks I also did some iperf tests with Suricata stopped(Netmap also stopped) with both drivers:

The tests were done from a LOCAL linux host to pfSense:

With Intel-ix-kmod driver version 3.3.14:
Client connecting to 172.18.0.12, TCP port 5201
TCP window size: 255 KByte (default)

[ 3] local 172.18.0.10 port 44654 connected with 172.18.0.12 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 670 MBytes 562 Mbits/sec

With FreeBSD 12.1 in-kernel driver 4.0.1-k:
Client connecting to 172.18.0.12, TCP port 5201
TCP window size: 255 KByte (default)

[ 3] local 172.18.0.10 port 44654 connected with 172.18.0.12 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 670 MBytes 562 Mbits/sec

This was not relevant to pin point the culprit, and I will try tommorrow to downgrade to pfSense 2.4.5 - FreeBSD 11.3 to see what speed will I get.

I would also like to thank @bmeeks for assistance on private chat.

I also uploaded the compiled driver for reference
if_ix_ko.zip

NollipfSense

@bmeeks said in Porting BGE Driver to IFLIB...:

@NollipfSense said in Porting BGE Driver to IFLIB...:

What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.

@NRgia and I have been in an extensive private chat today and may be on the way to a solution. He is trying a custom compile of a new driver from the lastest ports tree.

Awesome ... hooray, applauding!

NollipfSense

@NRgia said in Porting BGE Driver to IFLIB...:

The good one: Netmap works in Native mode, and it seems it doesn't use iflib

That means or suggest it didn't make it into the kernel ... doesn't it? That behavior seems to be the old method that should work with FreeBSD 11.3.