Porting BGE Driver to IFLIB...
-
I am embarking on a project into a dark place without a flashlight...porting the BGE driver to the IFlib. I have very limited knowledge on software development and is more of an end user as well as an open source enthusiast. I am also a longtime Mac person who likes to re-purpose Mac hardware. I have been wanting to use Suricata, and soon Snort inline mode with Netmap but the BGE driver isn't supported. So, I contacted both the Netmap creator as well as the BGE driver creator (he hasn't looked at it for over 15yrs) and shared my intention.
That led me to look at how the VMware driver was ported here: https://reviews.freebsd.org/D18761
Of course that immediately appeared overwhelming. FreeBSD is removing or has removed several old drivers (10/100mb) or limited used drivers from version 12. Also, the BGE driver 1000mb is not in version 12. I spoke here of a plan here:(https://forum.netgate.com/topic/144928/thunderbolt-pcie-enclosere-and-intel-nic); however, I am sure others could use the BGE driver as well.
So, I have access to the BGE source code...how to start?
-
Just to update thread, the Netmap developer is helping me...will report back once it's completed. I must say it's a steep learning curve for a neophyte.
-
Nice! Is there any public discussion of this anywhere we can follow?
Steve
-
@stephenw10 said in Porting BGE Driver to IFLIB...Help!:
Nice! Is there any public discussion of this anywhere we can follow?
Steve
Well, not yet Steve...it's overwhelming to say the least...so; I am taking baby steps.
-
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
1. Install FreeBSD (11 or 12) on a virtual machine. 2. Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed. 3. Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 4. Go to the Netmap tool: $ cd /netmap/apps/pkt-gen 5. Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx 6. Run Gmake, which converts the driver and launch cc to compile it: $ gmake 7. copy the file and place in your pfSense production box: /usr/local/bin - $ sudo cp pkt-gen /usr/local/bin
Your NIC driver would now be supported by Netmap so you can use Suricata and soon Snort in inline mode. Please note in the above step 7, you will be copying the file from your FreeBSD VM, so you may need a jump drive to transport. Also, please note that I did not need to do this as I took another option as stated here: https://forum.netgate.com/topic/144928/thunderbolt-pcie-enclosere-and-intel-nic
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
Correction per developer:
I'm sorry, but I think there has been a misunderstanding.
There are no "tools to enable to port any driver to iflib". The end result of the steps listed there is just that you have the pkt-gen test program built and installed.
You can read about pkt-gen here https://github.com/freebsd/freebsd/blob/master/tools/tools/netmap/pkt-gen.8
In any case, you don't really need to worry about iflib and native netmap support. Netmap will work on any NIC because it has a feature called "emulated netmap adapter. This means that you can use Suricata/Bro with netmap on any NIC. -
@NollipfSense said in Porting BGE Driver to IFLIB...:
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
- Install FreeBSD (11 or 12) on a virtual machine.
- Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed.
- Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap
- Go to the Netmap tool: $ cd /netmap/apps/pkt-gen
- Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx
- Run Gmake, which converts the driver and launch cc to compile it: $ gmake
Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:
Do you know what's missing?
Thanks -
@NRgia said in Porting BGE Driver to IFLIB...:
@NollipfSense said in Porting BGE Driver to IFLIB...:
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
1. Install FreeBSD (11 or 12) on a virtual machine. 2. Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed. 3. Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 4. Go to the Netmap tool: $ cd /netmap/apps/pkt-gen 5. Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx 6. Run Gmake, which converts the driver and launch cc to compile it: $ gmake
Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:
Do you know what's missing?
ThanksThe instructions you are following make no sense. There is no "conversion of a driver to iflib" possible by simply compiling and running
pkt-gen
. Thepkt-gen
program is just a demonstation utility (or think sample application) provided for netmap developers to test with. It just generates a packet stream for transmission using netmap. The interface parameter you provide on the command line is just to tell the utility which interface to send the packet stream out on.To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If you want a sample app that more closely mimics the inline mode of the IDS/IPS packages, you want to use the
bridge
sample application found in the same directory tree aspkt-gen
. It bridges a traffic stream between two interfaces, or an interface and the host stack, using netmap.The particular error you are hitting during the compilation of
pkt-gen
is likely due to a problem with include files and macro definitions in your FreeBSD environment. Specifically it is complaining about the syntax, but I suspect the compiler is getting confused by some missing macro definitions or else missing include files. But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib. -
I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.
You cannot just install drivers on pfSense,
you cannot do: "make install" - there is no make
Instead you compile a new driver from Intel site and then load the ko module.The issue here is, even after the latest driver is compiled, there will be no Native Netmap support. The driver must be compiled with iflib support.
We've discussed about this on my thread here:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0My understanding was that following the steps you explained will get me a driver with iflib support.
Compiling a new driver will get me with no Native Netmap Support
-
@bmeeks said in Porting BGE Driver to IFLIB...:
To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?
But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.
I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap. -
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?
But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.
I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap.To fix the Intel driver will require modifying the Intel driver's C source code. You could probably find examples by looking at the FreeBSD NIC driver source code commits on Github and finding some that say "porting driver to iflib" or something similar. Trying to make the changes yourself, unless you are a seasoned C programmer familiar with FreeBSD hardware driver interfaces, will be a challenge.
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
-
Here is an example from the FreeBSD source code tree of the ixgbe adapter being converted to use the iflib framework: https://github.com/freebsd/freebsd/commit/4fd3548cada3b9bc2f491612804009142dea73eb. Notice that 20 source code files were changed. The link shows the
diff
output highlighting the changes. -
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
The real incentive for getting open-source hardware drivers fixed lies with the manufacturers since they need wider compatibility of their product in order to sell more of it. So the more operating systems that have support for their hardware, the better for them profit-wise. Thus the incentive to fix the netmap problem would logically rest more with Intel than FreeBSD itself.
-
@bmeeks said in Porting BGE Driver to IFLIB...:
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
I spoke with him already:
Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0Any other replies asking for a new driver or so were met with silence.
I also contacted the FreeBSD intel-ix-kmod maintainer:
Believe me I tried :)
-
@bmeeks said in Porting BGE Driver to IFLIB...:
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.
I can see, I don't know C, only Java, and even if I knew, I must understand first what to change. But it will be time consuming, not a quick fix.
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
I spoke with him already:
Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0Any other replies asking for a new driver or so were met with silence.
I also contacted the FreeBSD intel-ix-kmod maintainer:
Believe me I tried :)
Hmm...curious since I saw where Eric specifically modified some of the ixgbe driver source files to port them to iflib. Of course that was in 2017.
-
What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.
@NRgia and I have been in an extensive private chat today and may be on the way to a solution. He is trying a custom compile of a new driver from the lastest ports tree.
-
I have managed to compile the new driver from the new version of intel-ix-kmod that Serghey Kozlvov updated for me.
I have good news and bad news:
The good one: Netmap works in Native mode, and it seems it doesn't use iflib
The bad one: The Speed didn't improve.Guided by @bmeeks I also did some iperf tests with Suricata stopped(Netmap also stopped) with both drivers:
The tests were done from a LOCAL linux host to pfSense:
- With Intel-ix-kmod driver version 3.3.14:
Client connecting to 172.18.0.12, TCP port 5201
TCP window size: 255 KByte (default)
[ 3] local 172.18.0.10 port 44654 connected with 172.18.0.12 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 670 MBytes 562 Mbits/sec- With FreeBSD 12.1 in-kernel driver 4.0.1-k:
Client connecting to 172.18.0.12, TCP port 5201
TCP window size: 255 KByte (default)
[ 3] local 172.18.0.10 port 44654 connected with 172.18.0.12 port 5201
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 670 MBytes 562 Mbits/secThis was not relevant to pin point the culprit, and I will try tommorrow to downgrade to pfSense 2.4.5 - FreeBSD 11.3 to see what speed will I get.
I would also like to thank @bmeeks for assistance on private chat.
I also uploaded the compiled driver for reference
if_ix_ko.zip - With Intel-ix-kmod driver version 3.3.14:
-
@bmeeks said in Porting BGE Driver to IFLIB...:
@NollipfSense said in Porting BGE Driver to IFLIB...:
What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.
@NRgia and I have been in an extensive private chat today and may be on the way to a solution. He is trying a custom compile of a new driver from the lastest ports tree.
Awesome ... hooray, applauding!
-
@NRgia said in Porting BGE Driver to IFLIB...:
The good one: Netmap works in Native mode, and it seems it doesn't use iflib
That means or suggest it didn't make it into the kernel ... doesn't it? That behavior seems to be the old method that should work with FreeBSD 11.3.
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
@NRgia said in Porting BGE Driver to IFLIB...:
The good one: Netmap works in Native mode, and it seems it doesn't use iflib
That means or suggest it didn't make it into the kernel ... doesn't it? That behavior seems to be the old method that should work with FreeBSD 11.3.
I have asked the maintainer of intel-ix-kmod here https://www.freshports.org/net/intel-ix-kmod to update the ports.
So at least for all the cards that are using this driver we now have 2 options to run NETMAP in Native mode:
- Compile from intel-ix-kmod port- And NETMAP will not use iflib
- Use the in-kernel driver, which may be old, that uses iflib
If I try the third version, downloading the driver from Intel site directly, and then compile, Netmap will not start in Native mode.
So you are right it will work with FreeBSD 11.3 also
-
Because I don't want to hijack @NollipfSense 's thread anymore I published the speed tests for pfSense 2.4.5-p1 and pfSense 2.5.0 on my initial thread here : https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0/40
-
@NRgia said in Porting BGE Driver to IFLIB...:
Because I don't want to hijack @NollipfSense 's thread anymore
I would not worry about it ... I left the thread open for others, such as yourself who may want to attempt the porting ... thank you for contributing!