Porting BGE Driver to IFLIB...
-
I am embarking on a project into a dark place without a flashlight...porting the BGE driver to the IFlib. I have very limited knowledge on software development and is more of an end user as well as an open source enthusiast. I am also a longtime Mac person who likes to re-purpose Mac hardware. I have been wanting to use Suricata, and soon Snort inline mode with Netmap but the BGE driver isn't supported. So, I contacted both the Netmap creator as well as the BGE driver creator (he hasn't looked at it for over 15yrs) and shared my intention.
That led me to look at how the VMware driver was ported here: https://reviews.freebsd.org/D18761
Of course that immediately appeared overwhelming. FreeBSD is removing or has removed several old drivers (10/100mb) or limited used drivers from version 12. Also, the BGE driver 1000mb is not in version 12. I spoke here of a plan here:(https://forum.netgate.com/topic/144928/thunderbolt-pcie-enclosere-and-intel-nic); however, I am sure others could use the BGE driver as well.
So, I have access to the BGE source code...how to start?
-
Just to update thread, the Netmap developer is helping me...will report back once it's completed. I must say it's a steep learning curve for a neophyte.
-
Nice! Is there any public discussion of this anywhere we can follow?
Steve
-
@stephenw10 said in Porting BGE Driver to IFLIB...Help!:
Nice! Is there any public discussion of this anywhere we can follow?
Steve
Well, not yet Steve...it's overwhelming to say the least...so; I am taking baby steps.
-
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
1. Install FreeBSD (11 or 12) on a virtual machine. 2. Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed. 3. Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 4. Go to the Netmap tool: $ cd /netmap/apps/pkt-gen 5. Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx 6. Run Gmake, which converts the driver and launch cc to compile it: $ gmake 7. copy the file and place in your pfSense production box: /usr/local/bin - $ sudo cp pkt-gen /usr/local/bin
Your NIC driver would now be supported by Netmap so you can use Suricata and soon Snort in inline mode. Please note in the above step 7, you will be copying the file from your FreeBSD VM, so you may need a jump drive to transport. Also, please note that I did not need to do this as I took another option as stated here: https://forum.netgate.com/topic/144928/thunderbolt-pcie-enclosere-and-intel-nic
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
Correction per developer:
I'm sorry, but I think there has been a misunderstanding.
There are no "tools to enable to port any driver to iflib". The end result of the steps listed there is just that you have the pkt-gen test program built and installed.
You can read about pkt-gen here https://github.com/freebsd/freebsd/blob/master/tools/tools/netmap/pkt-gen.8
In any case, you don't really need to worry about iflib and native netmap support. Netmap will work on any NIC because it has a feature called "emulated netmap adapter. This means that you can use Suricata/Bro with netmap on any NIC. -
@NollipfSense said in Porting BGE Driver to IFLIB...:
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
- Install FreeBSD (11 or 12) on a virtual machine.
- Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed.
- Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap
- Go to the Netmap tool: $ cd /netmap/apps/pkt-gen
- Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx
- Run Gmake, which converts the driver and launch cc to compile it: $ gmake
Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:
Do you know what's missing?
Thanks -
@NRgia said in Porting BGE Driver to IFLIB...:
@NollipfSense said in Porting BGE Driver to IFLIB...:
Just another update to this thread to share what I had learned. The first suggestion was for me to look at how the vmx(4) was ported to the IFLib: https://reviews.freebsd.org/D18761 That quickly became extremely overwhelming to say the least. That’s when the Netmap developer pointed out that he had provided tools to enable any driver to port to the IFLib, which supports Netmap…see man page: https://www.unix.com/man-page/freebsd/4/netmap/
These are the following steps:
1. Install FreeBSD (11 or 12) on a virtual machine. 2. Install these packages: Git, Gmake, and Gcc or Clang…note, if you install FreeBSD with SCR Tree, all those would already installed. 3. Get the Netmap tools: $ git clone https://github.com/luigirizzo/netmap 4. Go to the Netmap tool: $ cd /netmap/apps/pkt-gen 5. Run pkt-gen on NIC driver, example, in my case it is the Broadcom driver bge: #pkt-gen -I bge0 -f rx 6. Run Gmake, which converts the driver and launch cc to compile it: $ gmake
Hello @NollipfSense I tried to use your tutorial, for ixgbe driver.
I reached steps 6 and I have an issue, the commad pkt-gen -I ix0 -f rx or pkt-gen -I ixgbe0 -f rx will just print a list of pkt-gen commands. I tried will lower case "i" also.
If I skip to step 7 I will get:
Do you know what's missing?
ThanksThe instructions you are following make no sense. There is no "conversion of a driver to iflib" possible by simply compiling and running
pkt-gen
. Thepkt-gen
program is just a demonstation utility (or think sample application) provided for netmap developers to test with. It just generates a packet stream for transmission using netmap. The interface parameter you provide on the command line is just to tell the utility which interface to send the packet stream out on.To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If you want a sample app that more closely mimics the inline mode of the IDS/IPS packages, you want to use the
bridge
sample application found in the same directory tree aspkt-gen
. It bridges a traffic stream between two interfaces, or an interface and the host stack, using netmap.The particular error you are hitting during the compilation of
pkt-gen
is likely due to a problem with include files and macro definitions in your FreeBSD environment. Specifically it is complaining about the syntax, but I suspect the compiler is getting confused by some missing macro definitions or else missing include files. But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib. -
I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.
-
@NollipfSense said in Porting BGE Driver to IFLIB...:
I thought in his case, there is a driver ... the new Intel 25 released January 2020. He just need to install the new driver ... he won't be creating one so to speak.
You cannot just install drivers on pfSense,
you cannot do: "make install" - there is no make
Instead you compile a new driver from Intel site and then load the ko module.The issue here is, even after the latest driver is compiled, there will be no Native Netmap support. The driver must be compiled with iflib support.
We've discussed about this on my thread here:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0My understanding was that following the steps you explained will get me a driver with iflib support.
Compiling a new driver will get me with no Native Netmap Support
-
@bmeeks said in Porting BGE Driver to IFLIB...:
To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?
But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.
I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap. -
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
To actually convert a driver to use iflib, you will need to locate the driver's C source code in the FreeBSD source tree and make required changes there to replace certain routines with their iflib equivalents. This is not a trivial task, and is certainly not something an automated process is likely to accomplish for any driver.
If it can't be done by an automated process, then it requires, make config scripts, or tinkering with the driver code itself?
But I can tell you that you are wasting your time there within the netmap source code tree. The problems you are experiencing with slow speeds are within the NIC drivers themselves and their corresponding C source code. That's where the changes have to be made to convert over to iflib.
I can see by know that I hit a dead end, but what can I do, at least I'm trying to find a workaround at least.
My other choice will be to quit using Netmap.To fix the Intel driver will require modifying the Intel driver's C source code. You could probably find examples by looking at the FreeBSD NIC driver source code commits on Github and finding some that say "porting driver to iflib" or something similar. Trying to make the changes yourself, unless you are a seasoned C programmer familiar with FreeBSD hardware driver interfaces, will be a challenge.
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
-
Here is an example from the FreeBSD source code tree of the ixgbe adapter being converted to use the iflib framework: https://github.com/freebsd/freebsd/commit/4fd3548cada3b9bc2f491612804009142dea73eb. Notice that 20 source code files were changed. The link shows the
diff
output highlighting the changes. -
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
The real incentive for getting open-source hardware drivers fixed lies with the manufacturers since they need wider compatibility of their product in order to sell more of it. So the more operating systems that have support for their hardware, the better for them profit-wise. Thus the incentive to fix the netmap problem would logically rest more with Intel than FreeBSD itself.
-
@bmeeks said in Porting BGE Driver to IFLIB...:
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
I spoke with him already:
Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0Any other replies asking for a new driver or so were met with silence.
I also contacted the FreeBSD intel-ix-kmod maintainer:
Believe me I tried :)
-
@bmeeks said in Porting BGE Driver to IFLIB...:
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I suggest seeing if you can find the driver's maintainer in FreeBSD and find out from him or her why the 12.1 FreeBSD branch contains an older or "bad" version of that particular driver.
They aren't very responsive or helpful. I can throw some names here also from Intel, but they don't seem to care. I will try some more. Thank you
Checkout the Github mirror link I posted. You will see that making the changes is not a trivial task.
I can see, I don't know C, only Java, and even if I knew, I must understand first what to change. But it will be time consuming, not a quick fix.
-
@NRgia said in Porting BGE Driver to IFLIB...:
@bmeeks said in Porting BGE Driver to IFLIB...:
I think maybe I saw this guy referenced in some of your earlier posts, but he appears to be the active maintainer for the NIC stuff in FreeBSD: https://github.com/ricera. His name is Eric Joyner from Hillsboro, Oregon. Since Oregon is where Intel's research and development is located, I'm assuming this guy is possibly an Intel employee.
I spoke with him already:
Maybe you saw him mentioned in my other thread here, in the first posts:
https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0Any other replies asking for a new driver or so were met with silence.
I also contacted the FreeBSD intel-ix-kmod maintainer:
Believe me I tried :)
Hmm...curious since I saw where Eric specifically modified some of the ixgbe driver source files to port them to iflib. Of course that was in 2017.
-
What a dilemma this driver issue is, indeed! Drives me up the wall ... I can clearly see that pfSense hands-off approach is not against us ... it came with the building foundation. We should all tell Intel we're returning the hardware.