Netmap not supported for Intel X553 driver in pfSense 2.5.0

NRgia

@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.

All the users that are using pfSense will have nothing but a great respect for you as a person, and for your contribution.
But as it was with iOS and Android, Apple had just one phone released every year, and it was easy to customize and optimize it. The same thing here, I don't think it's Netgate job to customize third party boards, even if they're not Chinese knock offs, but their boxes I would like to believe they are doing some tinkering and optimizations, maybe compiling some drivers from time to time, it's their hardware. If I remember correctly a few years back the Denverton platform was supported only on Netgate's hardware.

So what am I looking for is, maybe some steps, to be able to compile an Intel driver with iflib support? On Linux I know how to compile a kernel, it shouldn't be that hard on FreeBSD...

As for pointing fingers...for my discussions with Intel, FreeBSD and some guys from Netmap, I tend to believe that FreeBSD should update the driver versions, with iflib support. It works on Linux, it worked before FreeBSD 11.3.

Unfortunately I cannot find a Bill Meeks on FreeBSD forums.

Any further help, links to where I should ask more questions will be much appreciated. I'm not referring only to you Bill, I'm asking in general.

Thank you

bmeeks

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

The problem you have uncovered is what I meant in my first post in this thread about netmap turning into a big disappointment for me. The support within the FreeBSD OS and within the various NIC drivers is hit-or-miss. And as is normal with these kinds of issues, you get some amount of finger-pointing between the FreeBSD folks, the netmap folks and the actual hardware vendors (Intel in this case). pfSense and the Suricata/Snort packages are victims in this game as they just depend on, and assume it exists, a healthy and functioning netmap kernel device working seamlessly with the hardware NIC driver.

All the users that are using pfSense will have nothing but a great respect for you as a person, and for your contribution.
But as it was with Apple and Android, Apple had just one phone released every year, and it was easy to customize and optimize it. The same thing here, I don't think it's Netgate job to customize third party boards, even if they're not Chinese knock offs, but their boxes I would like to believe they are doing some tinkering, it's they're hardware.

So what am I looking for is maybe some steps, to be able to compile an Intel driver with iflib support? On Linux I know how to compile a kernel, it shouldn't be that hard on FreeBSD...

As for pointing fingers...for my discussions with Intel, FreeBSD and some guys from Netmap, I tend to believe that FreeBSD should update the drivers versions, with iflib support. It works on Linux, it worked before FreeBSD 11.3.

Unfortunately I cannot find a Bill Meeks on FreeBSD forums.

Any further help, links to where I should ask more questions will be much appreciated. I'm not referring to you Bill, I'm asking in general.

Thank you

The pfSense source is available on Github here: https://github.com/pfsense/FreeBSD-src. There you will find the majority of what you need to compile a pfSense FreeBSD kernel. There are branches for both 2.4.5 and 2.5 at that link. Now setting up a builder is no trivial task and requires quite a bit of FreeBSD compiler "foo" to get it all working. You start by building a FreeBSD bare-bones machine and then create a pfSense "builder" on top of that. You will have to make some manual edits to various shell scripts and conf files to get it working. Several users over the years have tried this. A select few have been successful. Most have not. I have never attempted to build a full kernel. All I do is use my builder to create packages via Poudriere. That's a lot easier to set up.

The pfSense team has never shown much interest in hardware driver development, especially when it relates to specific packages. So while they would make sure a given Intel NIC driver worked properly in their custom hardware for normal operations, they would not necessarily invest the time and effort into making it 100% netmap compatible as that would only benefit the tiny percentage of users that might install and use one of the IDS/IPS packages with inline IPS mode. Without Suricata or Snort (with inline mode) installed, nothing in pfSense uses netmap, thus there is no big incentive for them to spend time and effort making some NIC driver work with netmap.

I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed. And as you have noted, the real issue seems to sit with the new iflib stuff in FreeBSD 12.1.

NRgia

@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed.

So even if, in the end I will buy support from Netgate for this particular issue, will it matter? Or if you're not the proper person to respond, I will leave this question pending.

It's very frustrating when something worked and then it will stop because some driver refactoring.

I will continue on FreeBSD forums, and update if I have something useful.

Thank you

bmeeks

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I'm going through this to illustrate why things are the way they are with regards to NIC drivers in pfSense. Not saying it is good or bad, just saying it is what it is for now. I think the bottom line is that, as of now, there is not enough of their "profitable user base" yelling about netmap support on their appliances. So there is no big push to get that fixed.

So even if, in the end I will buy support from Netgate for this particular issue, will it matter? Or if you're not the proper person to respond, I will leave this question pending.

It's very frustrating when something worked and then it will stop because some driver refactoring.

I will continue on FreeBSD forums, and update if I have something useful.

Thank you

I have no inside knowledge nor any special influence with the pfSense team. I'm just one of several volunteer package maintainers.

I don't think they just don't care, but rather they have a lot of things "on the stove" commercial wise at the moment with their appliances and TNSR and they don't have a lot of extra time or money to spend working on FreeBSD drivers unless there was a decent profit opportunity. And profit opportunity is hard to come by when you are talking about free open-source software ... .

NRgia

Sure, I understand. I'm glad that pfSense exists for free

NollipfSense

@bmeeks said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

they don't have a lot of extra time or money to spend working on FreeBSD drivers unless there was a decent profit opportunity. And profit opportunity is hard to come by when you are talking about free open-source software ... .

Good reality check, info here ... thanks for sharing.

NRgia

I did some 4 speedtests using iperf on both pfSense versions.
Tests were done on pfSense 2.4.5-p1 and pfSense 2.5.0 clean installs

iperf server was started on pfSense interface, and the client was a local Linux host, on the same LAN as pfSense

pfSense was installed on bare metal, no virtualization was used.

The results are as follows:

Test Results for pfSense 2.4.5-p1

With In-kernel Driver 3.3.12-k NETMAP disabled

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  289 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 48430 connected with 172.18.0.12 port 5201
write failed: Broken pipe
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 8.1 sec   885 MBytes   911 Mbits/sec

With Suricata enabled - Netmap Native mode

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 47494 connected with 172.18.0.12 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   348 MBytes   291 Mbits/sec

With Intel-ix-kmod driver 3.3.14 NETMAP disabled

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  298 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 47070 connected with 172.18.0.12 port 5201
write failed: Broken pipe
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 8.0 sec   885 MBytes   923 Mbits/sec

With Suricata enabled and Intel-ix-kmod driver 3.3.14 - Netmap Native mode

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  162 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 47262 connected with 172.18.0.12 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   349 MBytes   292 Mbits/sec

Test Results for pfSense 2.5.0 - latest snapshot

With In-kernel Driver 4.0.1-k NETMAP disabled

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  280 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 50368 connected with 172.18.0.12 port 5201
write failed: Broken pipe
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 8.5 sec   885 MBytes   876 Mbits/sec

With Suricata enabled - Netmap Native mode

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  187 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 50376 connected with 172.18.0.12 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   336 MBytes   282 Mbits/sec

With Intel-ix-kmod driver 3.3.14 NETMAP disabled

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  187 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 50444 connected with 172.18.0.12 port 5201
write failed: Broken pipe
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 8.2 sec   885 MBytes   905 Mbits/sec

With Suricata enabled and Intel-ix-kmod driver 3.3.14 - Netmap Native mode

------------------------------------------------------------
Client connecting to 172.18.0.12, TCP port 5201
TCP window size:  153 KByte (default)
------------------------------------------------------------
[  3] local 172.18.0.10 port 50530 connected with 172.18.0.12 port 5201
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   340 MBytes   285 Mbits/sec

As we can see after Netmap ( or in my case Suricata, that will start Netmap also) is started the speed penalty is immense.

trumee

I have a Supermicro motherboard 'A2SDi-4C-HLN4F' which uses X553 chipset. It is presently running Stable 2.4.5-p1 and Snort. Is this release affected by this issue?

bmeeks

@trumee said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I have a Supermicro motherboard 'A2SDi-4C-HLN4F' which uses X553 chipset. It is presently running Stable 2.4.5-p1 and Snort. Is this release affected by this issue?

Snort on pfSense-2.4.5 does not support netmap device operation, so no, the 2.4.5 release is not impacted. Snort on 2.4.5-RELEASE uses libpcap.

NRgia

@trumee said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I have a Supermicro motherboard 'A2SDi-4C-HLN4F' which uses X553 chipset. It is presently running Stable 2.4.5-p1 and Snort. Is this release affected by this issue?

My tests didn't include testing Snort on Stable 2.4.5. I was asked to install Snort on 2.5.0-devel by another user, to compare Snort vs Suricata, in the matter of speed and it was a little lower for me when I tested with Snort.

After some discussions with the guys that maintain Netmap, Intel drivers, Supermicro support, FreeBSD, I was directed to Suricata maintainers.

I took my time and tried various tutorials that optimize some networking parameters, but I got only small variances in performance like 30-40 Mbps.

My last try will be to have a chat with Suricata guys.
I hope they will not recommend me a Napatech card
Napatech products link , or something.
I will update if I find something of interest.

bmeeks

@NRgia said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

@trumee said in Netmap not supported for Intel X553 driver in pfSense 2.5.0:

I have a Supermicro motherboard 'A2SDi-4C-HLN4F' which uses X553 chipset. It is presently running Stable 2.4.5-p1 and Snort. Is this release affected by this issue?

My tests didn't include testing Snort on Stable 2.4.5. I was asked to install Snort on 2.5.0-devel by another user, to compare Snort vs Suricata, in the matter of speed and it was a little lower for me when I tested with Snort.

After some discussions with the guys that maintain Netmap, Intel drivers, Supermicro support, FreeBSD, I was directed to Suricata maintainers.

I took my time and tried various tutorials that optimize some networking parameters, but I got only small variances in performance like 30-40 Mbps.

My last try will be to have a chat with Suricata guys.
I hope they will not recommend me a Napatech card
Napatech products link , or something.
I will update if I find something of interest.

One issue that is likely at play with both Suricata and Snort (Snort on FreeBSD-11.x) is that on FreeBSD the netmap host stack originally exposed only a single ring. NIC drivers, on the other hand, pretty much uniformly expose multiple rings. The more rings you have, the higher the theoretical throughput can be.

The latest iteration of netmap on FreeBSD finally offers a multiple ring interface for the host stack. The host stack is the connection to the kernel itself. Most of the original implementations of netmap envisoned sending packets between two NIC interfaces directly (that is, without necessarily going through the kernel network stack). So to put this in Suricata terms, think of using two physical NICs and having Suricata sit between them policing traffic between the two NICs. In that scenario all rings available in the NIC drivers would be used.

But Suricata on pfSense needs to interract with the kernel network stack because we want to inspect traffic as it flows to and from the NIC to the pf firewall engine in the kernel. Also, we don't want to use up two valuable hardware NIC ports just to have an "in" and an "out" path. We want to use a single NIC for an interface.

Starting with FreeBSD-12 and the move to the iflib networking API, netmap now exposes a multi-ring netmap interface for the host stack. However, for the moment I don't believe Suricata 5.x is using that interface in order to maintain backwards compatibility with older netmap API versions.