netmap errors since 2.7.x

Cobrax2 · Dec 27, 2023, 7:34 AM

hi guys
I've never had problems on 2.6x, but since a few weeks when i did the upgrade, these appear in logs:

300.026045 [4335] netmap_transmit em1 full hwcur 726 hwtail 728 qlen 1021

i know they are not errors per se, but they mean that snort/suricata can't process fast enough the data, but it seems odd to happen at once, after the upgrade
also, cpu utilisation is not very high :(

any idea why this happens?
thanks
setup:
intel celeron g5905, 8gb ram, intel em gigabit nics

Cobrax2 · Dec 28, 2023, 6:39 AM

Actually the cpu never goes over 30-35% utilisation, and this happens when going over 40MB/s
So i dont think it is not capable of more, right?
The errors happen also when only watching something much less bw consuming like disney+ or netflix

bmeeks · Dec 28, 2023, 3:08 PM

Are you running Snort or Suricata on the box?

When you upgraded, did you remove the installed packages, perform the upgrade, then reinstall?

Or did you leave the packages as they were and let pfSense do its automatic update?

If you did the latter (let pfSense auto-update the packages), try removing the IDS/IPS package and then reinstalling it. That will make certain all the package files are current.

You are correct with your interpretation of that error message, but 40 megabits/second of traffic should not cause netmap to even breathe heavy , much less throw that buffer full error.

Cobrax2 · Dec 28, 2023, 3:31 PM

@bmeeks said in netmap errors since 2.7.x:

Are you running Snort or Suricata on the box?

When you upgraded, did you remove the installed packages, perform the upgrade, then reinstall?

Or did you leave the packages as they were and let pfSense do its automatic update?

If you did the latter (let pfSense auto-update the packages), try removing the IDS/IPS package and then reinstalling it. That will make certain all the package files are current.

You are correct with your interpretation of that error message, but 40 megabits/second of traffic should not cause netmap to even breathe heavy , much less throw that buffer full error.

I did automatic upgrade, had snort installed. When i discovered some time later the problem, did a manual reinstall of snort, no luck. Now i uninstalled snort, just to see if suricata has the same issue, and it does :{
The errors show at much lower bandwidth, as i said even watching netflix sometimes throw that error. But i was talking about 40 MB Megabytes there, just to show that it only brings the cpu to some 30% utilisation. Never went to 100%. The frequency though goes pretty fast to 3500 mhz. Maybe my issue is with the new power saving system? Speedsomething? Before i was with powerd and on balanced. Now tried it too and makes no difference. If i put it to max performance, the error seems to show less i think? Does the power management affect the nic too?
Thanks

bmeeks · Dec 28, 2023, 3:41 PM

I don't think the power daemon impacts the performance of a NIC. It might put the NIC in sleep mode after some extended period of inactivity, but certainly not when it was passing traffic.

Power saving mode will impact CPU clock speed.

There could be several issues at play in your setup.

First, the em NIC driver is somewhat generic in that it can work with several flavors of Intel NIC chipsets. But those individual chipsets do have internal differences in terms of queues, specific flags, and initialization requirements. It might be that the newer NIC driver code in the latest pfSense kernel (which comes from FreeBSD upstream) is not playing super well with your particular "flavor" of em NIC hardware.

Second, the Celeron is not a speed-demon chip. It only has 2 cores and 2 threads. Since it has to run pfSense itself, that does not leave a lot of capacity around for the IDS/IPS package to utilize.

What about trying another newer NIC card in the box? Intel is considered best, but a more recent Intel that FreeBSD 14 and above has excellent support for would be a better choice.

Cobrax2 · Dec 28, 2023, 3:59 PM

@bmeeks said in netmap errors since 2.7.x:

I don't think the power daemon impacts the performance of a NIC. It might put the NIC in sleep mode after some extended period of inactivity, but certainly not when it was passing traffic.

Power saving mode will impact CPU clock speed.

There could be several issues at play in your setup.

First, the em NIC driver is somewhat generic in that it can work with several flavors of Intel NIC chipsets. But those individual chipsets do have internal differences in terms of queues, specific flags, and initialization requirements. It might be that the newer NIC driver code in the latest pfSense kernel (which comes from FreeBSD upstream) is not playing super well with your particular "flavor" of em NIC hardware.

Second, the Celeron is not a speed-demon chip. It only has 2 cores and 2 threads. Since it has to run pfSense itself, that does not leave a lot of capacity around for the IDS/IPS package to utilize.

What about trying another newer NIC card in the box? Intel is considered best, but a more recent Intel that FreeBSD 14 and above has excellent support for would be a better choice.

Can i utilise the old driver somehow?
Lol, i know the cpu is not that fast, but it does the job {its a home setup, so i cant just change nics. They are intel pt i think older, yes} and i dont need that much speed anyway. But to throw errors while watching netflix? Its not 4k lol
Could the nic itself have failed?

bmeeks · Dec 28, 2023, 4:05 PM

@Cobrax2 said in netmap errors since 2.7.x:

Can i utilise the old driver somehow?

No, not with the newer kernel. Drivers are compiled for a specific kernel version. It is extraordinarily rare for an older driver to work with a newer kernel version.

@Cobrax2 said in netmap errors since 2.7.x:

Could the nic itself have failed?

I don't think so. Typically a failure would result in no link or no traffic passing at all. I can't envision a hardware failure mode that would simply result in reduced throughput.

PCI-X network cards are not very expensive. Might be worth trying a newer model NIC to see if that helps. But the better solution would be to step up to a more modern multi-core CPU at the same time.

Cobrax2 · Dec 28, 2023, 4:06 PM

@bmeeks so basically my only option would be to go back to 2.6x if i dont buy another nic. No way of using older driver?

Cobrax2 · Dec 28, 2023, 4:17 PM

@Cobrax2 But what about the cpu? Why is it not enough, if it only gets used to about 30%?

bmeeks · Dec 28, 2023, 4:18 PM

@Cobrax2 said in netmap errors since 2.7.x:

@bmeeks so basically my only option would be to go back to 2.6x if i dont buy another nic. No way of using older driver?

Correct. You could try copying over the driver from a 2.6 install, but I really doubt it would work.

Here are the relationships between pfSense Plus and pfSense CE releases and the underlying FreeBSD kernel version each is based upon. There is big jump from pfSense 2.6 CE to pfSense 2.7.2 CE.

https://docs.netgate.com/pfsense/en/latest/releases/versions.html

bmeeks · Dec 28, 2023, 4:27 PM

@Cobrax2 said in netmap errors since 2.7.x:

@Cobrax2 But what about the cpu? Why is it not enough, if it only gets used to about 30%?

The CPU is not necessarily the limit. Could be the way interrupts are being handled.

This is just the type of thing you can expect to happen as hardware ages and software moves on. Older hardware is just not as well supported. Changes made to support newer hardware can sometimes be detrimental to older hardware, but the developers either don't realize it (because they did not test the older hardware), or they make a judgement that fully supporting the new is worth sacrificing support (or optimized performance) of the old.

If you really want to keep this hardware, and it worked fine under 2.6, then install 2.6 CE and enjoy life . Yes, you won't be current, and any support issues you may have in the future will result in an immediate suggestion to "upgrade to the latest version". You just have to determine if it's worth updating to new hardware.

Cobrax2 · Dec 30, 2023, 5:39 AM

@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?

bmeeks · Dec 31, 2023, 5:29 AM

@Cobrax2 said in netmap errors since 2.7.x:

@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?

No, that error is not "normal". While an very infrequent logging of that error might happen, regular logging with traffic interruptions/slowdowns is not normal.

Have you disabled all the hardware offloading options for your NIC? That would include LRO and Checksum offloading as described here: https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html#network-interfaces.

I don't think we've discussed enabled rules, but minimizing the amount of rules will have a big impact on throughput when you have a marginal CPU.

Cobrax2 · Dec 31, 2023, 5:29 AM

@bmeeks said in netmap errors since 2.7.x:

@Cobrax2 said in netmap errors since 2.7.x:

@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?

No, that error is not "normal". While an very infrequent logging of that error might happen, regular logging with traffic interruptions/slowdowns is not normal.

Have you disabled all the hardware offloading options for your NIC? That would include LRO and Checksum offloading as described here: https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html#network-interfaces.

I don't think we've discussed enabled rules, but minimizing the amount of rules will have a big impact on throughput when you have a marginal CPU.

em2: flags=1028943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC,LOWER_UP> metric 0 mtu 1500
description: LAN
options=49120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NETMAP,HWSTATS,MEXTPG>
ether 00:1b:2184:d8
inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
inet6 fe80::21b:21ff:fecd:84d8%em2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

There are about 20-25k rules active, from em lists

bmeeks · Jan 1, 2024, 6:50 AM

@Cobrax2 said in netmap errors since 2.7.x:

There are about 20-25k rules active, from em lists

That's a somewhat high number of rules for a dual-core Celeron processor, but probably still manageable. What does the output of this command show?

grep netmap /var/log/dmesg.boot

This will print out the number of netmap queues (rings) the NIC driver supports.

Cobrax2 · Jan 1, 2024, 6:50 AM

@bmeeks said in netmap errors since 2.7.x:

@Cobrax2 said in netmap errors since 2.7.x:

There are about 20-25k rules active, from em lists

That's a somewhat high number of rules for a dual-core Celeron processor, but probably still manageable. What does the output of this command show?
grep netmap /var/log/dmesg.boot
This will print out the number of netmap queues (rings) the NIC driver supports.

Happy new year!
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024

Em0 is wan, em1 was lan before, and em2 is the new lan intel ct
Btw after just a day, the log is filled with those errors just from my wife watching disney+, so i didnt solve anything lol
Thanks!

bmeeks · Jan 1, 2024, 2:10 PM

@Cobrax2 said in netmap errors since 2.7.x:

em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024

Those particular NICs do not have many netmap queues (or rings). Most NICs today expose 4 queues for netmap to use. For example, here is the output of that command from my SG-5100 appliance:

igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
ix2: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: netmap queues/slots: TX 4/2048, RX 4/2048

There may some NIC hardware driver tunables that would improve performance a bit, but I have no idea which ones they may be for those particular NICs. Tunables are usually very hardware-specific. You could do some Google research to see if any pop up in a search.

The more queues, and the greater the number of slots in each queue, the more room the NIC has to store incoming packets. These are essentially buffers the kernel (and netmap, when using Inline IPS Mode in Suricata) can pull from. If the CPU horsepower is limited, larger queue counts with more slots per queue can help throughput.

With only a single queue on two of the NIC ports, your system has some potential choke points because of the limits of the dual-core Celeron. The problem is every single packet has to be run through the Suricata rules engine one-by-one. With lots of rules enabled, that is going to take some time. And during that Suricata processing time more packets are flying in from the wire and the NIC runs out of room to store them because the CPU has been busy running previous packets through all the Suricata rules and has not been able to keep netmap queues emptied to make room for new packets.

You can probably improve things substantially by cutting your number of enabled rules significantly. And I will be perfectly honest with you, on a home network where nearly 100% of traffic in and out is encrypted anyway (HTTPS, POP3S, IMAPS, SMTPS, etc.), running an IDS/IPS is almost pointless as it cannot look into packet payloads at all. It can only see IP header information - but no data. Your main methods of increased security on home networks are keeping all the installed software up-to-date, running a local AV client on internal hosts, and just generally paying attention to what you click on.

Cobrax2 · Jan 2, 2024, 2:24 PM

@bmeeks said in netmap errors since 2.7.x:

@Cobrax2 said in netmap errors since 2.7.x:

em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024

Those particular NICs do not have many netmap queues (or rings). Most NICs today expose 4 queues for netmap to use. For example, here is the output of that command from my SG-5100 appliance:
igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: netmap queues/slots: TX 4/1024, RX 4/1024
ix0: netmap queues/slots: TX 4/2048, RX 4/2048
ix1: netmap queues/slots: TX 4/2048, RX 4/2048
ix2: netmap queues/slots: TX 4/2048, RX 4/2048
ix3: netmap queues/slots: TX 4/2048, RX 4/2048
There may some NIC hardware driver tunables that would improve performance a bit, but I have no idea which ones they may be for those particular NICs. Tunables are usually very hardware-specific. You could do some Google research to see if any pop up in a search.

The more queues, and the greater the number of slots in each queue, the more room the NIC has to store incoming packets. These are essentially buffers the kernel (and netmap, when using Inline IPS Mode in Suricata) can pull from. If the CPU horsepower is limited, larger queue counts with more slots per queue can help throughput.

With only a single queue on two of the NIC ports, your system has some potential choke points because of the limits of the dual-core Celeron. The problem is every single packet has to be run through the Suricata rules engine one-by-one. With lots of rules enabled, that is going to take some time. And during that Suricata processing time more packets are flying in from the wire and the NIC runs out of room to store them because the CPU has been busy running previous packets through all the Suricata rules and has not been able to keep netmap queues emptied to make room for new packets.

You can probably improve things substantially by cutting your number of enabled rules significantly. And I will be perfectly honest with you, on a home network where nearly 100% of traffic in and out is encrypted anyway (HTTPS, POP3S, IMAPS, SMTPS, etc.), running an IDS/IPS is almost pointless as it cannot look into packet payloads at all. It can only see IP header information - but no data. Your main methods of increased security on home networks are keeping all the installed software up-to-date, running a local AV client on internal hosts, and just generally paying attention to what you click on.

Umm, tried to go back to 2.6.x but it seems that the old versions are unavailable for download? Wtf

bmeeks · Jan 2, 2024, 2:27 PM

@Cobrax2 said in netmap errors since 2.7.x:

Umm, tried to go back to 2.6.x but it seems that the old versions are unavailable for download? Wtf

They may not be there long, so grab a copy quickly from this link:

https://atxfiles.netgate.com/mirror/downloads/

There are 2.6.0, 2.7.0, 2.7.1, and 2.7.2 images posted at the link. Download the appropriate image for you (ISO or USB memstick) and make sure you save it in case you need to reinstall at some point in the future.

Be very careful installing/updating packages with any older version. Be sure you set the repo under SYSTEM > UPDATE > Update Settings to the appropriate version. Failure to do that will result in either the package installation failing, or worse, breaking the install completely by pulling down shared libraries compiled for newer pfSense versions.