netmap errors since 2.7.x
-
@Cobrax2 said in netmap errors since 2.7.x:
@bmeeks so basically my only option would be to go back to 2.6x if i dont buy another nic. No way of using older driver?
Correct. You could try copying over the driver from a 2.6 install, but I really doubt it would work.
Here are the relationships between pfSense Plus and pfSense CE releases and the underlying FreeBSD kernel version each is based upon. There is big jump from pfSense 2.6 CE to pfSense 2.7.2 CE.
https://docs.netgate.com/pfsense/en/latest/releases/versions.html
-
@Cobrax2 said in netmap errors since 2.7.x:
@Cobrax2 But what about the cpu? Why is it not enough, if it only gets used to about 30%?
The CPU is not necessarily the limit. Could be the way interrupts are being handled.
This is just the type of thing you can expect to happen as hardware ages and software moves on. Older hardware is just not as well supported. Changes made to support newer hardware can sometimes be detrimental to older hardware, but the developers either don't realize it (because they did not test the older hardware), or they make a judgement that fully supporting the new is worth sacrificing support (or optimized performance) of the old.
If you really want to keep this hardware, and it worked fine under 2.6, then install 2.6 CE and enjoy life
. Yes, you won't be current, and any support issues you may have in the future will result in an immediate suggestion to "upgrade to the latest version". You just have to determine if it's worth updating to new hardware. -
@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?
-
@Cobrax2 said in netmap errors since 2.7.x:
@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?
No, that error is not "normal". While an very infrequent logging of that error might happen, regular logging with traffic interruptions/slowdowns is not normal.
Have you disabled all the hardware offloading options for your NIC? That would include LRO and Checksum offloading as described here: https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html#network-interfaces.
I don't think we've discussed enabled rules, but minimizing the amount of rules will have a big impact on throughput when you have a marginal CPU.
-
@bmeeks said in netmap errors since 2.7.x:
@Cobrax2 said in netmap errors since 2.7.x:
@bmeeks well i changed the lan nic to a intel pro ct {still old, but probably newer?} And this one has 2 queues 4 workers according to suricata. It is somewhat better, the errors appear less frequent{i hope i am not imagining it}. But still there. Anything else i can do besides changing the cpu? Is it normal for it to do this?
No, that error is not "normal". While an very infrequent logging of that error might happen, regular logging with traffic interruptions/slowdowns is not normal.
Have you disabled all the hardware offloading options for your NIC? That would include LRO and Checksum offloading as described here: https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html#network-interfaces.
I don't think we've discussed enabled rules, but minimizing the amount of rules will have a big impact on throughput when you have a marginal CPU.
em2: flags=1028943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,PPROMISC,LOWER_UP> metric 0 mtu 1500
description: LAN
options=49120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,NETMAP,HWSTATS,MEXTPG>
ether 00:1b:21
84:d8
inet 192.168.10.1 netmask 0xffffff00 broadcast 192.168.10.255
inet6 fe80::21b:21ff:fecd:84d8%em2 prefixlen 64 scopeid 0x3
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>There are about 20-25k rules active, from em lists
-
@Cobrax2 said in netmap errors since 2.7.x:
There are about 20-25k rules active, from em lists
That's a somewhat high number of rules for a dual-core Celeron processor, but probably still manageable. What does the output of this command show?
grep netmap /var/log/dmesg.bootThis will print out the number of netmap queues (rings) the NIC driver supports.
-
@bmeeks said in netmap errors since 2.7.x:
@Cobrax2 said in netmap errors since 2.7.x:
There are about 20-25k rules active, from em lists
That's a somewhat high number of rules for a dual-core Celeron processor, but probably still manageable. What does the output of this command show?
grep netmap /var/log/dmesg.bootThis will print out the number of netmap queues (rings) the NIC driver supports.
Happy new year!
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024Em0 is wan, em1 was lan before, and em2 is the new lan intel ct
Btw after just a day, the log is filled with those errors just from my wife watching disney+, so i didnt solve anything lol
Thanks! -
@Cobrax2 said in netmap errors since 2.7.x:
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024Those particular NICs do not have many netmap queues (or rings). Most NICs today expose 4 queues for netmap to use. For example, here is the output of that command from my SG-5100 appliance:
igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 ix2: netmap queues/slots: TX 4/2048, RX 4/2048 ix3: netmap queues/slots: TX 4/2048, RX 4/2048There may some NIC hardware driver tunables that would improve performance a bit, but I have no idea which ones they may be for those particular NICs. Tunables are usually very hardware-specific. You could do some Google research to see if any pop up in a search.
The more queues, and the greater the number of slots in each queue, the more room the NIC has to store incoming packets. These are essentially buffers the kernel (and netmap, when using Inline IPS Mode in Suricata) can pull from. If the CPU horsepower is limited, larger queue counts with more slots per queue can help throughput.
With only a single queue on two of the NIC ports, your system has some potential choke points because of the limits of the dual-core Celeron. The problem is every single packet has to be run through the Suricata rules engine one-by-one. With lots of rules enabled, that is going to take some time. And during that Suricata processing time more packets are flying in from the wire and the NIC runs out of room to store them because the CPU has been busy running previous packets through all the Suricata rules and has not been able to keep netmap queues emptied to make room for new packets.
You can probably improve things substantially by cutting your number of enabled rules significantly. And I will be perfectly honest with you, on a home network where nearly 100% of traffic in and out is encrypted anyway (HTTPS, POP3S, IMAPS, SMTPS, etc.), running an IDS/IPS is almost pointless as it cannot look into packet payloads at all. It can only see IP header information - but no data. Your main methods of increased security on home networks are keeping all the installed software up-to-date, running a local AV client on internal hosts, and just generally paying attention to what you click on.
-
@bmeeks said in netmap errors since 2.7.x:
@Cobrax2 said in netmap errors since 2.7.x:
em0: netmap queues/slots: TX 1/1024, RX 1/1024
em1: netmap queues/slots: TX 1/1024, RX 1/1024
em2: netmap queues/slots: TX 2/1024, RX 2/1024Those particular NICs do not have many netmap queues (or rings). Most NICs today expose 4 queues for netmap to use. For example, here is the output of that command from my SG-5100 appliance:
igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 ix0: netmap queues/slots: TX 4/2048, RX 4/2048 ix1: netmap queues/slots: TX 4/2048, RX 4/2048 ix2: netmap queues/slots: TX 4/2048, RX 4/2048 ix3: netmap queues/slots: TX 4/2048, RX 4/2048There may some NIC hardware driver tunables that would improve performance a bit, but I have no idea which ones they may be for those particular NICs. Tunables are usually very hardware-specific. You could do some Google research to see if any pop up in a search.
The more queues, and the greater the number of slots in each queue, the more room the NIC has to store incoming packets. These are essentially buffers the kernel (and netmap, when using Inline IPS Mode in Suricata) can pull from. If the CPU horsepower is limited, larger queue counts with more slots per queue can help throughput.
With only a single queue on two of the NIC ports, your system has some potential choke points because of the limits of the dual-core Celeron. The problem is every single packet has to be run through the Suricata rules engine one-by-one. With lots of rules enabled, that is going to take some time. And during that Suricata processing time more packets are flying in from the wire and the NIC runs out of room to store them because the CPU has been busy running previous packets through all the Suricata rules and has not been able to keep netmap queues emptied to make room for new packets.
You can probably improve things substantially by cutting your number of enabled rules significantly. And I will be perfectly honest with you, on a home network where nearly 100% of traffic in and out is encrypted anyway (HTTPS, POP3S, IMAPS, SMTPS, etc.), running an IDS/IPS is almost pointless as it cannot look into packet payloads at all. It can only see IP header information - but no data. Your main methods of increased security on home networks are keeping all the installed software up-to-date, running a local AV client on internal hosts, and just generally paying attention to what you click on.
Umm, tried to go back to 2.6.x but it seems that the old versions are unavailable for download? Wtf
-
@Cobrax2 said in netmap errors since 2.7.x:
Umm, tried to go back to 2.6.x but it seems that the old versions are unavailable for download? Wtf
They may not be there long, so grab a copy quickly from this link:
https://atxfiles.netgate.com/mirror/downloads/
There are 2.6.0, 2.7.0, 2.7.1, and 2.7.2 images posted at the link. Download the appropriate image for you (ISO or USB memstick) and make sure you save it in case you need to reinstall at some point in the future.
Be very careful installing/updating packages with any older version. Be sure you set the repo under SYSTEM > UPDATE > Update Settings to the appropriate version. Failure to do that will result in either the package installation failing, or worse, breaking the install completely by pulling down shared libraries compiled for newer pfSense versions.
