pfBlockerNG-devel 2.2.5_30: IPv4 list: too many elements + high system load

getcom

Hello all,

last Friday I upgraded to pfSense 2.4.5 on one site and also switched to pfBlockerNG-devel.
pfSense is running on a netgate D1541 hardware which is strong enough...normally.

There are several issues:
There is one IPv4 list with ~ 1.7 million entries. With the previous pfBlockerNG version this list had sometimes over 2 million entries and I never had any problem with it.

Now I get this log entries:
There were error(s) loading the rules: /tmp/rules.debug:67: cannot define table pfB_octacryptIPv4complete_v4: too many elements. - The line in question reads [67]: table <pfB_octacryptIPv4complete_v4> persist file "/var/db/aliastables/pfB_octacryptIPv4complete_v4.txt"
@ 2020-04-01 03:23:38

This list shows 0 IP in the dashboard.

Is there a database limit in the new version?
For this list there are 110 single lists available. The effort is too high to split that into smaller lists and modify that on all sites.

Additionally there is a high CPU usage and system load up to 10 if it reloads the lists.
The process is pfctl and I saw CPU spikes up to 190%. I don`t know if this behavior comes from the larger lists or from GeoIP updates. Maybe I have to investigate to get more details of the root cause.

If the load is high, I get Nginx gateway timeouts and additionally both WAN gateways are showing a latency.
Working over VPN is then not really possible which is a problem at the moment because of Corona.
As a follow up problem other lists sometimes cannot be downloaded and I get something like this:
[ DNSBL_hpHosts - hpHosts ] Download FAIL [ 04/01/20 03:15:18 ]
[ DNSBL_ADs - hpHosts_ads ] Download FAIL [ 04/01/20 03:15:06 ]

Are these known issues with the devel version?
Is there something what I can try to do, e.g. delete the databases to refresh it?
Thank you in advance.

Ralf

Gertjan

@getcom said in pfBlockerNG-devel 2.2.5_30: IPv4 list: too many elements + high system load:

There were error(s) loading the rules

Check pfSense Advanced tab > Firewall and NAT > Firewall Max Table entries. It was raised some version ago.
You missed that ? ;)

1.7 million IP's to handle when the resolver starts or restarts ..... Systems spikes, slow DNS restart times, huge (massive) firewall rules (memory foot print) .... are you sure it's worth it ?

getcom

Firewall Max Table entries = 3 million and Im using this since four years now. This was never a problem in the past and this is one of the reasons why Im using a FreeBSD based firewall. pf can handle that and it is designed to do that. ipset/iptables in Linux will be slower and slower if using increasing IP lists, but not pf.
I never had any spikes like that before. It is a 8 core XEON CPU with 2 threads and 32 GB RAM. 12 GB is used for RAM disks and before upgrading to 2.4.5 the CPU load was 1-5% and 25% memory usage, now it is up to 100% and 65+% memory usage. I saw system loads over 20. These performance problems has nothing to do with my IP lists. In the past the lists also had 2 1/2 million entries without interesting system load.
From my perspective there is a bug in the 2.4.5 release which causes these performance problems.
Additionally there is a bug in pfBlockerNG-devel which cannot handle larger lists anymore.
Again: this was never a problem in the past with previous versions of pfS and pfB.
Additionally I never had any situation where pfSense was not reachable, IPsec VPN has problems, WAN connection has highest latency without any packet loss.
We have to investigate and find out what is going wrong. Discussions about list sizing does not help here and is also not the root cause for that.
As I read I`m not alone. There are also problems with fresh installed systems without any configured services. There are problems in CARP setups, in virtual environments and baremetal setups. It is unclear why it works for some users and for others not.

provels

@getcom said in pfBlockerNG-devel 2.2.5_30: IPv4 list: too many elements + high system load:

As a follow up problem other lists sometimes cannot be downloaded and I get something like this:
[ DNSBL_hpHosts - hpHosts ] Download FAIL [ 04/01/20 03:15:18 ]
[ DNSBL_ADs - hpHosts_ads ] Download FAIL [ 04/01/20 03:15:06 ]
Are these known issues with the devel version?

No, these lists have gone offline for the past few days.
https://forum.netgate.com/topic/151950/malwarebytes-feed-hphosts-offline

My understanding, possibly incorrect, is that Firewall Max Table entries must be at least double what your final count shows, since during updates both the new and old list counts exist in the table simultaneously.

getcom

@provels said in pfBlockerNG-devel 2.2.5_30: IPv4 list: too many elements + high system load:

My understanding, possibly incorrect, is that Firewall Max Table entries must be at least double what your final count shows, since during updates both the new and old list counts exist in the table simultaneously.

Each list has its own table and it will be processed one after the other. It should then > the doubled value of the biggest list.
I will try that and increase the value first to 3 1/2 million. This needs a reboot, which means this can be done only ooo in four hours.
Are DNSBL lists processed in a different way? The DNSBL_UT1 has >1.8 million entries and this list is working.

Ok., thank you for the offline hint of the lists...just discovered the download errors also on my own pfS which is still 2.4.4-P3.

Painkilla

I can 100% confirm this behavior. I have 32 CPU Cores with 64 GB RAM and the default Value from pfSense is set to 400000. When i try to activate only the default PR1 IPv4 List in pfBlockerNG, i get massively lags and all the symptoms described above. @getcom did setting the value higher resolve this problem?

getcom

@Painkilla said in pfBlockerNG-devel 2.2.5_30: IPv4 list: too many elements + high system load:

I can 100% confirm this behavior. I have 32 CPU Cores with 64 GB RAM and the default Value from pfSense is set to 400000. When i try to activate only the default PR1 IPv4 List in pfBlockerNG, i get massively lags and all the symptoms described above. @getcom did setting the value higher resolve this problem?

No, it does not.
My workaround is to switch back to pfS 2.4.4-P3. pfBlockerNG-devel is running now as usual. The system is rock solid running this combination.
The problem is somewhere in FreeBSD 11.3 and as an aftereffect also in pfS 2.4.5. pfBlockerNG-devel is definitely NOT the problem child.

Painkilla

Thank you for that information. I will downgrade now pfSense. Would you @getcom mind to set up a bug-report? Your reputation is surely better than mine and i expect you can describe the problem better i could ever do.