yet Another out-of-swap-space issue

[knight2f6]

hi everyone,
I looked at other postings and cannot find the answer here. I have a brand new NetGate 4100 for a small home office. 4G of RAM and 16G of SSD. everything is out of the box and standard install. I have not done anything special to the box. the volume of the traffic is minimal and I run about 4Mbps of traffic on the high end. I have clamav, netBlocker, and Suricata running on the box, all with standard configs. And yet, I am running into multiple issues.
My Clam AV dies every few days with an out of swap space error.

Nov 27 07:00:14 php 39596 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Nov 27 07:00:00 php 39596 [pfBlockerNG] Starting cron process.
Nov 27 06:43:51 kernel pid 46661 (clamd), jid 0, uid 106, was killed: out of swap space
Nov 27 06:00:17 php 79080 [pfBlockerNG] No changes to Firewall rules, skipping Filter Reload
Nov 27 06:00:00 php 79080 [pfBlockerNG] Starting cron process.

when I look at the memory usage around this time, the free memory is around 23% and after the crash it shoots up to 63%. But at no time it gets close to 0 that will even require using swap space.

looking at the clamAV logs everything looks normal at around this time.

[22.05-RELEASE][admin@pfSense.home.arpa]/var/log/clamav: ls -l
total 13
-rw-r----- 1 clamav clamav 106389 Nov 27 06:43 clamd.log
-rw-r----- 1 clamav clamav 87009 Nov 27 06:43 freshclam.log
[22.05-RELEASE][admin@pfSense.home.arpa]/var/log/clamav: tail clamd.log
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
SelfCheck: Database status OK.
Reading databases from /var/db/clamav/
[22.05-RELEASE][admin@pfSense.home.arpa]/var/log/clamav: tail freshclam.log
bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)

ClamAV update process started at Sun Nov 27 06:43:00 2022
daily database available for update (local version: 26732, remote version: 26733)
Testing database: '/var/db/clamav//tmp.469f484141/clamav-a27199040a65109130aaa19f30e85347.tmp-daily.cld' ...
Database test passed.
daily.cld updated (version: 26733, sigs: 2012873, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cvd database is up-to-date (version: 333, sigs: 92, f-level: 63, builder: awillia2)
Clamd successfully notified about the update.
[22.05-RELEASE][admin@pfSense.home.arpa]/var/log/clamav:

note that last update for both files were at 6:43, at the time of crash.

I have read other posting here about the out of swap space issues, but can't find anything that actually points to a solution.

with this being an out of the box issue at such a low traffic load, adding all the issues that I am having with Suricata, I am wondering if this was the right purchase. there seems to be a lot of issues here but admittedly I am new to PfSense. thought I give this forum a try first.

thanks for any help.

[johnpoz]

@knight2f6 said in yet Another out-of-swap-space issue:

with this being an out of the box issue

installing high requirement packages is not something I would consider out of the box..

proxy and suricata and clamav - not your hey lets use a couple of cpu cycles or memory sort of packages.

[knight2f6]

@johnpoz thanks John. I am with netGate support on a different thread and they are implying the same thing. So, I am thinking of RMA'ing the product and running it on my own hardware. what is your recommendation for CPU, RAM, SSD storage for someone who likes to play with different packages and run multiple packages simultaneously?
would this be a good option? t.ly/g_mV

[rcoleman-netgate]

@knight2f6 said in yet Another out-of-swap-space issue:

what is your recommendation for CPU, RAM, SSD storage for someone who likes to play with different packages and run multiple packages simultaneously?

As we noted in our reply to your tickets we do not support running those packages on the BASE hardware -- which is what you have -- due to the heavy writing that is done to the eMMC storage.

The 4100 is very capable of running the software, we just do not support the packages in TAC that you are running.

[knight2f6]

@rcoleman-netgate said in yet Another out-of-swap-space issue:

we do not support running those packages on the BASE hardware

Ryan, this implies that you do support running them on a "non-BASE" hardware. can you recommend a hardware that you have that supports running these packages? thanks

[rcoleman-netgate]

@knight2f6 I take it you did not read the link in my email:

This is the list of supported packages in TAC, and the limit of the support (if it is limited):

https://www.netgate.com/supported-pfsense-plus-packages

Missing from that list: ClamAV
Limited support but on the list: pfBlockerNG, Squid, Suricata.

That does not mean the PACKAGES cannot be run, that means there's either no or limited TECHNICAL support from Netgate TAC for those packages -- regardless of the level of TAC support you have (LITE, Professional, or Enterprise).

[knight2f6]

@rcoleman-netgate thanks

[stephenw10]

Here though you are exhausting the RAM so running from an SSD would not help directly. You still should run from SSD of course.

You would need to tun those packages more carefully to run them all. 4GB should be enough though unless you have chosen to load all the lists/signatures.

Steve

[knight2f6]

@stephenw10 thanks. I am actually running the bare minimal set of signatures. In the case of Suricata, I am only running the Fedora and nothing else. I have tried multiple configurations with same result. for most part on each package I only enabled the bare minimum settings. The memory never hits the 100%, it barely touches 80% before ClamAv crashes. See the original image.

I guess what I am not sure is how the system claims to run out of SWAP space if it still has 20% of free RAM. Unless of course, it runs out of the memory AND swap space in the 5 minutes interval that is between each sampling period for the graph. Although, I have looked at this graph multiple times during each crash and they all look the same. You would think out of the 12 or so crashes, one of them would show memory going down to 0.

[stephenw10]

Try running top -HaSP and the command line and see what's using memory.

[skogs]

The 4100 is certainly restricted in some of its capabilities, but it really ~should~ be capable of what you'd attempting...though clamd does eat quite a bit of extra...but I've got everything you've got running except for clam and I'm sitting at 15% of 8G memory usage...if I was restricted for 4G it would only be 30%.

soo...I have 8G because I had the same problem at 4G you have now. More RAM didn't fix it.

IIRC running suricata in anything other than 'Pattern Matcher Algorithm - Auto' will essentially slowly eat the system.
Maybe this is fixed in newer builds, but the older days any of the fancy tuning one might want to do pretty much did exactly the opposite of what one would expect.
Now that I've got suricata tamed...well the extra RAM just wastes electricity.

[bmeeks]

@skogs said in yet Another out-of-swap-space issue:

IIRC running suricata in anything other than 'Pattern Matcher Algorithm - Auto' will essentially slowly eat the system.

This +100

Never, ever, never touch the Pattern Matcher adjustment in Suricata or Snort!

I have contemplated removing that configuration parameter from the GUI package a number of times, but I elected to leave it just in case someone has gigs and gigs of extra RAM and wants to play around with it. But on average firewall systems, you never want to change it from the default. If you do, the result is likely to be voracious swallowing of RAM.

[knight2f6]

@bmeeks please qualify, what is gigs and gigs of extra RAM? 16, 32,, ... Just curious as I am planning my next hardware specs. thanks

[rcoleman-netgate]

@knight2f6 The more you put into it the less you will eventually get out of it. Loading the protections that are super uncommon or rare or you really wouldn't be a target for could make your system perform so slowly you'll realize that it's not worth the performance drop.

The bigger the list in pfBlockerNG the more RAM it consumes... but you probably don't need all that -- just a few nations and a few topic filters and leave it at that.

YMMV, of course, and this is not official advice but it is possible to force your firewall into submission and find it's use to be unbearable.

[bmeeks]

@knight2f6 said in yet Another out-of-swap-space issue:

@bmeeks please qualify, what is gigs and gigs of extra RAM? 16, 32,, ... Just curious as I am planning my next hardware specs. thanks

64 GB would be a starting point, and it goes up from there. Lots of other things can yield better performance improvements than tinkering with the Pattern Matcher algorithm will achieve, though. For example, having good NICs with multiple queues that the OS driver supports very well, having RSS enabled in the kernel and operating properly, and of course a multi-core high clock speed CPU. Tuning your ruleset and fiddling with core affinity settings in suricata.yaml can also help performance. But we are also starting to rapidly run out of pfSense territory and into special-purpose appliances with really fancy hardware (and the software to take full advantage of that fancy hardware) when you need to do these things.

And @rcoleman-netgate has a great point. It is very easy to pile stuff onto your firewall and overwhelm it. It's not always the best solution to run everything on the firewall itself. For home networks or small business networks, putting stuff on the firewall works (so long as you don't go wild). But as you get into larger networks with much higher volumes of traffic, splitting tasks off onto separate machines becomes a better solution.

Plus, installing lots of "other software packages" on your firewall drags in extra shared libraries and potentially a lot of exposure to new threats (via all those required shared libraries). This is particularly critical when the extra packages drag in something like Java (which UniFi Network Controller does). IMHO, Java has no place on anyone's firewall!

[johnpoz]

@bmeeks said in yet Another out-of-swap-space issue:

Java has no place on anybody's firewall!

Java has no business on anything to be honest ;) heheh

It runs on my unifi controller - but that is limited to my infrastructure vlan for the APs etc. and nothing else has access to it - nothing exposed to internet, and only I can access it from my management pc, etc.

And that is really the only thing that VM does - it just runs the controller software. No other anything is served off it, not even for local network.

[rcoleman-netgate]

@johnpoz Java has its business in my mug.

[johnpoz]

@rcoleman-netgate You wouldn't want that on/in your firewall either ;) hehehe

[rcoleman-netgate]

@johnpoz and that's the edited version! :D