swap_pager_getswapspace Failed

hugoeyng

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

bmeeks

@hugoeyng said in swap_pager_getswapspace Failed:

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

Well, you did not say if you had any installed packages running. It is possible, if you do, that one of them is exhausting RAM during some phase of its operation. One candidate would be the IDS/IPS packages (Snort or Suricata). I created and maintain those packages, so I know that during rule updates, for example, their RAM consumption can temporarily increase quite a bit. I believe the same may be true for some operating modes of pfBlockerNG-devel when using the new Python mode with unbound.

In the case of the IDS/IPS packages, the RAM usage is temporary, so unless you happened to be logged into the firewall at the exact instant the problem occurred, you would not see it.

Your second post mentioned the issue happening at the same time each day. That indicates to me a cron task is at fault. So the first step would be to see what is running at that time. You can install the Cron package to provide an easy-to-navigate GUI interface for managing cron tasks. Or, if you are good at the FreeBSD command-line, you can use the various shell tools to do the same thing.

Listing all of the currently running processes might reveal something, too. Try running this from a shell prompt:

ps -ax

or

top -aSH

It is a bit puzzling to me that your swap usage appears to be 80% when you are showing only about 22% of RAM currently utilized.

hugoeyng

@bmeeks said in swap_pager_getswapspace Failed:

@hugoeyng said in swap_pager_getswapspace Failed:

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

Well, you did not say if you had any installed packages running. It is possible, if you do, that one of them is exhausting RAM during some phase of its operation. One candidate would be the IDS/IPS packages (Snort or Suricata). I created and maintain those packages, so I know that during rule updates, for example, their RAM consumption can temporarily increase quite a bit. I believe the same may be true for some operating modes of pfBlockerNG-devel when using the new Python mode with unbound.

In the case of the IDS/IPS packages, the RAM usage is temporary, so unless you happened to be logged into the firewall at the exact instant the problem occurred, you would not see it.

Your second post mentioned the issue happening at the same time each day. That indicates to me a cron task is at fault. So the first step would be to see what is running at that time. You can install the Cron package to provide an easy-to-navigate GUI interface for managing cron tasks. Or, if you are good at the FreeBSD command-line, you can use the various shell tools to do the same thing.

Listing all of the currently running processes might reveal something, too. Try running this from a shell prompt:
ps -ax
or
top -aSH
It is a bit puzzling to me that your swap usage appears to be 80% when you are showing only about 22% of RAM currently utilized.

No Snort, no Suricata installed.

I will post some images:

top -aSH -o size

Installed packages:

Cron:

ps -ax
he.txt

Services runnig:

bmeeks

A few things I notice in those screenshots.

First, you have three instances of pcscd running. If you do not have any installed wireless cards in the firewall, I recommend stopping that service. Embedded within the link I provided earlier is a link to a pfSense patch that will disable the service and prevent it from restarting unless an actual wireless card is detected.

Secondly, clamav appears to be using quite a few resources, including RAM. Unless you have a full MITM (man-in-the-middle) encryption interception scheme configured so the files are decrypted on the firewall, scanning web and email files passing through the firewall is pointless as they are encrypted for the most part these days (via https web sites using SSL and email traffic travelling almost exclusively via TLS). So I would disable anti-virus on the firewall and instead put that resource on the endpoints (workstations and servers) in your network.

And finally, squid is taking up a fair amount of RAM. I think together all of these packages are periodically consuming all of the RAM and thus triggering the use of swap space. And the 4 GB of configured swap is not enough based on the error you receive.

As for the hourly trigger, I see only one cron task configured to run at 17 minutes past the hour, and that is the clamav update routine. That process might be the one that puts you over the top with respect to running out of RAM and also swap.

hugoeyng

@bmeeks said in swap_pager_getswapspace Failed:

A few things I notice in those screenshots.

First, you have three instances of pcscd running. If you do not have any installed wireless cards in the firewall, I recommend stopping that service. Embedded within the link I provided earlier is a link to a pfSense patch that will disable the service and prevent it from restarting unless an actual wireless card is detected.

Secondly, clamav appears to be using quite a few resources, including RAM. Unless you have a full MITM (man-in-the-middle) encryption interception scheme configured so the files are decrypted on the firewall, scanning web and email files passing through the firewall is pointless as they are encrypted for the most part these days (via https web sites using SSL and email traffic travelling almost exclusively via TLS). So I would disable anti-virus on the firewall and instead put that resource on the endpoints (workstations and servers) in your network.

And finally, squid is taking up a fair amount of RAM. I think together all of these packages are periodically consuming all of the RAM and thus triggering the use of swap space. And the 4 GB of configured swap is not enough based on the error you receive.

As for the hourly trigger, I see only one cron task configured to run at 17 minutes past the hour, and that is the clamav update routine. That process might be the one that puts you over the top with respect to running out of RAM and also swap.

I have no idea about why, when and who instaled pcscd.
I will try to understand https://redmine.pfsense.org/issues/12095

About squid I think that the below values can be critical (or became critical now for som razon).

bmeeks

pcscd is a native pfSense component (actually, it's part of FreeBSD). It is not something you installed. It comes with pfSense. But it currently has a memory leak problem that causes it to consume RAM and not return it to the operating system properly.

As for your squid configuration, you've told it to use 2 GB of RAM. The default for that parameter is 64 MB. That's a huge difference. Your selected value is many times larger than the suggested default.

I don't use squid, so I can't offer any guidance beyond simply saying that configuring a value that is so much larger than the default is potentially not optimum.

hugoeyng

@bmeeks

As I said before, I will try https://redmine.pfsense.org/issues/12095

About Squid I will reduce the amount of RAM, even I think that could not be the cause once it defined so since long time ago. But ...

About clamav, it is my existencial dilema. Keeping or not to keep installed.

This morning the error didn´t happened.

Thank you for all your answers. You helped a lot showing options and view points.

Best, regards.

stephenw10

It's almost certainly the leak in pcscd. You should apply that patch and reboot when you can.

You should pretty much never see swap use in pfSense. If you are seeing it regularly being used something is misconfigured / needs more tuning.

Steve

hugoeyng

@stephenw10 I restarted pfSense and the issue has gone. I did it around 3 hours ago.

I will monitor to find out if it was a "exception" or if it is a recurrent issue.

Thank you for your answer.

stephenw10

If you saw it once you will probably see it again. You shouldn't see any SWAP use until the RAM is used so it will likely take some days. I would apply that patch, it will be in the next release anyway.

Steve

hugoeyng

@stephenw10 Thank you Steve.

I will apply the patch.