swap_pager_getswapspace Failed

hugoeyng

System Logs is showing the message swap_pager_getswapspace(n): Failed.

"n" is an incremental number.

Well, I understood that swap area, defined to the system, has reached the total amount of space.

It started to happen after I increased users on VPN. (I thing so).

How can I increase the swap area without formatting an reinstalling?

hugoeyng

Curiously it occurs every day at the same hour (almost the same)

And, at this time, nobody is using the network (should not, at least).

bmeeks

Do you have any packages running such as Snort, Suricata or pfBlockerNG-devel? All of these can use up RAM space while performing certain tasks. It could be that is causing the swap issue.

There is also the known bug with pcscd and its memory leak documented here: https://redmine.pfsense.org/issues/12095.

As you may know, swap space is called upon whenever your firewall is out of free RAM. At that point, some of the contents of RAM are written to a temporary area of the disk to make room. The contents that get written to disk are processes that are currently "sleeping" and not actively executing. Then, when those tasks wake up and get ready to execute, their data is read back into RAM from the swap area. But this is extremely inefficient and will slow the firewall down to a crawl.

The answer to your question is that "yes", you can increase swap space by manually adding another disk and creating a FreeBSD swap partition on it, and then adding that to the available swap inventory. But that is not the solution you need. You need to find out what is causing your firewall to use swap at all. A properly functioning pfSense firewall should almost never use swap space, and certainly never use enough to actually run out of space.

hugoeyng

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

bmeeks

@hugoeyng said in swap_pager_getswapspace Failed:

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

Well, you did not say if you had any installed packages running. It is possible, if you do, that one of them is exhausting RAM during some phase of its operation. One candidate would be the IDS/IPS packages (Snort or Suricata). I created and maintain those packages, so I know that during rule updates, for example, their RAM consumption can temporarily increase quite a bit. I believe the same may be true for some operating modes of pfBlockerNG-devel when using the new Python mode with unbound.

In the case of the IDS/IPS packages, the RAM usage is temporary, so unless you happened to be logged into the firewall at the exact instant the problem occurred, you would not see it.

Your second post mentioned the issue happening at the same time each day. That indicates to me a cron task is at fault. So the first step would be to see what is running at that time. You can install the Cron package to provide an easy-to-navigate GUI interface for managing cron tasks. Or, if you are good at the FreeBSD command-line, you can use the various shell tools to do the same thing.

Listing all of the currently running processes might reveal something, too. Try running this from a shell prompt:

ps -ax

or

top -aSH

It is a bit puzzling to me that your swap usage appears to be 80% when you are showing only about 22% of RAM currently utilized.

hugoeyng

@bmeeks said in swap_pager_getswapspace Failed:

@hugoeyng said in swap_pager_getswapspace Failed:

@bmeeks thank you. Your answer is very clear for me and I totally agree that increasing swap area is not the solution.

The solution, as you said, is to find the what is the cause of this behavior and not trying to treat the effect.

As you can see in the image, there are 4GB RAM and, usually, 20-35% in use. So, there is no razon to the system using swap.

How could I check what process were running in the moment that the error message was logged.

Well, you did not say if you had any installed packages running. It is possible, if you do, that one of them is exhausting RAM during some phase of its operation. One candidate would be the IDS/IPS packages (Snort or Suricata). I created and maintain those packages, so I know that during rule updates, for example, their RAM consumption can temporarily increase quite a bit. I believe the same may be true for some operating modes of pfBlockerNG-devel when using the new Python mode with unbound.

In the case of the IDS/IPS packages, the RAM usage is temporary, so unless you happened to be logged into the firewall at the exact instant the problem occurred, you would not see it.

Your second post mentioned the issue happening at the same time each day. That indicates to me a cron task is at fault. So the first step would be to see what is running at that time. You can install the Cron package to provide an easy-to-navigate GUI interface for managing cron tasks. Or, if you are good at the FreeBSD command-line, you can use the various shell tools to do the same thing.

Listing all of the currently running processes might reveal something, too. Try running this from a shell prompt:
ps -ax
or
top -aSH
It is a bit puzzling to me that your swap usage appears to be 80% when you are showing only about 22% of RAM currently utilized.

No Snort, no Suricata installed.

I will post some images:

top -aSH -o size

Installed packages:

Cron:

ps -ax
he.txt

Services runnig:

bmeeks

A few things I notice in those screenshots.

First, you have three instances of pcscd running. If you do not have any installed wireless cards in the firewall, I recommend stopping that service. Embedded within the link I provided earlier is a link to a pfSense patch that will disable the service and prevent it from restarting unless an actual wireless card is detected.

Secondly, clamav appears to be using quite a few resources, including RAM. Unless you have a full MITM (man-in-the-middle) encryption interception scheme configured so the files are decrypted on the firewall, scanning web and email files passing through the firewall is pointless as they are encrypted for the most part these days (via https web sites using SSL and email traffic travelling almost exclusively via TLS). So I would disable anti-virus on the firewall and instead put that resource on the endpoints (workstations and servers) in your network.

And finally, squid is taking up a fair amount of RAM. I think together all of these packages are periodically consuming all of the RAM and thus triggering the use of swap space. And the 4 GB of configured swap is not enough based on the error you receive.

As for the hourly trigger, I see only one cron task configured to run at 17 minutes past the hour, and that is the clamav update routine. That process might be the one that puts you over the top with respect to running out of RAM and also swap.

hugoeyng

@bmeeks said in swap_pager_getswapspace Failed:

A few things I notice in those screenshots.

First, you have three instances of pcscd running. If you do not have any installed wireless cards in the firewall, I recommend stopping that service. Embedded within the link I provided earlier is a link to a pfSense patch that will disable the service and prevent it from restarting unless an actual wireless card is detected.

Secondly, clamav appears to be using quite a few resources, including RAM. Unless you have a full MITM (man-in-the-middle) encryption interception scheme configured so the files are decrypted on the firewall, scanning web and email files passing through the firewall is pointless as they are encrypted for the most part these days (via https web sites using SSL and email traffic travelling almost exclusively via TLS). So I would disable anti-virus on the firewall and instead put that resource on the endpoints (workstations and servers) in your network.

And finally, squid is taking up a fair amount of RAM. I think together all of these packages are periodically consuming all of the RAM and thus triggering the use of swap space. And the 4 GB of configured swap is not enough based on the error you receive.

As for the hourly trigger, I see only one cron task configured to run at 17 minutes past the hour, and that is the clamav update routine. That process might be the one that puts you over the top with respect to running out of RAM and also swap.

I have no idea about why, when and who instaled pcscd.
I will try to understand https://redmine.pfsense.org/issues/12095

About squid I think that the below values can be critical (or became critical now for som razon).

bmeeks

pcscd is a native pfSense component (actually, it's part of FreeBSD). It is not something you installed. It comes with pfSense. But it currently has a memory leak problem that causes it to consume RAM and not return it to the operating system properly.

As for your squid configuration, you've told it to use 2 GB of RAM. The default for that parameter is 64 MB. That's a huge difference. Your selected value is many times larger than the suggested default.

I don't use squid, so I can't offer any guidance beyond simply saying that configuring a value that is so much larger than the default is potentially not optimum.

hugoeyng

@bmeeks

As I said before, I will try https://redmine.pfsense.org/issues/12095

About Squid I will reduce the amount of RAM, even I think that could not be the cause once it defined so since long time ago. But ...

About clamav, it is my existencial dilema. Keeping or not to keep installed.

This morning the error didn´t happened.

Thank you for all your answers. You helped a lot showing options and view points.

Best, regards.

stephenw10

It's almost certainly the leak in pcscd. You should apply that patch and reboot when you can.

You should pretty much never see swap use in pfSense. If you are seeing it regularly being used something is misconfigured / needs more tuning.

Steve

hugoeyng

@stephenw10 I restarted pfSense and the issue has gone. I did it around 3 hours ago.

I will monitor to find out if it was a "exception" or if it is a recurrent issue.

Thank you for your answer.

stephenw10

If you saw it once you will probably see it again. You shouldn't see any SWAP use until the RAM is used so it will likely take some days. I would apply that patch, it will be in the next release anyway.

Steve

hugoeyng

@stephenw10 Thank you Steve.

I will apply the patch.