2.5 with many tunnels - Apply Changes fails

vergilis

After upgrading to 2.5 hitting Apply Changes takes a long time until the following message is displayed

504 Gateway Time-out
nginx

The firewall has about 50 tunnels defined.

jimp

Anything else running on there (other than IPsec) that might be consuming a lot of CPU time?

I haven't seen any reports of IPsec taking that much, but it's possible something else is contributing and causing it.

vergilis

There's HA proxy and Squid. There's very low utilization:

last pid: 97107;  load averages:  0.51,  0.50,  0.36  up 0+13:27:29    11:54:43
244 threads:   3 running, 225 sleeping, 16 waiting
CPU:  0.8% user,  0.0% nice,  1.3% system,  0.0% interrupt, 97.9% idle
Mem: 51M Active, 76M Inact, 234M Wired, 100M Buf, 4551M Free

Please let me know if you need any other information.

jimp

Anything in the system or IPsec logs at all?

There isn't much to go on here.

vergilis

It is really busy. Constantly get:

logfile turned over due to size>500K

Modifying the log setting will constantly get the same timeout. Perhaps that's the issue. How do I silence it?

vergilis

Is there a mapping of log entries in the UI so that I can turn down specific log entries after persistent retries?

app: applications other than daemons
asn: Low-level encoding/decoding (ASN.1, X.509 etc.)
cfg: Configuration management and plugins
chd: CHILD_SA/IPsec SA
dmn: Main daemon setup/cleanup/signal handling
enc: Packet encoding/decoding encryption/decryption operations
esp: libipsec library messages
ike: IKE_SA/ISAKMP SA
imc: Integrity Measurement Collector
imv: Integrity Measurement Verifier
job: Jobs queuing/processing and thread pool management
knl: IPsec/Networking kernel interface
lib: libstrongswan library messages
mgr: IKE_SA manager, handling synchronization for IKE_SA access
net: IKE network communication
pts: Platform Trust Service
tls: libtls library messages
tnc: Trusted Network Connect

This help link seems to go nowhere:
https://docs.netgate.com/pfsense/en/latest/book/ipsec/ipsec-advanced-settings.html

jimp

That link should go to https://docs.netgate.com/pfsense/en/latest/vpn/ipsec/advanced.html

There isn't a detailed list of what is logged where, though you can probably turn most things down to silent if you want to focus on just one part of the log.

See also https://wiki.strongswan.org/projects/strongswan/wiki/LoggerConfiguration

vergilis

So I guess on an related issue the ? go the wrong URL.

I was able to finally turn down the logging and I cannot find any errors in IPSec log.

Here are some entries in System log:

2021/02/18 13:38:18 [error] 51390#100109: *64336 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 5.6.7.8, server: , request: "POST /vpn_ipsec_settings.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "fwname.example.com:1234", referrer: "https://fwname.example.com:1234/vpn_ipsec_settings.php"

jimp

Can you watch the console as the firewall boots up and see how long it takes to configure IPsec at boot time?

vergilis

That is hard to do as it is an AWS instance, but I will try.

vergilis

It was stuck on this screen for about 4 minutes.

Please let me know if you need anything else.

jimp

That g_vfs_done line implies it's having problems communicating with its storage.

You may have some other problem in your hypervisor/guest config which is causing it to perform slower than it should, leading to your timerout.

vergilis

Only post 2.5 upgrade and only for IpSec tunnels?

jimp

Yes, it's common for hypervisors to need adjustments when moving from one FreeBSD version to another, depending on your settings.

It may not be related to IPsec at all, just that it's something that is time consuming and provokes the general slowness.

vergilis

Thank you.

What is your recommendation to correct this?

jimp

That depends on your hypervisor and guest settings, I don't have any general recommendations there other than to check what your hypervisor recommends for use with FreeBSD 12.2 (or at least 12.x).

vergilis

This is an AWS instance with your approved image and size. This is a c5n.large.

jimp

Then I suggest you redeploy it instead of upgrading in-place to see if the problem happens that way.

vergilis

When I load the XML file into a new instance, your software produces the same timeout issue and the XML never loads.

Any other ideas?

vergilis

Also, after a few attempts it loaded. It continues to hang in the same spot during boot up, and actually does not boot up at all. I don't think that its hardware or disk related.

Please let me know how you would like to proceed.