pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect

cyberteach

Hi folks-

I'm encountering a pfSense 2.5 (10.0.2.8) to pfSense 2.5 (10.0.2.15) IPsec tunnel connection failure. These are freshly deployed VMs in VirtualBox using the NAT Network for both WANs to simulate internet. The two pfSense 2.5 VMs can ping each other fine, but I haven't had luck with an IPsec tunnel using mostly default settings. When I press connect, it flashes for a sec then remains disconnected. Do my logs yield any clues?

Feb 28 06:03:42 	charon 	76827 	16[CFG] vici client 33 disconnected
Feb 28 06:03:42 	charon 	76827 	06[CFG] vici client 33 requests: list-sas
Feb 28 06:03:42 	charon 	76827 	14[CFG] vici client 33 registered for: list-sa
Feb 28 06:03:42 	charon 	76827 	16[CFG] vici client 33 connected
Feb 28 06:03:42 	charon 	76827 	14[CFG] vici client 32 disconnected
Feb 28 06:03:42 	charon 	76827 	16[CFG] vici client 31 disconnected
Feb 28 06:03:42 	charon 	76827 	16[CFG] vici initiate CHILD_SA 'con1000'
Feb 28 06:03:42 	charon 	76827 	16[CFG] vici client 32 requests: initiate
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici terminate IKE_SA 'con1000'
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici client 31 requests: terminate
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici client 32 registered for: control-log
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici client 32 connected
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici client 31 registered for: control-log
Feb 28 06:03:42 	charon 	76827 	09[CFG] vici client 31 connected
Feb 28 06:03:40 	charon 	76827 	14[CFG] vici client 30 disconnected
Feb 28 06:03:40 	charon 	76827 	07[CFG] vici client 30 requests: list-sas
Feb 28 06:03:40 	charon 	76827 	14[CFG] vici client 30 registered for: list-sa
Feb 28 06:03:40 	charon 	76827 	14[CFG] vici client 30 connected
Feb 28 06:03:38 	charon 	76827 	07[CFG] vici client 29 disconnected
Feb 28 06:03:38 	charon 	76827 	14[CFG] updated vici connection: con100000
Feb 28 06:03:38 	charon 	76827 	14[CFG] id = 10.0.2.15
Feb 28 06:03:38 	charon 	76827 	14[CFG] class = pre-shared key
Feb 28 06:03:38 	charon 	76827 	14[CFG] remote:
Feb 28 06:03:38 	charon 	76827 	14[CFG] id = 10.0.2.8
Feb 28 06:03:38 	charon 	76827 	14[CFG] class = pre-shared key
Feb 28 06:03:38 	charon 	76827 	14[CFG] local:
Feb 28 06:03:38 	charon 	76827 	14[CFG] if_id_out = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] if_id_in = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] proposals = IKE:AES_CBC_256/HMAC_SHA2_256_128/PRF_HMAC_SHA2_256/MODP_2048
Feb 28 06:03:38 	charon 	76827 	14[CFG] rand_time = 2880
Feb 28 06:03:38 	charon 	76827 	14[CFG] over_time = 2880
Feb 28 06:03:38 	charon 	76827 	14[CFG] rekey_time = 25920
Feb 28 06:03:38 	charon 	76827 	14[CFG] reauth_time = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] keyingtries = 1
Feb 28 06:03:38 	charon 	76827 	14[CFG] unique = UNIQUE_REPLACE
Feb 28 06:03:38 	charon 	76827 	14[CFG] childless = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] fragmentation = 2
Feb 28 06:03:38 	charon 	76827 	14[CFG] dpd_timeout = 60
Feb 28 06:03:38 	charon 	76827 	14[CFG] dpd_delay = 10
Feb 28 06:03:38 	charon 	76827 	14[CFG] encap = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] dscp = 0x00
Feb 28 06:03:38 	charon 	76827 	14[CFG] aggressive = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] mobike = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] ppk_required = 0
Feb 28 06:03:38 	charon 	76827 	14[CFG] ppk_id = (null)
Feb 28 06:03:38 	charon 	76827 	14[CFG] send_cert = CERT_SEND_IF_ASKED
Feb 28 06:03:38 	charon 	76827 	14[CFG] send_certreq = 1
Feb 28 06:03:38 	charon 	76827 	14[CFG] remote_port = 500
Feb 28 06:03:38 	charon 	76827 	14[CFG] local_port = 500
Feb 28 06:03:38 	charon 	76827 	14[CFG] remote_addrs = 10.0.2.15

Thanks!

jimp

To ensure you have all of the current known and fixed IPsec issues corrected, You can install the System Patches package and then create entries for the following commit IDs to apply the fixes:

ead6515637a34ce6e170e2d2b0802e4fa1e63a00 #11435
57beb9ad8ca11703778fc483c7cba0f6770657ac #11435
10eb04259fd139c62e08df8de877b71fdd0eedc8 #11442
ded7970ba57a99767e08243103e55d8a58edfc35 #11486
afffe759c4fd19fe6b8311196f4b6d5e288ea4fb #11487
2fe5cc52bd881ed26723a81e0eed848fd505fba6 #11488
f731957f945af90d6a75f0e33f91a440a6a55736 #11526

MarcO42

Hi,
I also need to install this one:

https://github.com/pfsense/pfsense/commit/4e5857b656c7bfd59efadbb9a124876a5516c7df.patch

and have to setup the Dead Peer Detection (DPD) to Delay 60 and Max failures 5.
Cheers
Marco

jimp

@marco42 said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

Hi,
I also need to install this one:

https://github.com/pfsense/pfsense/commit/4e5857b656c7bfd59efadbb9a124876a5516c7df.patch

That is the same as ead6515637a34ce6e170e2d2b0802e4fa1e63a00 which is in the list above already, but the commit you linked is to master and ead6515637a34ce6e170e2d2b0802e4fa1e63a00 is to RELENG_2_5_0 -- they both apply cleanly but it's better to use the one from the appropriate branch.

Morlock

Hi,

Just wanted to give some feedback that after the upgrade of one side of a VTI-connection between two pfSenses, IPsec failed with "trap not found, unable to acquire reqid".

Reconfiguration did not help, applying all the patches above did. I will revert to a VM snapshot anyway and wait for a maintenance release.

jimp

@morlock said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

"trap not found, unable to acquire reqid"

That isn't a fatal error, it's normal with VTI.

Morlock

@jimp Ok, then it didn't connect for some other reason that is not logged with my settings. I hadn't really time to look into it more detailed. I had applied the first six patches first, but only 731957f945af90d6a75f0e33f91a440a6a55736 eventually made a difference.

dyener

@jimp Thank you for compiling this convenient list of patches! Can I apply them to pfSense Plus 21.02 devices, or are they only for pfSense CE 2.5? I'm trying to make a basic IPsec tunnel between an SG-1100 and a homebuilt box, and cannot get it to work with those respective software versions. Forgive the question, as I have never tried using patches before. Also, do I need to revert the patches when the next upgrade becomes available, or is it safe to apply the upgrade on top of them? Thank you again!

ComputerFreek

Just a question, why is this not in an official fix or release yet? I have 3 firewalls spread out between my parents, my friends house and my own running the community edition. I don't understand why this hasn't been fixed yet. My ipsec tunnels are down even after a fresh install. I'm just confused why it is taking so long for this to become a "fix".

jimp

Making a new release takes time, effort, and testing. There are still numerous things we're actively investigating.

beejee

After upgraded to version 21.02-RELEASE-p1, my Netgate XG-7100 kept getting disconnected to a remote Cisco RV042 (even though it showed as connected). The IPsec tunnel had been working smoothly for a year without any hiccup now decided to act up. There is no pattern when it would stop the connection, most of the time just within couple minutes. I applied the 7 patches as advised about but still no luck. My tunnel is using IKEv1, AES_CBC (128), HMAC_SHA1_96, PRF_HMAC_SHA1, MODP_768. I am under the pressure to get it back on track. Please help! Thank you.

beejee

I finally got my IPsec tunnel to work without interruption on my Netgate XG-7100 by turning of the Hardware Crypto. I believe it is a work around solution since the Hardware Crypto has to be off.

jimp

@beejee said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

I finally got my IPsec tunnel to work without interruption on my Netgate XG-7100 by turning of the Hardware Crypto. I believe it is a work around solution since the Hardware Crypto has to be off.

Which hardware crypto option did you have enabled on there? If it was AES-NI, that sounds similar to an issue we're already tracking. If it's not, it could be a different problem. On the XG-7100 you can switch from AES-NI to QAT which should be equal to or faster in performance and potentially more stable as the issues we're aware of only affect AES-NI at the moment.

beejee

@jimp said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

Which hardware crypto option did you have enabled on there?

You were right. It was "AES-NI and BSD Cryto Device (aesni,cryptodev)" option. I believe it would be the best option for my appliance overall.

I will try the "QAT" as your suggested sometime tonight. Thank you!

beejee

@jimp said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

On the XG-7100 you can switch from AES-NI to QAT which should be equal to or faster in performance and potentially more stable

Yes my XG-7100 IPsec tunnel is working smoothly with "Intel QuickAssist (QAT)" on. My CPU Type is "Intel(R) Atom(TM) CPU C3558 @ 2.20GHz". I actually don't know where to check if my IPsec tunnel is really using the QAT since I didn't see it is listed anywhere in the Dashboard. The Dashboard mentioned that "AES-NI CPU Crypto: Yes (inactive)" instead.

jimp

The next update will show QAT on the dashboard properly, but for now you can check:

That the module is loaded:

: kldstat | grep qat
 5    1 0xffffffff84322000    146e0 qat.ko
 6    1 0xffffffff84337000    9f521 qat_c3xxxfw.ko

That the device is consuming interrupts (will increase as traffic is encrypted/decrypted through IPsec):

: vmstat -i | egrep 'total|qat'
interrupt                          total       rate
irq300: qat0                      489041          0

beejee

@jimp Awesome! I think my IPsec is really utilizing the QAT. Thank you so much!
I downloaded a large file through the tunnel and checked the stat:

:kldstat | grep qat
 3    1 0xffffffff83f22000    146e0 qat.ko
 4    1 0xffffffff83f37000    9f521 qat_c3xxxfw.ko

:vmstat -i | egrep 'total|qat'
interrupt                          total       rate
irq293: qat0                       43576          1
irq294: qat0                       27168          1
irq295: qat0                        7909          0
irq296: qat0                       18777          0

:vmstat -i | egrep 'total|qat'
interrupt                          total       rate
irq293: qat0                      396393          9
irq294: qat0                       39664          1
irq295: qat0                       19005          0
irq296: qat0                       47598          1

brians

@jimp Thanks I had problem with tunnel to a Cisco router where I could ping but then sending any traffic through would kill the tunnel. Switching my 5100 to QAT seems to have fixed this issue.

I have another tunnel between the SG-5100 and an older SG-2220.. enabling QAT on the 2200 is slower than using AES-NI - iperf3 test are 220Mbps with AES-NI and only around 100Mbps on QAT when I enable it on the SG-2220.

danjeman

@jimp Guessing the Dashboard display for QAT or other crypto modules didn't make it to 21.02.2 - at least my XG-7100's still show 'AES-NI CPU Crypto: Yes (inactive) - no mention of QAT anywhere that I can find on a Dashboard widget etc.

jimp

@danjeman said in pfSense 2.5 to pfSense 2.5 IPsec tunnel fails to connect:

@jimp Guessing the Dashboard display for QAT or other crypto modules didn't make it to 21.02.2 - at least my XG-7100's still show 'AES-NI CPU Crypto: Yes (inactive) - no mention of QAT anywhere that I can find on a Dashboard widget etc.

It's there on 21.05 snapshots, didn't make it into 21.02.2.