This is what the log shows when it comes back up:

Apr 4 09:26:29 mbsnet-pf1 charon: 06[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:26:36 mbsnet-pf1 charon: 12[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:26:42 mbsnet-pf1 charon: 05[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:26:49 mbsnet-pf1 charon: 15[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:26:58 mbsnet-pf1 charon: 10[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:26:59 mbsnet-pf1 charon: 14[IKE] <con19000|238> giving up after 5 retransmits Apr 4 09:26:59 mbsnet-pf1 charon: 14[IKE] <con19000|238> peer not responding, trying again (2/3) Apr 4 09:26:59 mbsnet-pf1 charon: 14[IKE] <con19000|238> initiating Main Mode IKE_SA con19000[238] to THEIR.IP.ADD.RESS Apr 4 09:26:59 mbsnet-pf1 charon: 14[ENC] <con19000|238> generating ID_PROT request 0 [ SA V V V V V ] Apr 4 09:26:59 mbsnet-pf1 charon: 14[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:27:03 mbsnet-pf1 charon: 13[IKE] <con19000|238> sending retransmit 1 of request message ID 0, seq 1 Apr 4 09:27:03 mbsnet-pf1 charon: 13[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:27:09 mbsnet-pf1 charon: 10[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:27:11 mbsnet-pf1 charon: 13[IKE] <con19000|238> sending retransmit 2 of request message ID 0, seq 1 Apr 4 09:27:11 mbsnet-pf1 charon: 13[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:27:20 mbsnet-pf1 charon: 12[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:27:24 mbsnet-pf1 charon: 12[IKE] <con19000|238> sending retransmit 3 of request message ID 0, seq 1 Apr 4 09:27:24 mbsnet-pf1 charon: 12[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:27:31 mbsnet-pf1 charon: 14[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:27:37 mbsnet-pf1 charon: 13[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:27:42 mbsnet-pf1 charon: 13[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:27:47 mbsnet-pf1 charon: 11[IKE] <con19000|238> sending retransmit 4 of request message ID 0, seq 1 Apr 4 09:27:47 mbsnet-pf1 charon: 11[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:27:49 mbsnet-pf1 charon: 06[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:28:04 mbsnet-pf1 charon: 09[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:28:17 mbsnet-pf1 charon: 11[NET] <241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (228 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 11[IKE] <241> THEIR.IP.ADD.RESS is initiating a Main Mode IKE_SA Apr 4 09:28:17 mbsnet-pf1 charon: 11[NET] <241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (120 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 05[NET] <241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (244 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 05[NET] <241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (260 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 05[NET] <241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (76 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 05[CFG] <241> looking for pre-shared key peer configs matching MY.IP.ADD.RESS...THEIR.IP.ADD.RESS[THEIR.IP.ADD.RESS] Apr 4 09:28:17 mbsnet-pf1 charon: 05[CFG] <241> selected peer config "con19000" Apr 4 09:28:17 mbsnet-pf1 charon: 05[IKE] <con19000|241> IKE_SA con19000[241] established between MY.IP.ADD.RESS[MY.IP.ADD.RESS]...THEIR.IP.ADD.RESS[THEIR.IP.ADD.RESS] Apr 4 09:28:17 mbsnet-pf1 charon: 05[ENC] <con19000|241> generating ID_PROT response 0 [ ID HASH ] Apr 4 09:28:17 mbsnet-pf1 charon: 05[NET] <con19000|241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (76 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 08[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (188 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 08[ENC] <con19000|241> parsed QUICK_MODE request 1473375317 [ HASH SA No ID ID ] Apr 4 09:28:17 mbsnet-pf1 charon: 08[IKE] <con19000|241> received 28800s lifetime, configured 0s Apr 4 09:28:17 mbsnet-pf1 charon: 08[ENC] <con19000|241> generating QUICK_MODE response 1473375317 [ HASH SA No ID ID ] Apr 4 09:28:17 mbsnet-pf1 charon: 08[NET] <con19000|241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (172 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 12[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (60 bytes) Apr 4 09:28:17 mbsnet-pf1 charon: 12[ENC] <con19000|241> parsed QUICK_MODE request 1473375317 [ HASH ] Apr 4 09:28:17 mbsnet-pf1 charon: 12[IKE] <con19000|241> CHILD_SA con19003{253} established with SPIs c80f364c_i 852f79ed_o and TS MY.LAN.IP.NET/24|/0 === THEIR.LAN.SUB.NET1/32|/0 Apr 4 09:28:18 mbsnet-pf1 charon: 07[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (76 bytes) Apr 4 09:28:18 mbsnet-pf1 charon: 07[ENC] <con19000|241> parsed INFORMATIONAL_V1 request 643855776 [ HASH D ] Apr 4 09:28:18 mbsnet-pf1 charon: 07[IKE] <con19000|241> received DELETE for ESP CHILD_SA with SPI f1221d88 Apr 4 09:28:18 mbsnet-pf1 charon: 07[IKE] <con19000|241> CHILD_SA not found, ignored Apr 4 09:28:19 mbsnet-pf1 charon: 08[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:28:26 mbsnet-pf1 charon: 15[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:28:29 mbsnet-pf1 charon: 16[IKE] <con19000|238> sending retransmit 5 of request message ID 0, seq 1 Apr 4 09:28:29 mbsnet-pf1 charon: 16[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:28:35 mbsnet-pf1 charon: 14[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:28:42 mbsnet-pf1 charon: 12[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {109} Apr 4 09:28:43 mbsnet-pf1 charon: 12[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {108} Apr 4 09:29:24 mbsnet-pf1 charon: 14[ENC] <con19000|241> parsed QUICK_MODE request 797297949 [ HASH SA No ID ID ] Apr 4 09:29:24 mbsnet-pf1 charon: 14[IKE] <con19000|241> received 28800s lifetime, configured 0s Apr 4 09:29:24 mbsnet-pf1 charon: 14[ENC] <con19000|241> generating QUICK_MODE response 797297949 [ HASH SA No ID ID ] Apr 4 09:29:24 mbsnet-pf1 charon: 14[NET] <con19000|241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (188 bytes) Apr 4 09:29:24 mbsnet-pf1 charon: 05[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (60 bytes) Apr 4 09:29:24 mbsnet-pf1 charon: 05[ENC] <con19000|241> parsed QUICK_MODE request 797297949 [ HASH ] Apr 4 09:29:24 mbsnet-pf1 charon: 05[IKE] <con19000|241> CHILD_SA con19001{255} established with SPIs c079570a_i b1d57c84_o and TS MY.LAN.IP.NET/24|/0 === THEIR.LAN.SUB.NET2/24|/0 Apr 4 09:29:24 mbsnet-pf1 charon: 05[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (76 bytes) Apr 4 09:29:24 mbsnet-pf1 charon: 05[ENC] <con19000|241> parsed INFORMATIONAL_V1 request 3723224733 [ HASH D ] Apr 4 09:29:24 mbsnet-pf1 charon: 05[IKE] <con19000|241> received DELETE for ESP CHILD_SA with SPI 834e133b Apr 4 09:29:24 mbsnet-pf1 charon: 05[IKE] <con19000|241> CHILD_SA not found, ignored Apr 4 09:29:29 mbsnet-pf1 charon: 09[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {109} Apr 4 09:29:30 mbsnet-pf1 charon: 06[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (188 bytes) Apr 4 09:29:30 mbsnet-pf1 charon: 06[ENC] <con19000|241> parsed QUICK_MODE request 3795121176 [ HASH SA No ID ID ] Apr 4 09:29:30 mbsnet-pf1 charon: 06[IKE] <con19000|241> received 28800s lifetime, configured 0s Apr 4 09:29:30 mbsnet-pf1 charon: 06[ENC] <con19000|241> generating QUICK_MODE response 3795121176 [ HASH SA No ID ID ] Apr 4 09:29:30 mbsnet-pf1 charon: 06[NET] <con19000|241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (188 bytes) Apr 4 09:29:30 mbsnet-pf1 charon: 13[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (60 bytes) Apr 4 09:29:30 mbsnet-pf1 charon: 13[ENC] <con19000|241> parsed QUICK_MODE request 3795121176 [ HASH ] Apr 4 09:29:30 mbsnet-pf1 charon: 13[IKE] <con19000|241> CHILD_SA con19000{256} established with SPIs ca872813_i a5f827a7_o and TS MY.LAN.IP.NET/24|/0 === THEIR.LAN.SUB.NET3/24|/0 Apr 4 09:29:30 mbsnet-pf1 charon: 13[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (76 bytes) Apr 4 09:29:30 mbsnet-pf1 charon: 13[ENC] <con19000|241> parsed INFORMATIONAL_V1 request 3250968574 [ HASH D ] Apr 4 09:29:30 mbsnet-pf1 charon: 13[IKE] <con19000|241> received DELETE for ESP CHILD_SA with SPI ddf8f1f4 Apr 4 09:29:30 mbsnet-pf1 charon: 13[IKE] <con19000|241> CHILD_SA not found, ignored Apr 4 09:29:44 mbsnet-pf1 charon: 11[IKE] <con19000|238> giving up after 5 retransmits Apr 4 09:29:44 mbsnet-pf1 charon: 11[IKE] <con19000|238> peer not responding, trying again (3/3) Apr 4 09:29:44 mbsnet-pf1 charon: 11[IKE] <con19000|238> initiating Main Mode IKE_SA con19000[238] to THEIR.IP.ADD.RESS Apr 4 09:29:44 mbsnet-pf1 charon: 11[ENC] <con19000|238> generating ID_PROT request 0 [ SA V V V V V ] Apr 4 09:29:44 mbsnet-pf1 charon: 11[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:29:48 mbsnet-pf1 charon: 14[IKE] <con19000|238> sending retransmit 1 of request message ID 0, seq 1 Apr 4 09:29:48 mbsnet-pf1 charon: 14[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:29:56 mbsnet-pf1 charon: 05[IKE] <con19000|238> sending retransmit 2 of request message ID 0, seq 1 Apr 4 09:29:56 mbsnet-pf1 charon: 05[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:30:09 mbsnet-pf1 charon: 11[IKE] <con19000|238> sending retransmit 3 of request message ID 0, seq 1 Apr 4 09:30:09 mbsnet-pf1 charon: 11[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:30:32 mbsnet-pf1 charon: 06[IKE] <con19000|238> sending retransmit 4 of request message ID 0, seq 1 Apr 4 09:30:32 mbsnet-pf1 charon: 06[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:31:14 mbsnet-pf1 charon: 15[IKE] <con19000|238> sending retransmit 5 of request message ID 0, seq 1 Apr 4 09:31:14 mbsnet-pf1 charon: 15[NET] <con19000|238> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (180 bytes) Apr 4 09:31:37 mbsnet-pf1 charon: 06[KNL] creating acquire job for policy MY.IP.ADD.RESS/32|/0 === THEIR.IP.ADD.RESS/32|/0 with reqid {109} Apr 4 09:32:30 mbsnet-pf1 charon: 07[IKE] <con19000|238> giving up after 5 retransmits Apr 4 09:32:30 mbsnet-pf1 charon: 07[IKE] <con19000|238> establishing IKE_SA failed, peer not responding Apr 4 09:33:37 mbsnet-pf1 charon: 05[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (188 bytes) Apr 4 09:33:37 mbsnet-pf1 charon: 05[ENC] <con19000|241> parsed QUICK_MODE request 506309004 [ HASH SA No ID ID ] Apr 4 09:33:37 mbsnet-pf1 charon: 05[IKE] <con19000|241> received 28800s lifetime, configured 0s Apr 4 09:33:37 mbsnet-pf1 charon: 05[ENC] <con19000|241> generating QUICK_MODE response 506309004 [ HASH SA No ID ID ] Apr 4 09:33:37 mbsnet-pf1 charon: 05[NET] <con19000|241> sending packet: from MY.IP.ADD.RESS[500] to THEIR.IP.ADD.RESS[500] (188 bytes) Apr 4 09:33:37 mbsnet-pf1 charon: 12[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (60 bytes) Apr 4 09:33:37 mbsnet-pf1 charon: 12[ENC] <con19000|241> parsed QUICK_MODE request 506309004 [ HASH ] Apr 4 09:33:37 mbsnet-pf1 charon: 12[IKE] <con19000|241> CHILD_SA con19002{258} established with SPIs c64c48a2_i ebe60b38_o and TS MY.LAN.IP.NET/24|/0 === THEIR.LAN.SUB.NET4/24|/0 Apr 4 09:33:37 mbsnet-pf1 charon: 12[NET] <con19000|241> received packet: from THEIR.IP.ADD.RESS[500] to MY.IP.ADD.RESS[500] (76 bytes) Apr 4 09:33:37 mbsnet-pf1 charon: 12[ENC] <con19000|241> parsed INFORMATIONAL_V1 request 2475751961 [ HASH D ] Apr 4 09:33:37 mbsnet-pf1 charon: 12[IKE] <con19000|241> received DELETE for ESP CHILD_SA with SPI 8ca208a7 Apr 4 09:33:37 mbsnet-pf1 charon: 12[IKE] <con19000|241> CHILD_SA not found, ignored

Hi,

So I ended up resolving this issue, for those who are interested it was an issue with the AT&T modem.

I have the Arris BGW-210 on both sides of the tunnel. The modem has a setting under Advanced Firewall called ESP ALG, this setting should be disabled if both sides of your tunnel are not behind NAT (pfSense has a public IP).

Thanks for your help getting this resolved, the tunnel is working great, I'm seeing over 300 mbps between the networks.

That depends on the mix of clients mostly.

What you are trying to do there typically requires the Cisco Anyconnect client on Windows anyway.

If you MUST try this, try IKEv2 but that will probably require the strongswan app on android.

There is no 100% universal solution unfortunately. The client support is too varied.

Yes, OpenVPN requires a client but in most cases it is free and your configuration will be substantially similar across any device it supports.

@blackbinary
Hey
This is possible, but you must make changes to the PFsense configuration files (responder side)
As a result of these changes, PFSense will create a config file (ipsec.conf) that will allow strongswan to accept connections from any ip address
Here is an example of how it looks in practice after the changes
In the settings section of Remote Gateway you enter "any"
and the necessary config is ready

c2a5f51c-aaa5-4e48-9629-6de183f1a0e5-image.png

If you write me in the chat your email, I'll send you an email with all instructions

Hello

Just noticed it breaks large packets of UDP :( hopefully we will get fix soon.
https://redmine.pfsense.org/issues/7801

@Konstanti
Ya, I can't tell you how many times I verified the IPSec settings
Magically, the connection was established last night as I left it on while doing some other work. When I returned to have another look, the connection was made. I tried this current configuration multiple times to no avail, so I am baffled as to what the resolution was

I'm booking a meeting with a guy at the other side to start pulling parts and pieces apart to determine the issue

One thing I noticed is that the initial attempts to connect were using port 4500 and the established tunnel is on 500 (I have no firewall logs blocking this and I have rules on WAN in place explicitly allowing UDP 500/4500 and ESP.

Perhaps their end isn't liking the 4500 (they told me they are good with the UDP 4500 mind you)

Sort of feels like Cisco just not wanting to play nice in the sandbox with the other kids.

I'll update with any resolution(s) or comments here

@Juan-Carlos-Gtz
Hey
You're only allowed TCP on the interface IPSEC Mex 2. Other protocols are prohibited. In order to use ping you need to enable ICMP.

pfs2.JPG azurepfs.png

Then post what you have because it most certainly does work.

Might be better to use FRR in place of OpenBGP.