High Avail secondary node IPs - How to find it

stephenw10

Hmm, well it's possible to set individual rules to not sync. That might have been done for some reason.

michmoor

@stephenw10

I thought the same as well but the "No XMLRPC Sync" checkbox is unchecked.

Logs show no errors. I even tried re-creating the interface and still no luck

stephenw10

No error on the Secondary either?

michmoor

@stephenw10

Figured it out.
The primary has OPT3 - IPsec1
The secondary has OPT4 - IPsec1

I was just looking at the IPsec1 identifier but clearly the OPT has to match. So i deleted the interface on the secondary and made sure that when it got recreated the OPT identifier matched. Once that's done Rules synced across.

The last thing I'm dealing with now is IPsec failover is very slow. An outage of over 3-4 minutes. CARP WAN address is being used but i do notice in my logs that the firewall keeps trying to initiate a connection using the WAN address and not the WAN CARP address. Is this normal?

Eventually, the firewalls speak and all is well.

stephenw10

What are they connected to?

If the IPSec is on the CARP VIP at this end the secondary should start trying to connect as soon as it fails over but the remote side will have to time-out before starts trying.

michmoor

@stephenw10 Remote side is me. How do i speed up the cutover? Is there a way?

stephenw10

So remote is a mobile IPSec client?

You can change the dpd settings for the client so it times out faster.

michmoor

@stephenw10 This is an IPsec site2site

stephenw10

Ok well if the other side is pfSense the default values are 10s and 5 failure so you can change that to, say, 5s and 3 failures.

michmoor

@stephenw10
Yep we are dealing with two pfSense boxes.
I am changing the value on both sides now and testing. Will let you know.

So something like this?

michmoor

@stephenw10 no deal.

It recovers...but slow.

It might make sense to create another IPsec tunnel to the Backup firewall.
Im thinking i can handle routing by placing both Master and Backup in a Gateway group and set Master as Tier 1 and Backup as Tier 2

edit: i actually don't know how that will work..
Any changes on the Backup will get overwritten...hmmm

stephenw10

Check the logs at both ends and see what's happening. Which end is delaying the failover.

How long does it actually take?

michmoor

@stephenw10
I initiated a failover on primary and on the secondary i went ahead and tried to establish a tunnel via CLI.

This is the output. The output is from the Backup pfsense trying to initiate a IKE P1 to my home pfsense

 sudo swanctl --initiate --ike con1
[IKE] retransmit 1 of request with message ID 0
[NET] sending packet: from 192.168.35.6[500] to 104.13.92.x[500] (464 bytes)
[IKE] retransmit 2 of request with message ID 0
[NET] sending packet: from 192.168.35.6[500] to 104.13.92.x[500] (464 bytes)
[IKE] retransmit 3 of request with message ID 0
[NET] sending packet: from 192.168.35.6[500] to 104.13.92.x[500] (464 bytes)
[IKE] retransmit 4 of request with message ID 0
[NET] sending packet: from 192.168.35.6[500] to 104.13.92.x[500] (464 bytes)
[IKE] retransmit 5 of request with message ID 0
[NET] sending packet: from 192.168.35.6[500] to 104.13.92.x[500] (464 bytes)
[IKE] giving up after 5 retransmits
[IKE] establishing IKE_SA failed, peer not responding
initiate failed: establishing IKE_SA 'con1' failed

Now just to let you know firewall at the location I'm managing is sitting behind a Cisco router that is performing NAT
192.168.35.6 is the NAT for the WAN VIP so the pfsense has a RFC1918 WAN address but the Cisco is doing the NAT.
For what its worth i do see translations on the Cisco so that's operating correctly.

Eventually the tunnel will restablish.

stephenw10

So 192.168.35.6 is the WAN CARP VIP for the HA pair? You shouldn't have to do anything at the CLI. When the VIP fails over the secondary should try to connect.

What is logged on the other side?

michmoor

@stephenw10
So for clarity

The NATs are like this

192.168.35.6 <> x.x.188.125 - CARP VIP
192.168.35.5 <> x.x.188.124 - Secondary WAN pfsense
192.168.35.4 <> x.x.188.123 - Primary WAN pfsense

So when I failover to backup what i see on my firewall is UDP/500 traffic coming from the Secondary WAN interface (not the CARP) which i found odd. I see that happening for a few times and then after awhile i see the CARP VIP finally reach out and establish a VPN

michmoor

@stephenw10
I think i figured out the problem.
The firewall is behind a NAT box
When i initiate a P1 connection its trying to talk out on port 500.

Obviously, this breaks IKE all together as after the translation, IKE drops packets.

NAT-T is set to Force. Yet its still trying to go out on port 500.
Any ideas as to why its doing that?

I have also restarted the IPsec daemon process but same results.

edit

Confirmed. This is for sure happening. On the Cisco i see the translations, Its trying on port 500 even though NAT-T is set to Force

udp 103.127.188.124:20402 192.168.35.5:20402 x.x.92.128:500 x.x.92.128:500

michmoor

@stephenw10

Ok...what a marathon day with IPsec.

I think i figured out the problem and now failover happens faster.

There was an Outbound NAT policy that said
Interface: WAN
Source: Any
Destination: Any
Nat Address: 'CARP WAN Address'

That seemed incorrect because my assumption is that any source address would include the firewall source address as well.
So i changed the source to 'LAN subnets' and things are looking much better.

Failover is quicker but i do find that in some cases i have to hop onto thee new Master firewall and initiate P1/P2 (it doesn't initiate right away sometimes).

viragomann

@michmoor said in High Avail secondary node IPs - How to find it:

There was an Outbound NAT policy that said
Interface: WAN
Source: Any
Destination: Any
Nat Address: 'CARP WAN Address'

That seemed incorrect because my assumption is that any source address would include the firewall source address as well.

That's correct. Outbound traffic from the firewall itself must not be natted to the CARP VIP.

If you have multiple local subnets and want to use any for the source you can override the default outbound NAT rule by additional ones for the firewall itself at the top of the rule set:

Also remember, that connections to port 500 must keep the port static.

stephenw10

Yup, that^.

Outbound NAT rules should almost never use source 'any'. Always define the subnets you actually need translation from to avoid over matching. IPSec is most commonly broken by that but other things can be.

michmoor

@stephenw10
So on my firewall, i have the following SNAT rules. Do i need to create one for NAT-T as shown in your picture @viragomann