All that matters is that they are syncing and are mostly identical.

Well if the identifier doesn't match the address used it will fail to match in IPsec.

It needs to be built with everything referencing the CARP VIP (or some other common identifier, like an FQDN that resolves to the CARP VIP.) If this is IKEv2 with a server certificate then the CN and SAN there needs to match whatever the client thinks it is connecting to or server certificate validation will fail.

@viragomann said in Slow IPsec / internet when using CARP:

Have you configured the Outbound NAT to use the WAN CARP VIP?

YEP

Why would sync interface traffic ever have to go out the WAN?

Yes, outbound NAT with source any is almost never right - especially to a CARP VIP.

Traffic from Localhost should NAT to the interface address

Traffic from inside hosts should:

Use the local interface CARP VIP as their default gateway Have outbound NAT to the WAN CARP VIP set.

Traffic from the sync interface should never need internet access.

I will try the setup in that page.
In addition, i have no idea whether my original setup is possible or not, may i have suggestion on it?
Thank you for your assistance.

So, in short, the 'seamless, state-table sync' functionality of CARP simply isn't going to work with computers using a VPN service? I'm willing to accept that. I just wanted to be sure.

I suppose one solution is to use the VPN apps (installed on the computers) as an alternative for those computers which must not loose connection on a fail over. I can't think of any that would fall into that category at the moment, but I might test it to know if it's an option. (Only drawback is that you use up a connection for a single computer, instead of many. Not a big deal, now that they give you 6 or so for $4 a month.

Two things I see:

Upstream is not responding at all when sourced from .164. Did you filter that packet capture on icmp? I would expect to see ARP or something there if not.

The replies to pings sourced from .163 should be destined to the CARP MAC address, not the interface MAC address.

It looks like something upstream does not like moving MAC addresses around like CARP does but just a guess at this point.

The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 224.0.0.18. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.

The ISP Layer 3 gear will get the CARP MAC in the "IS AT" response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.

The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.

No. DNS will return every A record for the fqdn but you need a specific one.

You could do it if your DNS was off the firewall using something like BIND views.

Queries for firewall-b.example.com return 192.168.1.3 if received from 192.168.1.0/24
Queries for firewall-b.example.com return 192.168.2.3 if received from 192.168.2.0/24
Etc.

But that seems like a lot of work when this is why people manage these things from a specific network.

@jimp Killing me softly with these words :)

Hello Derelict.
finally i was able to solve the problem due to the misconfigured XMLRPC SYNC section : the two pfsense had different password to admin username . in addition , in one of the servers the ip address in XMLRPC field was blank
thanks for your willing to help me with this issue

This is a UK ISP. They will not put /29 static routes on their edge equipment. That is not an option and I am aware that CARP will not work in this situation hence my question is asking about the alternative virtual IP setups.

What interested me is the IP Alias alternative to CARP listed on this page: https://www.netgate.com/docs/pfsense/firewall/virtual-ip-address-feature-comparison.html

It says it can be used by the firewall to bind/run services and it can be on a different subnet to the real interface IP.

So to explain a bit better, I was wanting to leave the /29 and CARP set up as they are. Then I wanted to add an IP Alias with the /30 on it and the ISP default gateway. The /29 default gateway will be the /30 therefore routing all traffic correctly. As far as I can tell, this should work but I don't have the capability to test and I was hoping someone who knew pfsense better than I would be able to confirm if there are any problems with doing this before I go and break everything on a live circuit...

Thanks,
Colin.

Maybe you can describe your setup a bit and phrase out some detailed question?

-Rico

Hi,

Didn't you seen the screenshots? Everything is explained. Outgoing traffic was done by the master and incoming (reply from server) was going trough the backup.

Finaly after one week of investigation - we've found the problem.

In the Virtual IP defined (used after in NAT 1..1) we've specified the "WAN" interface instead of the WAN CARP interface

I think it would be a great idea to put this information in the troubleshooting guide.

@derelict Understand thank you

Still not sure what you are asking.

There are no loops in an HA setup.

Seems like more of a question for Cisco.

I would say don't use '/' for the time being.

Allright, so that NAT's are not performed like firewall rules top to bottom

Yes. they are. Your any rule at the bottom catches everything so everything has NAT performed on it. The whole point is that some things should not have NAT to the CARP VIP performed on them.

In your example you are STILL natting for sources in 127.0.0.0/8, which is where traffic sourced from the firewall itself might be originating for NAT purposes, which is NOT the default configuration.

If I were you I would scrap what you have start over with a simple WAN/LAN setup on a test bench, read the docs again, and get a handle on what is involved here.

If NAT to the interface address doesn't work, then your /30 /28 scheme, which I said was broken out-of-the-gate, is incompatible with the upstream gear.

And this:

The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 224.0.0.18. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.

The ISP Layer 3 gear will get the CARP MAC in response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.

The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.

All references to ISP gear there should be interpreted in whatever is upstream of the two pfSense nodes in your environment. The Mikrotik and whatever else that is.

I am not sure if this is the best to go but I will increase the MTU to 9000 and see if all goes well :)

That is the last thing I would do if I was in your position.

I would get everything working perfectly and leave jumbo frames out of the picture.