You need a rule like that on the secondary for the initial sync. When that sync happens the rule on the sync interface on the Primary will sync to that interface so it also needs to be in place.

If the rule is on the sync interface on the primary and you end up with nothing on the sync interface on the secondary you likely have an interface mismatch.

Use Status > Interfaces on both to be sure they match. Everything on every interface has to match exactly

Example:
WAN Interface (wan, igb0)
LAN Interface (lan, igb1.223)
MGMT Interface (opt1, igb1.999)

All three elements must match (WAN, wan, igb0) (MGMT, opt1, igb1.999) in the same order.

You should be able to access it on its interface address(es) instead of the CARP VIP(s).

Yeah. Make a host-only network switch for each one and a pfSense interface. Make a pfSense interface for each subnet. Put the firewall rules on each so they pass the traffic.

If you don't want to make a bunch of vmware interfaces, then "tag" VLAN 4095 to a pfSense interface and the VLAN tags will be there so you can make pfSense interfaces on them.

Beware that vmware scrambles your interface order after you add your 6th interface or something.

Bottom line is putting all those subnets on one broadcast domain and expecting them to communicate is simply not the way to go. Each one should be on separate router interfaces so those networks can, you know, be routed to each other.

Regarding the intermittent behavior, you are probably running into issues with different behaviors regarding ICMP redirects which is one of the main problems with that sort of design. I would never expect that to work 100%.

Yea im not sure, coulda swore i set them the same on both. I went ahead and changed the user back to the one i wanted on the primary node. Did a force sync and it works now...heh. hey it works. Thanks for the help!!

@umademelosemyusernamepfsense said in Assigning uplinks to VIPs:

can you still take single out an address from the lot and masquerade it? Or do all have to be 1:1?

I'm not sure I understand your question.

Like I said earlier, I have 13 VIPs. One of them is our gateway. I could specify any of the others as gateways. I use NAT port forwards to connect some of those VIPs to internal servers such as our web server, Nextcloud server etc. I used to also run mail and DNS via NATs but I've scaled back lately and just have the one gateway and two web servers.

Maybe if you described the Big Picture of what you really want to accomplish from a high-level view. A lot of times, we get people who have dreamed up a half-baked solution and then they want specific help with each step when the better course would have been to ask for guidance about the project as a whole. I'm an intermediate-level brain here so maybe one of the bigger brains can see what you're trying to do.

Oh - Derelict, Thanks for pointing me in the right direction and warning me about the reaction I might get if I ask about CARP.

reberhar

@derelict correct.

Hi,
Any idea please ?

Outbound NAT must also be set to use the CARP VIPs.

It is perfectly normal for a traceroute response to appear to come from the interface address not the CARP VIP.

You'll probably need to perform troubleshooting steps to determine what is actually failing and we can go from there.

https://www.netgate.com/docs/pfsense/routing/connectivity-troubleshooting.html

Yes. Just like you would with an rfc1918 network.

If they routed 1.1.1.0/25 to you:

Interface: 1.1.1.1 /25
Usable: 1.1.1.2 - 1.1.1.126
They'd set 1.1.1.1 as the gateway.
Or you could configure DHCP to hand out the addresses if you wanted.
You could also just use a /26, /27, /28, /29, /30, /31 on the inside interface and use the rest of the space for other purposes.

@derelict
You know what, your right, sometimes the simplistic idea is the best one.

Thank you

As long as you are not using it with an HA cluster, you could add an IP alias or CARP VIP inside the proxy ARP range without (many) issues. I would use IP alias only, not CARP VIP. With a CARP VIP there is a potential that equipment on the segment would get different ARP responses for the address. IP alias would be the same as Proxy ARP.

If it's for HA, then toss out proxy ARP entirely.

@teamits
this is what support has suggested and i will be doing that. THANKS!

From support:
%(#000000)[ix0 and ix1 will sense interface down/up as they are discrete router interfaces.

SYNC on the switchport will not, but as it is not a CARP interface used to determine MASTER/BACKUP status (No CARP VIP on it) that will not affect the performance of the HA pair and it will failover normally. You would not want a failover event if the SYNC interface is disconnected anyway.

If you reassign the LAN interface to one of the ix interfaces, you should simply be able to create a new interface for SYNC using the lagg0.4091 interface that should be available for use/assignment after reassigning LAN to ix0 or ix1.

Then just number and add the firewall rules to SYNC interfaces on each side as usual. The default switch configuration should be adequate.]

Right. There was no CARP VIP on the secondary.

A simple edit/save of that VIP on the primary should have taken care of it.

Perhaps XMLRPC sync was not working/connected at the time that VIP was created or something else anomalous happened.

Moved to HA/Carp section.

Hi. Dont know if you re still on this but... i partially set up am ha environment using carp. However, I havemt been able to set up an ip sec vpn since I can’t put the same virtual public ip to both nodes. I have tried to fix fhis using azure load balancer but it is not working right.

@derelict said in High Availibility Failover stops SSH Session:

@vadim1 said in High Availibility Failover stops SSH Session:

VLAN200 tcp 10.10.231.252:38624 -> 10.10.231.253:519 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B
VLAN200 tcp 10.10.231.252:38624 -> 10.10.231.253:519 ESTABLISHED:ESTABLISHED 619.731 K / 619.733 K 34.29 MiB / 34.29 MiB

Those states are DHCP failover connections between the two firewalls and don't show anything about the SSH problems you are reporting.

before failover
Primary
VLAN20 tcp 10.10.190.5:17979 -> 10.10.224.1:22 ESTABLISHED:ESTABLISHED 180 / 116 14 KiB / 14 KiB
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 11.598 K / 11.597 K 657 KiB / 657 KiB
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B

BackUp

VLAN20 tcp 10.10.190.5:17979 -> 10.10.224.1:22 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 11.439 K / 11.438 K 648 KiB / 648 KiB

after failover

Primary

VLAN20 tcp 10.10.190.5:17979 -> 10.10.224.1:22 ESTABLISHED:ESTABLISHED 180 / 116 14 KiB / 14 KiB
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 11.765 K / 11.764 K 667 KiB / 667 KiB
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B

BackUp

VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 0 / 0 0 B / 0 B
VLAN20 tcp 10.10.231.253:30816 -> 10.10.231.252:520 ESTABLISHED:ESTABLISHED 11.758 K / 11.757 K 666 KiB / 666 KiB

If the DHCP servers are both set correctly (that setting should sync from primary to secondary), what do the clients report as their default gateway?

default via 10.10.231.254 dev ens160 proto dhcp metric 100
but using traceroute it is going through 10.10.231.253, is it the way it should work or does it has to go through 10.10.231.254?
traceroute to google.com (172.217.17.238), 30 hops max, 60 byte packets
1 localhost (10.10.231.253) 0.129 ms 0.157 ms 0.183 ms

Generally, with pfsync running, if the clients are set to use the CARP VIP as their default gateway and outbound NAT for that client network uses the WAN CARP VIP for outbound NAT, then they will have synced states and a failover will not break the client connections.

Looking at the states will not show the default gateway used but will show the outbound NAT used (if it is necessary to NAT).

OK, thanks!