Need clarify with CARP and multiple VLAN subnets

piwwo

Well I did that but now I have the problem that one of the interfaces doesn't get it's VIP synced.

This is what I did:

First configured the VIPs of all the VLAN interfaces with their according subnets 10.1, 192.168.30.1 ect. on the master.
Set the physically IPs to 10.2 ect on that box (master).
Made a dedicated pfSync interface with it's own subnet (192.168.255.1/24).

Installed the backup box.
Created all VLANs on that box (the page High Avail. Sync doesn't provide an option to sync VLANs configuration under system_hasync.php selectors), same VLAN IDs, same interface names, same subnets just with .3 instead of .2 "real" IP).
Hit the force sync button.

VLAN1 and VLAN3 gets synchronized, VIPs appears in the list.
VLAN4 produces an error claiming that there is "no matching interface for VIP 192.168.40.1 skipping".
The VIP gets created, with the same VHID as on the master, but it is not assigned to an interface.

When I assign the interface manually and force sync again, the assignment disappears remaining the VIP unassigned to an interface.

How are the interfaces matched, how does CARP create a certain VIP for an interface and why does it work with the first two but not the third?

Since two interfaces get synchronized but one doesn't, I think there is something different with that one interface but I can not find what it might be.

Derelict

Your interfaces need to match exactly at both the hardware and the pfSense level. igb0, igb1, igb2, ix0, ix1, etc. WAN, LAN, OPT1, OPT2, OPT3, etc.

Define everything on the secondary in exactly the same order as the primary.

piwwo

Ok. That might be the problem.

One is a soekris and the other is a pcengines. The soekrist names the interfaces em0-3 and the pcengines re0-2
The VLANs are on em2, em2_vlan3 and em2_vlan4 on the pcengines they are accordingly re2 for LAN, VLAN1 and VLAN2 and 3 are on re2_vlan2 and re2_vlan3
Could this be the issue?

Oh btw there was another question: I assume it's not possible to sync package configs within the CARP, or? So like Squid when I make a change to the proxy or blacklists I would have to do it on both right?

Derelict

Yeah. Hate to say it but, "good luck with that."

HA needs like hardware to properly sync both states and configs. Going down that road is setting yourself up for almost certain misery and despair.

piwwo

Yes I figured that out now, I hope it works now with two equal pcengines, thanks.

One last misunderstanding remains tho: Is there a way to synchronize not only pfsense internals config such as firewall rules, NAT, certs, Captive Portal ect but also installed packet settings such as Squid proxy settings, pfblocker, IP and domain blocklists, RADIUS accounts ect via XMLRPC? I assume I have to install all these packages manually before sync but I don't know if their settings will be transferred when they are installed. I'd need that especially for RADIUS so that I don't have to create all accounts twice.

If not via CARP XMLRPC is there maybe another protocol/tool that can sync these?

Derelict

Look for sync settings in the individual packages.

piwwo

I think this will do, thanks.

I also noticed something strange: The book suggests to set failover IP in the DHCP config to the IP of the backup system. I did that and suddenly DHCP did show weird behaviour. Known some MAC addresses got their IP number from DHCP others didn't, new clients did not get an answer at all from DHCP either. I double checked that I didn't accidentally select something like "Only the clients defined below will get DHCP leases from this server." or blocked DHCP in the firewall. After removing 192.168.10.3 from "Failover IP" field, DHCP worked normally again.

I can only guess that this has something to do with CARP and that I have the backup system up and fully synced and also attached to the same VLANs for this to work? The backup system is setup and synced just not yet plugged into the switch yet because I first wanted to test one firewall before plugging in the other to the switch and setting the according trunk ports for that firewall, that's why I am guessing so. I lack of insight into DHCP and CARP to answer this question my self tho.

piwwo

And today suddenly the DHCP gives out wrong routes, but only on one network the .10.0

enp2s0: rebinding lease of 192.168.10.10
enp2s0: leased 192.168.10.10 for 7200 seconds
enp2s0: adding route to 192.168.10.0/24
enp2s0: adding default route via 192.168.10.3
forked to background, child pid 28879

tcpdump shows that the dhcp on 192.168.10.3 which is the backup answers but not on 192.168.10.2 which is the master.

On the other network I get a correct default route to the CARP VIP (although there both 30.2 and 30.3 answer to DHCP requests, just .2 is the first reply I get.)

Derelict

That is how HA DHCP works. They share the load. Make sure both DHCP servers are configured to hand out the CARP VIP as the default gateway for the clients. These settings should sync primary -> secondary but it sounds like you should check them both.

piwwo

Yes I double checked that and everything gets synced (but for the failover peer IP which I then had to set manually ofcourse because this can't be synced).

I attached screenshots of the dhcp on VLAN1 - the other VLAN look the same just with .30 and .40 instead of .10

But on VLAN1 I get an answer from 192.168.10.3 pushing default route to 192.168.10.3 instead of the CARP VIP
On the other VLANs I get the proper CARP VIP for that VLAN.

dhcp_master-side.png_thumb

dhcp_backup-side.png_thumb

Derelict

Not really sure what you did but this works every time I try it. I generally follow the procedure outlined here: https://portal.pfsense.org/docs/book/highavailability/example-redundant-configuration.html

It isn't exactly synced but the XMLRPC is smart enough to do the right thing in that case, setting the peer IP address on the secondary to the LAN address of the primary.

The Failover Peer IP allows the daemon to communicate with the peer directly in this subnet to exchange data such as lease information. When the settings synchronize to the secondary, this value is adjusted automatically so the secondary points back to the primary.

I would packet capture to be sure the server you think is responding is actually the one responding (look at the MAC addresses.) You might also check for a DHCP static mapping that sets the wrong router.

I wouldn't make changes to it but the DHCP config file is /var/dhcpd/etc/dhcpd.conf. Search that for 192.168.10.3.

Is your My State/Peer State normal/normal in Status > DHCP Leases on both nodes?

piwwo

No the leases been not the same on both nodes. In fact, on the master there was no lease for the 192.168.10.0/24 network but on the backup there was one.

I stopped the dhcp on the master and looked with tcpdump on that master if it gets dhcp requests. There been some, but no reply (because it was disabled). However the PC sending the request got an answer - from the backup. So I checked the CARP status but backup was (correctly) in backup mode while master showed (correctly) master for all VLANs. I then disabled CARP on both, killed DHCP on both and started DHCP on master before enabling CARP on both again (master first). Then suddenly I got the correct route. After starting DHCP on the backup again and restarting DHCP on the PC in VLAN1, I got the correct default route pushed finally.

I have no slightest clue why this works out of a sudden after I disabled CARP and DHCP and enabled it again on both boxes. I would love to understand this behaviour to know a solution when the same happens again but I see no hint.

piwwo

Also is there a reason why both DHCP servers answer instead of the actual master (or backup in a failover situation)?

Derelict

Because that's the way ISC DHCPD works in failover mode.