Need clarify with CARP and multiple VLAN subnets

piwwo · Aug 11, 2016, 9:14 AM

Hello

I have the following setup

Netgear smart switch with pfSense as a router/firewall on port 1, tagged
Switch port 2 and 3 are vlan1 (preconfigured in the switch) and get dhcp for LAN from pfsense as 192.168.10.0/24 network. Their gateway is 192.168.10.1 (LAN port of pfsense). Firewall allows all communication on this network to the internet.
Switch port 4 is for the printer in untagged vlan3. Port 1 is tagged member of vlan3 too. pfSense gives dhcp to 192.168.30.0/24 network, firewall forbids any communication to the internet and only printer needed ports from other internal networks.
Switch port 5 and 6 are PG and SFW only, untagged member of vlan4. Switch port 1 tagged member of vlan4 too. Everything blocked but proxy on 3128, so everyone has to use proxy there. Squidguard for blocking adult, violence, gambling, and hacking/malware ect domains/urls. dhcp 192.168.40.0/24
pfSense VLAN configuration manages dhcp for different VLANs so everyone gets their own subnet using only one port on the pfSense thanks to the vlan tags.
every VLAN has their corresponding 192.168.10.1, .30.1, .40.1 as their dedicated default gateway.


VLAN1 (default VLAN in the switch)
    DHCP 192.168.10.0/24
    Default GW 192.168.10.1

(VLAN2 preconfigured in the switch, not used)    

VLAN3 (custom VLAN in the switch)
    DHCP 192.168.30.0/24
    Default GW 192.168.30.1

VLAN4 (custom VLAN in the switch)
    DHCP 192.168.40.0/24
    Default GW 192.168.40.1

Now I wanted to try setting up a CARP with a second pfSense running the same configuration redundant. Docs shows only with 1 LAN and no VLANs.

https://doc.pfsense.org/images/e/e7/CARP_Setup.png

When setting up the virtual IPs for that CARP on LAN side, documentation suggests setting up virtual IPs for LAN and for WAN. Since I have only one WAN, the configuration for WAN is no-brainer. But since every VLAN has it's own DHCP and gateway address, do I have to add these dedicated virtual IPs for every VLAN subnet?

I am a little confused here. Thank you for clarify.

Derelict · Aug 11, 2016, 9:25 AM

Yes.

You need to assign three addresses for every interface. Primary firewall interface, secondary firewall interface, and the CARP VIP. The DHCP server on each interface should assign the CARP VIP as the default gateway for the clients and possibly as the DNS server if you rely on that.

A normal convention is to assign x.x.x.2 to the primary firewall interface, x.x.x.3 to the secondary firewall interface, and x.x.x.1 as the CARP VIP.

piwwo · Aug 11, 2016, 9:41 AM

Thank you.

So I will have to setup something like this?


VLAN1 (default VLAN in the switch)
    DHCP 192.168.10.0/24
    Default GW 192.168.10.1

     pfSense 1 
         Interface IP 192.168.10.2
         CARP VIP 192.168.10.1
     pfSense 2
         Interface IP 192.168.10.3
         CARP VIP 192.168.10.1

VLAN3 (custom VLAN in the switch)
    DHCP 192.168.30.0/24
    Default GW 192.168.30.1

     pfSense 1 
         Interface IP 192.168.30.2
         CARP VIP 192.168.30.1
     pfSense 2
         Interface IP 192.168.30.3
         CARP VIP 192.168.30.1

VLAN4 (custom VLAN in the switch)
    DHCP 192.168.40.0/24
    Default GW 192.168.40.1

     pfSense 1 
         Interface IP 192.168.40.2
         CARP VIP 192.168.40.1
     pfSense 2
         Interface IP 192.168.40.3
         CARP VIP 192.168.40.1

(And the third Interface on both pfSense for pfSync network lets say 192.168.4.1 respective 192.168.4.2)

Derelict · Aug 11, 2016, 9:50 AM

That's about it, yes.

Make sure all your interfaces on both nodes are assigned in exactly the same order.

HA is extremely reliable if it is configured correctly. If not configured correctly, not so much.

Hands down, the best source of information about creating an HA cluster is in the pfSense book available with a gold membership here:

https://www.pfsense.org/our-services/gold-membership.html

piwwo · Aug 11, 2016, 11:55 AM

Yes I have the book but they only explain multi-WAN and a setup with DMZ which is similar but more complicated than what I want since it has two switches.

Derelict · Aug 11, 2016, 5:37 PM

No they don't.

https://portal.pfsense.org/docs/book/highavailability/example-redundant-configuration.html

Exactly what we're talking about here. Complete procedure for preparing a primary and bringing up a secondary.

piwwo · Aug 17, 2016, 12:28 PM

Well I did that but now I have the problem that one of the interfaces doesn't get it's VIP synced.

This is what I did:

First configured the VIPs of all the VLAN interfaces with their according subnets 10.1, 192.168.30.1 ect. on the master.
Set the physically IPs to 10.2 ect on that box (master).
Made a dedicated pfSync interface with it's own subnet (192.168.255.1/24).

Installed the backup box.
Created all VLANs on that box (the page High Avail. Sync doesn't provide an option to sync VLANs configuration under system_hasync.php selectors), same VLAN IDs, same interface names, same subnets just with .3 instead of .2 "real" IP).
Hit the force sync button.

VLAN1 and VLAN3 gets synchronized, VIPs appears in the list.
VLAN4 produces an error claiming that there is "no matching interface for VIP 192.168.40.1 skipping".
The VIP gets created, with the same VHID as on the master, but it is not assigned to an interface.

When I assign the interface manually and force sync again, the assignment disappears remaining the VIP unassigned to an interface.

How are the interfaces matched, how does CARP create a certain VIP for an interface and why does it work with the first two but not the third?

Since two interfaces get synchronized but one doesn't, I think there is something different with that one interface but I can not find what it might be.

Derelict · Aug 17, 2016, 3:15 PM

Your interfaces need to match exactly at both the hardware and the pfSense level. igb0, igb1, igb2, ix0, ix1, etc. WAN, LAN, OPT1, OPT2, OPT3, etc.

Define everything on the secondary in exactly the same order as the primary.

piwwo · Aug 17, 2016, 3:52 PM

Ok. That might be the problem.

One is a soekris and the other is a pcengines. The soekrist names the interfaces em0-3 and the pcengines re0-2
The VLANs are on em2, em2_vlan3 and em2_vlan4 on the pcengines they are accordingly re2 for LAN, VLAN1 and VLAN2 and 3 are on re2_vlan2 and re2_vlan3
Could this be the issue?

Oh btw there was another question: I assume it's not possible to sync package configs within the CARP, or? So like Squid when I make a change to the proxy or blacklists I would have to do it on both right?

Derelict · Aug 17, 2016, 11:39 PM

Yeah. Hate to say it but, "good luck with that."

HA needs like hardware to properly sync both states and configs. Going down that road is setting yourself up for almost certain misery and despair.

piwwo · Aug 19, 2016, 7:44 AM

Yes I figured that out now, I hope it works now with two equal pcengines, thanks.

One last misunderstanding remains tho: Is there a way to synchronize not only pfsense internals config such as firewall rules, NAT, certs, Captive Portal ect but also installed packet settings such as Squid proxy settings, pfblocker, IP and domain blocklists, RADIUS accounts ect via XMLRPC? I assume I have to install all these packages manually before sync but I don't know if their settings will be transferred when they are installed. I'd need that especially for RADIUS so that I don't have to create all accounts twice.

If not via CARP XMLRPC is there maybe another protocol/tool that can sync these?

Derelict · Aug 19, 2016, 3:06 PM

Look for sync settings in the individual packages.

piwwo · Aug 24, 2016, 3:02 PM

I think this will do, thanks.

I also noticed something strange: The book suggests to set failover IP in the DHCP config to the IP of the backup system. I did that and suddenly DHCP did show weird behaviour. Known some MAC addresses got their IP number from DHCP others didn't, new clients did not get an answer at all from DHCP either. I double checked that I didn't accidentally select something like "Only the clients defined below will get DHCP leases from this server." or blocked DHCP in the firewall. After removing 192.168.10.3 from "Failover IP" field, DHCP worked normally again.

I can only guess that this has something to do with CARP and that I have the backup system up and fully synced and also attached to the same VLANs for this to work? The backup system is setup and synced just not yet plugged into the switch yet because I first wanted to test one firewall before plugging in the other to the switch and setting the according trunk ports for that firewall, that's why I am guessing so. I lack of insight into DHCP and CARP to answer this question my self tho.

piwwo · Aug 25, 2016, 7:28 AM

And today suddenly the DHCP gives out wrong routes, but only on one network the .10.0

enp2s0: rebinding lease of 192.168.10.10
enp2s0: leased 192.168.10.10 for 7200 seconds
enp2s0: adding route to 192.168.10.0/24
enp2s0: adding default route via 192.168.10.3
forked to background, child pid 28879

tcpdump shows that the dhcp on 192.168.10.3 which is the backup answers but not on 192.168.10.2 which is the master.

On the other network I get a correct default route to the CARP VIP (although there both 30.2 and 30.3 answer to DHCP requests, just .2 is the first reply I get.)

Derelict · Aug 25, 2016, 4:00 PM

That is how HA DHCP works. They share the load. Make sure both DHCP servers are configured to hand out the CARP VIP as the default gateway for the clients. These settings should sync primary -> secondary but it sounds like you should check them both.

piwwo · Aug 26, 2016, 7:37 AM

Yes I double checked that and everything gets synced (but for the failover peer IP which I then had to set manually ofcourse because this can't be synced).

I attached screenshots of the dhcp on VLAN1 - the other VLAN look the same just with .30 and .40 instead of .10

But on VLAN1 I get an answer from 192.168.10.3 pushing default route to 192.168.10.3 instead of the CARP VIP
On the other VLANs I get the proper CARP VIP for that VLAN.

Derelict · Aug 26, 2016, 8:03 AM

Not really sure what you did but this works every time I try it. I generally follow the procedure outlined here: https://portal.pfsense.org/docs/book/highavailability/example-redundant-configuration.html

It isn't exactly synced but the XMLRPC is smart enough to do the right thing in that case, setting the peer IP address on the secondary to the LAN address of the primary.

The Failover Peer IP allows the daemon to communicate with the peer directly in this subnet to exchange data such as lease information. When the settings synchronize to the secondary, this value is adjusted automatically so the secondary points back to the primary.

I would packet capture to be sure the server you think is responding is actually the one responding (look at the MAC addresses.) You might also check for a DHCP static mapping that sets the wrong router.

I wouldn't make changes to it but the DHCP config file is /var/dhcpd/etc/dhcpd.conf. Search that for 192.168.10.3.

Is your My State/Peer State normal/normal in Status > DHCP Leases on both nodes?

piwwo · Aug 26, 2016, 8:38 AM

No the leases been not the same on both nodes. In fact, on the master there was no lease for the 192.168.10.0/24 network but on the backup there was one.

I stopped the dhcp on the master and looked with tcpdump on that master if it gets dhcp requests. There been some, but no reply (because it was disabled). However the PC sending the request got an answer - from the backup. So I checked the CARP status but backup was (correctly) in backup mode while master showed (correctly) master for all VLANs. I then disabled CARP on both, killed DHCP on both and started DHCP on master before enabling CARP on both again (master first). Then suddenly I got the correct route. After starting DHCP on the backup again and restarting DHCP on the PC in VLAN1, I got the correct default route pushed finally.

I have no slightest clue why this works out of a sudden after I disabled CARP and DHCP and enabled it again on both boxes. I would love to understand this behaviour to know a solution when the same happens again but I see no hint.

piwwo · Aug 26, 2016, 9:09 AM

Also is there a reason why both DHCP servers answer instead of the actual master (or backup in a failover situation)?

Derelict · Aug 26, 2016, 9:31 AM

Because that's the way ISC DHCPD works in failover mode.