pfSense HA LAN Interfaces Only
-
Hello;
I currently have 1 main bare metal pfsense firewall with the following:
-
WAN = PPPOE Single Static IP
-
LAN = 192.168.2.1/24
-
IOT = VLAN on LAN 10.10.10.1/24
-
NOT = VLAN on LAN 10.10.20.1/24
-
Cam = VLAN on LAN 10.10.30.1/24
I have just prepared a pfsense VM via unraid that I would like to use as a failover (not really like a HA since I only have the 1 WAN IP).
If and ever the main pf goes down, i am ok with a 10 min downtime while i swing the WAN cable over from main to my VM pf.
Question; I was able to get sync to work via the LAN interface - but I am reading this is not a good idea. Should i add a new interface to both systems set it up as SYNC and use that for HA?
2nd Question; with HA sync turned on i found i was getting all sorts of duplicate IP overlappings as both routers were on at the same time and i assume LAN was causing issues as both were in sync and both connected to my unifi switch. How do i make this work so that the main/master is only serving the network while my VM is on backup for LAN only. As for WAN, the VM will not have it connected until i need to ever swing it over.
Thank you very much!
-
-
@iptvcld said in pfSense HA LAN Interfaces Only:
Question; I was able to get sync to work via the LAN interface - but I am reading this is not a good idea.
Since you don’t need to sync states, but only the configuration, it‘s ok to do it on the LAN.
2nd Question; with HA sync turned on i found i was getting all sorts of duplicate IP overlappings as both routers were on at the same time and i assume LAN was causing issues as both were in sync and both connected to my unifi switch. How do i make this work so that the main/master is only serving the network while my VM is on backup for LAN only.
Disconnect the interfaces from the VM except the sync interface.
If you use LAN for sync you have to set the backup to another IP till you switch over. -
@viragomann So y other thought was to not connect the LAN interface cable to the VM until i ever need to and at that point I would swing over LAN and WAN..
But; i was wondering since I have a SYNC interface now (ordered the extra LAN cards) that I want to keep my LAN active in Master/Backup mode. Do I need to set CARP on the LAN interface for that to work and then change my DHCP gateway to the CARP VIP IP and the failover peer IP to the VM interface IP? Will this work?
As noted in my post i have 3 VLANS (IOT, NOT, CAM) all using the LAN interface, would I need to create CARP VIP's for those as well or just the main LAN?
-
I am also reading that my LAN interface and any VLAN interfaces have need to have 3 IP's assigned
Main: x.x.x.2 - VM: x.x.x.3 and CARP VIP x.x.x.1 and i need to do this for all my VLAN interfaces?
Issue with this is that I already have a heavy sever unit using x.x.x.3 and alot of item use that; what would i do in this case?
Thanks,
-
@iptvcld
If you set up CARP, there would be no need to disconnect or change the IP to switch over to the backup.
In this case you need to configure a CARP VIP for all used interfaces, for LAN and all VLAN.I am also reading that my LAN interface and any VLAN interfaces have need to have 3 IP's assigned
Main: x.x.x.2 - VM: x.x.x.3 and CARP VIP x.x.x.1 and i need to do this for all my VLAN interfaces?When you set up CARP, change all existing interface IPs to another one and set the CARP IP to that ones you used till now, since these have to be the gateways in your subnet. So you don't need to change the interface settings on the devices behind.
The real interface IPs can be any in the subnet. You can set them to whatever is not in used like .251 and .252.
-
@viragomann Thank you very much for the help; I am going to implement this today and will paste a screenshot and hopefully you will be able to let me know if i did it correct as a check point for me.
Question.. Outbound NAT Rule, i have been reading this needs to be updated as well - is this the case only if i was doing CARP for my WAN? In my case I am not going to have WAN connected in my failover router and will be swinging it over when in need.
-
@iptvcld said in pfSense HA LAN Interfaces Only:
Question.. Outbound NAT Rule, i have been reading this needs to be updated as well - is this the case only if i was doing CARP for my WAN? In my case I am not going to have WAN connected in my failover router and will be swinging it over when in need.
Exactly, it is not needed when you don't configure the WAN as CARP. As you stated above, you want to do a manual failover by pulling the WAN cable from the primary router and connect it to the secondary. So there is no need for CARP VIP.
-
@viragomann Thanks again, I am setting up my interfaces on the 2nd failover pf and since i have LAGG on my LAN (master pf) - i am trying to setup LAGG on my LAN on the failover but i just have the 1 card for lan on the failover. I was able to create a LAGG 1 the single card but what protocol would i use? I cannot do LACP because then my switch will need a group of 2 ports (min) - I am thinking Load Bal but i am not too sure.
Thank you!
-
I just swinged my WAN over and i getting this error
The link does not go up and i have verified the pppoe username and password that its the same on my master pf
-
@iptvcld
I would try FAILOVER. I think this should work with only one network port. -
@iptvcld
Any reason for using the Broadcom NIC? Do you passthrough the hardware?
If it's virtual set it to Intel E1000. -
@viragomann Thanks; i have selected failover and for the primary i changed it from Auto to my nic interface i will have my wan in. (even thought its a single card LAGG)
-
@viragomann Its a physical card pass-though on unraid but i think i figured out what the issue is That port may be defected on the card. Other port works ok . But now i need to find a different card as i needed both ports..
-
Alright card issue has been fixed..
Question; i noticed my non CARP Virtual IP's from master are not being synced over to failover.. Is this normal? Only when i create a CARP VIP, that is when it gets copied over. Thanks,
-
@iptvcld
Yes, interface settings cannot be synced. If they were, you would have the same IPs on master and backup, which wouldn’t work naturally. -
@viragomann thank you and that makes sense!
I am also finding that when i switch the master CARP to my failover that some devices i am no longer able to reach unless i reboot the device once whole on the master pf and then if i were hot it from the failover, that will work. Seems to be a one time thing per device though.. Maybe an option i am missing?
-
@iptvcld
A reboot should not be really necessary. But existing connections will timeout though, because the connection is on the other node and you don't sync states. -
@viragomann Thanks.. Does the High Availability Sync setting under System needs to also be turned on for the slave? I currently only have it configured and on the master and settings seem to be syncing fine. I just have been reading that on the slave you also need to enable with a check and select tje interface and pfsync Synchronize Peer IP only; dont touch the bottom portion and save. Is this true? and if so, how is sync working right now then?
Thank you!
-
@iptvcld
No, this is only needed for syncing states in both directions. But syncing states doesn't make sense in your case, cause your WAN has no CARP VIP.
So you can only sync settings and this is done from the master to the slave. -
@viragomann I am also syncing LAN interfaces as per below:
-
@iptvcld
Yeah, you can sync states if you want, but only connections passing CARP interfaces solely will benefit from it. Connections passing the WAN will stuck after failover anyway.
If states sync is on pfSense will also try to sync WAN states, of course. Don't know what happens on the other box, when the interface is not present.For syncing states it's recommended to use a separate sync interface.
-
@viragomann Thank you; yes i added a new card on both systems and created a new interface as SYNC and i am using that on both ends
-
@viragomann Your help along the way has been more much appreciated! I think i have it all set now and later on tonight i am going to test and fail it over to see how it all works.
-
This post is deleted! -
@viragomann Is it normal for the DHCP server to show the actual interface IP of the pf node or should it show the CARP LAN VIP IP (192.168.2.1)?
2.80 is master and 2.81 is slave (currently CARP is on Master) -
@viragomann sorry.. Was thinking how I can provide internet access to my backup node. Just for the purpose to keep the apps up to date as the master. Since there is no active wan when it's in slave mode.
-
@iptvcld said in pfSense HA LAN Interfaces Only:
Is it normal for the DHCP server to show the actual interface IP of the pf node
Yes, the DHCP server is the real interface IP.
You only have to enter others node IP at "Failover peer IP" in the DHCP server settings. This ensures that the server is exclusively running on the present master. -
@iptvcld said in pfSense HA LAN Interfaces Only:
Was thinking how I can provide internet access to my backup node. Just for the purpose to keep the apps up to date as the master. Since there is no active wan when it's in slave mode.
There is a way to go over the masters LAN to the internet:
Add the masters LAN IP as gateway on the secondary an set up a gateway group with the WAN DHCP gw as Tier 1 and the masters LAN as Tier 2. Set the gateway group as default gw.
But first of all disable the sync of routing settings on the masters System > HA page.Now the secondary goes out over the masters LAN interface to the internet, when the WAN gw isn't available.
-
@viragomann fantastic.. I will give this a shot today. So when I swing the wan over from primary to secondary node as a failover, this will restore internet to flow over the backup only?
And does this solution affect the primary wan internet in any negative way? Or will this just provide internet to the secondary node as a client for the purpose to update apps on the router.
Do I need to add additional CARP VIP for this? If master goes down my carp for the Lan interfaces will go master on the backup node. And I assume the secondary will loose access to the Internet right since it was routing from there (which is fine and makes since) at that point I would swing wan over.
-
@iptvcld said in pfSense HA LAN Interfaces Only:
I will give this a shot today. So when I swing the wan over from primary to secondary node as a failover, this will restore internet to flow over the backup only?
Yes, the gateway group cares that the the secondary use primarys LAN only if the WAN DHCP isn't available.
It's only to provide internet to the secondary and doesn't affect any other connections.
I guess, it's a good idea to set a public IP for monitoring (e.g. 8.8.8.8) in the LAN gateway settings. This way the secondary detects the gateway as offline in case you activate the CARP maintenance mode on the master and will switch over to the WAN gw.
-
@viragomann
I guess, it's a good idea to set a public IP for monitoring (e.g. 8.8.8.8) in the LAN gateway settings. = would I do this on the primary node or secondary node?This way the secondary detects the gateway as offline in case you activate the CARP maintenance mode on the master and will switch over to the WAN gw. = if I enable carp mant mode on the master node and I don't swing the wan fiber over to secondary node, it would simply mean no internet until that connection is made? What would happen if don't don't enable carp maint mode on master and just swing over wan to secondary? I guess my carp for Lan will still be master on my primary as Lan is still up.
I'm sorry I am just trying to get a full picture and you have been really great help.
-
@iptvcld said in pfSense HA LAN Interfaces Only:
I guess, it's a good idea to set a public IP for monitoring (e.g. 8.8.8.8) in the LAN gateway settings. = would I do this on the primary node or secondary node?
On the secondary. This gateway should only exist on the secondary at all.
f I enable carp mant mode on the master node and I don't swing the wan fiber over to secondary node, it would simply mean no internet until that connection is made?
Of course you have to connect the WAN cable to the secondary manually in this case to get internet.
What would happen if don't don't enable carp maint mode on master and just swing over wan to secondary?
Nothing. Failover only happens when a CARP interface failing is detected on the primary.
Pulling the WAN cable from the master and connect to the secondary will simple result in loosing internet. The primary will keep staying in master mode. -
@viragomann I hope I did this correctly - I also have a OpenVPN address in the group and not sure what Tier to assign to that. I leave it default Never (not sure if that should be something else as it left it out when i pressed save)
-
Master pf LAN Interface IP: 192.168.2.80
Backup pf LAN Interface IP: 192.168.2.81
LAN CARP = 192.168.2.1
Cant get the internet to flow to backup
-
@viragomann hmm. Can't seem to get to working. I have turned off HA static route sync from the master node and then on the backup node I have created a new gateway using LAN pointing to the LAN interface IP of the master node and then created a Gw Group with pppoe wan as tier 1 and my new gw as tier 2 and set this group as default but still no internet. I have also included pics of what I did. Any suggestions?
-
@iptvcld
The gateway and gateway group are looking correctly, but you have to troubleshoot the gateway monitoring problem.
Ensure that pinging 8.8.8.8 from 192.168.2.81 is allowed on the master and that there is an outbound NAT rule present on WAN for that source. -
@viragomann I did a test from my master to ping out to 8.8.8.8 and that works ok.
Also on master; this is what my outbound nat section looks like. I can see under source there is y LAN network 192.168.2.0/24
I am not too sure what else i should be looking at; would really appreciate your help. thank you!
-
@iptvcld
I think, I've forgotten something:
On the secondary you will need to add an outbound NAT rule to LAN for 127.0.0.0/8 and switch into hybrid mode. -
@viragomann Thanks; so on my backup PF i did the below:
This did not work still then i tried this
But this did not work as well.
-
@iptvcld
Keep the first one.Tried to restart the machine?
If it still doesn't take a packet capture on the LAN interface. You should see periodic pings to 8.8.8.8. As well you should see them on the masters LAN and also on the masters WAN if it works as expected.