HA Setup working by no internet access from LAN
-
I am very new to pfsense and I have come across the HA feature where I can use two pfsense boxes in parallel. I have two existing routers with pfsense 2.4.4 each that I am trying to setup in a HA setup. To create the second pfsense box I have just backed up the settings on the existing primary and restore these settings onto the backup server and changed the settings, host name and lane IPs and WAN IPs. individually these pfsense boxes are able to connect to the internet. What I am trying to achieve is to have an interrupted internet connection to my lam as I have some server serving up content on the internet but now and again I need to reboot the pf sense boxes which means if I continue with one , at the time of reboot my internet would be interrupted. Hence why I am trying to setup the HA.
The issue is as soon as I have configured the HA settings on the two boxes, my internet becomes very unstable or cannot connect to the internet at all from my LAN. could there be something I am doing wrong. I am following the steps outlined in the following URL;
https://www.netgate.com/docs/pfsense/highavailability/configuring-high-availability.html
Could someone please point me in the right direction?
-
@gsenzere said in HA Setup working by no internet access from LAN:
could there be something I am doing wrong.
If you have adhered strictly to the linked manual, it should word.
Do you have configured a dedicated sync network?
Double-check the settings for the state synchronization and the outbound NAT settings.
-
Thanks viragomann for your reply. I have a dedicated Sync network via re0 interfaces on each pfsense box. If you do not mind, how do I check the state synchronization?
On the primary box I have Synchronisation states checked, I have the synchronisation interface as the SYNC, which is my re0 and I have also defined the pfsense syncronisation peer IP defined as the SYNC IP of the secondary box. And I have a physical Ethernet cable connected between the primary re0 (SYNC) and secondary re0 (SYNC) interfaces.
On the outbound NAT I have tried to follow the instructions on the URL link and this is what I have done;
Selected Firewall >> NAT >> Outbound then checked "Manual Outbound NAT rule generation.
(AON - Advanced Outbound NAT)" and clicked save. After that I see the following mappings;
I have gone into edit each of the mappings and changed the "translation" "Address" to my CARP WAN VIP. Is this how its supposed to be done on the NAT outbound?
-
jimp did a great Hangout Video covering HA on pfSense 2.4 https://www.netgate.com/resources/videos/high-availability-on-pfsense-24.html
This will give you a great overview / tips and hints.-Rico
-
@gsenzere said in HA Setup working by no internet access from LAN:
I have gone into edit each of the mappings and changed the "translation" "Address" to my CARP WAN VIP. Is this how its supposed to be done on the NAT outbound?
Only change the translation address to the WAN VIP in the outbound NAT rules for internal networks (the last two in your screenshot, the other ones are for the pfSense box itself).
The state sync has also to be configured on the backup box.
-
Thsnks again. I will change only on the last two.
As for the state sync i also hsve the settings on the secondary box as well and the peer address is set to the sync interface ip address of the primary box. I will check on the NAT and let you know. -
@viragomann @Rico Thank you both for these pointers. I have tried the following;
I have rolled back to the two clean pfsense boxes without HA settings.
Have started entering the HA settings by setting up the SYNC interfaces on both boxes, then added rules to pass traffic between the two interfaces. Everything works as expected until I add the WAN and LAN VIPs and add the NAT outbound mappings. so as soon as I change the translation to my WAN VIP, I loose internet connection immediately. Even if I roll back the settings (remove the NAT outbound settings ) I still can't connect to the internet. I have also been following the hangout video. -
@viragomann @Rico I have found out something on the outbound NAT. As soon as I change the translation address on the mapping in red below, my internet connection does not work.
I also have another router on the WAN side. So I have a static IP from the ISP which gets to a Mikrotik router and then from here I have three IPs (primary WAN, secondary WAN and WAN SHARED VIP) going towards my HA setup. I do not have a switch in-between. Could this be an issue? -
You need to change that circled rule to the NAT to CARP VIP too.
As long as you have good connectivity to and from all three VIPs that should work fine with the upstream router.
-
@Derelict Thanks for replying. I have tried to change the translation address on this to my WAN VIP and as soon as i do this I have no internet on my LAN. Am I correct to change the translation adress to the WAN VIP?
-
Can you ping from the WAN VIP out to something like 8.8.8.8? You can set that as a ping source in Diagnostics > Ping for testing. You should be able to ping out to the internet sourcing from both interface addresses and the CARP VIP (from the node who holds the VIP in MASTER status).
If that's not working fix that first. It's fundamental to successful HA.
-
@Derelict Thanks for this I have checked this. I can ping the 8.8.8.8 from the two WANs but cannot ping it from the WAN VIP. This is how I created the WAN VIP;
Firewall >>Virtual IPs >> clicked ADD selected type as CARP, Interfaces as WAN, Address type as Single Type, for the address I selected a random IP/24 not in use but on the same subnet as the two WANs, put in a random Virtual IP Passwords that match, VHID of 30 (not in use on the network), a.f.left it at 1 and skew at 0. This is syncing to the second box as 100 as expected. and
as I mentioned earlier these three IPs go onto my ISP Mikrotik router. Could I be missing something here? -
Yeah, upstream layer 2 probably isn't doing the right thing with the CARP MAC or the ARP or something. Hard to say.
See this:
https://forum.netgate.com/topic/119864/carp-ha-sync-and-xmlrpc-sync-explained
-
And this:
The ISP Layer 2 device will see the CARP MAC as the source MAC in the CARP advertisements. They are sent to the Layer 2 Multicast address 01:00:5e:00:00:12 (all points multicast) to Layer 3 multicast address 224.0.0.18. That MAC address has to be added to the switch port's MAC address table based on those. This MAC address will change ports on a failover event. The ISP device must move the MAC address to the new port as any switch should.
The ISP Layer 3 gear will get the CARP MAC in response to ARP "WHO HAS" requests for the CARP VIP address. Their gear needs to do the right thing with it. The ARP reply from the WAN interface that is currently CARP MASTER will contain the CARP MAC in the ARP "IS AT" response. This ARP response will be sourced from the interface IP and MAC address.
The ISP Layer 3 gear also needs to honor the interface addresses that will ARP as normal. The ISP device will only ever see the interface MAC address on the port connected to that node.
All references to ISP gear there should be interpreted in whatever is upstream of the two pfSense nodes in your environment. The Mikrotik and whatever else that is.
