2 X PFsense HA with 2 X WAN is it possible?
-
The ALIX hardware was 32bit so wouldn't run 2.4.X. I assume you mean APU devices.
Yes, it's possible to run with one public IP but very much not recommended. Doing so mean the node which is backup will have no internet connectivity so cannot fetch firmware updates, bogon files etc.
https://www.netgate.com/docs/pfsense/book/highavailability/index.html#ip-address-requirements-for-carp
Steve
-
@stephenw10 Sorry you are absolutely right, APU devices.
I see what you mean. We would have to schedule repeat downtime to do software updates which wouldn't be too bad.
In scenario one, lets assume that WAN 1 is down and both boxes are up, WAN failover will work as normal (and not involve HA)?
In scenario two, lets say WAN1 is down and PFsense 1 fails simultaneously (A bad day!) then CARP would failover to PFsense2, which would then realise that WAN1 is unavailable, and then failover to WAN2 as the default route.
I assume it would just introduce more of a delay than if I had 3 hosts to play with on the WAN side.
How do I sync without a dedicated sync interface? As the boxes only have 3 interfaces each.
Edit: Sorry just spotted this in the docs that answers my last question:
In low throughput environments that aren’t security paranoid, use of the LAN interface for this purpose is acceptable.
-
Yes, the failover between WANs is independent of HA failover.
Yes, backup HA node will always see both WANs as down as it cannot ping a gateway until it becomes master. So there will be a delay at failover whilst the pings start working and the new master nodes decides which gateways are currently working.
You can sync across the LAN interface directly if you need to but it's better to use a dedicated interface. If you don't have a spare physical interface you can use a VLAN to keep it isolated.
Steve
-
@stephenw10 Thanks Steve Much apprecaited.
The only thing I need to figure out now is the VIP config whilst using the same Wan ip.
-
Sorry, missed you reply. Can you clarify, I'm not really sure what you're asking there?
Steve
-
Be sure to also set up at least one inside DNS server and configure the nodes to use that so when they are in BACKUP status they can at least resolve names.
You are far better off getting /29s from your providers there.
A /29 interface network generally does not even require any justification on the ISP's part.
I would insist on it or get connectivity elsewhere.
Opinion: If it is worth HA it is worth doing it right.
I also really don't like the HA SYNC being on LAN. See the sticky post in this category about pfsync being unauthenticated. If that is not at least an isolated VLAN you run the risk of LAN hosts being able to inject active states into your firewall's state table.
If the WANs aren't really high speed, I would consider using a managed outside switch with two VLANs for the WAN circuits and using OPT1 between the nodes for SYNC. The only real reason not to do that is you're now single point-of-failure on the outside switch.
-
Thanks both.
The guides and documentation on multi WAN HA failover all explain the setup using 3 x hosts on each WAN interface. (the guide mentions a 4th for 1:1 nat but we don't need this in our setup).
I have to figure out how to setup the CARP VIP by only using one WAN IP and I haven't fully got my head around how CARP is used in HA and why it needs 3 X IP's in the examples.
I 100% agree. Unfortunately these WAN connections are provided by a specialist provider as the customer is in the healthcare industry and these wan connections have direct peering with the government healthcare providers. In an idea world.
Also agree on the HA sync not being on the LAN. Its one of those cases where the hardware was specified and supplied, then the requirements changed in a short time period. I intend on using a VLAN for the HA sync.
Both LL are 100 over 1000, I am not sure if they intend on increasing this up to the link speed, but using a managed switch for WAN could work. Can we use CARP VIP with VLANS on Pfsense then?
A cold standby switch would take care of the single point of failure, but would remove the link redundancy across to the outside world, as you say.
-
You would usually add a small private subnet for each WAN that both nodes are in. That could also include the modem if it has a private IP. Then add the one public IP as a CARP VIP on that.
But just to be very clear; we don't recommend this!
Steve
-
Unfortunately these WAN connections are provided by a specialist provider as the customer is in the healthcare industry and these wan connections have direct peering with the government healthcare providers.
@everyonelovescheese Then they should understand that proper High-Availability (CARP/HSRP/VRRP) requires multiple IP addresses and that they are preventing their customers from implementing solutions that will provide the most reliable patient care.
Its one of those cases where the hardware was specified and supplied, then the requirements changed in a short time period.
I would push back in that case. You change the requirements, we get to spec new hardware.
So a healthcare provider cannot afford to upgrade from APUs? Really?
-
In an ideal world, yes.
However, we are supplying this, to another MSP, who is then providing this to their customer. I can only voice concerns, they are free to do what they like. We provide warnings when something is not done correctly but can only manage expectations.
-
Rather that cobble together some unsupported configuration, I would tell them their internet circuits are not compatible with pfSense High-Availablility. Here's the documentation stating such.
https://www.netgate.com/docs/pfsense/book/highavailability/index.html#ip-address-requirements-for-carp
If you really want to do it just configure the WAN interfaces with any old RFC1918 address schemes you want and put your side of the /31 as a CARP VIP on that.
Note also that when you fail over you might have to also wait for gateway monitoring to see that the gateways are now up on the new master node because dpinger will not be able to ping across the WANs without holding the single public address.
The only way to make this not the case is to disable gateway monitoring and then you won't have automatic WAN failover.
You probably also have to check the Use non-local gateway checkbox in the Advanced settings area of the gateway definition in order to add the other side of the /31 as the default gateway on the RFC1918-numbered WANs.
-
I have this exact same target setup at the moment except only one of my two ISPs is providing a single IP address.
Did you get anywhere with it?
I've spent hours looking for a decent example over the last few days and have finally came to the conclusion that I've got too much to do to keep banging my head against a wall trying to set up an unsupported and undocumented configuration. The book says it's "technically possible" but "not recommended" and there are no good articles showing how to do it.
I've persuaded my boss to just get some more IP addresses. I even used the "If it's worth doing HA, it's worth doing right" quote. Luckily for me, our ISP isn't charging too much and my boss just wants HA over and done with. However, I'd still like to know if you had any level of success with this setup.
I'm deleting my question about it as there is nothing useful in there.
-
@jmilne said in 2 X PFsense HA with 2 X WAN is it possible?:
I have this exact same target setup at the moment except only one of my two ISPs is providing a single IP address.
Did you get anywhere with it?
I've spent hours looking for a decent example over the last few days and have finally came to the conclusion that I've got too much to do to keep banging my head against a wall trying to set up an unsupported and undocumented configuration. The book says it's "technically possible" but "not recommended" and there are no good articles showing how to do it.
I've persuaded my boss to just get some more IP addresses. I even used the "If it's worth doing HA, it's worth doing right" quote. Luckily for me, our ISP isn't charging too much and my boss just wants HA over and done with. However, I'd still like to know if you had any level of success with this setup.
I'm deleting my question about it as there is nothing useful in there.
I did not want to risk deploying a solution that was not supported, so managed to get additional IP's.
however, its has opened a new issue, the ISP has provide publicly routable addresses which is fine, but then another set of private addresses for a link to a govmt service, so I have to figure out how to make that work with HA, should be possible with a static route, but not sure how I am going to have multiple ip addresses on one interface..
-
You'll have to give more details as to what and where these addresses are to get any feedback.
Public/private - doesn't really matter as long as the routing works while the node is the BACKUP. This is more true for things like accessing the internet (so the BACKUP node can look up DNS, check for updates, do NTP, etc.)
-
@derelict said in 2 X PFsense HA with 2 X WAN is it possible?:
You'll have to give more details as to what and where these addresses are to get any feedback.
Public/private - doesn't really matter as long as the routing works while the node is the BACKUP. This is more true for things like accessing the internet (so the BACKUP node can look up DNS, check for updates, do NTP, etc.)
Yes, I will create a new thread If I need assistance on that section of the config.
-
@everyonelovescheese :
Thanks for updating the thread with your final outcome i.e. getting new IPs. It helped me by closing the subject down at my end as not currently viable and allowing me to move on.
