BGP Routing vs ISP VRRP
-
Hi,
we have 2 1Gbit fiber connections (from the same ISP but but entering from different fibers into different sides to the building), each with a /29 transport network and a /28 inside net.
We want to build a HA Cluster with Carp and Multi WAN. Our ISP now offered two solutions. He can put two routers (for each WAN) in front of the pfsenses and let them do VRRP, so that we will have only one external Gateway via VIP.
Or we can omitt the Routers and do BGP Routing. What would be the best option here?
-
Why would you need BGP for that? Why not just treat them as dual WANs?
https://docs.netgate.com/pfsense/en/latest/book/multiwan/index.html
-
Because we want redundancy for the routed /28 inside IPv4 network in case one of the uplinks is down. On one of the uplinks we will get more public IPs in the future and we want them to work even if the uplink on which they are routed from our ISP goes down.
-
I would let them do it or put a pair of routers (not firewalls) in front and do it yourself.
Sorry, your OP made it sound like there was a different /28 for each /29.
You might be able to get FRR behaving acceptably but I am pretty fond of my BGP being on routers on the edge, not stateful firewalls.
-
Sorry perhaps I didn't explain that well:
We have two independent fibers, each with a /29 interconnect network. On one of the fibers we have several /28 networks (routed from the ISP to the CARP VIP of the /29 net) that we want to be redundant even if that uplink goes down. (On the other fiber we have a /28 network too, but we are not using this at the moment).
Basically it's a "Small WAN IP Subnet with Larger LAN IP Subnet" as described here: https://docs.netgate.com/pfsense/en/latest/book/firewall/methods-of-using-additional-public-ip-addresses.htmlThe pfSense machines are very powerful and it would be nice to not need additional routers but as you stated that would not be best practice in this situation?
Another question is how would we route an additional /28 network to a customer connected to the dmz switch when we want to give him the whole subnet (and not need to cut off IPs). Would we need another /29 interconnect network to route the /28 network to him. Could that be private IP adresses?
Thanks for all the help and advice!
-
You can do it, but if FRR/BGP decides it's time to move a route from one interface to the other, the ISP will start routing traffic for that to the other interface. All existing firewall states will break and everyone will need to reconnect, hit reload, etc.
You also have to be sure that, if you are using policy routing, that where pfSense sends the traffic out is the same interface the ISP will route the traffic back or you will have broken states/asymmetric routing there too. Or accept a default route from them which should always be on the same interface the routes back to you are one.
If the BGP is handled by a pair of routers ahead of the firewalls (so the firewalls have a single transit interface to the routers) then the routers can move the routes around however they want and the states will still work.
Not really a pfSense issue, more like a running BGP like this on a stateful firewall issue.
-
Thank you very much for pointing this out. Unfortunately our customer now doesn't want to put any more money on it (complicated situation) so we have to use the openbgp package on pfSense directly.
We only want to switch the WAN Uplink if WAN1 fails (BGP prepend), as this will hopefully not happen a lot it's ok for the users to reconnect etc. in this situation -
OK as stated here ( https://docs.netgate.com/pfsense/en/latest/packages/openbgpd-package.html) it's "best to have two BGP sessions with each provider, one from each firewall, and set the nexthop in the BGP network statement to a CARP IP on the interconnect subnet with that upstream provider."
I need to tell this our ISP, because they only know VRRP and there the IP goes to the active firewall and starts a new BGP session.
I just don't understand the "CARP Status IP" explanation in the openBGP package: "Used to determine the CARP status. When the CARP vhid is in BACKUP status, bgpd will not be started."
How can you have BGP sessions when bgpd is not started on the backup firewall? -
It's the same in the FRR package. When CARP vhid is in Backup status, frr will not be started, but the wiki says it's best to have four bgp sessions, two from each firewall.
I just don't know what is the best practice when you have two firewalls in HA, with two providers (each /29 transport network) and several routed inside networks from the first provider.
What is the best configuration in FRR regarding BGP sessions and failover time?
-
pfSense with HA is fundamentally active passive arrangement, one firewall is the master and the other is backup. It is not designed to be an active active configuration.
To try to make pfSense active active, with both pfSense firewalls online and routing etc is in my view far more trouble than it's worth.
Ask yourself these questions;
- What are you trying to protect against?
- What sort of outage can be tolerated (be realistic)?
- Can it be done in a simpler way?
Simpler is almost in all cases always better. Complexity generally increases the risks of downtime because there is more stuff that can (and does) go wrong.
-
@nzkiwi68 Thanks for your answer! I know that pfSense is designed as an active passive arrangement.
In the end I configured it this way. Only the active Firewall has a BGP session to each provider and it's working fine.Just stumbled over the explanation in the docs https://docs.netgate.com/pfsense/en/latest/packages/openbgpd-package.html and thought there should be active BGP sessions on both firewalls.
Anyway thanks for the help here!
