Pfsense setup with BGP - Carp on secondary also in master mode
-
We have 2 firewalls setup with BGP.
The secondary sees the WAN vip as master also.
So it starts the BGP service.I have tried to stop the BGP service and disable the carp on the secondary to remove the Master status but when I enable carp it puts itself in Master mode and turns on BGP.
This was not the case before we enabled BGP.
I have attached screen shots of our BGP configuration.
I would love some input, please.Primary BGP screen
Secondary BGP screen
Primary BGP raw screen
Secondary BGP raw screen
-
I don't think BGP is the issue here.
It sounds like you have a situation where both pfSense boxes think they are master, because they can't see each other via the CARP packets properly.-
Make sure the upstream switch isn't filtering CARP packets which are sent out as multicast to 224.0.0.18. Both pfSense boxes must be able to hear each other's packets.
-
Use packet capture to be sure nothing else is doing CARP or VRRP (ISP's equipment for example), on the same link, and if so, pick a different ID.
CARP and VRRP use the same underlying protocol number, but can't understand each other.
CARP and VRRP instances must use separate IDs to avoid collision.
-
-
I am sure you are right.
This is not a BGP issue but CARP.if the ISP is filtering and I can't get them to pass this traffic between both subnets.
Can I put a switch in front of this and plug both firewalls into the switch?I would use 2 switches for redundancy of course.
when I do a packet capture this is the only thing I see on the secondary.
Same thing on the primary just its own ip.21:19:34.871318 IP 4.16.115.210 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 100, authtype none, intvl 1s, length 36
So it looks like the broadcast is filtered?
Thank you for taking the time to answer.
-
By the way, the gateway on the secondary disappears after awhile.
but its still passing traffic.
-
Frequently ISPs will filter such multicast traffic so that it doesn't interfere with their systems. You would, as you suggested, have to put some switches in front of your setup to get around the filtering issue.
That being said, while I have not used pfSense with BGP, it would make more sense to me to have only XMLRPC sync occurring between the boxes to keep the configuration synchronized (at least the relevant parts), and not using CARP at all.
BGP will be used to route traffic based on reachability, and would announce your prefix via only one of the two pfSenses. If you use OSPF inward facing, your systems will use whichever box can actually reach the Internet.Maybe someone with more experience with this type of setup can chime in.
-
How would the external IP addresses live on both firewalls if you dont use CARP?
I really appreciate you taking the time to answer.
-
In this scenario, both routers are advertising the 4.15.227.0/25 subnet, only the secondary is advertising with an artificially more distant path, this wouldn't be chosen unless the primary is down.
The inside, when using OSPF, for example, the secondary advertises the default gateway with a less favorable metric than the primary, thus on your inside switches, you end up with two default routes, but only the best one would be used.
In the case that the primary pfsense goes down, then the secondary's routes become the only remaining routes on the ISP and Internally, and it keeps on working.You could also put a link between the two boxes to route traffic over it in the event that either just the inside or the outside link goes down on the primary, then the traffic would flow through the secondary on the cross-over link, or if your switches support multi chassis LAGG you could add redundancy that way too.
Because pfSense is a stateful firewall, under certain circumstances, the session would drop, but for web traffic it wouldn't be noticeable for the most part.
