Carp Failover and bridged Wan
-
What is the current situation with pfSense and getting failover with 2 firewalls and bridged Wans?
It seems there have been plenty of people trying to acheive this basic functionality over the years but no real solution as of yet.
Has anyone managed to get this to work reliably without using STP to acheive it? Maybe someone could write a package with scripts to accurately monitor the Carp interfaces and bring up the bridge on the failover firewall to avoid layer 2 loops? I think this function would be welcomed by alot of pfSense users.Any input would be apreciated because this is something i really need to get working soon or I will have to oabandon pfsense and go for a commercial solution ( and I REALLY love pfSense and don't want to change).
-
I don't see how it can be done without STP. That is the bridge way of not setting up a loop. I guess CARP would work the same way if you could stop the loop with STP.
-
Unfortunately I have set it up with STP but it just doesnt work reliable enough. Either STP fails to witch routes to the failover firewall or a loop crashes the whole network.
I read somewhere on these forums that someone had written a script that would monitor the carp interface and if the main firewall failed then the script would bring up the bridge and this would prevent a loop from occuring. Apparently it worked fairly ok but not sure of the exact instructions to get it set up.
-
I wouldn't rely on STP in such cases, too many possibilities for failure. Instructions here http://forum.pfsense.org/index.php/topic,4984.msg87793.html#msg87793
-
Thats exactly whati'm looking for. Thanks a mil.
Are there any improvements to this method that anyone is aware of? -
The link I posted is the best you can do. I think we're the only firewall in the world that accommodates such a setup (short of rolling your own) because of the risks and complications inherent in redundant bridging firewalls. It's best to re-engineer your network so you don't need bridging if you want redundant firewalls, though a lot of people do exactly as in that link with no issue.
-
Note that in those instructions, on 2.0.x you'll need to match on "vip" as the subsystem, and on 2.1 the subsystem should be "[a-z]+[0-9]+_vip[0-9]+"
As someone who ran a CARP+Bridge setup for several years, let me say it was always a headache. I moved that to a completely routed setup and never looked back. It was well worth the time it took to transition.
-
When you say moved to a completely routed setup, I don't know if that's possible for my setup. I have a /24 ip pool from my Data Centre but the only way I thought I could get my web servers to be firewalled and still use these public IPs as their primary interface was with a transparent bridge.
Is there another way? I don't want to switch to a Nat setup.
-
Why would you want your Web servers to have real IP addresses. I run all my web servers behind a NATed firewall. It is safer IMO and also works better in more solutions. I also changed over to split horizon DNS and made sure my web pages used DNS name or relative path for self referrals.
Like Jim said, it would be worth the time to transition to that. I am sure there is another solution out there as well. There are many ways to tackle this problem. -
Beg for an additional /29 from your DC.
Use the /29 for your CARP WANs and the DC/ISP gear
Use the /24 directly (you can route, not nat) on the LAN side
That's the typical datacenter style deployment for a firewall.
Things in your /24 use the firewall's CARP VIP IP in the /24 as their gateway. Your DC routes that /24 to your firewall's CARP VIP inside the /29.
No bridging required, much more reliable, none of the headaches.
-
I've managed to get a /29 subnet and here is what ive set up so far.
firewall 1:
Wan 200.200.200.2 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.1 (ip of existing /24 subnet)
Carp 172.17.1.1
Wan VIP 200.200.200.4
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on Wanfirewall 2:
Wan 200.200.200.3 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.2 (ip of existing /24 subnet)
Carp 172.17.1.2
Wan VIP 200.200.200.4
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on WanSo I set up a server and gave it an ip of 100.100.100.5 and a gateway of 100.100.100.3 (Lan Carp VIP)
I can get on the net and it works well for outgoing traffic. I have rebooted firewall 1 and firewall 2 switches to master in a few seconds. Internet still works and all looks good. Firewall 1 reboots and switches back to master and internet still works.
But then I realise there is a problem. Going to whatismyip.com shows my IP as 200.200.200.4 instead of 100.100.100.5
This is a problem for all my webservers that need to have their own ip display for outbound traffic.What have I done wrong?
I want this to be a fully routed setup and not just a Nat setup. How do I get my /24 ips to route correctly for outgoing traffic?Thanks
-
Sounds like you still have NAT on or atleast natting to the wrong IP. Check your outbound NAT. You might have to turn off auto and go from there.
-
I've checked outbound Nat and it's definitely on Manual. I used the default rule and set the translation interface to the vid ip as suggested by the docs.
Maybe this is where I'm going wrong?I've tried all the options from the translation drop down list and I even tried deleting all rules under manual outbound Nat in the hope it would route instead of Nat if no Nat rules were present but no luck. With no rules the test server can't even get out to the Internet.
-
Since you are using internet route-able IPs, you don't need to NAT at all. You just have to make sure that the IP address your ISP routes to for your /24 addresses is your CARP VIP interface. Why do you have a CARP of 172.17.1/24, this is a private IP. It should probably be something like:
Firewall1:
WAN 200.200.200.3
WAN CARP VIP: 200.200.200.2
LAN: 100.100.100.2
LAN CARP VIP: 100.100.100.1
AON disabled and all rules removed.
If you are using DHCP, you are going to have to change the default setting so that it uses the LAN CARP ip for Gateway and such.Firewall 2:
WAN: 200.200.200.4
WAN CARP: pulled from Primary
LAN: 100.100.100.3
LAN CARP: pulled from Primary
NAT: pulled from PrimaryYou will tell your ISP to route your /24 addresses to 200.200.200.2. Once that is complete, your should not have any problems.
-
Why do you have a CARP of 172.17.1/24, this is a private IP.
MAybe i explained my setup wrong. THe CARP ips i refer to is the private sync interface i.e a crossover cable between the two firewalls.
I have a CARP VIP added and it is one of the 3 ips from my /29 subnet.
firewall 1:
Wan 200.200.200.2 (200.200.200.1 GW of /29 subnet)
Lan 100.100.100.1 (ip of existing /24 subnet)
Carp 172.17.1.1 (this is private sync interface)
Wan VIP 200.200.200.4 (this is CARP IP on WAN from /29 range)
Lan VIP 100.100.100.3
AON Disabled and default rule set to use 200.200.200.4 as Nat interface on WanI havent yet asked the DC to route my /24 subnet to this CARP ip in the /29 because I wanted to be sure everything is wouking and i dont want any downtime.
The /29 and /24 are currently being presented on the same interface from the DC.
-
Ok so one I asked the DC to route the /24 to the CARP ip everything works. Outbound ips are showing correctly and I am very happy now :-)
Thanks for all the help people.
