CARP multi-master, splitting multi-IP WAN traffic
-
Hi.
I have 3 reverse proxy servers (Linux + nginx). Each of them has few public IPs and about 1Gbps of traffic to support. They do some business logic and pass requests down the road.
I'd like to hide those servers behind pfsenses for better firewall capabilities, because sometimes they are targets of DDoS attacks. Is there a way to have 3 pfsense instances in multi-master CARP configuration? Let say I have 9 public IPs. So what I would like is to each of these pfsenses to have 3 public IPs by default. When one goes down other two take those IPs.
It's like keepalived with VRRP. You can configure few virtual IPs, for each host add different weights and balance traffic between. If one host goes down, other compare their weights for IPs and choose new master for each.
Thanks in advance for any ideas.
-
@pfsenseuser1234 said in CARP multi-master, splitting multi-IP WAN traffic:
Is there a way to have 3 pfsense instances in multi-master CARP configuration?
No, pfSense has CARP implemented as active-standby constellation. There's no supported way to enable an active-active configuration. Also 3 pfSense nodes are not recommended due to the way configuration sync is working from primary to secondary node. You could daisy chain a third node to the second one but that constellation would bring more overhead and potential debug overhead then it would be useful.
CARP on OpenBSD has that ability (multi-master-mode) but AFAIK the FreeBSD implementation is lacking that feature (or it's not widely implemented) so there's no way I'm aware of the run such a config on pfSense.
That aside: how would run 3 proxies behind 3 firewalls help against a DDoS attack? As DDoS normally aims to exhaust your upstream, the number of hosts/services would be completely pointless as your upstream connection gets bombarded with requests? Curious as how that should help you with that :)
Cheers
-
@jegr Any other idea how can I achieve good HA in such configuration. I have three hardware nodes with proxmox. For now on each node there's (among others) Linux virtual machine with firewall and reverse_proxy (plus some business logic). All three VMs are connected with keepalived VRRP and split among themselves 9 public IPs. Why three? There's simply too much traffic for one (over 50k requests/minute, over 2Gbps of upload, over 45k TCP connections). Two is enough for now, third one is just to be safe.
Why so many public IPs? I have hundreds of domains. Each has two A records pointing to two IPs. If one VM is, for example target od DDoS*, browsers will try second IP after a few seconds. This way I can stay alive even before firewalls in datacater start to filter traffic. Also I would quickly run out of TCP ports to handle such traffic with just one IP.
I'm guessing pfsense can handle 50k req/min and I would be fine with simple active-standby approach. But what if I go to 100k by the end of the year? Or 150k. I'll probably hit some limit sooner or later. That's why I'm looking for multi-master firewall that will split traffic between all hardware nodes, filter it and then pass it to reverse proxies.
*Not only volumetric DDoS (>10Gbps to kill one VM), but also SYN floods, DNS amplifications, slow HTTP requests… I get them all.
-
@pfsenseuser1234 Why three? There's simply too much traffic for one (over 50k requests/minute, over 2Gbps of upload). Two is enough for now, third one is just to be safe.
No problem with that :)
I'm guessing pfsense can handle 50k req/min for now and I would be fine with simple active-standby approach. But what if I go to 100k by the end of the year? Or 150k. I'll probably hit some limit sooner or later. That's why I'm thinking about multi-master configuration.
I can understand the thought, but pfSense simply can't work in an active-active constellation.
*Not only volumetric DDoS (>10Gbps to kill one VM), but also SYN floods, DNS amplifications, slow HTTP requests… I get them all.
OK you could get various DDoS but the goal of all is clear: denial of service via massive flooding. Be it SYN, be it DNS, be it slow HTTPs. You can filter out some, but at the end your own firewall will give up or your upstream pipe is full. DDoS mitigation is nothing you can do on your end/site in a meaningful way. Yes you can somehow slow the process and try to mitigate the hit but with enough firepower / distribution you'll get nuked nonetheless. So if you get hit on a continuous fashion the only really helpful thing is bringing a CDN into play and (actually) try to hide the real endpoint addresses so they don't actually leak out (or the attacker will just ignore the CDN and hit the IP). That's the point where Cloudflare, Akamai or Stackpath etc. come into play.
Other then bringing a CDN into play and balancing your three proxies out for traffic reasons, I don't see a quick/simple way to bump that up further. But perhaps some other can chime in.
Cheers
