How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

In a large organisation though they may force all traffic through a proxy to decrypt it. In which case it could still be scanned.

But this mean at least 2 (HA, active-backup) IDS/IPS servers on each (!) LAN. So totally 2 proxy + 2 IDS/IPS + several switches on each of LANs.

stephenw10

Wouldn't really need more than one pair IMO. As long as all subnets have access to the proxy.

Sergei_Shablovsky

By the way, Gigamon’s TAPs and Packets Brokers looks VERY PROMISING to mirroring all traffic for future inspections…

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky

You will need to connect each firewall to each ISP's router.
This setup uses a single switch but you could use two with VRRP enabled if that is what you want.
This setup is using a LACP to the switches, but you could change that to use the 10G switch you mentioned.

Is that possible to use LACP also for CARP SYNC, if I have 2 interfaces of NICs for this purpose?

And what about using LACP from firewalls to upstream switches (igb2 - igb5) ?

P.S.
Sorry for late reply.
And THANK YOU SO MUCH for networking passion and patience!

stephenw10

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Is that possible to use LACP also for CARP SYNC, if I have 2 interfaces of NICs for this purpose?

You mean to use two links in a lagg for just the pfsync traffic? Yes, you can do that but it's probably not worth it IMO.

mcury

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Is that possible to use LACP also for CARP SYNC, if I have 2 interfaces of NICs for this purpose?

I don't think it is necessary, the SYNC interface doesn't use that much of traffic, as far as I'm concerned, firewall states and configuration changes only (If I'm wrong about this, please someone correct me).

And what about using LACP from firewalls to upstream switches (igb2 - igb5) ?

That would help only if your internet link is above 1Gbps. Although a single client would never go beyond 1Gbps anyway.
That setup above is considering intervlan traffic along with 1Gbps internet links, to don't bottleneck anything.

Another approach would be to update all the NICs in the computers to 2.5Gbps, get 2.5Gbps switches with 10Gbps uplink ports to the firewall, then connect the NAS to another 10Gbps port, use the remaining 2.5Gbps ports to connect to the ISP routers/gateways, if those have 2.5Gbps.
By doing like this, a single client would be able to reach 2.5gbps to the WAN.
You could also do LACP with 2.5Gbps ports.

And THANK YOU SO MUCH for networking passion and patience!

:) My pleasure

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Is that possible to use LACP also for CARP SYNC, if I have 2 interfaces of NICs for this purpose?

I don't think it is necessary, the SYNC interface doesn't use that much of traffic, as far as I'm concerned, firewall states and configuration changes only (If I'm wrong about this, please someone correct me).

My FIRST MAIN GOAL IS TO MAKE HARDWARE RESERVATION for all hardware links:

between pfSense boxes and switches that connected to it;
between pfSense boxes itself;

So the main “guide mantra” now: AVAILABILITY - FIRST, SECURITY - second, OBSERVABILITY (MONITORING & ALERTTING) - third.

And what about using LACP from firewalls to upstream switches (igb2 - igb5) ?

That would help only if your internet link is above 1Gbps. Although a single client would never go beyond 1Gbps anyway.
That setup above is considering intervlan traffic along with 1Gbps internet links, to don't bottleneck anything.

The pfSense itself connected to the nets by switches. So, for example, when offices nodes generate <1G at all, the web services generate between 3 and 7G depending on daytime.

So, the hardware doubled connection from pfSense to upstream switch (which directly connected to ISP’s aggregate switch) was not only as availability, but increase bandwidth.

Am I lost logic somewhere? ;)

Another approach would be to update all the NICs in the computers to 2.5Gbps, get 2.5Gbps switches with 10Gbps uplink ports to the firewall, then connect the NAS to another 10Gbps port, use the remaining 2.5Gbps ports to connect to the ISP routers/gateways, if those have 2.5Gbps.
By doing like this, a single client would be able to reach 2.5gbps to the WAN.
You could also do LACP with 2.5Gbps ports.

Yes, the next step in upgrading would be replacing existed downstream NIC’s on pfSense server to 10G-heads NICs and upstream NICs - to 40G or 20G-head NICs.

Am I looking at right direction? :)

mcury

@Sergei_Shablovsky I used to configure VRRP with Cisco Switches, Catalyst.
Something around 15 years ago, more or less, not sure anymore..

I'm getting really old hehe, that is not good..

Check if you can find VRRP switches, that supports 802.1ad (LACP), and go ahead, build the dream network :)

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

In a large organisation though they may force all traffic through a proxy to decrypt it. In which case it could still be scanned.

Even with 1k nodes in one branch, the inspecting of all incoming traffic would be challenge even for latest&greatest Intel CPU + Intel QAT, and in case client-server tunneling - not give us ability to inspect traffic between them at all.

In any case, decryption to inspect traffic - definitely is a very bad idea to implement exactly on border firewall: firewall need to take decisions very quickly, “on the fly”, and waiting the decryption results - is not possible, because shaping overall bandwidth in 10-15 times.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.

In IDS mode though you can just mirror the traffic to something external for analysis.

What about scheme, let’s call it “part of multi-layer defense strategy”

there are IDS/IPS system on separate server, that receives mirrored traffic from all of 4xWANs from switches above the pfSense-based firewall
2.
after some ACLs in IDS/IPS triggered and IDS/IPS adding IPs to blacklist
3.
some crone script on IDS/IPS server adding all banned/blacklisted IPs to the switches ACL’s

In this scheme we have constantly worked “traffic analyzing loop” which just detecting and blocking bad/potentially bad requests from coming to border pfSense-based firewall.
And taking off some loading from pfSense-based server’s hardware.

As fast example, this may be a guard from small DOS/DDOS or flooding attacks.

What is Your opinion on this?

stephenw10

That could work. It's unlikely to help much with a DOS/DDOS attack though because by the time that traffic hits the ACLs on a switch it's already used up bandwidth on the WAN.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

That could work. It's unlikely to help much with a DOS/DDOS attack though because by the time that traffic hits the ACLs on a switch it's already used up bandwidth on the WAN.

Thank You for answering and “network firewalling passion” :)

Agree, even in middle-size (mean 200-300 Gb/s) DDoS attack, no one edge FW/switch guard You from filling up Your 10Gb uplink to ISP: after incoming rate raising 9,7-10,3Gb/s (or some quantities of pps) automated alerting system just temporarily dev/null anything that going to You from outside.

But I asking about some kind of eliminating loading on edge FW: moving some portion of blocking rulesets from edge FT to uplink switches (especially they are great built on ASICs).

Anyway, Intel-based consumer servers with FreeBSD/RHEL able to crunching only up to 50-100Gbps depend on CPUs and RAM/PCIe speed, offloading scheme with DPDK - add extra 200-300Gbps to this, offloading scheme with Smart-NICs - give us another extra, than specialized NPU with custom silicon, etc…
This mean pfSense - up to 50-100 Gbps, TNSR - up to 400. (But in real life on non-synthetic tests the numbers would be like 60 and 200 respectively).

So, each 1Gb which would be dropped/rejected on uplink ASIC-based switch - would be VERY helpful.

Where am I wrong?

stephenw10

Well it would remove load from the firewall. So if you were under a DDoS attack and needed to still route between internal subnets that could be useful. But it wouldn't help with the attack itself much.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Well it would remove load from the firewall. So if you were under a DDoS attack and needed to still route between internal subnets that could be useful. But it wouldn't help with the attack itself much.

Agree! Anyway for middle/big DDoS better to deal on local ISPs + CloudFlare level. Here no room for edge FW… :)

Let me to note, if thinking in “Zero thrust” direction, also FW on end local node/service as “fine tuning firewalling” would be great, because each end node better know what particular (and how) need to be secured.

So at the end we build 3-layered (as minimum) defense:

ACLs on edge ASIC-based switches;
pfSense as edge FW;
PF/IPF/IPFW FW (sertificates, tokens, etc…) on end node/service;

What do You think about this 3-layered scheme, @stephenw10 ?