How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?

Sergei_Shablovsky

Dear pfSense Gurus!

Could You be so please to suggest how to make from scratch HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs that would be stable, with less possible issues with VPNs and HAproxy-balancing and monitoring (SNMP + by Prometheus exporter) ?

CONDITIONS:
1.
ISPs contracts that we have now determine 1 static “white” IP per 1 physical uplink.
2.
Now EACH of pfSense’s server have **4 x physical ports for WANs + 10 x physical ports for LANs + 1 physical port for HA sync.
3.
Now VLANS not used, but may be in a future.
4.
Not problem with installing and use 10G L2/L3 switches (we have it a lot on shelfs…) if needed in scheme.

Thank You all for DETAILED suggestions!

Have a nice sunny days!

mcury

@Sergei_Shablovsky

You will need to connect each firewall to each ISP's router.
This setup uses a single switch but you could use two with VRRP enabled if that is what you want.
This setup is using a LACP to the switches, but you could change that to use the 10G switch you mentioned.

One consideration since you are going to have only one public address per WAN, I would use internal IPs in the firewall, a /29 network for each WAN.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky

This setup uses a single switch but you could use two with VRRP enabled if that is what you want.
This setup is using a LACP to the switches, but you could change that to use the 10G switch you mentioned.

THANK YOU SO MUCH for spending time to create this initial scheme! Take my respect for this!

Please confirm that I understood Your scheme correctly:

this scheme more oriented on original Netgate appliances (where no such ports that I have on custom server with a lot of dedicated NICs);
so because of above, You using in this scheme only one L2 switch (and may be VLANs to dividing “server farm”, “other nets “ and “wifi access network”) in which all internal nets are connected.
And this L2 switch are important point of failure: if hangs or property functioning - ALL internal services/resources would be down;

Am I miss something?
Thank You again one time for passion!

mcury

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

THANK YOU SO MUCH for spending time to create this initial scheme! Take my respect for this!

You are welcome, as you can see, I love networking =)

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

this scheme more oriented on original Netgate appliances (where no such ports that I have on custom server with a lot of dedicated NICs);

This scheme will work with any device, a computer with 7 ethernet ports would work for that environment.
I'm not so experienced with HA, I mounted one that is working to this day, no problems at all, just remember to create the interfaces in backup pfsense because those are not created automatically.
I'm running IPsec VTIs with OSPF, VLANs... everything is working fine.

For the Switch, yes, if it goes down, all the network will be taken down with it.
What I did is to keep cold replacement in the site, every cable is tagged to help when things go south.

Here is the Netgate's documentation about HA:
https://docs.netgate.com/pfsense/en/latest/highavailability/index.html

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

THANK YOU SO MUCH for spending time to create this initial scheme! Take my respect for this!

You are welcome, as you can see, I love networking =)

Ok, so let’s dive in multi-WAN (4 x WANs) and 10 x LANs.

At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.

(And later setup on each of that WANs switches span port and connect all 4 span ports to dedicated server with snort/suricata on it.
The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)

Ok?

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

this scheme more oriented on original Netgate appliances (where no such ports that I have on custom server with a lot of dedicated NICs);

This scheme will work with any device, a computer with 7 ethernet ports would work for that environment.
I'm not so experienced with HA, I mounted one that is working to this day, no problems at all, just remember to create the interfaces in backup pfsense because those are not created automatically.
I'm running IPsec VTIs with OSPF, VLANs... everything is working fine.

For the Switch, yes, if it goes down, all the network will be taken down with it.
What I did is to keep cold replacement in the site, every cable is tagged to help when things go south.

Here is the Netgate's documentation about HA:
https://docs.netgate.com/pfsense/en/latest/highavailability/index.html

If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.

That’s correct?

mcury

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)

There is a long time that I don't run IPS/IDS, but as I see it, you would need to send a copy of the packets, mirroring a port from the switch, to the IDS/IPS.
You won't be able to kill connections, block hosts or anything like this, unless you run the IDS/IPS on the firewall interface (LAN would be better).

If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.

Usually, the ISP's device (router) have a builtin switch, 4 ports are common, you can use those ports and ignore that WAN switch.
However, sometimes these builtin switches causes problems and you may need a new switch.
Since you have only one public address, just configure that ISP router DMZ to point to your CARP VIP address.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)

There is a long time that I don't run IPS/IDS, but as I see it, you would need to send a copy of the packets, mirroring a port from the switch, to the IDS/IPS.
You won't be able to kill connections, block hosts or anything like this, unless you run the IDS/IPS on the firewall interface (LAN would be better).

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.

Usually, the ISP's device (router) have a builtin switch, 4 ports are common, you can use those ports and ignore that WAN switch.
However, sometimes these builtin switches causes problems and you may need a new switch.

On now we have 2 pair of uplinks (2 from one ISP, and 2 from another ISP), physically independent cabling from outside. (Even different independent cable entry to building to avoid simultaneous local damaging).

Since you have only one public address, just configure that ISP router DMZ to point to your CARP VIP address.

For now we have 1 “white” IP on EACH of uplink, so 4 fiber uplinks give us 4 IP, EACH PAIR IN SEPARATE ISP’s ADDRESS SPACE.

mcury

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.
You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.
I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.

Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.

I ABSOLUTELY AGREE with You according DRAMATICALLY DECREASED ROLE OF DPI like Snort/Suricata: significant changes of both packet’s payload, mix of protocol’s and new protocols (QUIC, HTTP/3) in nowadays internet are roots of this changes.

And there are several threads on this pfSense users forum according that nowadays Snort/Suricata become more and more useless for traffic from outside to inside.

You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.

And because of above ONLY DETECTING OF SOPHISTICATED ATTACKS, BLOCKING and ALERTING the operational stuff - are work for IDS/IPS on outside perimeter.

Another one place to successfully using IDS/IPS are inside organization’s perimeter to detecting HACKERS ATTACKS FROM INSIDE (from compromised stuff’s personal nodes).
(But in this my certain case - that is no such important.)

I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.

Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.

Agree!!!

stephenw10

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.

You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.

However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.html

Steve

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.

You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.

Thank You for suggestions!

But in this case I have DECREASING FAILOVER ABILITY because MAKING EXTRA POINT OF FAILURE: if one of two switches fail (PSU failure, or overheating because fan fails even the switch are have both redundant PSU and redundant fan),- I LOST 2(TWO) UPLINKS not one…

However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.html

So, in fact, in addition to ADDING SWITCH TO EACH WAN uplink, I NEED DOUBLING ALL LANS NICs heads (to working with LACP, STP - capable switches), yes?

stephenw10

If you have switches that support cross-chassis lagg then you can use that to provide redundancy, yes.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.

Is that mean that having hardware encryption/decryption accelerator (like Intel QAT) to decreasing loading on main pfSense gate-firewall and keeping IDS/IPS (like Snort + Suricata) on a separate server AFTER the main pfSense - is ONLY ONE WAY TO INSPECT INCOME/OUTCOME TRAFFIC (and also the traffic between LANs and inside each LAN)?
(Especially when QUIC becomes more and more popular.)

If this my tough are correct, where on Your scheme placing this dedicated IDS/IPS?

Thanks for Your patience and help!

stephenw10

The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.

Thank You.

How to create the same interaction between Snort/Suricata on external server and pfSense (like now happened when Snort/Suricata are a integrated part of pfSense in one same server) ?

stephenw10

Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.

In IDS mode though you can just mirror the traffic to something external for analysis.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.

In IDS mode though you can just mirror the traffic to something external for analysis.

Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?

stephenw10

It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.

You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.

At the first let me thank You so much for patience and help!

I know (mostly from @bmeeks post like this ) about that “impossibility” in current versions of CE and Plus. But periodically asking someone with much better experience with pfSense about latest changes, to not miss something important new…

You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.

I not switch to scheme with dedicated IDS/IPS in case using pfSense because prefer ”all blocking logs from one place” strategy to avoid dealing much with logs synchronization before importing to Prometheus’s data source by Loki…

I waiting that pfSense DevTeam making this ability, but that not happened until now, ok… may be time to deal with precisely log synchronisation.

bmeeks

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?

No, as @stephenw10 said, this is not possible in the packages (Snort or Suricata) and is likely to never be available. The internal technology just does not lend itself to that.

What you seem to want to implement is much better done using bare metal with a Linux OS and installation of the appropriate Suricata package for that Linux distro. You would need to configure and manage that Suricata instance completely from a command-line interface (CLI) on that bare metal machine. You could implement Inline IPS mode operation using two independent NIC ports (one for traffic IN and the other for traffic OUT). You would want to use the AF_PACKET mode of operation in Suricata. Details on that can be found here: https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html#af-packet-ips-mode. With proper hardware, such a setup could easily handle 10G data streams. You would never approach that on pfSense because the IDS/IPS packages on pfSense utilize the much slower host rings interface for one side of the packet path. The Linux AF_PACKET mode use two discrete hardware NIC interfaces and bypasses the much slower host rings pathway.