How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)

There is a long time that I don't run IPS/IDS, but as I see it, you would need to send a copy of the packets, mirroring a port from the switch, to the IDS/IPS.
You won't be able to kill connections, block hosts or anything like this, unless you run the IDS/IPS on the firewall interface (LAN would be better).

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.

Usually, the ISP's device (router) have a builtin switch, 4 ports are common, you can use those ports and ignore that WAN switch.
However, sometimes these builtin switches causes problems and you may need a new switch.

On now we have 2 pair of uplinks (2 from one ISP, and 2 from another ISP), physically independent cabling from outside. (Even different independent cable entry to building to avoid simultaneous local damaging).

Since you have only one public address, just configure that ISP router DMZ to point to your CARP VIP address.

For now we have 1 “white” IP on EACH of uplink, so 4 fiber uplinks give us 4 IP, EACH PAIR IN SEPARATE ISP’s ADDRESS SPACE.

mcury

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.
You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.
I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.

Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.

I ABSOLUTELY AGREE with You according DRAMATICALLY DECREASED ROLE OF DPI like Snort/Suricata: significant changes of both packet’s payload, mix of protocol’s and new protocols (QUIC, HTTP/3) in nowadays internet are roots of this changes.

And there are several threads on this pfSense users forum according that nowadays Snort/Suricata become more and more useless for traffic from outside to inside.

You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.

And because of above ONLY DETECTING OF SOPHISTICATED ATTACKS, BLOCKING and ALERTING the operational stuff - are work for IDS/IPS on outside perimeter.

Another one place to successfully using IDS/IPS are inside organization’s perimeter to detecting HACKERS ATTACKS FROM INSIDE (from compromised stuff’s personal nodes).
(But in this my certain case - that is no such important.)

I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.

Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.

Agree!!!

stephenw10

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.

You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.

However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.html

Steve

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.

You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.

Thank You for suggestions!

But in this case I have DECREASING FAILOVER ABILITY because MAKING EXTRA POINT OF FAILURE: if one of two switches fail (PSU failure, or overheating because fan fails even the switch are have both redundant PSU and redundant fan),- I LOST 2(TWO) UPLINKS not one…

However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.html

So, in fact, in addition to ADDING SWITCH TO EACH WAN uplink, I NEED DOUBLING ALL LANS NICs heads (to working with LACP, STP - capable switches), yes?

stephenw10

If you have switches that support cross-chassis lagg then you can use that to provide redundancy, yes.

Sergei_Shablovsky

@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.

I'm not the best user to answer that for you, really not, but here are my two cents:

Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.

Is that mean that having hardware encryption/decryption accelerator (like Intel QAT) to decreasing loading on main pfSense gate-firewall and keeping IDS/IPS (like Snort + Suricata) on a separate server AFTER the main pfSense - is ONLY ONE WAY TO INSPECT INCOME/OUTCOME TRAFFIC (and also the traffic between LANs and inside each LAN)?
(Especially when QUIC becomes more and more popular.)

If this my tough are correct, where on Your scheme placing this dedicated IDS/IPS?

Thanks for Your patience and help!

stephenw10

The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.

Thank You.

How to create the same interaction between Snort/Suricata on external server and pfSense (like now happened when Snort/Suricata are a integrated part of pfSense in one same server) ?

stephenw10

Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.

In IDS mode though you can just mirror the traffic to something external for analysis.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.

In IDS mode though you can just mirror the traffic to something external for analysis.

Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?

stephenw10

It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.

You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.

Sergei_Shablovsky

@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.

At the first let me thank You so much for patience and help!

I know (mostly from @bmeeks post like this ) about that “impossibility” in current versions of CE and Plus. But periodically asking someone with much better experience with pfSense about latest changes, to not miss something important new…

You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.

I not switch to scheme with dedicated IDS/IPS in case using pfSense because prefer ”all blocking logs from one place” strategy to avoid dealing much with logs synchronization before importing to Prometheus’s data source by Loki…

I waiting that pfSense DevTeam making this ability, but that not happened until now, ok… may be time to deal with precisely log synchronisation.

bmeeks

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?

No, as @stephenw10 said, this is not possible in the packages (Snort or Suricata) and is likely to never be available. The internal technology just does not lend itself to that.

What you seem to want to implement is much better done using bare metal with a Linux OS and installation of the appropriate Suricata package for that Linux distro. You would need to configure and manage that Suricata instance completely from a command-line interface (CLI) on that bare metal machine. You could implement Inline IPS mode operation using two independent NIC ports (one for traffic IN and the other for traffic OUT). You would want to use the AF_PACKET mode of operation in Suricata. Details on that can be found here: https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html#af-packet-ips-mode. With proper hardware, such a setup could easily handle 10G data streams. You would never approach that on pfSense because the IDS/IPS packages on pfSense utilize the much slower host rings interface for one side of the packet path. The Linux AF_PACKET mode use two discrete hardware NIC interfaces and bypasses the much slower host rings pathway.

Sergei_Shablovsky

@bmeeks said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?

No, as @stephenw10 said, this is not possible in the packages (Snort or Suricata) and is likely to never be available. The internal technology just does not lend itself to that.

Thank You so much for suggestions!

Ok, “never” mean “never”.

What you seem to want to implement is much better done using bare metal with a Linux OS and installation of the appropriate Suricata package for that Linux distro.

Which advantages Lunux (RHEL in my case) have in comparison with FreeBSD in this certain usecase?

You would need to configure and manage that Suricata instance completely from a command-line interface (CLI) on that bare metal machine.

From Your experience is great Ansible role exist for orchestrating the Suricata or Snort ?

You could implement Inline IPS mode operation using two independent NIC ports (one for traffic IN and the other for traffic OUT). You would want to use the AF_PACKET mode of operation in Suricata. Details on that can be found here: https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html#af-packet-ips-mode. With proper hardware, such a setup could easily handle 10G data streams. You would never approach that on pfSense because the IDS/IPS packages on pfSense utilize the much slower host rings interface for one side of the packet path. The Linux AF_PACKET mode use two discrete hardware NIC interfaces and bypasses the much slower host rings pathway.

So, in which place in scheme this dedicated server must be installed if I need to inspect/ blocking **both incoming traffic (on each of 4 uplinks) and inside traffic (on each of LANs (each of them connected to pfSense separate NICs port thru the hardware manageable switch) ?

bmeeks

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Which advantages Lunux (RHEL in my case) have in comparison with FreeBSD in this certain usecase?

Mainly because the upstream Suricata development team tests extensively on Linux and all of them use Suricata on Linux. The only thing they do for FreeBSD is compile Suricata on a dedicated single instance just to be sure it compiles successfully. They do not test or benchmark there other than to ensure basic functionality.

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

From Your experience is great Ansible role exist for orchestrating the Suricata or Snort ?

I've never used Ansible, so I don't feel qualified to answer this question. I would completely disregard Snort for this role as it is single-threaded and has no hope of matching performance with Suricata. That is unless you migrate to the new Snort3 binary.

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

So, in which place in scheme this dedicated server must be installed?

From your diagram at the top of this thread, it appears you would need at least 4 of these IDS appliances (and perhaps 8 for full redundancy). Each appliance would live on the link between your Internet facing switch and your pfSense firewalls.

The appliances would in effect perform as bridges copying traffic at wire speed between the two hardware NIC ports with Suricata sitting in the middle of the internal bridge path analyzing the traffic. Traffic that passed all the rules would be copied from the IN port directly to the OUT port while traffic that triggered a DROP rule would not be copied to the OUT port (and thus effectively dropped). I'm essentially describing a build-it-yourself version of this now discontinued Cisco/Sourcefire appliance family: https://www.cisco.com/c/en/us/support/security/firepower-8000-series-appliances/series.html.

Sergei_Shablovsky

@bmeeks said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

So, in which place in scheme this dedicated server must be installed?

From your diagram at the top of this thread, it appears you would need at least 4 of these IDS appliances (and perhaps 8 for full redundancy). Each appliance would live on the link between your Internet facing switch and your pfSense firewalls.

The appliances would in effect perform as bridges copying traffic at wire speed between the two hardware NIC ports with Suricata sitting in the middle of the internal bridge path analyzing the traffic. Traffic that passed all the rules would be copied from the IN port directly to the OUT port while traffic that triggered a DROP rule would not be copied to the OUT port (and thus effectively dropped).

Is this mean that possible using 2 of bare metal servers (for redundancy) and each one of servers have NICs with total 8 of hardware ports (4 for IN traffic and 4 for OUT)?

If You have some experience, which bare metal (in terms of CPU/NICs characteristics) capable to handle total 40Gb/s (4 x 10Gb/s + system Suricata overhead), 80 Gb/s (4 x 20Gb/s)?

I'm essentially describing a build-it-yourself version of this now discontinued Cisco/Sourcefire appliance family: https://www.cisco.com/c/en/us/support/security/firepower-8000-series-appliances/series.html.

Thank You so much! Promise to reading today evening’s;)

bmeeks

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Is this mean that possible using 2 of bare metal servers (for redundancy) and each one of servers have NICs with total 8 of hardware ports (4 for IN traffic and 4 for OUT)?

You may want to review the Suricata docs I linked earlier so that you fully understand how IPS mode works. When I say IN and OUT those are relative. Literally each NIC port is both an IN and an OUT simultaneously just like a NIC on a conventional workstation supports bi-directional traffic. But yes, you would want probably 8 total NIC ports for the IPS and then one additional NIC for a managment port. This management port would get an IP address. The other NIC ports would not get an IP assignment. More details below.

With a true Inline IPS configuration in Suricata, you specify in the YAML config file two port pairs for each Suricata instance. In the Suricata YAML configuration file you would specify an interface and a copy-interface for each traffic flow direction so that you can support bi-directional traffic. Here is an example config: https://docs.mirantis.com/mcp/q4-18/mcp-security-best-practices/use-cases/idps-vnf/ips-mode/afpacket.html.

This is the configuration from the link above:

af-packet:
- interface: eth0
threads: auto
defrag: yes
cluster-type: cluster_flow
cluster-id: 98
copy-mode: ips
copy-iface: eth1
buffer-size: 64535
use-mmap: yes
- interface: eth1
threads: auto
cluster-id: 97
defrag: yes
cluster-type: cluster_flow
copy-mode: ips
copy-iface: eth0
buffer-size: 64535
use-mmap: yes

Examine the YAML configuration above closely and notice how the eth0 and eth1 interfaces are each used twice in the configuration (as targets of the interface: parameter). Once as the "source" and again later on as the copy-iface or copy-to interface. This literally means traffic inbound on eth0 is intercepted by an af-packet thread, inspected by Suricata, then copied to eth1 for output or else dropped if the traffic matched a DROP rule. At the same time, a separate Suricata af-packet thread is reading inbound traffic from interface eth1, inspecting it, and copying it to eth0, the copy-iface, for output or else dropping it if the traffic matches a DROP rule signature. The above operations can be very fast because the network traffic completely bypasses the kernel stack.

In this design (or a similar design implemented with netmap on FreeBSD), you would need a totally separate NIC port for logging in and configuring and maintaining the IPS appliance. You would put an IP address on that separate NIC. But the NIC ports used for inline IPS operation would not get any IP configuration at all. Internally Suricata and the af-packet process treats them like a bridge (but do not configure them as an actual bridge!).

The above description is hopefully enough for you to formulate your own HA design in terms of hardware. With IPS operation, you would connect say eth0 to one of your WAN-side ports and eth1 to one of your pfSense firewall ports. Those two ports would literally be "inline" with the cable connecting your WAN switch port to your firewall port -- hence the name Inline IPS (inline intrusion prevention).

@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

If You have some experience, which bare metal (in terms of CPU/NICs characteristics) capable to handle total 40Gb/s (4 x 10Gb/s + system Suricata overhead), 80 Gb/s (4 x 20Gb/s)?

I don't have any personal experience with hardware. There are lots of thread discusussions and examples to be found with a Google search for "high throughput or high speed or high traffic" Suricata operation. You might consider up to 4 server appliances. It would all depend on the capacity of the chosen hardware. NICs with plenty of queues will help. Run Suricata in the workers threading mode. That will create a single thread per CPU core (and ideally per NIC queue). Each single thread would handle a packet from ingestion to output on a given NIC queue. With lots of queues, CPU cores, and Suricata threads you get more throughput. There are commercial off-the-shelf appliances for this, but of course they are not cheap .

Follow-up Caveat: Suricata cannot analyze encrypted traffic. Since nearly 90% or more of traffic on the Internet these days is encrypted, Suricata is blind to a lot of what crosses the perimeter link. You can configure fancy proxy servers to implement MITM (man-in-the-middle) interception and decryption/re-encryption of such traffic, but that carries its own set of issues. Depending on what you are hoping to scan for, it could be that putting the security emphasis on the endpoints (workstations and servers) instead of the perimeter (firewall) is a much better strategy with a higher chance of successfully intercepting bad stuff.

Sergei_Shablovsky

@bmeeks said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:

Follow-up Caveat: Suricata cannot analyze encrypted traffic. Since nearly 90% or more of traffic on the Internet these days is encrypted, Suricata is blind to a lot of what crosses the perimeter link. You can configure fancy proxy servers to implement MITM (man-in-the-middle) interception and decryption/re-encryption of such traffic, but that carries its own set of issues. Depending on what you are hoping to scan for, it could be that putting the security emphasis on the endpoints (workstations and servers) instead of the perimeter (firewall) is a much better strategy with a higher chance of successfully intercepting bad stuff.

At the first let me say BIGGEST THANKS FOR SO DETAILED ANSWERING and passion to help me resolving the case.

So, because SSL/TLS1.3 connections become standard by default in most common used desktop and mobile browsers (and even search systems exclude sites w/o SSL from their ranking and search results) and QUIC at all become more and more popular on all server OSs and web-servers,- is that mean that EOL date for IDS/IPS w/o mitm come close and close? And not only for outside incoming traffic.
Even inside of organisation’s security perimeter would be no place for Suricata/Snort.

And only what Security Admin may doing would be:

fresh updates for applications and OSs;
planning internal infrastructure and intrusion monitoring well;
extensively using AI for monitoring hardware and apps, anomalies real-time finding (and alerting);
creating great firewall’s rules;

Glad to read Your opinion about that.

stephenw10

In a large organisation though they may force all traffic through a proxy to decrypt it. In which case it could still be scanned.