How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?
-
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
THANK YOU SO MUCH for spending time to create this initial scheme! Take my respect for this!
You are welcome, as you can see, I love networking =)
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
this scheme more oriented on original Netgate appliances (where no such ports that I have on custom server with a lot of dedicated NICs);
This scheme will work with any device, a computer with 7 ethernet ports would work for that environment.
I'm not so experienced with HA, I mounted one that is working to this day, no problems at all, just remember to create the interfaces in backup pfsense because those are not created automatically.
I'm running IPsec VTIs with OSPF, VLANs... everything is working fine.For the Switch, yes, if it goes down, all the network will be taken down with it.
What I did is to keep cold replacement in the site, every cable is tagged to help when things go south.Here is the Netgate's documentation about HA:
https://docs.netgate.com/pfsense/en/latest/highavailability/index.html -
@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
THANK YOU SO MUCH for spending time to create this initial scheme! Take my respect for this!
You are welcome, as you can see, I love networking =)
Ok, so let’s dive in multi-WAN (4 x WANs) and 10 x LANs.
At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.
(And later setup on each of that WANs switches span port and connect all 4 span ports to dedicated server with snort/suricata on it.
The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)Ok?
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
this scheme more oriented on original Netgate appliances (where no such ports that I have on custom server with a lot of dedicated NICs);
This scheme will work with any device, a computer with 7 ethernet ports would work for that environment.
I'm not so experienced with HA, I mounted one that is working to this day, no problems at all, just remember to create the interfaces in backup pfsense because those are not created automatically.
I'm running IPsec VTIs with OSPF, VLANs... everything is working fine.For the Switch, yes, if it goes down, all the network will be taken down with it.
What I did is to keep cold replacement in the site, every cable is tagged to help when things go south.Here is the Netgate's documentation about HA:
https://docs.netgate.com/pfsense/en/latest/highavailability/index.htmlIf I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.
That’s correct?
—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)
There is a long time that I don't run IPS/IDS, but as I see it, you would need to send a copy of the packets, mirroring a port from the switch, to the IDS/IPS.
You won't be able to kill connections, block hosts or anything like this, unless you run the IDS/IPS on the firewall interface (LAN would be better).If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.
Usually, the ISP's device (router) have a builtin switch, 4 ports are common, you can use those ports and ignore that WAN switch.
However, sometimes these builtin switches causes problems and you may need a new switch.
Since you have only one public address, just configure that ISP router DMZ to point to your CARP VIP address. -
@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
The question are how dedicated IPS/IDS server would instruct LEAD pfSense to act accordingly events/alerts in snort/suricata?)
There is a long time that I don't run IPS/IDS, but as I see it, you would need to send a copy of the packets, mirroring a port from the switch, to the IDS/IPS.
You won't be able to kill connections, block hosts or anything like this, unless you run the IDS/IPS on the firewall interface (LAN would be better).Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.
If I understand scheme Diagram of Multi-WAN HA with DMZ correctly, I need extra 10 switches (10G ports all),- ONE 10G SWITCH for ONE LAN PORT.
Usually, the ISP's device (router) have a builtin switch, 4 ports are common, you can use those ports and ignore that WAN switch.
However, sometimes these builtin switches causes problems and you may need a new switch.On now we have 2 pair of uplinks (2 from one ISP, and 2 from another ISP), physically independent cabling from outside. (Even different independent cable entry to building to avoid simultaneous local damaging).
Since you have only one public address, just configure that ISP router DMZ to point to your CARP VIP address.
For now we have 1 “white” IP on EACH of uplink, so 4 fiber uplinks give us 4 IP, EACH PAIR IN SEPARATE ISP’s ADDRESS SPACE.
-
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.
I'm not the best user to answer that for you, really not, but here are my two cents:
Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.
You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.
I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.dead on arrival, nowhere to be found.
-
@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.
I'm not the best user to answer that for you, really not, but here are my two cents:
Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.I ABSOLUTELY AGREE with You according DRAMATICALLY DECREASED ROLE OF DPI like Snort/Suricata: significant changes of both packet’s payload, mix of protocol’s and new protocols (QUIC, HTTP/3) in nowadays internet are roots of this changes.
And there are several threads on this pfSense users forum according that nowadays Snort/Suricata become more and more useless for traffic from outside to inside.
You will get a lot of false positives, a lot of tweaking is necessary, so choose your IPS/IDS rules carefully.
And because of above ONLY DETECTING OF SOPHISTICATED ATTACKS, BLOCKING and ALERTING the operational stuff - are work for IDS/IPS on outside perimeter.
Another one place to successfully using IDS/IPS are inside organization’s perimeter to detecting HACKERS ATTACKS FROM INSIDE (from compromised stuff’s personal nodes).
(But in this my certain case - that is no such important.)I would also make sure the server is always updated and patched.
Perhaps, run Ninjaone, PRTG or something like that to monitor the server.Edit:
Syslog server is really important.
Backups, I can't stress enough how important backups are.Agree!!!
-
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.
You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.
However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.htmlSteve
-
@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
At the first step I need for 1 switch (let’s say 10G ports and 20/40G uplink port) for EACH OF WANs.
You don't actually need a separate switch for each WAN. It can just be a port VLAN group on the switch for example. It just needs to separate the WANs at layer 2.
Thank You for suggestions!
But in this case I have DECREASING FAILOVER ABILITY because MAKING EXTRA POINT OF FAILURE: if one of two switches fail (PSU failure, or overheating because fan fails even the switch are have both redundant PSU and redundant fan),- I LOST 2(TWO) UPLINKS not one…
However you might want to use at least two switches to provide some redundancy there:
https://docs.netgate.com/pfsense/en/latest/highavailability/layer-2-redundancy.htmlSo, in fact, in addition to ADDING SWITCH TO EACH WAN uplink, I NEED DOUBLING ALL LANS NICs heads (to working with LACP, STP - capable switches), yes?
-
If you have switches that support cross-chassis lagg then you can use that to provide redundancy, yes.
-
@mcury said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Sorry for my possible mistakes in explanation: now only one unresolved problem exist,- HOW TO make pfSense to listen (and act accordingly) IDS/IPS snort/suricata on dedicated server.
I'm not the best user to answer that for you, really not, but here are my two cents:
Just enable Snort/Suricata, run it on the LAN side.
Snort/Suricata will listen on all VLANs if configured in an interface, so I would enable it only in a dedicated interface for the server, not running other VLANs.
Snort/Suricata won't be able to see the payload of the packets since 99,99% of the Internet today is encrypted.Is that mean that having hardware encryption/decryption accelerator (like Intel QAT) to decreasing loading on main pfSense gate-firewall and keeping IDS/IPS (like Snort + Suricata) on a separate server AFTER the main pfSense - is ONLY ONE WAY TO INSPECT INCOME/OUTCOME TRAFFIC (and also the traffic between LANs and inside each LAN)?
(Especially when QUIC becomes more and more popular.)If this my tough are correct, where on Your scheme placing this dedicated IDS/IPS?
Thanks for Your patience and help!
-
The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.
-
@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
The only way to inspect encrypted traffic is to run the IPS where it is decrypted. So that's either on the server endpoint directly or by proxying all the traffic and decrypting it there.
Thank You.
How to create the same interaction between Snort/Suricata on external server and pfSense (like now happened when Snort/Suricata are a integrated part of pfSense in one same server) ?
-
S Sergei_Shablovsky referenced this topic on
-
Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.
In IDS mode though you can just mirror the traffic to something external for analysis.
-
@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Well there's no way to pass firewall blocks from something external if you're using it in IPS mode. At least not yet.
In IDS mode though you can just mirror the traffic to something external for analysis.
Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.
You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.
-
@stephenw10 said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
It's not possible to do it directly if you need blocking mode because both Snort and Suricata packages include custom code to interact with pf that only work locally.
At the first let me thank You so much for patience and help!
I know (mostly from @bmeeks post like this ) about that “impossibility” in current versions of CE and Plus. But periodically asking someone with much better experience with pfSense about latest changes, to not miss something important new…
You could set it up in in-line mode as, for example, a bridged device of some sort in front of pfSense. There is could block traffic directly as I understand it but I've never tried to do that.
I not switch to scheme with dedicated IDS/IPS in case using pfSense because prefer ”all blocking logs from one place” strategy to avoid dealing much with logs synchronization before importing to Prometheus’s data source by Loki…
I waiting that pfSense DevTeam making this ability, but that not happened until now, ok… may be time to deal with precisely log synchronisation.
-
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?No, as @stephenw10 said, this is not possible in the packages (Snort or Suricata) and is likely to never be available. The internal technology just does not lend itself to that.
What you seem to want to implement is much better done using bare metal with a Linux OS and installation of the appropriate Suricata package for that Linux distro. You would need to configure and manage that Suricata instance completely from a command-line interface (CLI) on that bare metal machine. You could implement Inline IPS mode operation using two independent NIC ports (one for traffic IN and the other for traffic OUT). You would want to use the AF_PACKET mode of operation in Suricata. Details on that can be found here: https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html#af-packet-ips-mode. With proper hardware, such a setup could easily handle 10G data streams. You would never approach that on pfSense because the IDS/IPS packages on pfSense utilize the much slower host rings interface for one side of the packet path. The Linux AF_PACKET mode use two discrete hardware NIC interfaces and bypasses the much slower host rings pathway.
-
@bmeeks said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Sorry for my understanding, I try to re-formulate my question: how to Snort/Suricata on separate bare metal server may interacts with pfSense like internal “packaged” version does?
Need to pulling out Snort, Suricata, ntopng to separate node.
Is that possible at all?No, as @stephenw10 said, this is not possible in the packages (Snort or Suricata) and is likely to never be available. The internal technology just does not lend itself to that.
Thank You so much for suggestions!
Ok, “never” mean “never”.
What you seem to want to implement is much better done using bare metal with a Linux OS and installation of the appropriate Suricata package for that Linux distro.
Which advantages Lunux (RHEL in my case) have in comparison with FreeBSD in this certain usecase?
You would need to configure and manage that Suricata instance completely from a command-line interface (CLI) on that bare metal machine.
From Your experience is great Ansible role exist for orchestrating the Suricata or Snort ?
You could implement Inline IPS mode operation using two independent NIC ports (one for traffic IN and the other for traffic OUT). You would want to use the AF_PACKET mode of operation in Suricata. Details on that can be found here: https://docs.suricata.io/en/latest/setting-up-ipsinline-for-linux.html#af-packet-ips-mode. With proper hardware, such a setup could easily handle 10G data streams. You would never approach that on pfSense because the IDS/IPS packages on pfSense utilize the much slower host rings interface for one side of the packet path. The Linux AF_PACKET mode use two discrete hardware NIC interfaces and bypasses the much slower host rings pathway.
So, in which place in scheme this dedicated server must be installed if I need to inspect/ blocking **both incoming traffic (on each of 4 uplinks) and inside traffic (on each of LANs (each of them connected to pfSense separate NICs port thru the hardware manageable switch) ?
-
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
Which advantages Lunux (RHEL in my case) have in comparison with FreeBSD in this certain usecase?
Mainly because the upstream Suricata development team tests extensively on Linux and all of them use Suricata on Linux. The only thing they do for FreeBSD is compile Suricata on a dedicated single instance just to be sure it compiles successfully. They do not test or benchmark there other than to ensure basic functionality.
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
From Your experience is great Ansible role exist for orchestrating the Suricata or Snort ?
I've never used Ansible, so I don't feel qualified to answer this question. I would completely disregard Snort for this role as it is single-threaded and has no hope of matching performance with Suricata. That is unless you migrate to the new Snort3 binary.
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
So, in which place in scheme this dedicated server must be installed?
From your diagram at the top of this thread, it appears you would need at least 4 of these IDS appliances (and perhaps 8 for full redundancy). Each appliance would live on the link between your Internet facing switch and your pfSense firewalls.
The appliances would in effect perform as bridges copying traffic at wire speed between the two hardware NIC ports with Suricata sitting in the middle of the internal bridge path analyzing the traffic. Traffic that passed all the rules would be copied from the IN port directly to the OUT port while traffic that triggered a DROP rule would not be copied to the OUT port (and thus effectively dropped). I'm essentially describing a build-it-yourself version of this now discontinued Cisco/Sourcefire appliance family: https://www.cisco.com/c/en/us/support/security/firepower-8000-series-appliances/series.html.
-
@bmeeks said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
@Sergei_Shablovsky said in How to make HA on 2 pfSense on bare metal WITH 4 x UPLINKS WANs ?:
So, in which place in scheme this dedicated server must be installed?
From your diagram at the top of this thread, it appears you would need at least 4 of these IDS appliances (and perhaps 8 for full redundancy). Each appliance would live on the link between your Internet facing switch and your pfSense firewalls.
The appliances would in effect perform as bridges copying traffic at wire speed between the two hardware NIC ports with Suricata sitting in the middle of the internal bridge path analyzing the traffic. Traffic that passed all the rules would be copied from the IN port directly to the OUT port while traffic that triggered a DROP rule would not be copied to the OUT port (and thus effectively dropped).
Is this mean that possible using 2 of bare metal servers (for redundancy) and each one of servers have NICs with total 8 of hardware ports (4 for IN traffic and 4 for OUT)?
If You have some experience, which bare metal (in terms of CPU/NICs characteristics) capable to handle total 40Gb/s (4 x 10Gb/s + system Suricata overhead), 80 Gb/s (4 x 20Gb/s)?
I'm essentially describing a build-it-yourself version of this now discontinued Cisco/Sourcefire appliance family: https://www.cisco.com/c/en/us/support/security/firepower-8000-series-appliances/series.html.
Thank You so much! Promise to reading today evening’s;)
