Suricata behind HA Proxy - Only run in IDS mode

michmoor · Mar 26, 2024, 7:32 PM

Greetings,
Just want to confirm my set up here.
I have an interface that is my DMZ that HA proxy will initiate traffic towards for from Internet-bound IPs.

Most of the backend communications is on port 80,. Is this is the logical place to have suricata running? I believe it is as i cant put Suricata anywhere else as it will be all TLS flows.
This brings up two issues.

The source IP of any traffic will be the firewall itself (HA Proxy).
I cant run Suricata in IPS mode as any block will block traffic from the firewall itself. Am I really limited to IDS mode if i run it behind HA Proxy?

bmeeks · Mar 26, 2024, 7:50 PM

@michmoor said in Suricata behind HA Proxy - Only run in IDS mode:

I cant run Suricata in IPS mode as any block will block traffic from the firewall itself. Am I really limited to IDS mode if i run it behind HA Proxy?

If you use inline IPS mode, then you should be okay as that mode does not block an entire IP address. It simply drops particular offending traffic flows (but not every flow on the interface). Inline IPS mode does not block by blocking an IP address -- only Legacy Mode does that.

If you are forced to use Legacy Mode due to some other dependency, then it is true that IDS mode is what you need (then with no ability to block traffic).

michmoor · Mar 26, 2024, 8:00 PM

@bmeeks
I should mention this is a trunk port
So it looks like this

igc2.14

I can still run INLINE, right? I know there was some issue about sub-interfaces and it not being a real interface.

bmeeks · Mar 26, 2024, 8:15 PM

@michmoor said in Suricata behind HA Proxy - Only run in IDS mode:

@bmeeks
I should mention this is a trunk port
So it looks like this
igc2.14
I can still run INLINE, right? I know there was some issue about sub-interfaces and it not being a real interface.

The GUI code will work under the cover to run Suricata on the parent interface instead of the VLAN. VLANs are not well supported with netmap. There is also an issue with how netmap parses VLAN interface names in FreeBSD, so if a true VLAN interface is passed to netmap it will throw a syntax error on the name and bail out. A fix has been submitted upstream for that, but I'm not sure that fix has migrated into pfSense yet.

You can try to see if it works. Worst that can happen is it won't work at all (as in Suricata won't start up).

michmoor · Mar 26, 2024, 8:50 PM

@bmeeks
I'll take the win when i can. Rule reload complete. So far its enabled on a sub-interface.

So to confirm, Suricata is running on the parent interface?
Suricata is looking specifically at flows matching that interface or ALL flows on the parent. Confused on what will actually trigger.

I am using SID Mgmt to manage the categories. I noticed that they are set to drop as that's what i want but under the Categories list they are just checked off.
For example,: attack_response rules don't show that its enabled by SID mgmt conf files.

Maybe just GUI representation. But the rules are set to drop so that's good.

bmeeks · Mar 26, 2024, 8:50 PM

@michmoor said in Suricata behind HA Proxy - Only run in IDS mode:

ALL flows on the parent.

Suricata by default puts the monitored interface in promiscuous mode, so it is seeing every packet that hits the interface.

But how the enabled rules actually respond to that traffic is controlled by the content of the $HOME_NET and $EXTERNAL_NET variables. Most rules use those two variables along with a direction operator to put conditions around when they trigger. The IP addresses in those variables will be pulled from how the VLAN interface is configured in pfSense.

bmeeks · Mar 26, 2024, 8:53 PM

@michmoor said in Suricata behind HA Proxy - Only run in IDS mode:

Maybe just GUI representation. But the rules are set to drop so that's good.

You can always see the actual rules' text by going to the RULES tab and selecting Active Rules in the Category drop-down selector. If you have lots of enabled rules, it will take several seconds for the rules to fully populate on the screen. Then you can click on any individual rule and view the literal text for that rule in Suricata.

michmoor · Mar 26, 2024, 9:03 PM

@bmeeks
ahh spoke to soon. Starting to error out.

Based on your previous warning this is probably expected. I will have to operate on IDS mode for now.

Mar 26 17:01:58 GAFW kernel: 918.125691 [1843] netmap_ring_reinit        called for igc2 RX2
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359760] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 918.140234 [1843] netmap_ring_reinit        called for igc2 RX0
Mar 26 17:01:58 GAFW suricata[8453]: [359812] <Error> -- igc2^: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 84 rh 210 rc 210 rt 184 hc 183 ht 184
Mar 26 17:01:58 GAFW kernel: 918.150166 [1843] netmap_ring_reinit        called for igc2 RX6
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 918.172061 [1843] netmap_ring_reinit        called for igc2 RX0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 918.191211 [1843] netmap_ring_reinit        called for igc2 RX0
Mar 26 17:01:58 GAFW suricata[55451]: [358283] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[55451]: [358283] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW php-fpm[1922]: /rc.newwanip: Removing static route for monitor 8.8.8.8 and adding a new route through 104.13.92.1
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW php-fpm[1922]: /rc.newwanip: Removing static route for monitor 8.8.4.4 and adding a new route through 10.2.0.1
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[55451]: [358283] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 918.310742 [1803] nm_rxsync_prologue        igc2 RX0: fail 'head < kring->nr_hwcur && head > kring->nr_hwtail' h 455 c 455 t 451 rh 455 rc 455 rt 451 hc 590 ht 451
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359719] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW arpwatch[27291]: b0:09:da:27:8a:5d sent packet not ETHERTYPE_IP (0x4c5)
Mar 26 17:01:58 GAFW suricata[55451]: [358302] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[55451]: [358302] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW suricata[8453]: [359742] <Error> -- igc2: error reading netmap data via polling: No error: 0
Mar 26 17:01:58 GAFW kernel: 918.863809 [1803] nm_rxsync_prologue        igc2 RX1: fail 'head < kring->nr_hwcur && head > kring->nr_hwtail' h 665 c 665 t 664 rh 665 rc 665 rt 664 hc 686 ht 664

bmeeks · Mar 26, 2024, 9:10 PM

As I mentioned, the netmap device and VLANs really do not like each other .

Inline IPS Mode should really only be used with true native physical interfaces (no VLANs, no Bridges, and no LAGGs).

If you have enough physical interfaces on your firewall box, why use a VLAN in the first place? Just install a dedicated switch for the DMZ and connect it directly to a physically-mapped interface on pfSense.

It seems folks today want to immediately implement VLANs, but that brings a level of complexity into the design as well as potentially causing compatibility issues (the netmap and inline IPS mode incompatibility being one example). Often simply using a separate switch and a dedicated physical interface on the firewall can work just fine (and be quite a bit simpler to implement and maintain).

michmoor · Mar 27, 2024, 1:18 PM

@bmeeks said in Suricata behind HA Proxy - Only run in IDS mode:

Often simply using a separate switch and a dedicated physical interface on the firewall can work just fine (and be quite a bit simpler to implement and maintain).

For sure in much smaller environments especially one that i am working on right now. I have a 6100 so i do have a spare interface.
But its not uncommon to have a single interface trunking multiple zones. Company i work for now has multiple DMZ zones off a parent interface. Sometimes design dictates choices.

michmoor · Mar 27, 2024, 9:37 PM

@bmeeks
I really appreciate your help with answer my questions yesterday.

Had to change a few backend services to listen on port 80 or another custom port (if its docker) that still sends data in the clear; in my case port 5100

So to answer anyone else's questions that stumble upon this post.

Have Suricata listen in on the backend communications where the servers are located and make sure that backend communication is in clear text and not https. Some folks like to enable https on the backend as well but then you hurt your IDS chances of scanning payloads and alerting.
As of now netmap only works on physical interfaces - no vlans. no trunking. I ended up breaking traffic flow to my DMZ segment yesterday due to netmap errors.