Suricata Inline dropping some HTTPS

rmhco

Hello All,
Bit puzzled on this one, running Suricata in inline mode in a decently sized environment with ET PRO rules. When I switched over from Legacy to Inline mode (thanks bmeek!), not immediately but after a recent reboot, HTTPS traffic to www.google.com started being dropped; if there are other sites affected I'm unaware. Nothing is logged by Suricata, pfBlocker or the firewall itself; QUIC is working.

I had to turn off the TLS parser which is not ideal… Anyone having similar issues? Ideas?

Other settings:

TLS logging/storing: on or off makes no difference
Pattern Matcher: Hyperscan
Detect-Engine Profile: High
IPS Policy Selection: Connectivity
IPS Policy Mode: Policy
Flow/Stream options are default, could be increased system if beefy

bmeeks

If you look on the ALERTS tab do you see any entries in red text? Those would indicate DROPPED connections which is the way blocks work with Inline IPS mode. The BLOCKS tab is only relevant when using Legacy Mode blocking. If HTTPS Google is indeed getting dropped, then you should be seeing red text lines on the ALERTS tab showing what rule triggered.

Bill

rmhco

Thanks for the response and all your work on this package bmeeks.

I don't blame you for asking but, no, no events on the alerts tab, blocks tab, or on the remote syslog server from barnyard2. No error messages in suricata.log or local syslog, but I do see entries on the tls.log referencing the google IP addresses clients are trying to visit.

I'm left assuming it has to be the parser?

I did a packet capture on a client workstation and another capture on the firewall's WAN line hoping to identify which packets are being dropped but for some reason I was only able to capture inbound traffic on the firewall making the test less useful, I've included screenshots anyways.

My guess is the clients TLS Client Hello is not being delivered because of the multiple retransmission requests the clients sends afterward and the eventual FIN-ACK response.

onyxfire

Are you using VLAN interfaces?

I ask because I was not getting any alerts at all once I switched to inline mode but packets were clearly being dropped but not logged by suricata in the alerts.log and some sites even just plain refused to load at all. I have my routers setup in a "router-on-a-stick" type config where all traffic from internal networks comes in just the one physical interface but with separate VLAN interfaces tagged by 802.1.Q VLANs on the switch.

Suricata looks at your MTU on the interface it is running on and adds 16 for its "Bucket length" (like snaplen on snort but not configurable on suricata). This makes the "snaplen" of data for each packet 1516 if you haven't changed from the default MTU of 1500. This leaves out the VLAN information which makes it almost impossible for the "stream" processor in suricata to actually tell which packets are related to which streams on which VLAN. This can cause website issues, no alerts on pfsense GUI, etc… The fix is to change your MTU to 1502 which makes suricata again add 16 giving you a total of 1518 "snaplen" which will then be large enough to include the VLAN information.

To fix it on pfsense:

Create another interface for the physical (parent) interface to your VLAN interfaces that suricata is listening on if you don't already have a non-VLAN interface configured for it (i.e. igb1 or whatever yours is)
Make sure you enable this interface or all VLAN traffic will cease to flow to that interface. You do NOT have to assign it an IP address, just make sure the enabled box is checked.
Set its MTU to 1502 (this has to be done or you can't set the VLANs to an MTU of 1502. They are considered child interfaces and you can't set the child higher than the parent)
Go into each of your VLAN interfaces on that physical interface and set them to an MTU of 1502
Restart suricata

If you are not using VLANs, it may still be beneficial though I haven't tested it in that setup. That being said, security onion (popular linux security distro) has made it their default for a while now to prevent issues with their distro using suricata and snort. It should be a no impact change (with the exception of temporary outage when the interfaces are configured) due to the fact that any devices connecting will use Path MTU Discovery (PMTUD) to discover the appropriate packets size to send.

Other than that, you could try putting it in non-blocking mode again and see if when you visit google you get any alerts (if you haven't already confirmed that as it seems you have already done some investigation).

bmeeks

@onyxfire, thank you for the heads-up about the snaplen parameter and its impact on VLAN traffic detection. It is in fact a configurable parameter for Legacy Mode (the mode that uses PCAP), so I will add a GUI item for setting that in the next Suricata update that I am working on. That will let the user configure their own custom Snap Length for situations involving VLAN-tagged traffic on an interface using Legacy Mode blocking.

I don't currently see a similar configuration item for the Netmap mode used for Inline IPS in the Suricata documentation or source code.

Bill

onyxfire

@bmeeks

No problem. The documentation for 1:1 comparison of snort.conf to suricata.yaml (https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Snortconf_to_Suricatayaml#Snaplen) says it is always automatically at "full snaplen" for suricata but I think this only applies to Inline or if no MTU is detected using ioctl for pcap mode. This is where I got started looking at how it gets it and found out it adds 16 to the interface MTU. Basically it looks for the MTU of the interface and, if it cannot find one using ioctl, it automatically sets to 1518 or "full length" which would work. That being said, it CAN find the MTU of 1500 on most interfaces and so defaults to 1500+16. I see where they added the ability for you to configure it in pcap mode here: https://github.com/OISF/suricata/pull/287/commits/1feb84916c43666f02efe33d46debcf0d92834a0

As a side note, Snort should probably be defaulted to 1518 snaplen regardless in pfsense to mitigate any issues with VLAN users on the snort side (won't hurt anything but will definitely help. It will just make sure it works properly and, if they aren't using any VLANs, will just use slightly more resources but should be a negligible amount). You could then also make it configurable in the GUI in case someone has different needs or needs to adjust it (jumbo frames scenarios, etc.) as you mentioned.

On the suricata side, maybe the GUI should only give you the option to set snaplen for pcap mode and hide it otherwise with a note about the MTU and VLANs because I don't see it anywhere being able to do it in the suricata.yaml for Inline mode either and it might confuse those who don't know it won't work in that mode.

Here is a discussion from when security onion distro was talking about the issue and they describe it pretty well. In the end, if you look at github, they decided to go ahead and make it the default about 2 years ago to resolve this issue. You probably can't do what they did and force all interfaces to use MTU 1502 when switching suricata to inline mode but a warning in the GUI would probably suffice.

https://groups.google.com/forum/#!topic/security-onion/94s7beFDMU0

Thanks for always being so responsive on the forums.

bmeeks

@onyxfire:

@bmeeks

No problem. The documentation for 1:1 comparison of snort.conf to suricata.yaml (https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Snortconf_to_Suricatayaml#Snaplen) says it is always automatically at "full snaplen" for suricata but I think this only applies to Inline or if no MTU is detected using ioctl for pcap mode. This is where I got started looking at how it gets it and found out it adds 16 to the interface MTU. Basically it looks for the MTU of the interface and, if it cannot find one using ioctl, it automatically sets to 1518 or "full length" which would work. That being said, it CAN find the MTU of 1500 on most interfaces and so defaults to 1500+16. I see where they added the ability for you to configure it in pcap mode here: https://github.com/OISF/suricata/pull/287/commits/1feb84916c43666f02efe33d46debcf0d92834a0

As a side note, Snort should probably be defaulted to 1518 snaplen regardless in pfsense to mitigate any issues with VLAN users on the snort side (won't hurt anything but will definitely help. It will just make sure it works properly and, if they aren't using any VLANs, will just use slightly more resources but should be a negligible amount). You could then also make it configurable in the GUI in case someone has different needs or needs to adjust it (jumbo frames scenarios, etc.) as you mentioned.

On the suricata side, maybe the GUI should only give you the option to set snaplen for pcap mode and hide it otherwise with a note about the MTU and VLANs because I don't see it anywhere being able to do it in the suricata.yaml for Inline mode either and it might confuse those who don't know it won't work in that mode.

Here is a discussion from when security onion distro was talking about the issue and they describe it pretty well. In the end, if you look at github, they decided to go ahead and make it the default about 2 years ago to resolve this issue. You probably can't do what they did and force all interfaces to use MTU 1502 when switching suricata to inline mode but a warning in the GUI would probably suffice.

https://groups.google.com/forum/#!topic/security-onion/94s7beFDMU0

Thanks for always being so responsive on the forums.

We are on the same page … :D. I found those same links in my quick searching. Good tip about hiding the SNAPLEN setting when using Inline IPS Mode in Suricata. I will also add the parameter to Snort as well. It defaults to using the MTU when snap length is not specified. I'm working on some common updates to both packages (Snort and Suricata).

Bill

rmhco

@onyxfire:

Are you using VLAN interfaces?

I ask because I was not getting any alerts at all once I switched to inline mode but packets were clearly being dropped but not logged by suricata in the alerts.log and some sites even just plain refused to load at all. I have my routers setup in a "router-on-a-stick" type config where all traffic from internal networks comes in just the one physical interface but with separate VLAN interfaces tagged by 802.1.Q VLANs on the switch.

Suricata looks at your MTU on the interface it is running on and adds 16 for its "Bucket length" (like snaplen on snort but not configurable on suricata). This makes the "snaplen" of data for each packet 1516 if you haven't changed from the default MTU of 1500. This leaves out the VLAN information which makes it almost impossible for the "stream" processor in suricata to actually tell which packets are related to which streams on which VLAN. This can cause website issues, no alerts on pfsense GUI, etc… The fix is to change your MTU to 1502 which makes suricata again add 16 giving you a total of 1518 "snaplen" which will then be large enough to include the VLAN information.

To fix it on pfsense:

Create another interface for the physical (parent) interface to your VLAN interfaces that suricata is listening on if you don't already have a non-VLAN interface configured for it (i.e. igb1 or whatever yours is)

Make sure you enable this interface or all VLAN traffic will cease to flow to that interface. You do NOT have to assign it an IP address, just make sure the enabled box is checked.

Set its MTU to 1502 (this has to be done or you can't set the VLANs to an MTU of 1502. They are considered child interfaces and you can't set the child higher than the parent)

Go into each of your VLAN interfaces on that physical interface and set them to an MTU of 1502

Restart suricata

If you are not using VLANs, it may still be beneficial though I haven't tested it in that setup. That being said, security onion (popular linux security distro) has made it their default for a while now to prevent issues with their distro using suricata and snort. It should be a no impact change (with the exception of temporary outage when the interfaces are configured) due to the fact that any devices connecting will use Path MTU Discovery (PMTUD) to discover the appropriate packets size to send.

Other than that, you could try putting it in non-blocking mode again and see if when you visit google you get any alerts (if you haven't already confirmed that as it seems you have already done some investigation).

Thank you. Appreciate the response.

No VLAN's on this line (WAN) but I've adjusted the MTU per your suggestion.

igb0: flags=28943 <up,broadcast,running,promisc,simplex,multicast,ppromisc>metric 0 mtu 1516
	options=5400a8 <vlan_mtu,jumbo_mtu,vlan_hwcsum,vlan_hwtso,netmap,txcsum_ipv6>ether XX:XX:XX:XX:XX:X
	hwaddr XX:XX:XX:XX:XX:X
	inet6 XXX::XXXX:XXXX:XXXX:XXXX%igb0 prefixlen 64 scopeid 0x1 
	inet XXX.XXX.XXX.XXX netmask 0xffffff00 broadcast XXX.XXX.XXX.XXX
	 <snip>...</snip> 
	inet XXX.XXX.XXX.XXX netmask 0xffffff00 broadcast XXX.XXX.XXX.XXX
	nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active</full-duplex></performnud,auto_linklocal></vlan_mtu,jumbo_mtu,vlan_hwcsum,vlan_hwtso,netmap,txcsum_ipv6></up,broadcast,running,promisc,simplex,multicast,ppromisc>

I've also tried putting the interface in non-blocking mode* without luck. No alerts generated for traffic associated with this issue and HTTPS access to www.google.com was uninterrupted.

I did notice one thing new, after clearing out the tls.log, I realized certificates are not being recorded to the tls.log which reinforces my hypothesis that the issue occurs prior to the completion of the TLS handshake.

* Connected to www.google.com (172.217.12.68) port 443 (#0)
* Server aborted the SSL handshake
* Closing connection 0
curl: (35) Server aborted the SSL handshake

So for now it looks like I'm stuck with the TLS parser set to "detection-only", hopefully someone else can confirm this issue so I don't have to continue to question my own sanity.

*@bmeeks, unchecking the Block Offenders checkbox if inline mode was previously selected, results in [ERRCODE: SC_ERR_INITIALIZATION(45)] - No interface found in config for netmap, I had to switch back to legacy mode then uncheck the box.

bmeeks

@rmhco:

*@bmeeks, unchecking the Block Offenders checkbox if inline mode was previously selected, results in [ERRCODE: SC_ERR_INITIALIZATION(45)] - No interface found in config for netmap, I had to switch back to legacy mode then uncheck the box.

Thanks for the heads up on that issue. I'm working on a Suricata GUI package update now, so I will test this out and fix any logic issues. What it should do, when selecting no blocking, is switch over to PCAP mode and not use Netmap. Your error indicates the code may not be doing what I intended.

Bill

crept

I have a similar issue with a single HTTPS website. (for now)

Website will not load when Suricata is enabled on the WAN interface (Inline or Legacy).
Been running Suricata in Inline mode for over a year.
No Alerts are generated other than ET CINS Active Threat Intelligence Poor Reputation IP XX
(I've disabled these to see if it's a bad rep IP of the web host, but still will not load)
If I disable Block Offenders, website will load.

Other things to note:
Firefox gets stuck "Performing a TLS handshake to centennialplastics.com"
Chrome "ERR_TIMED_OUT"
tls.log shows:

04/13/2018-21:50:47.752526 47.147.85.134:49742 -> 72.9.159.115:443 TLS: Subject='CN=centennialplastics.com' Issuerdn='C=US, ST=TX, L=Houston, O=cPanel, Inc., CN=cPanel, Inc. Certification Authority' SHA1='bb:8e:b3:2d:e4:f8:b6:db:20:16:1e:0c:a1:bb:2b:50:94:12:9e:c6' SNI='centennialplastics.com' SERIAL='00:BB:D7:82:E3:F2:E9:B0:36:2C:97:38:EF:3D:9D:A6:92' VERSION='TLS 1.2' NOTBEFORE='2018-01-28T00:00:00' NOTAFTER='2018-04-28T23:59:59'

I'm hoping it's something I've missed. (No VLANS in this environment and tested MTU 1502 with no success)

NollipfSense

Good info Onyxfire!