Suricata Blocking WAN IP Address

Eboman

Hi folks,

This was partially covered in a previous posting….but the forum suggested I start a "new" topic, when I clicked to reply on the previous one (because of the age). No solution was provided in the last post, so here we go.......

Here's my issue. I have Suricata set to "block" on both LAN and WAN.

Recently, on a few random occasions, Suricata will block my WAN address. And it's actually blocking it....just just an Alert. I usually find out because all of the VOIP phones go dead and the family starts complaining "the internet is broke again!!" I have my VOIP provider set to text me if connection ever goes down, and it's been happening at very unusual times. The most recent was today and I was the only person home and not doing anything that would go through the router.

I've checked my Suricata.log file and under the WAN adapter, it shows this:

adding firewall interface em0 IPv4 address 70.x.x.x to automatic interface IP Pass List.

and LAN side….

adding firewall interface em0 IPv4 address 70.x.x.x to automatic interface IP Pass List

70.x.x.x is my WAN ip address given to me by DHCP from Cox Communications.

Here's a copy/paste of the most recent Alert (that subsequently generated the block).

03/17/2018
13:54:05 2 UDP Attempted Denial of Service 23.247.97.8
53932 70.161.x.x
123 1:2017919
ET DOS Possible NTP DDoS Inbound Frequent Un-Authed MON_LIST Requests IMPL 0x03

And sure enough…under the Blocks tab....there's my IP Address. As soon as I removed it from the "block" list, everything promptly went back to normal and I got a nice alert from my VOIP provider "service restored".

Got any ideas? It only recently started -- about 2 weeks ago. Prior to this, Suricata worked like a champ!! I'm stumped, other than to disable Suricata. But I certainly don't want to do that. Until now, I've been adding a ton of Suppress rules every time it's been getting blocked. I even went so far as to completely re-install pfSense and re-configure it. I did that last night, while the rest of the family were out of town. No issues all night long, until this one popped up today.

I did a Google search using my WAN IP Address and it doesn't appear to be on any "lists".

Got any ideas? I'm stumped!!

Thanks!
Eboman

Eboman

Just to add to the confusion….Suricata DID also block the outside IP address as well (earlier, it had both on there...but I took the block off my my WAN IP address already). Here's copy/paste from my current "block" list.

1 23.247.97.8
ET DOS Possible NTP DDoS Inbound Frequent Un-Authed MON_LIST Requests IMPL 0x03 - 03/17/2018-13:54:05

bmeeks

What was the time stamp on those log entries? How does the time stamp on those entries match up with the time of the alert and block? Does your WAN IP address change frequently?

As a side note, that set of rules has very limited usefullness on a Home Internet connection. In your case it is almost certainly a false positive. Unless you are running a public NTP server, then you don't need that rule at all.

Bill

bmeeks

Follow up test results –

I configured a test scenario in a VMware lab environment to test if the automatic Pass List feature that prevents blocking the WAN IP address of the firewall works. I could not reproduce your issue of blocking the WAN IP. I had a pfSense firewall running Suricata configured for Legacy Mode blocking with no passlist defined (meaning there was no automatic pass list being generated). Suricata was configured to block BOTH alert IP addresses (SRC and DST).

I then scanned the WAN IP of this virtual machine from a Kali Linux host in the same subnet as the WAN IP. I got immediate blocks on the IP address of the Kali Linux host, but no blocks on the WAN IP of the firewall being scanned. There were no blocks on the firewall being scanned because the automatic WAN IP pass list feature of Suricata prevented it.

So Suricata will not block your currently active WAN IP (at least when used in the default configuration). If it would allow the WAN IP to be blocked, then my test would have resulted in an immediate block of both the Kali Linux box IP and the WAN IP of my test firewall victim. I only got blocks on the Kali Linux host's IP address.

Bill

Eboman

I don't know…..
It just happened again.

My IP address does not change at all.

Here's the most recent one.

03/17/2018
21:36:14 3 UDP Generic Protocol Command Decode 60.191.29.20
27637 70.161.x.x
80 1:2200075
SURICATA UDPv4 invalid checksum
03/17/2018
21:33:49 3 UDP Generic Protocol Command Decode 60.191.29.20
27637 70.161.x.x
53 1:2200075
SURICATA UDPv4 invalid checksum

It just happened less than 10 minutes ago. Weird.

Next time, I'll take a snapshot of the "Block" page and post it up.

bmeeks – thanks for the advice. Times all jive. and IP address has been the same all day. I appreciate you running the test. You're correct -- it should NOT happen. I am using Legacy mode, by the way.

Is there some way to "manually" add my WAN IP address to Suricata to be on the Pass List?

I broke my machine! Damn!

Thanks again,
Eboman

Eboman

Hmmm…..I might be on to something.
I just went into the Pass List section and it was completely blank. No lists at all.

So I clicked "add", made sure I included all of the Auto-Generated Lists and I'll see what it does.

Now I have one list in my listnames...passlist_30503.

Fingers crossed.

Thanks again everyone. I'll let you know if this fixed it. Fun to learn something new.

Eboman

Eboman

Ehh…come to find out there IS a default PassList. And my WAN IP address was in it -- and it was enabled under Networks Suricata Should Inspect and Protect. It was not previously sent to "none"...it was on "default".

For kicks, I'm going to change it from Default to the new Pass List I created (which is identical to the default one).

More to come....

Thanks,
Eboman

bmeeks

You should not be getting those UDP checksum alerts. Go to SYSTEM > ADVANCED and then the NETWORKING tab in the pfSense menu. Scroll down to the Network Interfaces section and be sure the three checkboxes to disable Hardware Checksum Offloading, Hardware TCP Segmentation Offloading and Hardware Large Receive Offloading are checked. You want those three features disabled when using an IDS/IPS.

Even still, you should not be getting blocks from those alerts even when you receive them. Have you checked for a possible zombie Suricata process? Run this command from the CLI and be sure you see only a single instance of Suricata for each enabled interface –

ps -ax | grep suricata

If the output of that command shows any "identical lines", then kill the duplicate process or else reboot your firewall to start fresh.

Is your setup a standard LAN/WAN situation with pfSense doing all the routing and maybe just a cable modem on the WAN side? Is there anything unusual like maybe you are running in bridge mode or something?

Bill

Eboman

Hi bmeeks,

Yes. I have all of the three checkboxes "checked" to disable the checking (did that when I set up Suricata, based on all of the guides I read).

I actually JUST now did a reset of the machine I've got pfSense running to see what the command you asked me to run would show. Here it is:

64917 - Ss 0:20.52 /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/sur
72457 - Ss 0:19.25 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc/sur
98399 - Ss 0:26.08 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc/sur
99743 - S 0:00.00 sh -c ps -ax | grep suricata 2>&1
99923 - S 0:00.00 grep suricata

And yes, it's a simple setup. Cable modem into WAN…..only one LAN port. Both on Intel Cards.

Thanks again for helping me with this.

Eboman

Eboman

Weird.
Rebooted again and this is what I got:

64582 - Ss 0:22.10 /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/suri
67499 - Ss 0:22.15 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc/suri
94199 - S 0:00.00 sh -c ps -ax | grep suricata 2>&1
94789 - S 0:00.00 grep suricata

No blocks since I changed from default to the custom pass list though.

Eboman

Son of a gun.
It happened again. While I was in the shower. Nobody else home (not that it would make a difference).

Came out of the shower to a text message and no Netflix.

I attached a screen shot.

I ran the command again but got the same result.

1391 - S 0:00.00 sh -c ps -ax | grep suricata 2>&1
1580 - S 0:00.00 grep suricata
64582 - Ss 2:49.60 /usr/local/bin/suricata -i em0 -D -c /usr/local/etc/su
67499 - Ss 2:59.49 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc/su

So evidently, my "custom" pass list doesn't help any.

I'm starting to get bummed out.

Eboman

Suricata.jpg_thumb

Eboman

Going to disable "live-swap" of rules.
At this point…I'm going through every tab to see if I missed something.

Bill,
Do you have a link to any IPS testing websites I could try to use to test different settings? Basically, I want to manually generate an Alert, instead of waiting for one to "happen".
I used to have one bookmarked years ago...the first time I started playing with firewall stuff.

Thanks!
Eboman

Eboman

I think I "might" be on to something.

Here's the Suricata log that just now was generated (when I changed the setting, I manually stopped Suricata on the WAN Interface and restarted it:

18/3/2018 – 22:39:39 - <info>-- Threshold config parsed: 31 rule(s) found
18/3/2018 -- 22:39:39 - <info>-- 26443 signatures processed. 1190 are IP-only rules, 6677 are inspecting packet payload, 15956 inspect application layer, 100 are decoder event only
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface em0 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 – 22:39:49 - <info>-- alert-pf -> adding firewall interface em0 IPv4 address 70.161.x.x to automatic interface IP Pass List.</info>
18/3/2018 – 22:39:49 - <info>-- alert-pf -> adding firewall interface em1 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface em1 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf -> adding firewall interface em1.2 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 -- 22:39:49 - <info>-- alert-pf output device (regular) initialized: block.log
18/3/2018 -- 22:39:49 - <info>-- Pass List /usr/local/etc/suricata/suricata_28752_em0/passlist parsed: 14 IP addresses loaded.
18/3/2018 -- 22:39:49 - <info>-- Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses.
18/3/2018 -- 22:39:49 - <info>-- alert-pf output initialized, pf-table=snort2c block-ip=both kill-state=on block-drops-only=off
18/3/2018 -- 22:39:49 - <info>-- fast output device (regular) initialized: alerts.log
18/3/2018 -- 22:39:49 - <info>-- http-log output device (regular) initialized: http.log
18/3/2018 -- 22:39:49 - <info>-- Using 1 live device(s).
18/3/2018 -- 22:39:49 - <info>-- using interface em0
18/3/2018 -- 22:39:49 - <info>-- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
18/3/2018 -- 22:39:49 - <info>-- Found an MTU of 1500 for 'em0'
18/3/2018 -- 22:39:49 - <info>-- Set snaplen to 1524 for 'em0'
18/3/2018 -- 22:39:49 - <info>-- RunModeIdsPcapAutoFp initialised
18/3/2018 -- 22:39:49 - <notice>-- all 3 packet processing threads, 2 management threads initialized, engine started.
18/3/2018 -- 22:40:01 - <info>-- No packets with invalid checksum, assuming checksum offloading is NOT used

Here's the previous one…from earlier, when I rebooted the machine (notice something missing?):

18/3/2018 – 19:03:33 - <info>-- Threshold config parsed: 31 rule(s) found
18/3/2018 -- 19:03:33 - <info>-- 26588 signatures processed. 1190 are IP-only rules, 6703 are inspecting packet payload, 16086 inspect application layer, 100 are decoder event only
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface em0 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface em1 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface em1 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf -> adding firewall interface em1.2 IPv6 address xxx to automatic interface IP Pass List.
18/3/2018 -- 19:03:48 - <info>-- alert-pf output device (regular) initialized: block.log
18/3/2018 -- 19:03:48 - <info>-- Pass List /usr/local/etc/suricata/suricata_28752_em0/passlist parsed: 10 IP addresses loaded.
18/3/2018 -- 19:03:48 - <info>-- Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses.
18/3/2018 -- 19:03:48 - <info>-- alert-pf output initialized, pf-table=snort2c block-ip=both kill-state=on block-drops-only=off
18/3/2018 -- 19:03:48 - <info>-- fast output device (regular) initialized: alerts.log
18/3/2018 -- 19:03:48 - <info>-- http-log output device (regular) initialized: http.log
18/3/2018 -- 19:03:48 - <info>-- Using 1 live device(s).
18/3/2018 -- 19:03:48 - <info>-- using interface em0
18/3/2018 -- 19:03:48 - <info>-- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
18/3/2018 -- 19:03:48 - <info>-- Found an MTU of 1500 for 'em0'
18/3/2018 -- 19:03:48 - <info>-- Set snaplen to 1524 for 'em0'
18/3/2018 -- 19:03:48 - <info>-- RunModeIdsPcapAutoFp initialised
18/3/2018 -- 19:03:48 - <notice>-- all 3 packet processing threads, 2 management threads initialized, engine started.
18/3/2018 -- 19:05:27 - <info>-- No packets with invalid checksum, assuming checksum offloading is NOT used

I'm guessing that since it's already in the "pass list" the WAN IP address should not be getting blocked, unless the "pass list" automatically clears itself and regenerates with the new data every time the system is restarted.

At the moment, everything is working as it should. I've been generating Alerts, but it's not blocking the WAN IP address. I don't know if it's because I turned off the "live swap" option. I don't think so, since I haven't updated rules recently. I have a funny feeling it's because THIS time starting up, Suricata recognized my WAN IP address.

Many times, when I turn on the pfSense machine, I have to "reboot" my cable modem to get a WAN IP address. It's been like that forever (going back to 2 different ASUS routers….ALWAYS had to reboot the modem to get an WAN IP address after rebooting the Router). I assumed the Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses was to monitor for changes.

Bill,
Do you think I'm on to something here?

Thanks!
Eboman</info></notice></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></notice></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info>

bmeeks

Yes, this last scenario you posted would be the source of your WAN IP blocks. Suricata is starting up and not finding your WAN IP active. It does create a thread to monitor for IP address changes on all active firewall interfaces, so later, when your WAN IP comes up it should get added to the list automatically and you should see a log entry for it. Maybe that is not happening in your case, though.

I suggest putting your cable modem and pfSense box both on the same UPS, and then never turn either of them off. Powering your pfSense box up and down should not be necessary unless you are making a hardware change, or it needs to reboot to install an update. I have my system configured as described: both the cable modem and pfSense firewall are on the same UPS and they never get powered off. Always on.

Bill

Eboman

Fantastic!!!

Thanks again for all of the help Bill.

I had recently taken my pfSense build out of a little tower case and stuck it in a rack-mount case and got a nice cabinet to keep it all in (switches/patch panel/PDU/etc). Of course, every time I moved something, I had to power-down everything and do my little modem trick (which would explain why it only recently started blocking the WAN IP). However, it's FINALLY all together and all on one UPS for modem/pfSense (using apcupsd which is nice). SmartUPS is on the Windows 2016 Server/switches/etc.

Mystery solved. I thought I was going crazy! hahaha!! It was one of those "it can't possibly be happening…but IS happening" scenarios that we all love so much.

Thanks again,
Eboman