(SOLVED) Suricata Interfaces have to be manually Restarted

Raffi_

Hi @bmeeks,
I recently noticed a new problem in suricata 4.1.4_3. This may have been occuring before the update as well, I can't be sure. I'm able to start Suricata on both my LAN and OPT1 interfaces, both are /24 interfaces. After some time, it seems to stop on both interfaces and I have to manually restart it. When this happens, Suricata is still running according to the Status => Services page.

I have a bunch of errors in my suricata.log as seen below and it keeps going on with similar patterns. I'm not sure if these specific errors are the cause or not. The interface which is referred to below as em1 is my 4G LTE failover WAN which is never really used. That interface has an IP which changes frequently.

12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:42:42 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:05 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Warning> -- [ERRCODE: SC_WARN_UNCOMMON(230)] - alert-pf -> Firewall interface IP change notification thread received an invalid IP address via kernel routing message socket.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> Received notification of IP address change on interface em1.
12/6/2019 -- 23:43:10 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
12/6/2019 -- 23:43:10 - <Info> -- alert-pf -> added address blank to automatic firewall interface IP Pass List.
13/6/2019 -- 00:30:53 - <Notice> -- Signal Received.  Stopping engine.
13/6/2019 -- 00:30:53 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2 
13/6/2019 -- 00:30:53 - <Info> -- time elapsed 37603.703s
13/6/2019 -- 00:30:53 - <Info> -- (RX#01-igb1) Packets 3195608, bytes 1697670558
13/6/2019 -- 00:30:53 - <Info> -- (RX#01-igb1) Pcap Total:3195598 Recv:3195598 Drop:0 (0.0%).
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output inserted 0 IP address blocks
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output processed 3 alerts
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output inserted 0 IP address blocks
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output processed 3 alerts
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output inserted 0 IP address blocks
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output processed 3 alerts
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output inserted 0 IP address blocks
13/6/2019 -- 00:30:53 - <Info> -- alert-pf output processed 3 alerts
13/6/2019 -- 00:30:53 - <Info> -- Alerts: 0

One thing I notice is that the one line below seems to be occurring during some kind of update routine. At least the time stamp would lead me to believe that.
13/6/2019 -- 00:30:53 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2

Currently, I haven't manually restarted the interfaces because I read in one of your other posts that the logs will be overwritten. So let me know if there is anything else you need.

Thanks,
Raffi

bmeeks

I feel certain this is related to a bug I accidentally introduced when I fixed Redmine Issue #9031. I found the bug this morning and am finishing testing of the fix, then I will post for the pfSense team to review and merge.

The bug will let Suricata delete an active memory pointer under certain conditions. Once that happens, weird stuff will occur up to and including a Suricata process crash when that deleted pointer is referenced elsewhere in the code. This bug only impacts Legacy Mode blocking operation.

Raffi_

Thanks for the quick response Bill! No problem, hope that update does the trick. I read in other threads about issues with blocks not working and wasn't sure if it was related so I figured I would start a new thread. Thanks for being on top of it, great work.

Raffi

bmeeks

I feel bad because this bug is all on me. I did not completely read and understand all of the features of a built-in Suricata binary function I am using to store Pass List and firewall interface IP addresses. The feature is a Radix Tree that is used to store IP addresses for high speed lookups. A parameter I thought was optional is not always optional. That was the cause of the Redmine Issue #9031 bug. I fixed that one by passing in a pointer value instead of NULL for that parameter. I overlooked the little fact that the Radix Tree code would, when removing an IP from the tree, automatically free the memory pointed to by the parameter. The proper way to make the API call is to provide your own deletion function that the Radix Tree can call. Fixed that mistake.

Look for a new package hopefully a little later today.

Raffi_

Thanks for the explanation. A little over my head, but I understand it for the most part :)
It would be nice if there was a way to roll back a package in cases like this I guess. Don't beat yourself up over it. You're doing your best and being responsive to the issues. We all make mistakes, at least you're owning up to it and making it right very quickly.

bmeeks

The fix for this bug is posted for the pfSense team to review and merge. I've asked them to expedite this one, so keep checking for a new Suricata package to show up in PACKAGE MANAGER either later today or early tomorrow.

The pull request for the fix is here: https://github.com/pfsense/FreeBSD-ports/pull/652.

Raffi_

Awesome. You're the man Bill! Thanks again :)

Raffi_

Hi Bill, thanks for the quick update.
I uninstalled 4.1.4_3 and installed 4.1.4_4. It seemed ok for a while. After some time I saw the same tree null errors and I also noticed one of the alerts that came up did not get blocked. The alert prior to it did come up in the block log however so I wonder if these tree null errors have something to do with alerts not being blocked.

I also ran a force update to see if my theory on the update was right. As soon as I did that, Suricata stopped running on both interfaces and that [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2, line came up again.

Let me know if you need any more info.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Hi Bill, thanks for the quick update.
I uninstalled 4.1.4_3 and installed 4.1.4_4. It seemed ok for a while. After some time I saw the same tree null errors and I also noticed one of the alerts that came up did not get blocked. The alert prior to it did come up in the block log however so I wonder if these tree null errors have something to do with alerts not being blocked.

I also ran a force update to see if my theory on the update was right. As soon as I did that, Suricata stopped running on both interfaces and that [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2, line came up again.

Let me know if you need any more info.

This is really puzzling. The "tree is null" error should be impossible in the new binary. I'm wondering if you actually received the newest binary. It really sounds like you are still running the "unfixed" binary. Look at the file date and time for this file for me:

/usr/local/bin/suricata

It should yesterday's date. The values on my test machine are June 13, 2019 and the file size is 4340864 bytes.

kiokoman

sorry for the intrusion, but , this only apply to pfsense 2.4 ? because i don't see a 4.1.4_4 on my pfsense 2.5, the last available version is still 4.1.4_3

bmeeks

@kiokoman said in Suricata Interfaces have to be manually Restarted:

sorry for the intrusion, but , this only apply to pfsense 2.4 ? because i don't see a 4.1.4_4 on my pfsense 2.5, the last available version is still 4.1.4_3

The new version is supposed to be in both places. I'm wondering if something went weird with the package building and distribution in the 2.5-DEVEL tree. I have a 2.5-DEVEL virtual machine that is current and it also can't see the newest posted version of the Suricata package. However, if you browse the actual repository of posted packages the lastest version shows up there.

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Hi Bill, thanks for the quick update.
I uninstalled 4.1.4_3 and installed 4.1.4_4. It seemed ok for a while. After some time I saw the same tree null errors and I also noticed one of the alerts that came up did not get blocked. The alert prior to it did come up in the block log however so I wonder if these tree null errors have something to do with alerts not being blocked.

I also ran a force update to see if my theory on the update was right. As soon as I did that, Suricata stopped running on both interfaces and that [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2, line came up again.

Let me know if you need any more info.

This is really puzzling. The "tree is null" error should be impossible in the new binary. I'm wondering if you actually received the newest binary. It really sounds like you are still running the "unfixed" binary. Look at the file date and time for this file for me:
/usr/local/bin/suricata
It should yesterday's date. The values on my test machine are June 13, 2019 and the file size is 4340864 bytes.

Sorry for the trouble. Here is what mine looks like. It has the same date as yours but the size is a bit bigger.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

@bmeeks said in Suricata Interfaces have to be manually Restarted:
@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Hi Bill, thanks for the quick update.
I uninstalled 4.1.4_3 and installed 4.1.4_4. It seemed ok for a while. After some time I saw the same tree null errors and I also noticed one of the alerts that came up did not get blocked. The alert prior to it did come up in the block log however so I wonder if these tree null errors have something to do with alerts not being blocked.

I also ran a force update to see if my theory on the update was right. As soon as I did that, Suricata stopped running on both interfaces and that [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2, line came up again.

Let me know if you need any more info.

This is really puzzling. The "tree is null" error should be impossible in the new binary. I'm wondering if you actually received the newest binary. It really sounds like you are still running the "unfixed" binary. Look at the file date and time for this file for me:
/usr/local/bin/suricata
It should yesterday's date. The values on my test machine are June 13, 2019 and the file size is 4340864 bytes.
Sorry for the trouble. Here is what mine looks like. It has the same date as yours but the size is a bit bigger.

The larger size is puzzling. Is your hardware a Netgate appliance or do you run on a generic Intel AMD64 type machine?

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

@Raffi_ said in Suricata Interfaces have to be manually Restarted:
@bmeeks said in Suricata Interfaces have to be manually Restarted:
@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Hi Bill, thanks for the quick update.
I uninstalled 4.1.4_3 and installed 4.1.4_4. It seemed ok for a while. After some time I saw the same tree null errors and I also noticed one of the alerts that came up did not get blocked. The alert prior to it did come up in the block log however so I wonder if these tree null errors have something to do with alerts not being blocked.

I also ran a force update to see if my theory on the update was right. As soon as I did that, Suricata stopped running on both interfaces and that [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2, line came up again.

Let me know if you need any more info.

This is really puzzling. The "tree is null" error should be impossible in the new binary. I'm wondering if you actually received the newest binary. It really sounds like you are still running the "unfixed" binary. Look at the file date and time for this file for me:
/usr/local/bin/suricata
It should yesterday's date. The values on my test machine are June 13, 2019 and the file size is 4340864 bytes.
Sorry for the trouble. Here is what mine looks like. It has the same date as yours but the size is a bit bigger.
The larger size is puzzling. Is your hardware a Netgate appliance or do you run on a generic Intel AMD64 type machine?

I forgot to include the system info. It's not a netgate appliance, it's a custom build.
system specs.JPG

bmeeks

Let's compare MD5 hashes to see if your binary is the same as mine. Run this command from a shell prompt on the firewall:

md5 -q /usr/local/bin/suricata

The output should be:

cc5200e8369def9268b9e30c0c3f41c6

Let me know what you get.

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

Let's compare MD5 hashes to see if your binary is the same as mine. Run this command from a shell prompt on the firewall:
md5 -q /usr/local/bin/suricata
The output should be:
cc5200e8369def9268b9e30c0c3f41c6
Let me know what you get.

Here is what I got.

c962d5d995867c5baf3136035a34fac7

bmeeks

Okay. Thanks for the information. I've sent an email to my pfSense developer contact who works with me on the Suricata and Snort packages asking him to investigate. It's obvious the binary files are different. I would expect them to be the same.

I will wait to hear from him before taking any further action. The bug is fixed on my end so far as I can tell. I definitely saw the problem in the source code and fixed it there. So I'm wondering why the fix seems to be missing for users.

Raffi_

Thanks for the help Bill. I'll keep any eye out for an update on this. In the meantime, I'll try uninstalling and installing again just for the sake of being persistent. If anything changes on my end I'll let you know.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Thanks for the help Bill. I'll keep any eye out for an update on this. In the meantime, I'll try uninstalling and installing again just for the sake of being persistent. If anything changes on my end I'll let you know.

I doubt that will make any difference so long as it still downloads and installs the same binary. It really looks like to me that the binary built as suricata-4.1.4_2 is not "really" my new 4.1.4_2. It appears to still have the old version of my custom blocking plugin patch. That's the only way I can explain the "tree is null" error. I very explicitly fixed that error ... .

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Thanks for the help Bill. I'll keep any eye out for an update on this. In the meantime, I'll try uninstalling and installing again just for the sake of being persistent. If anything changes on my end I'll let you know.

I doubt that will make any difference so long as it still downloads and installs the same binary. It really looks like to me that the binary built as suricata-4.1.4_2 is not "really" my new 4.1.4_2. It appears to still have the old version of my custom blocking plugin patch. That's the only way I can explain the "tree is null" error. I very explicitly fixed that error ... .

Yup, uninstall and install of 4.1.4_4 gave me the same md5 results as before.

kiokoman

the update appeared to me now on my pf 2.5

[2.5.0-DEVELOPMENT][root@pfSense.localdomain]/root: md5 -q /usr/local/bin/suricata
cc5200e8369def9268b9e30c0c3f41c6

bose301s

I am seeing odd behavior, Suricata was running fine for awhile but when I got home from work around 6 I noticed the WAN instance had stopped, when I restarted it I lost all internet connectivity on all the devices on my network, once I stopped it everything immediately worked again, wondering if this is related to these bugs.

bmeeks

@bose301s said in Suricata Interfaces have to be manually Restarted:

I am seeing odd behavior, Suricata was running fine for awhile but when I got home from work around 6 I noticed the WAN instance had stopped, when I restarted it I lost all internet connectivity on all the devices on my network, once I stopped it everything immediately worked again, wondering if this is related to these bugs.

Yes, if you actually still have the non-patched binary running it will cause a crash. As I just posted in another thread, something unusual happened during the posting of the update and the new binary was actually missing my bug fix. You should get the MD5 hash I posted up above (and the same one user @kiokoman just posted about for pfSense-2.5.

bose301s

@bmeeks OK, I got the same md5 as Raffi, should I reinstall, I already tried that but obviously didn't work as I still get the wrong md5. I don't want to go to 2.5 devel, I am on 2.4.4 p3.

bmeeks

Just sit tight and try again tomorrow. I reported the issue to the pfSense developer who posts my Snort and Suricata packages to the repository. Since the new file showed up in pfSense-2.5 DEVEL a couple of hours ago, I suspect the correct file will get posted to 2.4.4 RELEASE pretty soon.

bmeeks

Is this problem with no blocking and/or having to manually restart Suricata still occurring? I have been unable to reproduce this, and I've tried in both a 2.4.4 RELEASE virtual machine and a 2.5 DEVEL virtual machine. The blocking module now correctly senses and registers firewall interface IP changes without generating a "tree is null" error and without those invalid kernel IP message errors.

The MD5 hash values for the Suricata binary are different for pfSense 2.4.4 versus 2.5. This is due to the differences in the compiling libraries, but both versions do contain my fix for the problems above.

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

Is this problem with no blocking and/or having to manually restart Suricata still occurring? I have been unable to reproduce this, and I've tried in both a 2.4.4 RELEASE virtual machine and a 2.5 DEVEL virtual machine. The blocking module now correctly senses and registers firewall interface IP changes without generating a "tree is null" error and without those invalid kernel IP message errors.

The MD5 hash values for the Suricata binary are different for pfSense 2.4.4 versus 2.5. This is due to the differences in the compiling libraries, but both versions do contain my fix for the problems above.

Hi @bmeeks, unfortunately the problem is still around for me. I tried uninstall and reinstall this morning to see if anything changed, but the md5 is still the incorrect one I posted before. The behavior is still the same as well. I even tried to un/reinstall just before posting this to be sure nothing has changed since earlier this morning. The md5 is still not correct for me.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

@bmeeks said in Suricata Interfaces have to be manually Restarted:

Is this problem with no blocking and/or having to manually restart Suricata still occurring? I have been unable to reproduce this, and I've tried in both a 2.4.4 RELEASE virtual machine and a 2.5 DEVEL virtual machine. The blocking module now correctly senses and registers firewall interface IP changes without generating a "tree is null" error and without those invalid kernel IP message errors.

The MD5 hash values for the Suricata binary are different for pfSense 2.4.4 versus 2.5. This is due to the differences in the compiling libraries, but both versions do contain my fix for the problems above.

Hi @bmeeks, unfortunately the problem is still around for me. I tried uninstall and reinstall this morning to see if anything changed, but the md5 is still the incorrect one I posted before. The behavior is still the same as well. I even tried to un/reinstall just before posting this to be sure nothing has changed since earlier this morning. The md5 is still not correct for me.

The MD5 hash you posted last Friday (the one ending in fac7) is the same one I have on my 2.4.4 virtual machine, and that's the version I can't see the problem with. So we need to determine what is different between your setup and my virtual machine.

For starters, is your WAN IP a PPPoE interface or is it a DHCP or static IP setup?

Raffi_

@bmeeks said in Suricata Interfaces have to be manually Restarted:

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

@bmeeks said in Suricata Interfaces have to be manually Restarted:

Is this problem with no blocking and/or having to manually restart Suricata still occurring? I have been unable to reproduce this, and I've tried in both a 2.4.4 RELEASE virtual machine and a 2.5 DEVEL virtual machine. The blocking module now correctly senses and registers firewall interface IP changes without generating a "tree is null" error and without those invalid kernel IP message errors.

The MD5 hash values for the Suricata binary are different for pfSense 2.4.4 versus 2.5. This is due to the differences in the compiling libraries, but both versions do contain my fix for the problems above.

Hi @bmeeks, unfortunately the problem is still around for me. I tried uninstall and reinstall this morning to see if anything changed, but the md5 is still the incorrect one I posted before. The behavior is still the same as well. I even tried to un/reinstall just before posting this to be sure nothing has changed since earlier this morning. The md5 is still not correct for me.

The MD5 hash you posted last Friday (the one ending in fac7) is the same one I have on my 2.4.4 virtual machine, and that's the version I can't see the problem with. So we need to determine what is different between your setup and my virtual machine.

For starters, is your WAN IP a PPPoE interface or is it a DHCP or static IP setup?

Ah ok, that's good to know.

My secondary WAN IP is configured via DHCP from my Netgear LB1120 4G LTE modem. This modem is configured in bridge mode. That's the one which changes periodically. It is setup as part of a gateway group for failover. Let me know if you need anything else.

Thanks,
Raffi

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

@bmeeks said in Suricata Interfaces have to be manually Restarted:

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

@bmeeks said in Suricata Interfaces have to be manually Restarted:

Is this problem with no blocking and/or having to manually restart Suricata still occurring? I have been unable to reproduce this, and I've tried in both a 2.4.4 RELEASE virtual machine and a 2.5 DEVEL virtual machine. The blocking module now correctly senses and registers firewall interface IP changes without generating a "tree is null" error and without those invalid kernel IP message errors.

The MD5 hash values for the Suricata binary are different for pfSense 2.4.4 versus 2.5. This is due to the differences in the compiling libraries, but both versions do contain my fix for the problems above.

Hi @bmeeks, unfortunately the problem is still around for me. I tried uninstall and reinstall this morning to see if anything changed, but the md5 is still the incorrect one I posted before. The behavior is still the same as well. I even tried to un/reinstall just before posting this to be sure nothing has changed since earlier this morning. The md5 is still not correct for me.

The MD5 hash you posted last Friday (the one ending in fac7) is the same one I have on my 2.4.4 virtual machine, and that's the version I can't see the problem with. So we need to determine what is different between your setup and my virtual machine.

For starters, is your WAN IP a PPPoE interface or is it a DHCP or static IP setup?

Ah ok, that's good to know.

My secondary WAN IP is configured via DHCP from my Netgear LB1120 4G LTE modem. This modem is configured in bridge mode. That's the one which changes periodically. It is setup as part of a gateway group for failover. Let me know if you need anything else.

Thanks,
Raffi

And this is an IPv4 address? I'm trying to understand why I can't see the issue in my testing. I simulate WAN IP changes by going to the firewall interface in pfSense and changing the IP. That triggers the same code as a DHCP change would your box, but it works in my test machine.

Raffi_

Yes, it's an IPV4 address. Are there any other logs that might help?

This isn't something a system reboot might fix? I like to avoid that, but I might give it a try if all else fails.

To clarify, I'm not sure if I might have two different problems at least according to the suricata error logs.
Problem 1, when my WAN2 IP changes, I get the tree null errors. I'm not sure how that error manifests itself in terms of software issues. Is that what prevented my alert from not being blocked?

Problem 2, when suricata performs a rules update, the status on both my LAN and OPT1 interface go from the green check to the red x and each has to be manually restarted. That was indicated in the log by the "[ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2"

In case it's relevant, here are my packages.

bose301s

Mine seems to be working now, will be keeping an eye on it.

digdug3

I have the same problem, Suricata is not restarting after an update on all the interfaces. Also tried reinstalling the package.
I can start the interfaces manually.

MD5: c962d5d995867c5baf3136035a34fac7

bmeeks

@digdug3 said in Suricata Interfaces have to be manually Restarted:

I have the same problem, Suricata is not restarting after an update on all the interfaces. Also tried reinstalling the package.
I can start the interfaces manually.
MD5: c962d5d995867c5baf3136035a34fac7

What messages do you see in the suricata.log file for the interfaces that do not restart? I must know what errors are printing to even know where to begin looking for the problem.

bmeeks

@Raffi_ said in Suricata Interfaces have to be manually Restarted:

Yes, it's an IPV4 address. Are there any other logs that might help?

This isn't something a system reboot might fix? I like to avoid that, but I might give it a try if all else fails.

To clarify, I'm not sure if I might have two different problems at least according to the suricata error logs.
Problem 1, when my WAN2 IP changes, I get the tree null errors. I'm not sure how that error manifests itself in terms of software issues. Is that what prevented my alert from not being blocked?

Problem 2, when suricata performs a rules update, the status on both my LAN and OPT1 interface go from the green check to the red x and each has to be manually restarted. That was indicated in the log by the "[ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2"

In case it's relevant, here are my packages.

Let's first make sure that you don't have a duplicate or zombie Suricata process running. Stop all Suricata instances via the GUI controls, and then open a shell session on the firewall and run this command:

ps -ax | grep suricata

You should see only a single line display (the grep command). If you see any other lines with suricata in them, then kill those processes and try restarting Suricata from the GUI.

Also, at any time in the past have you run a command to manually install Suricata from the shell prompt? Something like pkg install pfSense-pkg-suricata? Don't do that now, but if you ever did it in the past I found when testing a Snort change a while back that the old copy of the binary installed by the manual pkg install process hangs around even when you do a delete and reinstall from the GUI.

bmeeks

@bose301s said in Suricata Interfaces have to be manually Restarted:

Mine seems to be working now, will be keeping an eye on it.

Thank you for the feedback. Glad it seems to be working for you. As I mentioned in a previous post, I can't make it fail for me on either a 2.4.4. or a 2.5 pfSense machine.

digdug3

@bmeeks
Here is the log:

18/6/2019 -- 06:13:53 - <Notice> -- Signal Received.  Stopping engine.
18/6/2019 -- 06:13:53 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2 
18/6/2019 -- 06:13:53 - <Info> -- time elapsed 29316.584s
18/6/2019 -- 06:13:53 - <Info> -- (RX#01-vtnet1) Packets 433047, bytes 110221491
18/6/2019 -- 06:13:53 - <Info> -- (RX#01-vtnet1) Pcap Total:433049 Recv:433049 Drop:0 (0.0%).
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output inserted 252 IP address blocks
18/6/2019 -- 06:13:53 - <Info> -- alert-pf output processed 329 alerts
18/6/2019 -- 06:13:53 - <Info> -- Alerts: 0
18/6/2019 -- 06:13:53 - <Info> -- cleaning up signature grouping structure... complete
18/6/2019 -- 06:13:53 - <Notice> -- Stats for 'vtnet1':  pkts: 433047, drop: 0 (0.00%), invalid chksum: 0

And another one:

18/6/2019 -- 06:13:55 - <Notice> -- Signal Received.  Stopping engine.
18/6/2019 -- 06:13:55 - <Error> -- [ERRCODE: SC_ERR_PCAP_DISPATCH(20)] - error code -2 
18/6/2019 -- 06:13:55 - <Info> -- time elapsed 29297.555s
18/6/2019 -- 06:13:55 - <Info> -- (RX#01-vtnet0) Packets 19190012, bytes 10423765489
18/6/2019 -- 06:13:55 - <Info> -- (RX#01-vtnet0) Pcap Total:19194208 Recv:19189936 Drop:4272 (0.0%).
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output inserted 0 IP address blocks
18/6/2019 -- 06:13:55 - <Info> -- alert-pf output processed 2 alerts
18/6/2019 -- 06:13:55 - <Info> -- Alerts: 0
18/6/2019 -- 06:13:55 - <Info> -- cleaning up signature grouping structure... complete
18/6/2019 -- 06:13:55 - <Notice> -- Stats for 'vtnet0':  pkts: 19190012, drop: 4272 (0.02%), invalid chksum: 0

And here the manual start of that last interface:

18/6/2019 -- 08:28:37 - <Notice> -- This is Suricata version 4.1.4 RELEASE
18/6/2019 -- 08:28:37 - <Info> -- CPUs/cores online: 4
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for MPM
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for SPM
18/6/2019 -- 08:28:37 - <Info> -- HTTP memcap: 67108864
18/6/2019 -- 08:28:37 - <Notice> -- using flow hash instead of active packets
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
--- CUT ---
18/6/2019 -- 08:28:37 - <Info> -- alert-pf output device (regular) initialized: block.log
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Pass List /usr/local/etc/suricata/suricata_38383_vtnet0/passlist parsed: 31 IP addresses loaded.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Created firewall interface IP change monitor thread for auto-whitelisting of firewall interface IP addresses.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf -> Firewall interface IP address change notification monitoring thread started.
18/6/2019 -- 08:28:37 - <Info> -- alert-pf output initialized, pf-table=snort2c  block-ip=both  kill-state=on  block-drops-only=on
18/6/2019 -- 08:28:37 - <Info> -- fast output device (regular) initialized: alerts.log
18/6/2019 -- 08:28:37 - <Info> -- http-log output device (regular) initialized: http.log
18/6/2019 -- 08:28:37 - <Info> -- stats output device (regular) initialized: stats.log
18/6/2019 -- 08:28:37 - <Info> -- Syslog output initialized
18/6/2019 -- 08:28:37 - <Info> -- SSSE3 support not detected, disabling Hyperscan for SPM
18/6/2019 -- 08:28:37 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'sip_method'.
--- CUT ---
18/6/2019 -- 08:28:39 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert udp $EXTERNAL_NET any -> $SIP_SERVERS $SIP_PORTS (msg:"PROTOCOL-VOIP Known SIP scanner User-Agent detected"; flow:to_server; sip_header; content:"User-Agent: sip-scan"; fast_pattern:only; metadata:policy max-detect-ips drop, service sip; reference:url,blog.kolmisoft.com/sip-attack-friendly-scanner/; classtype:attempted-recon; sid:48309; rev:2;)" from file /usr/local/etc/suricata/suricata_38383_vtnet0/rules/suricata.rules at line 21407
18/6/2019 -- 08:28:39 - <Info> -- 2 rule files processed. 22211 rules successfully loaded, 203 rules failed
18/6/2019 -- 08:28:39 - <Warning> -- [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2000419, gid 1: unknown rule
--- CUT ---
18/6/2019 -- 08:28:39 - <Warning> -- [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2200004, gid 1: unknown rule
18/6/2019 -- 08:28:39 - <Info> -- Threshold config parsed: 16 rule(s) found
18/6/2019 -- 08:28:40 - <Info> -- 22223 signatures processed. 11 are IP-only rules, 8729 are inspecting packet payload, 15311 inspect application layer, 0 are decoder event only
18/6/2019 -- 08:28:40 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'ET.GenericPhish_Excel' is checked but not set. Checked in 2023046 and 0 other sigs
--- CUT ---
18/6/2019 -- 08:28:40 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.docm' is checked but not set. Checked in 43975 and 1 other sigs
18/6/2019 -- 08:28:44 - <Info> -- Using 1 live device(s).
18/6/2019 -- 08:28:44 - <Info> -- using interface vtnet0
18/6/2019 -- 08:28:44 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
18/6/2019 -- 08:28:44 - <Info> -- Set snaplen to 1518 for 'vtnet0'
18/6/2019 -- 08:28:44 - <Info> -- RunModeIdsPcapAutoFp initialised
18/6/2019 -- 08:28:44 - <Notice> -- all 5 packet processing threads, 2 management threads initialized, engine started.

Had to cut some "flowbit error messages" and "pass list" to keep the post short

bmeeks

@digdug3: I suspect you may have the same underlying problem as another user in a similar thread. He had multiple instances of Suricata running on the same interface, and they were running with memory resident copies of the old binary.

Those PCAP_DISPATCH errors look suspiciously like what I would expect if multiple Suricata instances were trying to execute on the same interface. PCAP might get confused. You can walk through the steps below to check for and kill any duplicate runaway processes, or you can try rebooting the firewall.

Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.
Open a shell prompt on the firewall and execute this command:

ps -ax | grep suricata

It should return only a single line showing the grep command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the single grep command.
Now execute this command at the shell prompt:

ls /usr/local/bin/suricata*

It should find no file by that name. If it does, then delete the file it finds.
Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.

digdug3

@bmeeks Thanks for the info. Indeed there were some old instances still running after package deletion. Had to kill them using

kill -9 PID

A normal kill did not work.

After package reinstalling all interface came up automatically (as expected).
For your info, the running processes after package deletion:

ps -ax | grep suricata

76850  -  INs      6:14.59 /usr/local/bin/suricata -i vtnet1 -D -c /usr/local/etc/suricata/suricata_22491_vtnet1/suricata.yaml --pidfile /var/run/suricata_vtnet122491.pid
77805  -  INs     17:18.04 /usr/local/bin/suricata -i vtnet0 -D -c /usr/local/etc/suricata/suricata_38383_vtnet0/suricata.yaml --pidfile /var/run/suricata_vtnet038383.pid
78845  -  INs     15:48.93 /usr/local/bin/suricata -i vtnet3 -D -c /usr/local/etc/suricata/suricata_21975_vtnet3/suricata.yaml --pidfile /var/run/suricata_vtnet321975.pid
79709  -  INs      5:29.14 /usr/local/bin/suricata -i vtnet0.10 -D -c /usr/local/etc/suricata/suricata_4336_vtnet0.10/suricata.yaml --pidfile /var/run/suricata_vtnet0.104336.pid
22120  0  S+       0:00.00 grep suricata

And the ls:

ls /usr/local/bin/suricata*
ls: No match.

I think this fixes the not starting of the interfaces, if not, I'll let you know. Thank you.

Raffi_

@bmeeks Success! Thanks a lot for all the help on this. I was having the exact same issue as @digdug3. Thanks to the info you posted about killing the processes that solved it.

And thank you @digdug3 for the hint on how to kill those processes. I was having the same issue as you with killing them.

FYI, I had a bunch of the processes running which had to be killed. I did not grab the output but it was something like 5 per interface. I think each time I uninstalled and reinstalled, it started new processes for each interface.

I can confirm with pretty high confidence that both of my issues are solved.

Previously when I forced an update, that would cause the issue of suricata not starting back up on the interfaces.
Now, when I do that there is no error in the log and suricata was still running on both interfaces.

The other tree null issues is gone as well. Previously I was able to reproduce that by rebooting the 4G LTE modem and when it got a new address, those errors would show up.
Now, there are no errors related to that.

I'll mark this as solved.

Thanks again!
Raffi