everyday at 6am suricata crashes

Euman

Suricata will crash at 6am everyday, here are all the details I have and /var/crash is empty so no dump files.

when maxmind geolite2 database updates then this crash occurs.

Jun 21 06:00:03 kernel pid 82689 (suricata), jid 0, uid 0: exited on signal 10 (core dumped)
Jun 21 06:00:02 php-cgi 42554 [Suricata] Cleaning up temp files after GeoLite2-Country database update.
Jun 21 06:00:02 php-cgi 42554 [Suricata] GeoLite2-Country database update completed.
Jun 21 06:00:02 php-cgi 42554 [Suricata] Moving new database to /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb...
Jun 21 06:00:02 php-cgi 42554 [Suricata] Extracting new GeoLite2-Country database from the archive...
Jun 21 06:00:02 php-cgi 42554 [Suricata] New GeoLite2-Country IP database gzip archive successfully downloaded.
Jun 21 06:00:01 php-cgi 42554 [Suricata] Downloading new GeoLite2-Country IP database...
Jun 21 06:00:01 php-cgi 42554 [Suricata] A new GeoLite2-Country IP database is available.
Jun 21 06:00:01 php-cgi 42554 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...

it's become hugely frustrating, any help would be very appreciated.

my system detail:
Version 23.05-RELEASE (amd64)
built on Mon May 22 15:04:36 UTC 2023
FreeBSD 14.0-CURRENT
official Netgate 6100 appliance

Euman

This post is deleted!

pst

@Euman check if you have a file /root/suricata.core

The system puts program core dumps in /root/ by default (on my system anyway)

Euman

hello, @pst thank you for your responce.. /root/suricata.core && /root/php-fpm.core are there but the files are empty

pst

@Euman what is the output of "sysctl kern.coredump"? if it is 1 then core dumps should be created. Other things to check:

"ulimit -c" shows max size of core dumps
is there enough disk space to write the core dump?
if the time stamp on suricata.core is not matching the expeced time of the last crash check if the system has put the file elsewhere: find / -name suricata.core

that's all I can think of at the moment.

Euman

@pst Thank you for your response and here are the answers to your questions

sysctl kern.coredump
kern.coredump: 1

ulimit -c
unlimited

disk space
Mount Used Size Usage
/ 1.7G 441G 0% of 441G (zfs)
/cf 6.8M 439G 0% of 439G (zfs)
/eepf 96K 439G 0% of 439G (zfs)
/home 120K 439G 0% of 439G (zfs)
/tmp 456K 3.4G 0% of 3.4G (tmpfs)
/var 100M 3.4G 3% of 3.4G (tmpfs)
/var/cache/pkg 225M 439G 0% of 439G (zfs)
/var/db/pkg 7.2M 439G

find / -name suricata.core
/root/suricata.core

pst

@Euman that looks good to me. I don't have any other ideas at the moment. Hopefully someone else can chip in.

Euman

@pst You've been a great help and in this process I have discovered what appear to be other issues

PHP ERROR: Type: 1, File: /usr/local/www/diag_edit.php, Line: 55, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 648482848 bytes)
which may be the reason the core file is empty but that is above my pay grade.

also to note: the only way to get Suricata working again w/o a system reboot is to copy the interface setup and start it then remove the old one. None of the buttons to shutdown or restart Suricata service bring it back online.

Thank you again.

NogBadTheBad

@Euman if there isn't enough space for the core dump you could create a symbolic link to /dev/null

[23.05-RELEASE][admin@pfsense]/root: ls -alg
total 64
drwxr-xr-x   5 root  wheel   512 Jun 21 16:47 .
drwxr-xr-x  24 root  wheel  1024 Jun 11 22:59 ..
drwx------   2 root  wheel   512 Feb 17  2021 .cache
-rw-r--r--   2 root  wheel  1023 May 22 16:57 .cshrc
-rw-r--r--   1 root  wheel     0 Jun 11 22:59 .hushlogin
-rw-r--r--   1 root  wheel    80 May 22 16:57 .k5login
-rw-------   1 root  wheel    82 May 24 10:23 .lesshst
-rw-r--r--   1 root  wheel   328 May 22 16:57 .login
-rw-------   1 root  wheel  1848 Dec 27  2020 .lsof_pfsense
-rw-r--r--   2 root  wheel  1140 Jun 11 22:59 .profile
-rw-------   1 root  wheel  1024 Apr 10 16:57 .rnd
-rw-------   1 root  wheel    68 Jun 11 22:57 .sh_history
-rw-r--r--   1 root  wheel  2090 Jun 11 22:59 .shrc
drwx------   2 root  wheel   512 May  4 08:40 .ssh
-rw-r--r--   1 root  wheel  3348 Jun 11 22:59 .tcshrc
lrwxr-xr-x   1 root  wheel     9 Nov 24  2022 ntopng.core -> /dev/null
-rw-r--r--   1 root  wheel   512 Mar 10 13:11 packetcapture.cap
-rw-r--r--   1 root  wheel     0 Mar 10 13:11 packetcapture.start
drwxr-xr-x   2 root  wheel   512 Sep 13  2021 scripts
lrwxr-xr-x   1 root  wheel     9 Nov 24  2022 snort.core -> /dev/null
[23.05-RELEASE][admin@pfsense]/root: 

ln -s /dev/null ./suricata.core << should do it

Euman

@NogBadTheBad

Here is the official Crash reporter diagnostics:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/sources/FreeBS

Crash report details:

PHP Errors:
[21-Jun-2023 08:02:05 US/Pacific] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 648482848 bytes) in /usr/local/www/diag_edit.php on line 55
[21-Jun-2023 08:02:36 US/Pacific] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 648482848 bytes) in /usr/local/www/diag_edit.php on line 55
[21-Jun-2023 08:07:14 US/Pacific] PHP Fatal error: PHP Request Shutdown: Cannot use output buffering in output buffering display handlers in Unknown on line 0
[21-Jun-2023 08:13:09 US/Pacific] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 648482848 bytes) in /usr/local/www/diag_edit.php on line 55

No FreeBSD crash data found.

These showed up when I tried viewing the core dumps in browser gui, I will hop over to a console and pull from there if possible. currently away from console and will take a moment.
Can I ask why I would need to create symbolic links and tbh, I've always used /dev/null as a blackhole/trash repository

Euman

this should be separate issues.. Suricata's crash and the GUI PHP issues

bmeeks

Your PHP errors stem from trying to load a file that is too big to fit into the finite amount of RAM the PHP process reserves. That is a "normal" error message when attempting to load and view very large files. You can't load up and view a core dump file in the GUI. You will need to export the file off the firewall and load it into another editor (or post its contents back here).

Your Suricata problem is not related to that PHP error at all. A Signal 10 error is a BUS FAULT. That was a common error on 32-bit ARM hardware, but should almost never happen on Intel/AMD64 hardware. Your system detail says you have an SG-6100.

The GeoLite2 database update appears to be completing as the last log message shows the temp subdirectory used during that process being cleaned up.

Euman

I'm going on the record here. I had Suricata use Wan interface because I have this stupid AT&T internet and the gateway had to be setup with Use non-local gateway

Use non-local gateway through interface specific route. This will allow use of a gateway outside of this interface's subnet.
This is usually indicative of a configuration error, but is required for some scenarios.

I have a block of IP's and the Wan I setup is one of those in the block of ip's I purchased from AT&T
Wan is supposed to block by default inbound activity so Herein lies my mistake.

I have just now shifted from using Suricata on Wan to the 4 LAN's my Netgate 6100 has. I do hope this doesn't prove to be wrong and fixes this particular issue.

Euman

Thank you all for your help and apologies for my ignorance and making an issue where misconfiguration was the issue.

bmeeks

@Euman said in everyday at 6am suricata crashes:

Thank you all for your help and apologies for my ignorance and making an issue where misconfiguration was the issue.

The Signal 10 Bus Error is very unusual. I would essentially expect to never see that error on Intel/AMD hardware, no matter what interface you run Suricata on.

That error usually means the running code attempted to access memory on a non-word aligned boundary. But Intel hardware always automatically fixes up such access by converting unaligned memory accesses into a series of sequential reads followed by bit shifting to properly align the resulting read data.

Euman

@bmeeks How can I help debug this further? I'm a novice programmer (in advance) and your msg read like the titanic is sinking.

Euman

suricata.core is definitely 633276.00 kb.. I will download and gather any detail

bmeeks

@Euman said in everyday at 6am suricata crashes:

@bmeeks How can I help debug this further? I'm a novice programmer (in advance) and your msg read like the titanic is sinking.

Not easily debugged without compiling your own binary will full debugging enabled. That really can't be done unless you create a pfSense package builder.

There can be a few other things that generate a Signal 10 Bus Error, but non-aligned memory access is the most common. It's also possible some obscure hardware fault has occurred that some piece of Suricata code is tickling just right to trigger the problem.

While I don't know precisely how many SG-6100 users are out there running Suricata with the GeoIPLite option enabled, if there was a widespread problem I would expect to have seen at least a few other similar posts. Not seeing any yet from other users would be a trend favoring a potential hardware issue of some type in your setup.

bmeeks

@Euman said in everyday at 6am suricata crashes:

suricata.core is definitely 633276.00 kb.. I will download and gather any detail

That file will be a binary memory image you would load into the GDB debugger package (which you would need to install separately). But since the production pfSense packages are compiled WITHOUT debugging info or symbols, the utility of the core dump can be limited.

Euman

apparently the "save button" for the dialog window that I had opened for the file (0) zero'd the file contents, I lost it.. sighs heavily!

Old people mice clicking should be denied..