SG-5100 Firewall logs dissapearing

azdeltawye

Hello, I'm having a strange issue where my firewall logs are automatically being cleared.

Disk usage is at only 40% so storage shouldn't be the cause. I tried clearing all the logs anyway to see if that would help but did not. System and DHCP logs are accumulating normally as you would expect but the Firewall logs are clearing every couple hours or so. I even checked the logs in the Shell: clog /var/log/filter.log | filterparser.php
and it only had three entries for the past few days..

Over the past few months I have pruned my logging by turning off default deny blocking, etc.. such that I typically only get about 50 or so entries per day. Mostly reject packets from pfblocker and vpn pass rules..

My system is just running a small home network with 30 - 40 clients, 5 Vlans & 2 OpenVPN servers. I only have a couple 'high burden' packages installed: pfblockerng_devl & snort and other simple packages like openvpn_client, and avahi...

Speaking of packages, I did notice that this firewall logging anomaly seemed to coincide with the installation of the package 'softflowd'. I have since removed the softflowd package but that had no effect...

Any help would be appreciated..

Here's a screenshot of my System Information:

stephenw10

We're going to need an example of how you are checking, what you see and what you expect to see.
If you're really only seeing 50 entries a day in the firewall log (which is very low) that should maintain somewhere in the 100 days region even with the default log size.

Steve

azdeltawye

@stephenw10
Thanks for the reply.

I normally check the firewall logs through the GUI. I conducted a little experiment to show you whats going on:

1/24/21 11:15 - Checked firewall log and it was empty. So I enabled 'Log firewall default blocks' in Status/System Logs/Settings to harvest some noise.
1/24/21 11:30 - Disabled 'Log firewall default blocks' and checked logs; 449 firewall log entries.
1/24/21 12:00 - Checked firewall log: one pfblocker log added, 450 total
1/24/21 13:00 - Checked firewall log: only shows pfblock log, default blocks gone!
1/24/21 13:15 - Logged into VPN server via mobile device over LTE.
1/24/21 13:30 - Checked firewall log: VPN pass logged, pfblock log gone.

azdeltawye

So as you can see, the firewall log keeps automatically clearing itself. And, like I said before, all the other Systems logs do not get reset and continue to log events normally.

Looking at the System/General log, nothing seems to be out of the ordinary or indicate any kind of reset condition.

I Tried stopping and restarting the System Logger Daemon. We'll see if that helps...

stephenw10

Hmm, weird. Almost looks like what you might see with a very low log rotation timer set. Except you can't set that in 2.4.5p1, that's a new feature in 2.5.
Check the actual log file if it;s still happening: /var/log/filter.log
See if it's actually removeing the log data or just filtering the GUI view.

Steve

azdeltawye

@stephenw10
Yes, strange indeed...

Yeah checking the log file via Shell only shows the last several events generated within the last couple of hours. Currently nothing in the log with a timestamp before Jan 25 08:43:06.

stephenw10

Hmm, the log files in 2.4.5p1 are circular with a fixed size, the default being 512K. That is usually ~4000 log lines. It's not rotated based on time/date.

Do you have any other packages that may be interfering with that?

Steve

azdeltawye

@stephenw10 said in SG-5100 Firewall logs dissapearing:

Do you have any other packages that may be interfering with that?

Steve

I wouldn't even know how to begin to answer that question... I'm pretty new to pfsense...

However, as I mentioned earlier, this problem seemed to start after I installed the 'softflowd' package. My intention was to send NetFlow data to a separate server running ntop. I was unable to get this working so I aborted the effort and removed the package..

Here are all the packages I have installed:

stephenw10

Mmm, I can't really imagine any package the might either.

So to be clear the actual log file is only a few bytes in size?

As I say it should be 512KB. That could be the timespan you're seeing if it was just the gui filtering logs.

Steve

jimp

If the entries cannot be parsed, they are not shown/counted, so it's possible there is data in the log file but it's not what the system expected.

What is in the log file at the time? (clog /var/log/filter.log)

stephenw10

I guess something could be filling the log file with unreadable data that clog doesn't show. The filesize should still be 512K though if that was the case.

azdeltawye

Thanks for the help guys. Really appreciate it!!

So it appears the log file is not empty at all:

-rw-------   1 root  wheel  511488 Jan 26 12:07 filter.log

This is confirmed with command: clog /var/log/filter.log

This is the first 7 lines of the file which has 3347 lines.

22,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:11 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:14 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:18 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:22 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:24 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,
Jan 26 09:48:25 pfSense filterlog: 64,,,11000,igb0,match,block,in,6,0xe0,0x00000,1,Options,0,136,fe80::29e:1eff:fe59:822,ff02::16,HBH,RTALERT,0x0000,PADN,

However when I try this command: clog /var/log/filter.log | filterparser.php

It only shows one line, which is the same as the GUI:

Jan 26 11:54:59 block igb0 UDP 80.82.65.90:45735 73.24.XXX.XXX:123

stephenw10

Mmm, that's fun. It is just the logs rotating pull out the old log lines at least.

So multicast IPv6. Is igb0 your WAN?

Do you have 'Log packets blocked by 'Block Bogon Networks' rules' unset in the log settings?

Steve

azdeltawye

@stephenw10
Yes, igb0 is the WAN.

No, Block Bogons is checked.

Also, I had 'GUI Log Entries' set to 2000 originally but changed to 500. Didn't seem to help..

serbus

Hello!

Your filter.log lines dont look like mine...the RTALERT and PADN looks odd...

Found this in the way-back-machine...

https://forum.netgate.com/topic/102426/my-firewall-log-is-getting-trimmed

If those lines are being ignored and you are getting a lot of them, they could be flushing out the "valid" lines.

Maybe try bumping up your log file size in Status -> System Logs -> Settings and see if you are able to retain more of the important log lines.

John

jimp

That is almost certainly the case. The log parser is dropping/ignoring those lines, and there aren't many/any others left to show.

Find whatever is causing those log messages and fix it, and then the log will be more useful.

stephenw10

Yes, though that may not be possible if it's something WAN side.

You could probably also add a custom block rule on WAN to catch that and not log it so it doesn't fill the log.

Steve

azdeltawye

Well I think that was it!

I disabled 'Log packets blocked by Block Bogon Networks rules' at 14:05 today. I just checked the filter log file and the last RTALERT and PADN entry occurred exactly at 14:06:01. Nothing but valid firewall events after that... Up until that point it was logging about 230 of those offending messages per hour.

The funny thing is, I've always had that Bogon logging option enabled and never had a problem until now.. My ISP is Comcast and like the mention in bug report #3494, Comcast appears to send ICMP6 Multicast Listener Report messages out on their system which get flagged as Bogon traffic by pfSense. I guess Comcast must have made some changes recently that increased the flow of this type of traffic...

Anyway, glad we got to the bottom of it. Thanks again for all the help! No way I could have figured this out on my own...