Tons sshguard log entries and its not enabled

geovaneg

Yes:

[2.6.0-RELEASE][admin@pfsense]/var/log: ls -tal | grep filter
-rw------- 1 root wheel 585448 Jul 8 12:48 filter.log
-rw------- 1 root wheel 110227 Jul 8 12:47 filter.log.0.bz2
-rw------- 1 root wheel 77990 Jul 8 12:44 filter.log.1.bz2
-rw------- 1 root wheel 101983 Jul 8 12:42 filter.log.2.bz2
-rw------- 1 root wheel 78511 Jul 8 12:39 filter.log.3.bz2
-rw------- 1 root wheel 88127 Jul 8 12:37 filter.log.4.bz2
-rw------- 1 root wheel 109005 Jul 8 12:35 filter.log.5.bz2
-rw------- 1 root wheel 82652 Jul 8 12:32 filter.log.6.bz2

johnpoz

@geovaneg your going to need to limit how much your logging, or go with a much larger size log file.. maybe 10M vs 1, or even like 100M? That seems like a lot of logging ;)

geovaneg

thank you to reply @johnpoz

I found it strange that for some time after I change the settings (about a day) the problem stopped but returns later. Also, in previous versions the amount of logs was the same and the problem did not occur.

Do you have any idea why this change in behavior?

stephenw10

The logs switched from the circular log format back to the default BSD format. That means they have to be rotated and sshguard restarts whenever any log is rotated.
https://docs.netgate.com/pfsense/en/latest/releases/2-5-0.html?highlight=clog#logging

Steve

geovaneg

@stephenw10 said in Tons sshguard log entries and its not enabled:

sshguard

Thank you @stephenw10

I understood.

But the previous version of this server was already 2.5.1 and it didn't have the excess of sshguard logs, with the same volume of logs and I imagine with the same rotation frequency of them.

This is what struck me as strange.

johnpoz

@geovaneg can you not limit the stuff your logging in some way? And also up the log file size?

So at a 2 min rotation for 1M and only keeping what 6 rotations - the amount of logs is only the last 12 minutes anyway - how is that even useful?

edit: my math off, with current log and 6 rotated (0-6 = 7) at most you would have like 16 minutes ;)

edit2: none of what is being logged is noise that you could just not log? I would think you would want atleast 24 hours of logs? Mine go back like 3 days and only storing 5 rotations. Are you also sending to say syslog as well?

stephenw10

Yeah you should be using an external syslog server there.

In 2.5.1 the same applies so it can really only have been that the logs were not rotating. Eother the logs were far larger or there was far less to log.

Steve

geovaneg

@stephenw10

Hi Guys.

Yes, logs are sent to a remote syslog server.
This PfSense server serves a large wireless network and I need to retain the fw logs for possible future audits.
I still have 4 other VPN servers that I haven't updated to version 2.6.0. They are in 2.5.2. I must face the same problem with them as they produce a lot of logs.

I ran a test just now:
I manually reset the syslog, via graphical console: Status->Services->Syslogd<Restart Service>

Jul 8 15:21:59 pfsense syslogd: exiting on signal 15
Jul 8 15:21:59 pfsense syslogd: kernel boot file is /boot/kernel/kernel

We can see that the files keep rotating:

[2.6.0-RELEASE][admin@pfsense]/var/log: ls -tal | grep filter
-rw------- 1 root wheel 940974 Jul 8 15:35 filter.log
-rw------- 1 root wheel 116009 Jul 8 15:34 filter.log.0.bz2
-rw------- 1 root wheel 127082 Jul 8 15:32 filter.log.1.bz2
-rw------- 1 root wheel 114071 Jul 8 15:30 filter.log.2.bz2
-rw------- 1 root wheel 92094 Jul 8 15:28 filter.log.3.bz2
-rw------- 1 root wheel 101016 Jul 8 15:26 filter.log.4.bz2
-rw------- 1 root wheel 93269 Jul 8 15:24 filter.log.5.bz2
-rw------- 1 root wheel 103993 Jul 8 15:22 filter.log.6.bz2

However, there are no more "sshguard" messages in the system logs until I forced an event into the log (logout and login to the web interface), which seems to have "waked up" the problem again.:

Jul 8 15:18:00 sshguard 25670 Exiting on signal.
Jul 8 15:18:00 sshguard 10773 Now monitoring attacks.
Jul 8 15:20:00 sshguard 10773 Exiting on signal.
Jul 8 15:20:00 sshguard 95347 Now monitoring attacks.
Jul 8 15:21:59 syslogd exiting on signal 15
Jul 8 15:21:59 syslogd kernel boot file is /boot/kernel/kernel
Jul 8 15:27:45 nginx 2022/07/08 15:27:45 [error] 2540#100160: send() failed (54: Connection reset by peer)
Jul 8 15:40:28 php-fpm 89546 /index.php: User logged out for user 'admin' from: remote-machine (Local Database)
Jul 8 15:40:31 php-fpm 89546 /index.php: Successful login for user 'admin' from: remote-machine (Local Database)
Jul 8 15:40:31 sshguard 13794 Now monitoring attacks.
Jul 8 15:42:00 sshguard 13794 Exiting on signal.
Jul 8 15:42:00 sshguard 93750 Now monitoring attacks.
Jul 8 15:44:00 sshguard 93750 Exiting on signal.

Could you help me understand why manual reset interrupts sshguard log records for a while, even though there is rotation?

Thanks

stephenw10

Hmm, that looks more a bug in sshguard that stops it running....

stephenw10

Yeah, the processes should look like:

root    77695   0.0  0.1  11384  2824  -  Ss   13:03     0:00.24 |-- /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.p
root    63461   0.0  0.1  11524  3048  -  Is   20:41     0:00.00 | `-- /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root    63916   0.0  0.1  10620  2224  -  I    20:41     0:00.00 |   |-- /bin/cat
root    63956   0.0  0.2  17420  4420  -  IC   20:41     0:00.00 |   |-- /usr/local/libexec/sshg-parser
root    64030   0.0  0.1  12072  2796  -  IC   20:41     0:00.00 |   |-- /usr/local/libexec/sshg-blocker
root    64102   0.0  0.2  11524  3052  -  I    20:41     0:00.00 |   `-- /bin/sh /usr/local/sbin/sshguard -i /var/run/sshguard.pid
root    64566   0.0  0.1  11492  3044  -  I    20:41     0:00.00 |     `-- /bin/sh /usr/local/libexec/sshg-fw-pf

But after restarting syslogd manually:

root    38082   0.0  0.1  11372  2820  -  Ss   20:44     0:00.00 |-- /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf

And logging out and back in restarts it:

Jul 8 20:49:47 	sshd 	77257 	Received disconnect from 172.21.16.5 port 58362:11: disconnected by user
Jul 8 20:49:47 	sshd 	77257 	Disconnected from user admin 172.21.16.5 port 58362
Jul 8 20:49:47 	sshguard 	26522 	Now monitoring attacks.
Jul 8 20:49:52 	sshd 	27315 	Accepted keyboard-interactive/pam for admin from 172.21.16.5 port 58364 ssh2

Hmm

stephenw10

Ok, it's not really a bug because the process is restarted as soon as there is a login attempt. And it still locks you out after three bad attempts.

johnpoz

@stephenw10 you could even call it a feature ;)

Why run a process if there is nothing to do.. Until there is a login attempt why monitor a log for something that isn't happening ;)

stephenw10

Mmm, but that begs the question, why run it continually at all?

geovaneg

@johnpoz

Following this line of reasoning, I wonder: is it really necessary for sshguard to keep recording in the system log each time it restarts along with rotate, if it is really protecting the ssh service?
Is there another unwanted side effect of the rotate occurring every 2 minutes? It wouldn't matter much to me, as long as it doesn't fill up my filesystem, as the logs are sent remotely.
Isn't there a way to disable sshguard restart log in the system logs?

geovaneg

@stephenw10 said in Tons sshguard log entries and its not enabled:

Mmm, but that begs the question, why run it continually at all?

exactly!

stephenw10

In some systems the CPU required to compress the logs when rotates can become significant. It's worth disabling the compression if that's the case.

johnpoz

@stephenw10 said in Tons sshguard log entries and its not enabled:

It's worth disabling the compression if that's the case.

Yeah I would turn that off, especially if your only going to keep 6 logs, and they at most could be 1M..

Also if your on zfs believe its recommended not to compress them because zfs is already compressing, etc.

Now if you were say logging files were set for say 100M and storing lots of them rotated, and you were tight on space compression might make sense.

If your logging remotely - would it be possible to just disable local logging? There is an option for that I believe.

geovaneg

@johnpoz @stephenw10

Good morning guys.

I've never had any CPU problems with this machine. It is a VM hosted on VMWare with good hardware features.

My file system is UFS and I don't intend to change that now.

I don't know if there is an option to disable logs locally. Even if there is, I have no intention of doing that, because we usually do a daily overview of the system logs, via the web console, to assess the latest events and the overall health of the system. Remote logs are primarily intended for audits.

Talking to you, I decided to implement some of your suggestions:

I kept disabled the log packets matched from the default block rules in the ruleset to reduce the amount of system logs. It is worth remembering that these logs were enabled in version 2.5.1 and I didn't experience any issues;
I increased the size of the log files to 10 MB;
Even though I didn't have any CPU problems, I disabled file compression. CPU is never too much ;
As the local logs don't interest me much, I reduced the retention to only 4 files in the log rotation.

Result:

All this is giving me less than an hour of peace, without the "sshguard" generating restart logs.
The main drawback of this is that it can fill the system logs page on the web interface with this filth and distract me from some really important event during my daily inspections.

-rw------- 1 root wheel 5554233 Jul 11 09:53 filter.log
-rw------- 1 root wheel 10558634 Jul 11 09:43 filter.log.0
-rw------- 1 root wheel 10289157 Jul 11 09:19 filter.log.1
-rw------- 1 root wheel 10304545 Jul 11 08:47 filter.log.2
-rw------- 1 root wheel 10279009 Jul 11 07:40 filter.log.3

Thank you for your suggestions and if you have any more they are welcome.

Thanks

Geovane

johnpoz

@geovaneg said in Tons sshguard log entries and its not enabled:

because we usually do a daily overview of the system logs,

You mean a last 16 minutes or so overview ;) Well your other logs could be longer I guess.. Just the one that cycles ever 2 minutes is pretty useless..

geovaneg

Good Morning @johnpoz !

Firewall logs are not mixed with system logs in the web interface.