SG1100 - Disk Full - Help
-
Hello, everyone.
Today one of our SG1100 reaches 108% of the disk and stops.
We have syslogng installed sending logs to a remote SOC.
I changed it to a new one SG1100 and restore the old to the factory default.
The factory default didn´t cleaned up the disk, how do i wipe the disk entirely?Thank you.
-
You can re-install clean to wipe the drive. Open a ticket with us to get the recovery image if you need it: https://go.netgate.com/
But it will happen again if you have something configured to use space with no limit. Probably syslog-ng.
Steve
-
@stephenw10 Hello Stephen, you are right, the cause was Syslog-ng it created a file with 7.8GB
even with Max Archives 2 set-up. -
That doesn't set a size limit though. You can only set the log rotation frequency. If you are logging that much with daily rotation you should definitely be using a dedicated syslog server.
Steve
-
@stephenw10 Ok, sorry for boring you but how do i set a file size limit in syslog, this will be very usefull here, right now one of them (we have 7 SG1100) is at 60% of disk.
Thank You. -
@adrianoebm said in SG1100 - Disk Full - Help:
@stephenw10 Ok, sorry for boring you but how do i set a file size limit in syslog
Here :
First, get the size of the space left of your drive.
With the "Log rotate size (bytes)" you can set a file size. The default 500 Kbytes is a good start.
Check yourself and have a look at the flog files here : /var/log/
With the "log rotation count" you select how many (compressed) files are kept on the disk.
Again : always keep an eye on the remain partition space.
You installed pfSense packages that tend to eat up all space ? O, but now double de guard.
Very Frequently !!
Or script it and do something like this and be warned when, fo example, 20 % or less is left : this is a graph of the used disk space of my pfSense @work. -
That's the normal system logging not the logs from the syslog-ng package which I believe was the problem here. There is no gui setting for the log file size there.
Steve
-
@stephenw10 you are right, i didn't find this option under syslog-ng settings and i can't figure out why our files are so huge.
Answering your question we have 5 ipsecs, one openvpn, zabbix package, syslog-ng and watchdog. -
@gertjan nice graphics is it Zabbix?
-
@adrianoebm Munin.
@adrianoebm said in SG1100 - Disk Full - Help:
i can't figure out why our files are so huge
Euh ....
Read the log files ? They are there to be read by you. If not, why logging in the first place ? ( why installing syslog-ng ? ) When you read them, one of the first things you'll know is : who fills them up.
Processes inform the admin in real time what they are doing. They don't use the screen, but a log file.
Up to you to decide what gets logged. For every process.
Up to you to decide when to throw away the logs, or use a tool to deal with them, by sending them to a long storage device (NAS, another syslog server, to rotate them etc).Just IMHO : the log files are the most important files on systems like pfSense.
-
I would guess it's because you're sending all firewall logs to it and logging all blocked traffic on WAN. That can be a lot of logs!
-
@stephenw10 These are the settings that our SOC suggests.
Last week they disable de vpn events too in order to reduce the file size.
Instead of racking my brain trying to understand why it fills the disk or not, I configured our zabbix to send us alerts via Telegram when the reamaining space is 20%, apparently it's under control now.
It wasn't exactly clear to me how to limit the file size in syslog-ng but it´s ok. -
@gertjan Our SOC says they read the logs but they can´t figure out why sometimes these files reaches 8GB.
-
By looking at the logs, one of the first things you'll find out is : what's in them.
Just an example :
Check this option :
From now on, every blocked pcket on your WAN will get logged.
Don't be surprised to see hundreds of logs lines per second .....I know, this is a silly example. But as any other log line, easy to recognize.
If, for any reason, the content of the log files is totally non-comprehensible, you might consider stopping logging al together. That would solve the issue.
-
@gertjan I definitely agree with you. :)
-
@gertjan said in SG1100 - Disk Full - Help:
By looking at the logs, one of the first things you'll find out is : what's in them.
Yeah, if those are the things you're sending then it's almost certainly the firewall logs filling it.
Really you should not be storing that data on the eMMC in an 1100. Apart from anything else you are going to be significantly increasing the write wear on the storage. That level of logging should be exported off he firewall to a dedicated log server.
It could be that is what's intended and it's storing them locally unintentionally.Steve
