504 Gateway Time-out on Dashboard and Firewall System Logs

jimp

If you need a reliable audit trail you should never rely on the logs being on the firewall itself. You need a proper syslog server for that. 1GB logs are huge, and on 2.5.0 they get compressed and rotated, so even more are kept. You have to look not just at <name>.log files but also <name>.<number>.bz2 for compressed rotated archives.

Clear all of those out manually from the shell, or try going straight to /status_logs_settings.php and reset them there, and set a much more sane log size.

If you must keep large file sizes for logs, disable compression as that will drastically slow down as the file sizes get large.

We don't put limits on the sizes since the performance varies widely by hardware and there is no way to calculate what is "too large" for a given setup.

dobrosavljevic

Thanks for the advice!

Not all of our clients that are using a Netgate firewall have the budget for a dedicated syslog server so we wanted to utilize as much of the infrastructure that they do have for an audit trail.

I guess I didn't think 1GB log file sizes are particularly large and I thought a 7100 would be able to handle them. Thanks for clarifying that I was in fact wrong! 8)

Manually deleting the rotated filter.log.*.bz files as well allowed me to load up the Firewall page.

I turned off compression (under Satus > System Logs > Settings> and lowered the firewall log size to 500Gb to see if the device could handle that better. I'll play with the log size until I find a size that will allow the firewall to still work well and keep some of the audit trail.

Gertjan

Log files are often referred to when it is 'late' or 'to late.
Like : "what happened to the disk that isn't accessible any more ?".
The logs with possible answers are .... on that disk.
So humanity invented the syslog server, to be the witness of bad events.

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

dedicated syslog server

It's not a question of funds. A 30 $ (example) Rasberry Pi could do the job perfectly. You could mount it IN the "7100".
These do not have xxx Tb of disk space, but I presume that if you want to keep xxxx Tb size log files, some investments have to be made ( 50 $ ? ).

Also : my Syno NAS has a syslog app - could be used also.

dobrosavljevic

Yea, I understand the point of an audit trail. We do want logs so we can tell what happened after the fact. Things happen that you can't predict or prevent and it's sometimes important to be able to tell what transpired during an unwanted event.

I guess I don't understand how a Pi would be a better syslog server then an XG-7100 that has an Intel(R) Atom(TM) CPU C3558 @ 2.20GHz processor, that otherwise isn't utilized all that much, and 30 GB of storage space.

A budget often doesn't just mean the cost of hardware. A budget also includes the time to initially set something up and maintain and monitor it. Maintaining a network with additional complexity, however slight that complexity seems, adds costs of time and money and hence increases the total cost of ownership over the lifetime of the network.

All I am saying is that I'd like to get the most out of the hardware that is installed and it seems reasonable to expect that the firewall itself should be able to keep some of the audit trail that a network might need.

jimp

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

I guess I didn't think 1GB log file sizes are particularly large and I thought a 7100 would be able to handle them. Thanks for clarifying that I was in fact wrong! 8)

On the old clog-based logs that was the total size of the single log, but on 21.02/2.5.0 and later that's the size at which the log is rotated, plus a number of rotated logs are kept.

So essentially at 1GB you're actually keeping 8GB of logs (main log + 7 rotated logs) -- since the rotated logs are compressed and take up less space, that isn't factored into the initial calculation.

jimp

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

I guess I don't understand how a Pi would be a better syslog server then an XG-7100 that has an Intel(R) Atom(TM) CPU C3558 @ 2.20GHz processor, that otherwise isn't utilized all that much, and 30 GB of storage space.

It's not about how beefy the hardware is, it's the logical duty separation. A syslog server would have persistent long-term log storage that isn't on the device being monitored. It could receive logs from multiple devices (other routers, switches, devices, servers, etc) and with the right software could correlate and report on the log data.

The firewall is a firewall, let it be a firewall.

dobrosavljevic

Thanks for the feedback everyone. It helps clarify my thinking and assumptions! I haven't had a chance to apply these changes for the firewall logs to the other xg-7100 that we manage that also had the Dashboard loading problems. I'll report back if this resolves that issue as well.

jimp

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

Thanks for the feedback everyone. It helps clarify my thinking and assumptions! I haven't had a chance to apply these changes for the firewall logs to the other xg-7100 that we manage that also had the Dashboard loading problems. I'll report back if this resolves that issue as well.

If you have the firewall log widget on the dashboard, it likely would solve the problem.

dobrosavljevic

@gertjan said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

Also : my Syno NAS has a syslog app - could be used also.

Thanks for this tip by the way. We will definitely use the synology for logging with the clients where we have one implemented.

dobrosavljevic

@jimp said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

If you have the firewall log widget on the dashboard, it likely would solve the problem.

This was totally the case. Reset the local logging settings to factory defaults and setup a Synology as a syslog server and we are back to normal.