504 Gateway Time-out on Dashboard and Firewall System Logs

[jimp]

Without more information it's hard to say what might be happening.

Do you have a lot of routes in your routing table? (from BGP, for example) -- that could affect the dashboard.

Did you alter your log settings in the past to use gigantic log file sizes? -- that could affect the system logs.

What size are the log files in /var/log/?

[dobrosavljevic]

I am not sure how to judge how many routes are too many. We have a single 8 IP address block that routes through a single upstream router. Mostly it's port forwarders that we have setup and it's maybe three or four per IP. I wouldn't think that would be too much for this machine to handle but I guess I'll let you correct me if I am wrong.

We did adjust the size of the firewall logs because we wanted to have an audit trail for outbound connections to refer to in case we need to respond to a cyber attack incident after the fact. We would like to be able to tell where malicious traffic was going to in case the attack was successful on of our internal devices.

We changed only the firewall logs to rotate them out up to 4 times when they reached 1Gb in size. We didn't receive a warning on the interface when making the change that this could cause the Web GUI to become unresponsive.

To try and troubleshoot the problem we simply removed the big filter.log file and tried to go to the web GUI with the same result, the page was just trying to load and then eventually timed out. To be clear this only happens to the "Firewall" tab under Status -> System Logs.

The total size of the /var/log directory after that deletion is 708Mb.

Now that the page keeps timing out we are also unable to change those values back to the defaults for the firewall logs to see if that will make any difference.

A curious thing happened as well after deleting the filter.log file my assumption was that the system would just recreate the file if it sees that it was missing. Not only did not do that but after manually creating the filter.log file using touch the system didn't seem to populate it with new data either.

I guess my next thing to try would be to restart the appliance to see if there is any change in behavior after the deletion of the file but I won't be able to do that until later tonight when it's not being used in production.

[jimp]

If you need a reliable audit trail you should never rely on the logs being on the firewall itself. You need a proper syslog server for that. 1GB logs are huge, and on 2.5.0 they get compressed and rotated, so even more are kept. You have to look not just at <name>.log files but also <name>.<number>.bz2 for compressed rotated archives.

Clear all of those out manually from the shell, or try going straight to /status_logs_settings.php and reset them there, and set a much more sane log size.

If you must keep large file sizes for logs, disable compression as that will drastically slow down as the file sizes get large.

We don't put limits on the sizes since the performance varies widely by hardware and there is no way to calculate what is "too large" for a given setup.

[dobrosavljevic]

Thanks for the advice!

Not all of our clients that are using a Netgate firewall have the budget for a dedicated syslog server so we wanted to utilize as much of the infrastructure that they do have for an audit trail.

I guess I didn't think 1GB log file sizes are particularly large and I thought a 7100 would be able to handle them. Thanks for clarifying that I was in fact wrong! 8)

Manually deleting the rotated filter.log.*.bz files as well allowed me to load up the Firewall page.

I turned off compression (under Satus > System Logs > Settings> and lowered the firewall log size to 500Gb to see if the device could handle that better. I'll play with the log size until I find a size that will allow the firewall to still work well and keep some of the audit trail.

[Gertjan]

Log files are often referred to when it is 'late' or 'to late.
Like : "what happened to the disk that isn't accessible any more ?".
The logs with possible answers are .... on that disk.
So humanity invented the syslog server, to be the witness of bad events.

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

dedicated syslog server

It's not a question of funds. A 30 $ (example) Rasberry Pi could do the job perfectly. You could mount it IN the "7100".
These do not have xxx Tb of disk space, but I presume that if you want to keep xxxx Tb size log files, some investments have to be made ( 50 $ ? ).

Also : my Syno NAS has a syslog app - could be used also.

[dobrosavljevic]

Yea, I understand the point of an audit trail. We do want logs so we can tell what happened after the fact. Things happen that you can't predict or prevent and it's sometimes important to be able to tell what transpired during an unwanted event.

I guess I don't understand how a Pi would be a better syslog server then an XG-7100 that has an Intel(R) Atom(TM) CPU C3558 @ 2.20GHz processor, that otherwise isn't utilized all that much, and 30 GB of storage space.

A budget often doesn't just mean the cost of hardware. A budget also includes the time to initially set something up and maintain and monitor it. Maintaining a network with additional complexity, however slight that complexity seems, adds costs of time and money and hence increases the total cost of ownership over the lifetime of the network.

All I am saying is that I'd like to get the most out of the hardware that is installed and it seems reasonable to expect that the firewall itself should be able to keep some of the audit trail that a network might need.

[jimp]

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

I guess I didn't think 1GB log file sizes are particularly large and I thought a 7100 would be able to handle them. Thanks for clarifying that I was in fact wrong! 8)

On the old clog-based logs that was the total size of the single log, but on 21.02/2.5.0 and later that's the size at which the log is rotated, plus a number of rotated logs are kept.

So essentially at 1GB you're actually keeping 8GB of logs (main log + 7 rotated logs) -- since the rotated logs are compressed and take up less space, that isn't factored into the initial calculation.

[jimp]

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

I guess I don't understand how a Pi would be a better syslog server then an XG-7100 that has an Intel(R) Atom(TM) CPU C3558 @ 2.20GHz processor, that otherwise isn't utilized all that much, and 30 GB of storage space.

It's not about how beefy the hardware is, it's the logical duty separation. A syslog server would have persistent long-term log storage that isn't on the device being monitored. It could receive logs from multiple devices (other routers, switches, devices, servers, etc) and with the right software could correlate and report on the log data.

The firewall is a firewall, let it be a firewall.

[dobrosavljevic]

Thanks for the feedback everyone. It helps clarify my thinking and assumptions! I haven't had a chance to apply these changes for the firewall logs to the other xg-7100 that we manage that also had the Dashboard loading problems. I'll report back if this resolves that issue as well.

[jimp]

@dobrosavljevic said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

Thanks for the feedback everyone. It helps clarify my thinking and assumptions! I haven't had a chance to apply these changes for the firewall logs to the other xg-7100 that we manage that also had the Dashboard loading problems. I'll report back if this resolves that issue as well.

If you have the firewall log widget on the dashboard, it likely would solve the problem.

[dobrosavljevic]

@gertjan said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

Also : my Syno NAS has a syslog app - could be used also.

Thanks for this tip by the way. We will definitely use the synology for logging with the clients where we have one implemented.

[dobrosavljevic]

@jimp said in 504 Gateway Time-out on Dashboard and Firewall System Logs:

If you have the firewall log widget on the dashboard, it likely would solve the problem.

This was totally the case. Reset the local logging settings to factory defaults and setup a Synology as a syslog server and we are back to normal.