Logging still not working properly with v2.5
-
One the most anticipated features of the v2.5 upgrade was to finally have logs show more than 30mins of history.
Yes we like to log lots. And outputting to syslog is just too messy. The Pfsense GUI is a nice search tool.I'm glad they moved from CLOG, but search history is still not showing up in the GUI with new format.
It seems there is a BUG in this release.For whatever reason, the GUI search for FIREWALL NORMAL and DYNAMIC views, still cut off the logs during in a search.
The search results only show a few hours of data. Almost as if there is some kind of hard limit set on the search history.
The data is there in the var\logs folder, so the history is available.
We tried with setting on "bzip2" and also with "none" compression.
We also tried with logs sizes at 512k, 1024k, and 999k, to test for any rotation issues.
Even with a Log rotation size of value maxed out at "999999999", the data in the file is there, but the GUI will only display part of the history.
And no this has nothing to do with the Log Entries setting. WE tried different values from 10-5000.The best way to actually see this problem. And replicate it.
-set log file compression to NONE, with 1024 Rotation log size, and set log rotation to 99
-also use the Syslog format
-accumulate a few days of logs (make sure you're logging at lot of things to fill them up)
-View the accumulated log files using via the GUI
-Diagnostics => Edit File => Browse => /var/log
-You should see a bunch of filter.log files
-click on the last one to see the oldest rotated entries and note the date and time
-Now Click on the FIREWALL link in logs
-do a date search in the GUI in the TIME field eg"2021-03-08 17"
-see if the oldest hour ( in this case 17) shows up in the search
-if you get a blank result - up the time value to 18 or 19, or 20, etc
-keep upping it as you do a new search each time
-eventually you will get a result.
-this will be the OLDEST returned value the GUI supplies
-You will see the data is truncated form the log filter files
-wait a few mins and refresh the page, keep the same search values.
-You will see the data start to disappear as it gets truncated.So is this a BUG in v2.5, or am I missing something here?
The filer log data is there, but the GUI still truncates it. -
The code gathers all of the log filenames and runs the contents of them ALL through grep looking for your filter string, and then takes the last N lines of that result and displays it in the GUI (where N=your current GUI line limit).
The only bug I could find is https://redmine.pfsense.org/issues/11639 but the worst outcome of that is some entries being out of order, not missing.
If you have a lot of data it is going to get cycled out over time as new data comes in. Even with the improved log behavior on the firewall in 2.5.0 if you care about persistence you must export the logs to a syslog server and retain them there.
-
I'm sure there is a bug in the logging retrieval, and it's not that one for sure.
Because the issue comes up even with a single log file, so being out of order doesn't apply.And nothing to do with the GUI line limit either.
Since it won't even display a single entry if its passed a certain amount of actual log entries in the filter.log file.
So if there is a limit, its not in the GUI, but the actual parsing of the filter.log file itself.Simple example:
(you can try this yourself. Use none compression so you manually look at the log files later.)
Eg.- Look for a recent IP you know is in the logs (9.9.9.9 or whatever), that has only a few entries (maybe 3 total)
- GUI search the IP, and it will return all the 3 total entries.
- Wait a few days to fill up the log, then do the same GUI search.
- Result will be blank
- Goto var\logs\filterlogs and check the files.
- You will see the 3 logs entries are still there in one of the log files. (they did not get rotate out)
- But they will not show up in the GUI.
So that means the logs files (at least the filter.log files), are not fully parsed.
Or parsed to only a certain amount of logs entries.
If there is a hard limit, then its at the parser level, not GUI.I'm surprised no one has said anything yet, maybe it's too early after the v2.5 release.
But were seeing this on multiple setups.thanks
-
@nz said in Logging still not working properly with v2.5:
Or parsed to only a certain amount of logs entries.
If there is a hard limit, then its at the parser level, not GUI.Hello!
Maybe explore here
John
-
The way the filter log is handled is different than every other log. Since it must be parsed and searched by field, it can't use the same
greptechnique, so you are only searching within the configured number of entries (+100 for a buffer in case some entries cannot be parsed).There may be some room for refinement there but it's still not "broken" and it's still true that if you care about long term log contents you must send them to a proper syslog server.
-
Thanks to Serbus for posting the link to the code location for this.
I can confirm this is the actual hard limit on the search parser.
And the problem with the GUI search results./* Safety belt to ensure we get enough lines for filtering without overloading the parsing code */
if ($filtertext) {
$tail = 10000;
}
Which incidentally is the same thing used on the Firewall Summary view.We ran tests multiple times, on a single filter.log file set at it's max limit size of "999999999", so we don't bring file rotation into this.
Currently this log file contains about 141k entries.We searched by date to display the last possible entries the GUI will actually show.
(eg starting with blank search results and working up incrementally, by adding 1 hour for each new search, until we get a hit)So out of 141082 log entries, the parser only goes back as far as 131080.
Which is roughly 10K, as in your "safetly belt limit."This kind of negates the release of these new logging features.
Since the GUI will never fully search all the logs, because it will never go past 10K entries.Can we have this limit removed? Or make it user configurable?
Or only have it limited to the Firewall Summary view, instead of the entire GUI search function?thx
-
@nz said in Logging still not working properly with v2.5:
Thanks to Serbus for posting the link to the code location for this.
I can confirm this is the actual hard limit on the search parser.
And the problem with the GUI search results./* Safety belt to ensure we get enough lines for filtering without overloading the parsing code */
if ($filtertext) {
$tail = 10000;
}
Which incidentally is the same thing used on the Firewall Summary view.We ran tests multiple times, on a single filter.log file set at it's max limit size of "999999999", so we don't bring file rotation into this.
Currently this log file contains about 141k entries.We searched by date to display the last possible entries the GUI will actually show.
(eg starting with blank search results and working up incrementally, by adding 1 hour for each new search, until we get a hit)So out of 141082 log entries, the parser only goes back as far as 131080.
Which is roughly 10K, as in your "safetly belt limit."This kind of negates the release of these new logging features.
Since the GUI will never fully search all the logs, because it will never go past 10K entries.Can we have this limit removed? Or make it user configurable?
Or only have it limited to the Firewall Summary view, instead of the entire GUI search function?thx
I also would like to see this limit adjustable/removed so we can use the firewall logs properly. Only 30 minutes of logging is available which renders the firewall log useless.
-
@remie2000 said in Logging still not working properly with v2.5:
I also would like to see this limit adjustable/removed so we can use the firewall logs properly. Only 30 minutes of logging is available which renders the firewall log useless.
What are your syslog settings ?
I don't know if this is default :
but 2000 line would show up - that's more then a week for me, as I don't log every packets that hits the wall.
You have really more then 2000 lines in 30 minutes ? Or about a firewall log line every second.
With that rate, and you want some some history in the GUI, let's say : one day, the browser would have to open a file that has at least 86400 lines x 60 chars a line, that's 5 mega bytes. It can be done with a browser, but would use a text editor for that.Btw :
I'm also sending my logs to a syslogger on my LAN, so I have at the logs at my disposal.
And if something happens to pfSense, I have the logs saved, for later inspection if needed. -
Hi,
Due to compliance rules I need to log everything that hits the firewall.
When I do a filter on src or dst IP I only get hits from the last 30 minutes because of the amount of logging. I have increased the size of the logs to 200mb in pfsense and I would suspect that pfsense does the filtering for me (not the browser) and serve me the hits that matches my search query. I have plenty of resources available on the firewall (Dell PowerEdge R340 with 32 GB RAM and a Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz (6 core / 12 threads) ) So I don't mind that my machines needs to some more working going trough the logs.I'm also testing with a syslog server and I get my data there but I cannot search easily trough the logs. What are you using and how can you search for a simple src, dst and port match? It could be a workaround for the limitation that has been built into pfsense.
Thanks for your reply :-)
- Remie
-
@remie2000 said in Logging still not working properly with v2.5:
Due to compliance rules I need to log everything that hits the firewall.
So your not from planet earth. Or you have to oblige to rules created by people that really don't know what they talk about.
Really, 'they' asked you to store all incoming data in a file on a disk ?
Throw this in Google : " can I record a DDOS attack ? "
The TCO would be millions.If you want to be part of a public network you should accept that every member of that network can communicate with you when they want.
Up to you to try to record to record every packet that's thrown at you.
Start by buying a box with very good cigars.Edit : OVH, in France, co - developed an anti DOS system - called "VAC". The price tag is/was mind blowing. And they don't record, just keep traffic in memory, the time it takes to judge if it's part of a DOS or not.
