How to search firewall logs
-
Hi all.
I'm not sure if this has been discussed already but all my searches seem to describe where to find the log files and descriptions of what the logs are. I know about using the filters but the problem is that their searching is very limited in the time period. Even an IP that I know should be there from yesterday doesn't seem to show up when I use the GUI.
Perhaps I should explain what I would like to do. I have pfSense setup for a client that is also running an email server. I have configured it so that there are various blocklists that get downloaded and used in addition to only allowing certain countries. My difficulties come in when I get an email from one of their staff that says "such and such a person has received this delay message" or "failure message" which is a day or two old. To troubleshoot I would like to be able to go to the log and search for the IP associated with their MX records or from the email headers but it never seems to be there and I need to employ other methods to find out what happened. Without the router I would just go to my email server and find the email address of the sender or receiver and what happened.
Details about my configuration: I have lots of RAM, HD space, and Processor so I've set the log file sizes to what I perceive to be quite large. In the Settings under General Logging Options I've set the log file size to 700,000,000 bytes and I've reset the log files so they are working with the new size. I've only included the commas here for easier reading. Based on the numbers in the Dashboard I could make them much larger and still be okay if that's the issue.
Can anyone tell me if it's possible to do the searching that i want from the GUI or if I'm going to have to setup a Syslog server or something?
Any help is appreciated.
Thanks
Joe
-
Pfsense doesn't keep that detailed of a log, in regards to this. If you look at the raw logs on the console you can see for yourself. The firewall log seems to be good to track connections (but not a whole lot of detail). Postfix (or whichever similar thing you are using) will keep logs that are probably much more interesting to you.
If you are using the mailscanner package, it has its own logs; I wasn't super impressed with its level of customizability so I haven't used it in awhile, not sure if the logs are searchable in the GUI.
-
You're never going to see more than 50 entries from the GUI.Shell in and use either ee /var/log/filter.log or clog /var/log/filter.log | grep IP_address if you need to see more. For me, the default 512K of log shows roughly 1 hour of use. You have yours set to 700MB, and that's per log file, so the actual space required would be 20 x 700 MB = 14 GB.https://doc.pfsense.org/index.php/Adjusting_the_Size_of_Log_Files
https://doc.pfsense.org/index.php/Why_can't_I_view_view_log_files_with_cat/grep/etc%3F_(clog)
-
You're never going to see more than 50 entries from the GUI.
Unless of course you go to "Status->System logs->Settings" and set the "GUI Log Entries to Display" to say, 2000.
This will definitely slow down the display of log pages, but can be quite useful in some scenarios.For larger reviews, the "clog" comments are the way to go.
-
Thank you KOM I will take a look at using ee or clog it seems like that will allow me to do the searches I need.
I'm thinking "time" might not be the best way to express the log file limits because that will change depending on the amount of traffic. 1 day of traffic for you might be 2 or 3 days for someone else or vice versa. I have the email reports package and I had it send me what it had which should be a full day but not in this case because I reset the logs yesterday evening. The email size will be larger than the actual file because it also includes an RRD image of the traffic but probably not by too much. The email is 16megs, spans "Mar 10 20:39:32" to "Mar 11 14:10:55" and is almost exactly 100,000 lines (that's one line per entry at 160 bytes per entry). That's just over 17 hours or approximately (less than) 1 meg per hour. At that rate I should be able to get somewhere around 700 hours of logging or almost a month which will do nicely now that I know how to search it. :) Obviously, on a busy or slow month that could change considerably.
Anyway, I just included all that stuff to help anyone that comes along later that wants to try calculating things. Lots of rounding and estimating going on so try to use your own numbers if you can.
Note, I saved all the lines into a text file and that's only 15.7 megs.
Thanks for your help everyone.
Joe