How to troubleshoot why we are missing logs?
-
We run pfSense in front of our WiFi-infrastructure and serve between 10k-20k clients during the day. The aim is to log all connections in order to be able to determine who did what on the network. Since we have an address pool of external addresses, we log both traffic incoming on the LAN iface as well as outgoing on the WAN interface, in order to match LAN ip/port to WAN ip/port.
The actual way this is accomplished is using floating match rules, generating the following pf code:
match log on { $WAN } inet from $nat_pool to any tracker 1465913559 label "USER_RULE" match log on { $LAN } inet from any to !$LAN_RANGE tracker 1469087402 label "USER_RULE"We also save logs using syslog-ng on the actual boxes themselves as well as forward to a syslog server (logstash).
The trouble is, we've had two incidents where we've been unable to uniquely match an abuse report to the logs. What we've been able to do in these situations so far is to use our passive fiber tap logs where the reported connection does show up, gather additional connections and match those to logs that show up in pfsense. This is certainly not ideal, as it shows that pf isn't logging a connection that we can prove happened, so what could be the cause of such a situation? We need to be certain in pfSenses ability to log everything, for analysis and to follow up on reported abuse.
I appreciate any pointers from the community, thanks.
-
I'm pretty sure PFSense has a max log size and loops back around, over-writing past logs. If you want lopg term storage, have logs sent to another logging server.
-
I'm pretty sure PFSense has a max log size and loops back around, over-writing past logs. If you want lopg term storage, have logs sent to another logging server.
Right, that's what we do. In the OP I mention that we run syslog-ng on the boxes themselves to be able to save the logs for longer as well as an ELK instance on an external server. The clogs aren't used at all, they don't keep more than about 10 minutes of logs @100MB during peak times. The point is, we're missing logs and to me it seems like an issue with pf nog logging, what could the reason be? Are we hitting some sort of celiling in terms of logging capacity? Is pf associating the connection with something that's already been logged? It shouldn't, but could that be what's happening? Surely if pf is able to classify a connection it should also be able to log it. The CPUs aren't even close to being fully utilized.
This has me a bit stumped, and considering the logs are missing on both syslog instances it would seem the issue lies somewhere with pf and/or the syslog code.
-
Do you have multiple networks on your LAN using IP Alias VIPs?
You might be hitting this:
https://redmine.pfsense.org/issues/6799
Privacy Policy · Cookie Policy