grep for all MACs from a certain day?

RedBeardCowboy

Is there a default limitation to the DHCP server logs on pfsense? I'm trying to use the filter to find all MAC addresses that requested an address on a certain day.

Which log file should I look in?

-rw-r--r--  1 root   wheel   43391 Jul 21  2018 bsdinstall_log
-rw-------  1 root   wheel  511488 Feb 21 20:19 dhcpd.log
-rw-r--r--  1 root   wheel    9964 Jan 30 09:35 dmesg.boot
-rw-------  1 root   wheel  511488 Feb 21 20:26 filter.log
-rw-------  1 root   wheel  511488 Jan 30 09:35 gateways.log
-rw-------  1 root   wheel  511488 Jul 21  2018 ipsec.log
-rw-------  1 root   wheel  511488 Jul 21  2018 l2tps.log
-rw-r--r--  1 root   wheel       0 Jul 21  2018 lastlog
drwxr-xr-x  2 root   wheel     512 Jul 21  2018 nginx
-rw-------  1 root   wheel  511488 Feb 21 20:25 nginx.log
drwxr-xr-x  2 root   wheel     512 Jul 21  2018 ntp
-rw-------  1 root   wheel  511488 Jan 30 09:47 ntpd.log
-rw-------  1 root   wheel  511488 Feb 21 16:51 openvpn.log
-rw-------  1 root   wheel  511488 Jul 21  2018 poes.log
-rw-------  1 root   wheel  511488 Jul 21  2018 portalauth.log
-rw-------  1 root   wheel  511488 Jul 21  2018 ppp.log
drwxr-xr-x  2 redis  redis     512 May 23  2018 redis
-rw-------  1 root   wheel  511488 Jul 21  2018 relayd.log
-rw-------  1 root   wheel  511488 Dec 16 14:40 resolver.log
-rw-------  1 root   wheel  511488 Feb 20 18:56 routing.log
-rw-------  1 root   wheel  511488 Feb 21 20:18 system.log
-rw-------  1 root   wheel   11254 Jan 30 09:35 userlog
-rw-r--r--  1 root   wheel     394 Feb 21 20:18 utx.lastlogin
-rw-------  1 root   wheel    1584 Feb 21 20:18 utx.log
-rw-------  1 root   wheel  511488 Jul 21  2018 vpn.log
-rw-------  1 root   wheel  511488 Jul 21  2018 wireless.log

I tried:

grep "Feb 12" /var/log/filter.log
grep "Feb 12" /var/log/system.log
grep "Feb 12" /var/log/dhcpd.log

Where'd I go wrong?

Thanks in advance!

Gertjan

@redbeardcowboy said in grep for all MACs from a certain day?:

Where'd I go wrong?

Hint 1 : check the file size of most those log files.
511488 bytes ... how can this happen ?

Hint 2: most firewalls are small boxes with limit disk storage. Logs files that grow all day could be handled by something like logrottae .... pfSense has none.

Hint 3 : because it's just a half mega, did you checked what's in these file ?
Noop, you would have found that these files are binary (not much text in them) so grep can't make any soup of it.

Hint 4: pfsense log

Grimson

RTFM: https://docs.netgate.com/pfsense/en/latest/book/monitoring/system-logs.html
https://docs.netgate.com/pfsense/en/latest/monitoring/working-with-binary-circular-logs-clog.html

RedBeardCowboy

@gertjan said in grep for all MACs from a certain day?:

@redbeardcowboy said in grep for all MACs from a certain day?:

Where'd I go wrong?

Hint 1 : check the file size of most those log files.
511488 bytes ... how can this happen ?

Hint 2: most firewalls are small boxes with limit disk storage. Logs files that grow all day could be handled by something like logrottae .... pfSense has none.

Hint 3 : because it's just a half mega, did you checked what's in these file ?
Noop, you would have found that these files are binary (not much text in them) so grep can't make any soup of it.

Hint 4: pfsense log

@grimson said in grep for all MACs from a certain day?:

RTFM: https://docs.netgate.com/pfsense/en/latest/book/monitoring/system-logs.html
https://docs.netgate.com/pfsense/en/latest/monitoring/working-with-binary-circular-logs-clog.html

First, let me say thanks for the replies and the links. clog - who'd have thunk it.

Second, I guess I'm out of luck since there's no log rotate. What a failure to not handle this differently since it's just text and you could at least compress and archive, by default. I'm used to enterprise firewalls, and I guess that's what I get for assumptions.

Lastly, what is it about IT people that make so many become jerks. I'm not sure you even know you're doing it sometimes, it's like 70% have turrets or something. Those "hint" and "RTFM" comments... seriously? Do you keyboard commandos act this condescending to people in person? Perhaps you might reflect on the unnecessary hostility.

If you've nothing good to say, say nothing.

Grimson

@redbeardcowboy said in grep for all MACs from a certain day?:

Second, I guess I'm out of luck since there's no log rotate. What a failure to not handle this differently since it's just text and you could at least compress and archive, by default. I'm used to enterprise firewalls, and I guess that's what I get for assumptions.

RTFM some more then: https://docs.netgate.com/pfsense/en/latest/monitoring/copying-logs-to-a-remote-host-with-syslog.html

RedBeardCowboy

@grimson said in grep for all MACs from a certain day?:

@redbeardcowboy said in grep for all MACs from a certain day?:

Second, I guess I'm out of luck since there's no log rotate. What a failure to not handle this differently since it's just text and you could at least compress and archive, by default. I'm used to enterprise firewalls, and I guess that's what I get for assumptions.

RTFM some more then: https://docs.netgate.com/pfsense/en/latest/monitoring/copying-logs-to-a-remote-host-with-syslog.html

That comment was to mean that I'm out of luck, in that the topic was "finding all MACs from a certain day" which is no longer in the logs due to the lack of logrotate.

I'm aware of how syslog works.

This is just what you get with free shit.

Gertjan

Ok.
So you know how logs are stored.
pfSense gets installed on very small devices, so the (initial) limited log storage is understandable.
Up to you to size up the log file size to give space for some more history.
Then a grep => awk (today) => grep (MAC) will do.

tim.mcmanus

@redbeardcowboy said in grep for all MACs from a certain day?:

@grimson said in grep for all MACs from a certain day?:

@redbeardcowboy said in grep for all MACs from a certain day?:

Second, I guess I'm out of luck since there's no log rotate. What a failure to not handle this differently since it's just text and you could at least compress and archive, by default. I'm used to enterprise firewalls, and I guess that's what I get for assumptions.

RTFM some more then: https://docs.netgate.com/pfsense/en/latest/monitoring/copying-logs-to-a-remote-host-with-syslog.html

That comment was to mean that I'm out of luck, in that the topic was "finding all MACs from a certain day" which is no longer in the logs due to the lack of logrotate.

I'm aware of how syslog works.

This is just what you get with free shit.

This is just what you get when you don't do your homework before deploying a solution. If logs were important, why didn't you have a strategy? Or even look at that part of the documentation?

Part of my "definition of done" is deploying with a logging solution in place; for everything. Logs are important. Setting up a linux syslog server is pretty easy, I'm sure you could find a VM template that's already baked for the task.

I too work in enterprise. I never look at device logs. I have a monitoring solution in place for that so I can do event correlation; essentially what you're trying to do.

I think SecurityOnion might be able to do this as well as a ton of other things. Set that up as part of your monitoring system, and you'll be well on your way to designing networks for the enterprise!

Derelict

Yes. Logging to an external logging system is expected if you want more than basic debugging tools.

I have used many "enterprise" firewalls and we never depended on them to store anything but the most cursory of logs on the devices themselves. We always logged to something external if anything historical was desired.