Filter System Logs
-
Hi there, I have been trying to troubleshoot an event that happened a few months ago, essentially 1 of our 2 power circuits went down in a datacenter and we have 2 PF Sense boxes in HA. Our PDU has a switch in it which should have been redundant, however the datacenter says they only lost one of the power circuits. I have been trying to dig through the logs to provide them insight into this as we should not have lost connectivity if this was the case. If anyone has any tips or tricks to filter for lost connectivity or power outages that would be super helpful. IThanks in advance!
-
You had one node on each circuit? Or both on a UPS?
What would have failed if the 2nd circuit went down?
You would see interface events logged if the link actually went down to a switch.
Steve
-
We have a PDU with a smart switch which has 2 separate electrical circuits plugged into it (in the event one goes down, it should auto swap to the circuit without missing a beat)
-
So nothing should have gone down? Everything is into that PDU? All devices connected to all the ports?
If so that sounds like the PDU failover didn't work as planned. When was it last tested?
Steve
-
Correct, nothing should have gone down. I already have an open ticket with the Datacenter in regards to this. I have only worked for this company 2 months and the previous guy that could answer that question is no longer with the company. I am unfamiliar with PFSense logging, which is why I opened a ticket asking for help on the logging to see if it was network or power. I need insight into this while the datacenter get's back to me. They closed my ticket and didn't answer my question so I opened another one :)
-
Well you can certainly look at the system logs at the time if they still exist. If it was a power issue you will see the firewall rebooted. If it was some sort of network issue you might see a link status event or possibly a gateway alarm is the link stayed up but lost upstream connectivity somehow.
You can download the complete system log by going to Diagnostics > Command prompt and first executing:
clog /var/log/system.log > /tmp/systemlog.txtAnd then downloading
/tmp/systemlog.txtfrom the download field on that page.By default the logs are ~500K though so may not cover much time if they are busy.
Steve
-
Thank you for the tip Steve I will do this now, I appreciate the help.
-
The logs only go to Jan 7 2019. Do the previous months get cached in another file location or purged by default?
-
Unfortunately not. The logs are a circular format to limit their size so they continually overwrite the old data once they are full.
You can increase the size of the logs in the log settings quite significatly. They are only 512K by default.
Really though you should be exporting the logs if you need long term log storage:https://docs.netgate.com/pfsense/en/latest/monitoring/copying-logs-to-a-remote-host-with-syslog.html
Steve
-
Thanks for the insight Steve, this information you provided me saved me lots of time. Appreciate it, the previous guy had put a SYSlog server into place, but the license had expired so I lost out on that end as well lol. Still no word from the data center.
Chris
