Syslog service doesn't start correctly in 26.03-RELEASE
-
Re: Syslog service in pfSense v2.8.1 often stop itself
Hello, I've read through the above topic, and it seems like I'm having a similar issue.
I had a power outage yesterday, and obviously pfsense (4100 appliance) is going to boot up before my hypervisor, which also needs to boot a graylog VM.
At about the time when power came back, pfsense just didn't start syslog at all.
Could the watchdog a solution for this problem?
... SNIP ... Apr 17 12:28:20 kernel igc2.14: promiscuous mode disabled Apr 17 12:28:20 syslogd kernel boot file is /boot/kernel/kernel ... ^ Today I went into syslog settings (status_logs_settings.php) and clicked save, so it started logging again, otherwise, there are no logs ever since it booted yesterday, not in pfsense nor in graylog Apr 16 12:14:59 check_reload_status 660 Starting packages Apr 16 12:14:59 php-fpm 1531 NOTICE Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 10.0.12.1 -> 10.0.12.1 - Restarting packages. Apr 16 12:14:59 php-fpm 88465 NOTICE Skipping STARTing packages process because previous/another instance is already running Apr 16 12:14:57 check_reload_status 660 Reloading filter Apr 16 12:14:57 check_reload_status 660 Starting packages Apr 16 12:14:57 php-fpm 96196 NOTICE Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 10.0.20.1 -> 10.0.20.1 - Restarting packages. Apr 16 12:14:56 php-fpm 37996 ERROR No default gateway found Apr 16 12:14:55 check_reload_status 660 Reloading filter Apr 16 12:14:55 check_reload_status 660 Reloading filter Apr 16 12:14:55 check_reload_status 660 Starting packages ... SNIP ... Power came back -
Is this repeatable if you just reboot?
Using the service watchdog should really only be for diagnosing a problem. Running it continuously can result in unexpected/unintended restarts for some services.
-
@stephenw10 Apologies for the delay, for some reason I didn't get an email for your reply. I modified my notifications settings so I should be getting new replies.
As for the reboot, how do you mean? If I reboot pfsense or the graylog VM?
I think the issue is: if the remote syslog server is unavailable when syslogd starts, it just exits and never tries to reconnect.
-
That can certainly happen. If so then just rebooting pfSense would never hit that?
I assume it keeps logging locally though? It can stop sending to remote servers but usually only if they respond with a refusal. If that happens it should be logged locally.
-
If so then just rebooting pfSense would never hit that?
Likely not.
I assume it keeps logging locally though?
Negative, here's a gap where it happened again
Same gap in Graylog (I restarted logging again today)
-
Hmm, I haven't seen that before. That's just using a syslog server configured in the main pfSense log settings? Not using syslog-ng?
-
Yes, the built in syslog. I modify the settings in status_logs_settings.php
-
Nothing unusual or custom configured? Failing to replicate it here so far....
-
pfsense is going to boot before the hypervisor, so that means DNS is going to be broken, and the Graylog server is going to be broken, when pfsense tries to connect. Unsure what else might be triggering the issue.
If you could test those two cases on your end, I'll could also try rebooting pfsense after a power outage event (after the VMs are booted) to see if that fixes the issue, but I got a UPS in the meantime, so a power outage will likely never occur spontaneously for me from now on.
-
Ah, no DNS there could be an issue. Let me see....
Can you test it with an IP there instead?
-
Using host names is nearly always better, but could this be an exception ? : use an IP here, and now pfSense syslog doesn't need to rely on 'dns' to function.
@FoolCoconut said in Syslog service doesn't start correctly in 26.03-RELEASE:
power outage will likely never occur spontaneously for me from now on
An UPS can give you the possibility to handle a controlled shutdown.
Your issue is startup related
If the USP is smart enough, maybe it can activate the power outlets in a sequence, with a delay between them ? (just an idea).
What about this pfSense package :
and have exit execute a small shell script that delays for a minute or so, and then restart 'syslog' ?
This probably somewhat broken as you will miss the initial pfSense boot up logs.
The only real solution is (imho) : make sure the syslog server is up when pfSense boots.I use this :
where 192.168.1.4 is my Syno NAS with syslog server capabilities.
My Syno NAS obtains an MAC Lease static IPv4 using DHCPv4, and pfSense is the DHCPv4 server.
Afaik, I don't miss any (boot ) logs ...
( Note to myself : I should re check this ) -
Yeah I use an IP there for test syslog setups there but using an FQDN should work.
How do you have DNS configured in pfSense? External server only?
-
converted to an IP, maybe that will help prevent the issue down the road.
DNS is forwarded to an unbound server which runs on the VMs, so yeah, external only.
-
Your issue is startup related
Yeah, and the simultaneous startup happens after a power outage (but pfsense wins obviously)
If the USP is smart enough, maybe it can activate the power outlets in a sequence, with a delay between them ? (just an idea).
Won't work, hypervisors need networking in order to start properly.
and have exit execute a small shell script that delays for a minute or so, and then restart 'syslog' ?
I suppose that could be a workaround, but I'm not too keen on doing workarounds for critical infra such as core routers/firewalls.
-
Hmm, failing to replicate this so far. With no DNS at all and a syslog server defined as an FQDN it still completes the boot. The syslog daemon is still running.
Seems like it must require some other factor from your config.
-
Unclear for now. Let's keep the thread open, and I'll update if the issue ever occurs again (and I have more details)
-
Just to be clear is this new behaviour you saw in 26.03 or just the first time you saw it?
-
Just the first time I saw it.
