Syslog service in pfSense v2.8.1 often stop itself

provels

@AlexanderK At least with Watchdog you'll get informed when it happens. That will give you something to work with to try tracking it down in the logs. That said, mine stopped again a few days ago, coincidentally at 00:15 again (marked XXXX). Perhaps it is some timing issue in syslogd that works only 95% of the time? There re plenty of instances of syslogd exiting and successfully restarting a second later, but sometimes it doesn't without a kick in the pants. In any case, I'll stick with watchdog for the present, won't hurt. Just adding to the conversation, not a big issue for me.

XXXX Oct 14 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:45:53	syslogd		sendto: Connection refused
Oct 8 02:45:53	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:45:52	syslogd		exiting on signal 15
Oct 8 02:45:22	syslogd		sendto: Connection refused
Oct 8 02:45:22	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:44:27	syslogd		exiting on signal 15
Oct 8 02:44:21	syslogd		sendto: Connection refused
XXXX Oct 3 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Sep 30 17:09:30	syslogd		sendto: Connection refused

stephenw10

If it's actually external you can add the work-around stateless floating rules to prevent the connection refused message.

provels

@stephenw10 If directed at me, it's not external. It's a second HDD internal to the FW, mounted as a directory on the system SSD. Not complaining.

stephenw10

No sorry that was at the previous poster. The workaround rule won't work for traffic to syslog-ng locally.

provels

Stopped again this AM at 00:15, random interval. Maybe something to do with daily log rotation, GZipping the log, dunno. Just info, not an issue for me anyway.

6 Matched General Log Entries. (Maximum 500)
Oct 23 00:15:02	php-cgi	95349	notify_monitor.php: Message sent to provels
Oct 23 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 14 00:15:03	php-cgi	10330	notify_monitor.php: Message sent to provels
Oct 14 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 3 00:15:03	php-cgi	55524	notify_monitor.php: Message sent to provels
Oct 3 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel

The top of today's default.log.

Oct 23 00:00:00 fw syslog-ng[13248]: Configuration reload request received, reloading configuration;
Oct 23 00:00:00 fw syslog-ng[13248]: Configuration reload finished;
Oct 23 00:10:00 fw syslog-ng[13248]: Log statistics; processed='destination(_DEFAULT)=183', dropped='global(internal_source)=0', processed='global(internal_source)=183', queued='global(internal_source)=0', processed='global(msg_clones)=0', processed='source(_DEFAULT)=183', processed='src.internal(_DEFAULT#0)=183', processed='global(sdata_updates)=0', stamp='src.internal(_DEFAULT#0)=1761195600', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=178', processed='center(queued)=183', processed='center(received)=183', queued='global(scratch_buffers_bytes)=0'
Oct 23 00:15:02 localhost syslogd: restart
Oct 23 00:15:02 localhost syslogd: kernel boot file is /boot/kernel/kernel
Oct 23 00:15:02 localhost php-cgi[95349]: notify_monitor.php: Message sent to provels

aldomoro

Hi

We use Graylog as remote syslog. If server with Graylog has outage, e.g. is restarted due to updates, syslogd is stopped in pfSense 2.8.1. We did not have this issue in v2.8.0.

Aldomoro

stephenw10

Yes, that's the bug discussed here. The workaround rules will prevent it. https://redmine.pfsense.org/issues/16362#note-5

provels

@aldomoro Possibly the best use of Service Watchdog. Maybe the only one! :)

geovaneg

Hi,

Same problem here:

"Nov 2 22:00:02 pfsense syslogd: sendto: Connection refused" (system.log)

PfSense CE 2.8.1, remote logging enabled.

Anothers instances 2.8 running OK.

Workaround: whatchdog

Thanks.

Geovane

tyros

Same problem here on 2.8.1

stephenw10

Applying the workaround firewall rules will prevent it seeing the refusals so will not stop.

geovaneg

Hi,

Apparently the rule isn't working because the traffic counters aren't incrementing. There's a "let out anything IPv4 from firewall host itself" rule with higher precedence that seems to be capturing UDP traffic to the remote syslog server, even though the new rule is of the "floating" type.

@28 pass out inet all flags S/SA keep state (if-bound) allow-opts label "let out anything IPv4 from firewall host itself" ridentifier 1000003613
@46 pass quick inet proto udp from (self:3) to 10.0.1.19 no state label "USER_RULE: rule to avoid syslog stop bug" label "id:1762800608" ridentifier 1762800608

Geovane

mcury

@geovaneg said in Syslog service in pfSense v2.8.1 often stop itself:

Apparently the rule isn't working because the traffic counters aren't incrementing.

Yeap, they should increment.

geovaneg

This is a VPN server located in the DMZ... It has no LAN interface, only a WAN and IPSEC interface, and the counters are not incrementing despite continuous traffic to the log server.

I might be forgetting something obvious, but I reviewed the settings and tested it more than twice.

Geovane

geovaneg

In any case, the watchdog isn't the perfect solution, but it did the job.

thanks

Geovane

stephenw10

Looks like you might have the source port set to 514 instead of the destination.

In your first screenshot it's not shown as an OUT rule also but it looks like you corrected that.

taz3146

Same ongoing issue, remote syslog enabled, it seems rather random, but mostly when the logging machine is down, which is a linux vm on proxmox host.

Bria1972

@slu said in Syslog service in pfSense v2.8.1 often stop itself:

@jrey years ago there was a p1 release:
https://docs.netgate.com/pfsense/en/latest/releases/2-3-5-p1.html

Thanks for the source