Syslog service in pfSense v2.8.1 often stop itself

sokeada

Dear All,

After upgraded from pfSense from v2.8.0 to v2.8.1, I've noticed that syslog service often stop itself and need to start manually each time it stop. This issue never happen in pfSense v2.8.0.

Status from pfSense GUI
pfsense log service.JPG

Status from monitoring system
pfsense log service stop.JPG

w0w

@sokeada
And what do the system logs show on pfSense itself, not on the monitoring system?

slu

@sokeada I saw this also on our system, but still no idea what cause the stop.

sokeada

@w0w looks like no much detail in log. The top one is when syslog stop and the second one is after I clicked start syslog service manually.

slu

@sokeada
we enabled remote logging to our syslog server and maybe this is the issue if the server is not available.

This can explain why we saw this only after reboot (so far).

sokeada

@slu yeah and it happened to me second times already since I upgraded to v2.8.1 from the first day of v2.8.1 released.

sokeada

@slu said in Syslog service in pfSense v2.8.1 often stop itself:

@sokeada
we enabled remote logging to our syslog server and maybe this is the issue if the server is not available.

This can explain why we saw this only after reboot (so far).

This is make sense, I enabled remote logging as well.

tsmalmbe

I just wrote this https://forum.netgate.com/topic/198793/syslog-fails-on-2.8.1-when-remote-syslog-server-goes-down

Maybe an admin can combine these threads. I think we are seeing the same issue.

sokeada

@tsmalmbe said in Syslog service in pfSense v2.8.1 often stop itself:

I just wrote this https://forum.netgate.com/topic/198793/syslog-fails-on-2.8.1-when-remote-syslog-server-goes-down

Maybe an admin can combine these threads. I think we are seeing the same issue.

I just saw your post, yeah looks like we are on the same page. Hope to get fixed soon.

slu

@sokeada said in Syslog service in pfSense v2.8.1 often stop itself:

This is make sense, I enabled remote logging as well.

That explain also why my other boxes without remote logging doesn't hit this issue.

sokeada

@slu said in Syslog service in pfSense v2.8.1 often stop itself:

@sokeada said in Syslog service in pfSense v2.8.1 often stop itself:

This is make sense, I enabled remote logging as well.

That explain also why my other boxes without remote logging doesn't hit this issue.

yeah.

myke30

Hello Everyone,
I have exactly the same issue only with pfsense 2.8.1 ( 2.8.0 is okay) with the same logs :

"sendto: Connection refused " on the syslogd process and the service stopped after that ( i think)

The remote logging is enabled.

I hope we'll be a patch :-)

Best Regards.

w0w

https://redmine.pfsense.org/issues/16362

sendto: Connection refused

Do you have any idea why it can be refused? I see this only when booting and network not initiated yet or whatever, never happened when everything is up and running.

stephenw10

Typically it seems to be when the syslog server is cycling in some way. Archiving perhaps? Seems like an issue for a syslog server to periodically stop accepting logs! But enough people are seeing it that I guess it's a thing.
This looks to be fixed by changes that went into syslogd recently. I can't replicate it in 25.11.

tsmalmbe

@stephenw10 It is not an "issue" for syslogservers to stop accepting logs for the duration of a restart. It is a common fact that this may happen, all syslog receivers are not (or cannot be) clusters. And while this is all UDP, i am (still) a bit baffled why the pfSense syslog would actually even care - it should just spit UDP in that direction and let the receiver worry about ...well, the receiving part.

slu

@stephenw10 said in Syslog service in pfSense v2.8.1 often stop itself:

Typically it seems to be when the syslog server is cycling in some way. Archiving perhaps?

In our case we saw this multiple times after reboot and pfSense is ready before all the KVM guests are online for example the syslog server...

Not sure this also happens in 2.8.0, maybe possible.

stephenw10

Oh it absolutely shouldn't. It's a bug and it's now fixed. https://reviews.freebsd.org/D51995

stephenw10

@slu said in Syslog service in pfSense v2.8.1 often stop itself:

Not sure this also happens in 2.8.0, maybe possible.

Yes, it's in 2.8.0 and 2.8.1 and 25.07. Unfortunately.

slu

@stephenw10
mhm, I try the "Service Watchdog" at the moment, maybe a workaround?
Will see..

tsmalmbe

ChatGPT overview of the change, seems legit (of course caveat emptor for LLM...)

After D51995, all of the still-fatal cases are local to the pfSense box (syslogd/process/kernel socket). None of the remaining fatal errors are caused by the remote syslog host; the remote/network-state errors were reclassified as transient and no longer make the destination “dead.”

What can still make syslogd drop the destination (and why)

Local to the pfSense box (syslogd / socket / config):

EBADF – invalid/closed descriptor used for sendmsg(). Programming/state issue on the sender.
man.freebsd.org
EACCES – permission denied (e.g., trying to send to a broadcast address without SO_BROADCAST, or lacking permission on a UNIX-domain socket path). Sender-side socket option or filesystem perms.
ENOTSOCK – fd is not a socket. Sender bug/misconfiguration.
EFAULT – bad user-space buffer/pointer given to sendmsg(). Sender bug.
EMSGSIZE – message too large for the socket/protocol to send atomically (e.g., oversize UDP/UNIX-dgram). Sender data/MTU limits at the local stack boundary—not the remote host.
Any other unexpected errno not on the new whitelist (e.g., EINVAL, EAFNOSUPPORT, EDESTADDRREQ, ENOTCONN)—all indicate a local misuse/state problem.

Dependent on the remote syslog host or wider network?

None of the still-fatal ones. Host/network conditions like refused connection, no route, host down/unreachable, address not available, buffer pressure, or EAGAIN were explicitly moved to the “transient, keep retrying” bucket and no longer cause F_UNUSED.