[Negate 6100] No logging post 23.01 upgrade

dono

Hi. I have a Netgate 6100 running the latest 23.01-RELEASE (amd64).
I recently noticed that all logs have stopped updating since February 18th. I am fairly confident that is the exact date that I updated to the newest 23.01 release.

Other than this, everything else seems to be functional as normal.
Is this a known issue with the recent upgrade?
How can I re-enable the logging?

Best regards.

stephenw10

No that's not something I've seen. Is syslogd running in Status > Services?

I would first try resetting the logs in: Status > System Logs > Settings

Steve

dono

@stephenw10 Thanks for the response.

Is syslogd running in Status > Services?

No, it is not. I am not currently using syslog.

I would first try resetting the logs in: Status > System Logs > Settings

Other than clearing all of the logs, no change. Normally I expect several blocks a second. I have waited about an hour since clearing.

Best regards.

heper

@dono there will be no logging if syslogd is not running....

dono

Hi @heper.

@dono there will be no logging if syslogd is not running....

Hmm. I meant to say that I am not doing remote logging.
However, I did not disable the service either. And now that I try running it, it remains unstarted. (And there is nothing in the logs.) Are there other logs that I can check as for why the service does not start?

Best regards.

stephenw10

Hmm, normally I'd say check the logs but....
So try starting it manually at the cli. The command used depends on the settings you have but this should work:

/usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf

It might throw an error at the command line. Or if it starts correctly then I'd suggest you have some setting that is invalid in 23.01. Though I have no idea what that could be.

dono

Hi @stephenw10,

It might throw an error at the command line.

It did work. There were no command line errors.
I then tried stopping and starting via the UI, but it again failed, though.
And again I re-starated via the command line.

Or if it starts correctly then I'd suggest you have some setting that is invalid in 23.01.
Though I have no idea what that could be.

I compared a config backup from before the upgrade and now.
Other than a few recent firewall rule changes, nothing really stands out.
For reference, here are the <syslog /> settings.

<syslog>
		<filterdescriptions>1</filterdescriptions>
		<nentries>500</nentries>
		<logcompressiontype>none</logcompressiontype>
		<format>rfc3164</format>
		<rotatecount></rotatecount>
		<sourceip>lan</sourceip>
		<ipproto>ipv4</ipproto>
		<auth></auth>
		<dpinger></dpinger>
		<system></system>
		<ntpd></ntpd>
		<reverse></reverse>
		<logconfigchanges>enabled</logconfigchanges>
		<vpn></vpn>
		<resolver></resolver>
	</syslog>

If there are any other specific settings that may be useful I can share them.

Thank you and best regards.

stephenw10

Hmm, OK using your config directly it still starts fine for me. The resulting command line is:

/usr/sbin/syslogd -O rfc3164 -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf -b 192.168.241.1

Where 192.168.241.1 is my LAN interface IP. Perhaps yours is trying to bind to an IP that doesn't exist?

It does seem odd to see that set when there are no remote syslog servers configured. You might just try removing the sourceip and ipproto lines from your config dircetly.

dono

Hi @stephenw10,

Thanks for the continued feedback. It is much appreciated.

Where 192.168.241.1 is my LAN interface IP. Perhaps yours is trying to bind to an IP that doesn't exist?

My LAN interface does not have an IP address. Only the VLANs connected to it have IPs. This was working previously though. Does the logic try to bind to the LAN IP only now?

Best regards.

stephenw10

At some point you had configured an external syslog server and set a source IP of 'LAN'. For some reason in 23.01 it's trying to bind to that IP even though you no longer have a remote server set. And presumably that did not happen in 22.05. That appears to be a bug.
So either remove the config line that sets it manually.
Or enable a remote server temporarily and set the source to 'any', then remove the remote server IP.

I've confirmed that as a bug here. I'll open a report...
https://redmine.pfsense.org/issues/14120

michmoor

@stephenw10 I may be running into this as well. With Suricata installed I enable EVE json logs to be generated and sent to my remote syslog. This isn’t happening.
Although this maybe package related I wanted to bring up that I’ve seen thisnbehav

stephenw10

Did that log an error? Or prevent syslogd starting?

michmoor

@stephenw10 Nope no error. I still think it’s package related perhaps.
Suricata alerts I get a syslog entry.
EVE json logs are not generated and not sent to pfsense syslog.

dono

Hi @stephenw10,

That was it! Thank you very much.
After clearing <sourceip />, I am able to stop / start and restart the syslog service again.

At some point you had configured an external syslog server and set a source IP of 'LAN'.

Indeed, back in late 2021 when I initially setup this 6100 I did configure an external server. At that time I was doing inter-VLAN routing on a L3 switch so the VLANs did not yet exist on the FW. This was extremely fast but was not possible to control traffic between the VLANs. So I changed the design to route between the VLANs on the FW; it was during this transition that I disabled the IP address on the LAN interface. It must have remained in the <syslog /> configuration.

Thank you very much for helping me resolve this.
It is much appreciated.

stephenw10

@michmoor said in [Negate 6100] No logging post 23.01 upgrade:

@stephenw10 Nope no error. I still think it’s package related perhaps.
Suricata alerts I get a syslog entry.
EVE json logs are not generated and not sent to pfsense syslog.

Probably just similar to this then but as you say that code will all be in the package. Better to open a new thread for that in the IDS/IPS section.