25.07 unbound - pfblocker - python - syslog
-
Hmm, are the logs duplicated both locally and in the remote syslog server?
It 'feels' like maybe Unbound is sending logs independently as well as the local syslogd service. Though I've never seen that before. Or have any idea how it could happen.
I agree the remote syslog server temporarily going down should not stop the service sending. Can you see open states for it when it fails? Is it in a local subnet? In the ARP table?
-
-
said in 25.07 unbound - pfblocker - python - syslog:
In the case of filterdns and unbound the local log file only has a single entry -
They are only duplicated in the syslog not the local file.
@stephenw10 said in 25.07 unbound - pfblocker - python - syslog:
Or have any idea how it could happen.
the only thing, I can think of is that (supported by many web articles) comes back to a problem with the handlers
said in 25.07 unbound - pfblocker - python - syslog:
"duplicate syslog messages in Python can occur when multiple handlers are attached to the same logger or when log messages propagate through multiple loggers"
I would imagine that it is perhaps not only a python issue, but rather a problem with the way these two application (filterdns and unbound) are initializing their log handlers ?.
-
Not sure why that would make any difference since the only ones selected are for the services running on the netgate.
the unselected ones have never been selected, no point. these are not used
Netgate Nexus - not used
DHCP (is not running at all on the netgate, never has, DHCP is handled on a different server)
PPP - not used
Captive Portal - Not used.
Routing Daemon Events - Not used
Wireless - not usedin fact the associated local files (for example dhcp.log, wireless.log, ppp.log etc and the others are all zero bytes on disk).
I can add this to the list of things to try however, Thanks
-
Step 1: uncheck logging for DNS Events (nothing bad happened)
Step 2: stop the syslog server machine (nothing bad happened)
Step 3: force a restart of ntpd because this would generate logging to the now offline server. (nothing bad happened (syslogd did not stop running)
Step 4: start syslog server machine
Step 5: force another restart of ntpd (nothing logged to syslog) it says the syslog is running (there is a entry in the arp table, and states exist)
Step 6: restart syslogd - logging starts (maybe if I had waited longer before doing this restart it may have eventually stopped the service)
Step 7: restart ntpd - now logs the startup on the syslog (no duplicates here never have been)
Step 8: recheck the logging for DNS events (nothing changes)
Step 9: restart unbound (unbound and startup messages as noted above all duplicated)It seems it is a "new feature" in 25.07 for sure.
Guess I have to spin up a version 2.8 virtual and see if I can recreate it there.
It is pretty basic logging -
My test 2.7 version still works as expected.I guess I could also reboot back to the boot environment - but it was working fine there, so likely a waste of time. All I have to do is look back to the syslog history and I can see the messages are not duplicated before the update.
-
@jrey said in 25.07 unbound - pfblocker - python - syslog:
Step 5: force another restart of ntpd (nothing logged to syslog) it says the syslog is running (there is a entry in the arp table, and states exist)
It still logs locally though? Just not sending to the remote syslog server?
-
said in 25.07 unbound - pfblocker - python - syslog:
Everything ... I can add this to the list of things to try however, Thanks
WTH - (I may have used stronger language here)
but selecting "Everything" and restarting syslogd and unbound seems to have stopped the duplicates.
So now the question is why ? The previously unselected items on the list are not even running.. (their associated log files are still all zero bytes. ). Must be in the syslog handler init (order things start maybe?) or top level when everything is selected vs individual items and you end up with a duplicate handler ?
and since that seemed to work, I've shut the syslog server down yet again to see when it comes back up is the netgate resumes without manual intervention on syslog and unbound.
the answer to that is apparently no - because the syslogd service died
required a restart of both the syslogd and unbound services to make things "normal" again
-
Hmm, WTH indeed!
Good clue though.
Seems like that's probably independent of syslogd dying locally.
-
@stephenw10 said in 25.07 unbound - pfblocker - python - syslog:
It still logs locally though? Just not sending to the remote syslog server?
yes - still locally - local logging hasn't changed or stopped
yes not sending to syslog (ie didn't recover after the syslog server came back up)
after switching to "everything". the duplicates have stopped, but syslogd just outright died again with I took the syslog server down
I restarted both syslogd and unbound still no duplicates at this point.
something is still not playing nice together. syslog itself or syslog/unbound
and looking at the live stream of data coming into the syslog server
I suddenly remembered why I have turned off "everything" and selected the individual services I hadI'm now seeing a /usr/sbin/cron message for newsyslog (checking every minute if it need to roll local files over ) and the application nginx is sending it's log file when I navigate the any web page.
So "everything" likely includes some things that are not included/available in individual selection.
changing anything on the selection list and hitting save, requires a manual restart of unbound looking at the live stream on the syslog server other events are still coming through
I temporarily changed the setting from Everything back to my list so I could grab the appropriate pfSense.conf file that is generated for syslog.d
and then changed it back to Everything -I was looking for a specific reference to either cron and/or nginx logging - leaving the system on the dashboard is currently flooding the syslog stream with the everything being done. But hey no duplicates.
Still looking for clues as to why it might hang/die if the remote server goes off line. And if I could specifically stop the nginx messaging that would be cool,
pondering which is worse actually --- duplicate messages from filterdns/unbound or 100's of nginx messages / minute I will simply never use. sitting on the dashboard = bad, sitting on any static page = no messages
the /usr/sbin/cron (messages) are 1 per minute minumum more if other cron jobs run. /usr/sbin/newsyslog every minute seems a bit aggressive to determine if log files need to roll over.the hanging/dead syslogd is still going to be a problem. can likely "fix" that temporarily with a monitor on the service -
in hind sight I now say to myself you had to push upgrade didn't you. The system was working perfectly with 255+ days of uptime and zero issues. The other side says where is the fun in that.
-
For what it is worth
After sitting turned off for almost 260 days, I started and upgraded a virtual from 2.7.x install to 2.8
the same "duplicate" syslog problem exists there.
when I had the same individual syslog options checked there as I did on my production box) which is the way the virtual was last turned off.duplicates from unbound
changed setting to "everything".
Duplicates gone
and "bonus" cron and nginx messages
