Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Syslog service in pfSense v2.8.1 often stop itself

    Scheduled Pinned Locked Moved General pfSense Questions
    64 Posts 18 Posters 10.1k Views 16 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • provelsP Offline
      provels @stephenw10
      last edited by provels

      @stephenw10 Got a notification that syslogd was restarted at 00:15 today. Looks like the previous default.log gzipped at 23:50, so had that been what stopped syslogd, watchdog would have caught it a minute later.
      SYSTEM LOG from last night to present

      Oct 3 03:07:59	php-fpm	70563	/index.php: Successful login for user 'admin' from: 192.168.0.82 (Local Database)
Oct 3 03:01:00	root	93209	rc.update_bogons.sh is sleeping for 30028
Oct 3 03:01:00	root	92013	rc.update_bogons.sh is starting up.
Oct 3 01:01:00	php-cgi	63710	rc.dyndns.update: phpDynDNS (mydom.ddns.net): No change in my IP address and/or 25 days has not passed. Not updating dynamic DNS entry.
Oct 3 00:15:03	php-cgi	55524	notify_monitor.php: Message sent to admin@mydom.net OK
Oct 3 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 2 21:47:53	php-fpm	44887	/index.php: User logged out for user 'admin' from: 192.168.0.82 (Local Database)

      SYSLOG-NG

      Oct 3 00:00:00 fw syslog-ng[50722]: Configuration reload finished;
Oct 3 00:00:00 fw syslog-ng[50722]: Configuration reload request received, reloading configuration;
Oct 3 00:10:00 fw syslog-ng[50722]: Log statistics; processed='destination(_DEFAULT)=319', dropped='global(internal_source)=0', processed='global(internal_source)=319', queued='global(internal_source)=0', processed='global(msg_clones)=0', processed='source(_DEFAULT)=319', processed='src.internal(_DEFAULT#0)=319', processed='global(sdata_updates)=0', stamp='src.internal(_DEFAULT#0)=1759467600', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=312', processed='center(queued)=319', processed='center(received)=319', queued='global(scratch_buffers_bytes)=0'
Oct 3 00:15:02 localhost syslogd: kernel boot file is /boot/kernel/kernel
Oct 3 00:15:02 localhost syslogd: restart
Oct 3 00:20:00 fw syslog-ng[50722]: Log statistics; processed='destination(_DEFAULT)=359', dropped='global(internal_source)=0', processed='global(internal_source)=320', queued='global(internal_source)=0', processed='global(msg_clones)=0', processed='source(_DEFAULT)=359', processed='src.internal(_DEFAULT#0)=320', processed='global(sdata_updates)=0', stamp='src.internal(_DEFAULT#0)=1759468200', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=313', processed='center(queued)=359', processed='center(received)=359', queued='global(scratch_buffers_bytes)=0'

      Peder

      MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
      BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

      1 Reply Last reply Reply Quote 0
      • stephenw10S Offline
        stephenw10 Netgate Administrator
        last edited by

        Hmm, well if syslog-ng is restarting that would certainbly explain why syslogd sees the refusals and hence ends up stopping. But I don't know why syslog-ng would be doing that,

        1 Reply Last reply Reply Quote 0
        • D Offline
          DBMandrake @sokeada
          last edited by DBMandrake

          @sokeada Noticed this problem a couple of times myself over the last two weeks.

          I don't use syslog-ng but do log System Events, General Authentication Events and VPN Events to a remote syslog server on a LibreNMS server.

          Although I don't have the data to confirm it, after reading this thread the failures very likely correlate with a reboot of that remote server.

          1 Reply Last reply Reply Quote 1
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            The workaround firewall rules should work for you there.

            1 Reply Last reply Reply Quote 0
            • A Offline
              AlexanderK
              last edited by

              i am facing the same issue.
              Any fix expept from watchdog and rules?

              provelsP 1 Reply Last reply Reply Quote 0
              • provelsP Offline
                provels @AlexanderK
                last edited by provels

                @AlexanderK At least with Watchdog you'll get informed when it happens. That will give you something to work with to try tracking it down in the logs. That said, mine stopped again a few days ago, coincidentally at 00:15 again (marked XXXX). Perhaps it is some timing issue in syslogd that works only 95% of the time? There re plenty of instances of syslogd exiting and successfully restarting a second later, but sometimes it doesn't without a kick in the pants. In any case, I'll stick with watchdog for the present, won't hurt. Just adding to the conversation, not a big issue for me.

                XXXX Oct 14 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:45:53	syslogd		sendto: Connection refused
Oct 8 02:45:53	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:45:52	syslogd		exiting on signal 15
Oct 8 02:45:22	syslogd		sendto: Connection refused
Oct 8 02:45:22	syslogd		kernel boot file is /boot/kernel/kernel
Oct 8 02:44:27	syslogd		exiting on signal 15
Oct 8 02:44:21	syslogd		sendto: Connection refused
XXXX Oct 3 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Sep 30 17:09:30	syslogd		sendto: Connection refused

                Peder

                MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                1 Reply Last reply Reply Quote 0
                • stephenw10S Offline
                  stephenw10 Netgate Administrator
                  last edited by

                  If it's actually external you can add the work-around stateless floating rules to prevent the connection refused message.

                  provelsP 1 Reply Last reply Reply Quote 0
                  • provelsP Offline
                    provels @stephenw10
                    last edited by

                    @stephenw10 If directed at me, it's not external. It's a second HDD internal to the FW, mounted as a directory on the system SSD. Not complaining.

                    Peder

                    MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                    BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S Offline
                      stephenw10 Netgate Administrator
                      last edited by

                      No sorry that was at the previous poster. The workaround rule won't work for traffic to syslog-ng locally. 😕

                      1 Reply Last reply Reply Quote 0
                      • provelsP Offline
                        provels
                        last edited by

                        Stopped again this AM at 00:15, random interval. Maybe something to do with daily log rotation, GZipping the log, dunno. Just info, not an issue for me anyway.

                        6 Matched General Log Entries. (Maximum 500)
Oct 23 00:15:02	php-cgi	95349	notify_monitor.php: Message sent to provels
Oct 23 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 14 00:15:03	php-cgi	10330	notify_monitor.php: Message sent to provels
Oct 14 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel
Oct 3 00:15:03	php-cgi	55524	notify_monitor.php: Message sent to provels
Oct 3 00:15:02	syslogd		kernel boot file is /boot/kernel/kernel

                        The top of today's default.log.

                        Oct 23 00:00:00 fw syslog-ng[13248]: Configuration reload request received, reloading configuration;
Oct 23 00:00:00 fw syslog-ng[13248]: Configuration reload finished;
Oct 23 00:10:00 fw syslog-ng[13248]: Log statistics; processed='destination(_DEFAULT)=183', dropped='global(internal_source)=0', processed='global(internal_source)=183', queued='global(internal_source)=0', processed='global(msg_clones)=0', processed='source(_DEFAULT)=183', processed='src.internal(_DEFAULT#0)=183', processed='global(sdata_updates)=0', stamp='src.internal(_DEFAULT#0)=1761195600', queued='global(scratch_buffers_count)=0', processed='global(payload_reallocs)=178', processed='center(queued)=183', processed='center(received)=183', queued='global(scratch_buffers_bytes)=0'
Oct 23 00:15:02 localhost syslogd: restart
Oct 23 00:15:02 localhost syslogd: kernel boot file is /boot/kernel/kernel
Oct 23 00:15:02 localhost php-cgi[95349]: notify_monitor.php: Message sent to provels

                        Peder

                        MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                        BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                        1 Reply Last reply Reply Quote 1
                        • A Offline
                          aldomoro
                          last edited by

                          Hi

                          We use Graylog as remote syslog. If server with Graylog has outage, e.g. is restarted due to updates, syslogd is stopped in pfSense 2.8.1. We did not have this issue in v2.8.0.

                          Aldomoro

                          provelsP 1 Reply Last reply Reply Quote 0
                          • stephenw10S Offline
                            stephenw10 Netgate Administrator
                            last edited by

                            Yes, that's the bug discussed here. The workaround rules will prevent it. https://redmine.pfsense.org/issues/16362#note-5

                            1 Reply Last reply Reply Quote 0
                            • provelsP Offline
                              provels @aldomoro
                              last edited by

                              @aldomoro Possibly the best use of Service Watchdog. Maybe the only one! :)

                              Peder

                              MAIN - pfSense+ 25.07.1-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                              BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                              1 Reply Last reply Reply Quote 0
                              • G Offline
                                geovaneg
                                last edited by geovaneg

                                Hi,

                                Same problem here:

                                "Nov 2 22:00:02 pfsense syslogd: sendto: Connection refused" (system.log)

                                PfSense CE 2.8.1, remote logging enabled.

                                Anothers instances 2.8 running OK.

                                Workaround: whatchdog

                                Thanks.

                                Geovane

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.