• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

syslog buffer

Scheduled Pinned Locked Moved General pfSense Questions
9 Posts 3 Posters 742 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M
    marcolefo
    last edited by marcolefo Nov 17, 2022, 4:18 PM Nov 17, 2022, 4:17 PM

    Hi
    I have configured a remote syslog server on my pfsense.
    It using udp/514 and I send everything.

    The remote log server receive the logs, that's working fine.

    But on pfsense the /var/log/system.log is flooded by this message

    syslogd: sendto: No buffer space available
    

    I have a lot of free space on my disk.
    I have googled the message but I don't find anything interesting.

     netstat -m show
    35505/29524/65029 mbufs in use (current/cache/total)
    35178/23814/58992/1001064 mbuf clusters in use (current/cache/total/max)
    2131/12294 mbuf+clusters out of packet secondary zone in use (current/cache)
    0/879/879/500532 4k (page size) jumbo clusters in use (current/cache/total/max)
    0/0/0/444915 9k jumbo clusters in use (current/cache/total/max)
    0/0/0/333688 16k jumbo clusters in use (current/cache/total/max)
    79254K/58525K/137779K bytes allocated to network (current/cache/total)
    0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
    0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
    0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
    0/0/0 requests for jumbo clusters denied (4k/9k/16k)
    2 sendfile syscalls
    2 sendfile syscalls completed without I/O request
    0 requests for I/O initiated by sendfile
    0 pages read by sendfile as part of a request
    4 pages were valid at time of a sendfile request
    0 pages were valid and substituted to bogus page
    0 pages were requested for read ahead by applications
    0 pages were read ahead by sendfile
    0 times sendfile encountered an already busy page
    0 requests for sfbufs denied
    0 requests for sfbufs delayed
    
     pkg info pfSense
    pfSense-2.6.0
    Name           : pfSense
    Version        : 2.6.0
    Installed on   : Wed Nov  2 11:11:06 2022 CET
    Origin         : security/pfSense
    Architecture   : FreeBSD:12:amd64
    Prefix         : /usr/local
    Categories     : security kld
    Licenses       : APACHE20
    Maintainer     : renato@pfsense.com
    WWW            : https://www.pfsense.org/
    Comment        : Meta package to install pfSense required ports
    Annotations    :
    	FreeBSD_version: 1203500
    	build_timestamp: 2022-10-05T22:02:31+0000
    	built_by       : poudriere-git-3.3.99.20220831
    	flavor         : php74
    	port_checkout_unclean: no
    	port_git_hash  : dbe27cbde8df
    	ports_top_checkout_unclean: yes
    	ports_top_git_hash: ff3049a2f3c5
    	repo_type      : binary
    	repository     : pfSense
    Flat size      : 10.2KiB
    Description    :
    Meta package to install pfSense required ports
    
    WWW: https://www.pfsense.org/
    

    Have you an idea ?

    1 Reply Last reply Reply Quote 0
    • B
      bmeeks
      last edited by Nov 17, 2022, 5:38 PM

      This appears to be a NIC or network-level issue. The clue is the mention of mbufs in the log messages. It might be you need to do some tuning using sysctl parameters for the NIC.

      You may find some help in the official Netgate documentation here: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html.

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Nov 17, 2022, 9:53 PM

        Is that the only error logged?

        Where is your syslog server, local to the firewall?

        Steve

        M 1 Reply Last reply Nov 18, 2022, 8:31 AM Reply Quote 1
        • B
          bmeeks
          last edited by Nov 17, 2022, 11:01 PM

          @marcolefo, you are in very good hands with @stephenw10, so I will hand this off to him ... 🙂.

          1 Reply Last reply Reply Quote 1
          • M
            marcolefo @stephenw10
            last edited by marcolefo Nov 18, 2022, 8:32 AM Nov 18, 2022, 8:31 AM

            Thanks @bmeeks ;)

            Hi @stephenw10. Thanks for your help.

            The rsyslogd server is on another server (Debian 11)

            In /var/log/system.log it's alternating with sshguard :

            Nov 18 08:30:00  sshguard[46079]: Exiting on signal.
            Nov 18 08:30:00  sshguard[63791]: Now monitoring attacks.
            Nov 18 08:33:00  sshguard[63791]: Exiting on signal.
            Nov 18 08:33:00  sshguard[89997]: Now monitoring attacks.
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:33:39  syslogd: sendto: No buffer space available
            Nov 18 08:35:00  sshguard[89997]: Exiting on signal.
            Nov 18 08:35:00  sshguard[445]: Now monitoring attacks.
            Nov 18 08:37:00  sshguard[445]: Exiting on signal.
            Nov 18 08:37:00  sshguard[3441]: Now monitoring attacks.
            Nov 18 08:39:00  sshguard[3441]: Exiting on signal.
            Nov 18 08:39:00  sshguard[17002]: Now monitoring attacks.
            

            Today (since midnight) 264 messages from sshguard and only 10 from syslogd
            (yesterday 509 and 682)

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by Nov 18, 2022, 1:05 PM

              The sshguard messages are inconvenient but expected. They are triggered when the service restarts each time the logs are rotated. You are seeing that every 3mins so one of your logs is very busy, probably the firewall logs if it's a publicly accessible WAN. You can mitigate that by increasing the log size and/or adding block rules without logging for commonly blocked traffic.

              The messages from syslog show that because for some reason it's unable to send messages and has filled the send buffer. You will commonly see that at boot before the interfaces are brought up and syslog can start sending. So I would be looking for something blocking that traffic.
              Where is the syslog server in relation to the firewall? Is it on the same local subnet? Routed via a VPN perhaps? If it was and the VPN dropped out for a few seconds that's what you would see.

              Steve

              M 1 Reply Last reply Nov 18, 2022, 3:26 PM Reply Quote 0
              • M
                marcolefo @stephenw10
                last edited by Nov 18, 2022, 3:26 PM

                @stephenw10

                Okay for sshguards. It's of course filter.log :

                -rw-------   1 root    wheel    5.6M Nov 18 15:47 filter.log
                -rw-------   1 root    wheel    713K Nov 18 15:47 filter.log.0.bz2
                -rw-------   1 root    wheel    780K Nov 18 15:45 filter.log.1.bz2
                -rw-------   1 root    wheel    699K Nov 18 15:27 filter.log.10.bz2
                -rw-------   1 root    wheel    719K Nov 18 15:25 filter.log.11.bz2
                -rw-------   1 root    wheel    746K Nov 18 15:23 filter.log.12.bz2
                -rw-------   1 root    wheel    744K Nov 18 15:21 filter.log.13.bz2
                -rw-------   1 root    wheel    682K Nov 18 15:19 filter.log.14.bz2
                -rw-------   1 root    wheel    695K Nov 18 15:17 filter.log.15.bz2
                -rw-------   1 root    wheel    801K Nov 18 15:15 filter.log.16.bz2
                -rw-------   1 root    wheel    724K Nov 18 15:13 filter.log.17.bz2
                -rw-------   1 root    wheel    746K Nov 18 15:11 filter.log.18.bz2
                -rw-------   1 root    wheel    703K Nov 18 15:09 filter.log.19.bz2
                -rw-------   1 root    wheel    718K Nov 18 15:43 filter.log.2.bz2
                -rw-------   1 root    wheel    720K Nov 18 15:07 filter.log.20.bz2
                -rw-------   1 root    wheel    710K Nov 18 15:05 filter.log.21.bz2
                -rw-------   1 root    wheel    782K Nov 18 15:03 filter.log.22.bz2
                -rw-------   1 root    wheel    787K Nov 18 15:01 filter.log.23.bz2
                -rw-------   1 root    wheel    771K Nov 18 14:59 filter.log.24.bz2
                -rw-------   1 root    wheel    729K Nov 18 14:57 filter.log.25.bz2
                -rw-------   1 root    wheel    779K Nov 18 14:55 filter.log.26.bz2
                -rw-------   1 root    wheel    808K Nov 18 14:53 filter.log.27.bz2
                -rw-------   1 root    wheel    764K Nov 18 14:51 filter.log.28.bz2
                -rw-------   1 root    wheel    805K Nov 18 14:49 filter.log.29.bz2
                -rw-------   1 root    wheel    767K Nov 18 15:41 filter.log.3.bz2
                -rw-------   1 root    wheel    785K Nov 18 15:39 filter.log.4.bz2
                -rw-------   1 root    wheel    855K Nov 18 15:37 filter.log.5.bz2
                -rw-------   1 root    wheel    726K Nov 18 15:35 filter.log.6.bz2
                -rw-------   1 root    wheel    813K Nov 18 15:33 filter.log.7.bz2
                -rw-------   1 root    wheel    769K Nov 18 15:31 filter.log.8.bz2
                -rw-------   1 root    wheel    768K Nov 18 15:29 filter.log.9.bz2
                

                The log rotation size is fixed at 100000000 bytes (~100 MiB) but it rotates at 12M. But ok I will see that later.

                The rsyslog server is on a VLAN which is routed by the pfsense. So pfsense have an interface on the VLAN of the rsyslog server.
                I have noticed that the slave pfsense send mail to inform that he is master (but the Carp status on GUI say slave). Perhaps there is a link ?

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by Nov 18, 2022, 4:06 PM

                  If the traffic to the syslog server is being NAT'd to the VIP then it would be unable to send if the other node took over as master. That would be logged on both nodes though.

                  M 1 Reply Last reply Nov 18, 2022, 4:20 PM Reply Quote 1
                  • M
                    marcolefo @stephenw10
                    last edited by Nov 18, 2022, 4:20 PM

                    @stephenw10 I will look at my CARP problem and then I will look if the syslog is solved.

                    Thanks a lot for your time

                    1 Reply Last reply Reply Quote 1
                    9 out of 9
                    • First post
                      9/9
                      Last post
                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                      This community forum collects and processes your personal information.
                      consent.not_received