syslog buffer

marcolefo

Hi
I have configured a remote syslog server on my pfsense.
It using udp/514 and I send everything.

The remote log server receive the logs, that's working fine.

But on pfsense the /var/log/system.log is flooded by this message

syslogd: sendto: No buffer space available

I have a lot of free space on my disk.
I have googled the message but I don't find anything interesting.

 netstat -m show
35505/29524/65029 mbufs in use (current/cache/total)
35178/23814/58992/1001064 mbuf clusters in use (current/cache/total/max)
2131/12294 mbuf+clusters out of packet secondary zone in use (current/cache)
0/879/879/500532 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/444915 9k jumbo clusters in use (current/cache/total/max)
0/0/0/333688 16k jumbo clusters in use (current/cache/total/max)
79254K/58525K/137779K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters delayed (4k/9k/16k)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
2 sendfile syscalls
2 sendfile syscalls completed without I/O request
0 requests for I/O initiated by sendfile
0 pages read by sendfile as part of a request
4 pages were valid at time of a sendfile request
0 pages were valid and substituted to bogus page
0 pages were requested for read ahead by applications
0 pages were read ahead by sendfile
0 times sendfile encountered an already busy page
0 requests for sfbufs denied
0 requests for sfbufs delayed

 pkg info pfSense
pfSense-2.6.0
Name           : pfSense
Version        : 2.6.0
Installed on   : Wed Nov  2 11:11:06 2022 CET
Origin         : security/pfSense
Architecture   : FreeBSD:12:amd64
Prefix         : /usr/local
Categories     : security kld
Licenses       : APACHE20
Maintainer     : renato@pfsense.com
WWW            : https://www.pfsense.org/
Comment        : Meta package to install pfSense required ports
Annotations    :
	FreeBSD_version: 1203500
	build_timestamp: 2022-10-05T22:02:31+0000
	built_by       : poudriere-git-3.3.99.20220831
	flavor         : php74
	port_checkout_unclean: no
	port_git_hash  : dbe27cbde8df
	ports_top_checkout_unclean: yes
	ports_top_git_hash: ff3049a2f3c5
	repo_type      : binary
	repository     : pfSense
Flat size      : 10.2KiB
Description    :
Meta package to install pfSense required ports

WWW: https://www.pfsense.org/

Have you an idea ?

bmeeks

This appears to be a NIC or network-level issue. The clue is the mention of mbufs in the log messages. It might be you need to do some tuning using sysctl parameters for the NIC.

You may find some help in the official Netgate documentation here: https://docs.netgate.com/pfsense/en/latest/hardware/tune.html.

stephenw10

Is that the only error logged?

Where is your syslog server, local to the firewall?

Steve

bmeeks

@marcolefo, you are in very good hands with @stephenw10, so I will hand this off to him ... .

marcolefo

Thanks @bmeeks ;)

Hi @stephenw10. Thanks for your help.

The rsyslogd server is on another server (Debian 11)

In /var/log/system.log it's alternating with sshguard :

Nov 18 08:30:00  sshguard[46079]: Exiting on signal.
Nov 18 08:30:00  sshguard[63791]: Now monitoring attacks.
Nov 18 08:33:00  sshguard[63791]: Exiting on signal.
Nov 18 08:33:00  sshguard[89997]: Now monitoring attacks.
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:33:39  syslogd: sendto: No buffer space available
Nov 18 08:35:00  sshguard[89997]: Exiting on signal.
Nov 18 08:35:00  sshguard[445]: Now monitoring attacks.
Nov 18 08:37:00  sshguard[445]: Exiting on signal.
Nov 18 08:37:00  sshguard[3441]: Now monitoring attacks.
Nov 18 08:39:00  sshguard[3441]: Exiting on signal.
Nov 18 08:39:00  sshguard[17002]: Now monitoring attacks.

Today (since midnight) 264 messages from sshguard and only 10 from syslogd
(yesterday 509 and 682)

stephenw10

The sshguard messages are inconvenient but expected. They are triggered when the service restarts each time the logs are rotated. You are seeing that every 3mins so one of your logs is very busy, probably the firewall logs if it's a publicly accessible WAN. You can mitigate that by increasing the log size and/or adding block rules without logging for commonly blocked traffic.

The messages from syslog show that because for some reason it's unable to send messages and has filled the send buffer. You will commonly see that at boot before the interfaces are brought up and syslog can start sending. So I would be looking for something blocking that traffic.
Where is the syslog server in relation to the firewall? Is it on the same local subnet? Routed via a VPN perhaps? If it was and the VPN dropped out for a few seconds that's what you would see.

Steve

marcolefo

@stephenw10

Okay for sshguards. It's of course filter.log :

-rw-------   1 root    wheel    5.6M Nov 18 15:47 filter.log
-rw-------   1 root    wheel    713K Nov 18 15:47 filter.log.0.bz2
-rw-------   1 root    wheel    780K Nov 18 15:45 filter.log.1.bz2
-rw-------   1 root    wheel    699K Nov 18 15:27 filter.log.10.bz2
-rw-------   1 root    wheel    719K Nov 18 15:25 filter.log.11.bz2
-rw-------   1 root    wheel    746K Nov 18 15:23 filter.log.12.bz2
-rw-------   1 root    wheel    744K Nov 18 15:21 filter.log.13.bz2
-rw-------   1 root    wheel    682K Nov 18 15:19 filter.log.14.bz2
-rw-------   1 root    wheel    695K Nov 18 15:17 filter.log.15.bz2
-rw-------   1 root    wheel    801K Nov 18 15:15 filter.log.16.bz2
-rw-------   1 root    wheel    724K Nov 18 15:13 filter.log.17.bz2
-rw-------   1 root    wheel    746K Nov 18 15:11 filter.log.18.bz2
-rw-------   1 root    wheel    703K Nov 18 15:09 filter.log.19.bz2
-rw-------   1 root    wheel    718K Nov 18 15:43 filter.log.2.bz2
-rw-------   1 root    wheel    720K Nov 18 15:07 filter.log.20.bz2
-rw-------   1 root    wheel    710K Nov 18 15:05 filter.log.21.bz2
-rw-------   1 root    wheel    782K Nov 18 15:03 filter.log.22.bz2
-rw-------   1 root    wheel    787K Nov 18 15:01 filter.log.23.bz2
-rw-------   1 root    wheel    771K Nov 18 14:59 filter.log.24.bz2
-rw-------   1 root    wheel    729K Nov 18 14:57 filter.log.25.bz2
-rw-------   1 root    wheel    779K Nov 18 14:55 filter.log.26.bz2
-rw-------   1 root    wheel    808K Nov 18 14:53 filter.log.27.bz2
-rw-------   1 root    wheel    764K Nov 18 14:51 filter.log.28.bz2
-rw-------   1 root    wheel    805K Nov 18 14:49 filter.log.29.bz2
-rw-------   1 root    wheel    767K Nov 18 15:41 filter.log.3.bz2
-rw-------   1 root    wheel    785K Nov 18 15:39 filter.log.4.bz2
-rw-------   1 root    wheel    855K Nov 18 15:37 filter.log.5.bz2
-rw-------   1 root    wheel    726K Nov 18 15:35 filter.log.6.bz2
-rw-------   1 root    wheel    813K Nov 18 15:33 filter.log.7.bz2
-rw-------   1 root    wheel    769K Nov 18 15:31 filter.log.8.bz2
-rw-------   1 root    wheel    768K Nov 18 15:29 filter.log.9.bz2

The log rotation size is fixed at 100000000 bytes (~100 MiB) but it rotates at 12M. But ok I will see that later.

The rsyslog server is on a VLAN which is routed by the pfsense. So pfsense have an interface on the VLAN of the rsyslog server.
I have noticed that the slave pfsense send mail to inform that he is master (but the Carp status on GUI say slave). Perhaps there is a link ?

stephenw10

If the traffic to the syslog server is being NAT'd to the VIP then it would be unable to send if the other node took over as master. That would be logged on both nodes though.

marcolefo

@stephenw10 I will look at my CARP problem and then I will look if the syslog is solved.

Thanks a lot for your time