Remote syslog strangeness
bladesm last edited by
Posted in general since it doesn't seem to fit any of the specific categories. Apologies for the long, rambling post, but I wanted to cover all the bases and explain how much troubleshooting I have done.
I am attempting to set up a pfsense firewall (2.2.4 i386, freshly upgraded this morning from another recent build) to send logs to a remote syslog server. The syslog server is running rsyslog and is accepting logs from a number of other hosts on the network quite happily.
I have configured a remote syslog target on the firewall via the web interface and told it to log everything but nothing is received at the syslog server, but all events are stored quite happily in the local logs.
I did some digging into the problem. I can SSH in to the firewall and use the logger utility to send logs to the local store (logger blah) and to send logs to the remote server (logger -h syslog.internal blah). When sending locally, the logs are received locally. When sending remotely, the logs are received locally and on the remote server.
I run tcpdump on the firewall and on the syslog server and, when log events are generated internally on the firewall, I can see packets leaving the firewall - but the tcpdump on the syslog server does not show them arriving.
I can use nc on the firewall to manually send garbage to the syslog server, which the syslog server receives. Pinging from the firewall to the syslog server and vice versa works fine. I can see syslog messages from hosts outside the network being passed on by the firewall to the syslog server without any problems.
I have reset the log files; rebooted the firewall; turned logging to everything, to single facilities, to all facilities; specified the syslog server by hostname (DNS is working) and by IP address; specified a :port in the syslog server and not specified a port; checked that the /cf/conf/config.xml contains the right hostname/IP address (it does); checked that /var/etc/syslog.conf contains the right hostname/IP address (it does).
If I configure pfsense to send logs to an additional server on which I have a simple nc listener running, I can see logs coming in, but the same logs which should be sent to the syslog server are seemingly not received.
The only difference I can note between the internal pfsense-generated syslog messages and those generated by either logger or by a remote system being forwarded on via the firewall are that the pfsense-generated messages are coming from source port 514 while others come from random high ports.
My next step will be to sacrifice a small animal in the name of some ancient God. I've done everything else I can think of, so it's probably worth a shot. In the interests of saving the lives of small animals, if anyone has any suggestions, I'll be more than happy to hear them.
*** Edit ***
Looks like it might be some rsyslog / Ubuntu / apparmour strangeness. I used ncat from a third server to try to send messages to the syslog server on 514 with source port 514, and the messages did not show up in tcpdump. When I allow ncat to randomly select the source port, the message gets through.
I will carry on digging into the strangeness, but I wonder, is there any way to configure pfsense to use random high source ports when generating messages?
*** Edit 2 ***
After a reasonable amount of debugging, I have tracked down the problem - but not found a true solution. It would appear that packets with a matching source and destination port are not handled well and appear to be discarded. I tested between the firewall (running pfsense 2.2.4) and an Ubuntu 14.04 server (kernel 3.13.0-55), and reliably reproduced the symptoms.
Thinking back to my computer science networking classes some 20 years ago, I can't seem to recall any rules within IPv4 stating that packets from one host to another must not have matching source and destination ports, so I can only put this down to… experience, I guess.
In the way of a solution, I have moved my syslog server onto a high port and the logs are once more flowing. In the longer term, it might be nice if pfsense's syslog agent were to send packets with a randomly-selected high port rather than 514, unless there is some good reason for doing it this way.