Alix remote Syslog not working after reboot

[The Computer Guy]

I've seen a couple of posts on this, but I'm still getting this issue.

I have 3 x pfSense firewalls running on Alix boards and all three have the same issue, I use 2 x Alix boards in rackmount cases, one runs pfSense, and the other runs Voyage Linux and acts as a syslog and webserver for the pfSense box.

If the pfSense box reboots, syslog stops working, the first two messages under the logs are -

May 11 19:38:50 syslogd: sendto: Network is unreachable
May 11 19:38:50 syslogd: kernel boot file is /boot/kernel/kernel

I had a look in /etc/rc.bootup and noticed that the following code is quite close to the end of the file, however, it still doesn't work until I go into the settings page, take the tick out of Send to remote server, put the tick back in there and click on save.

/* Give syslogd a kick after everything else has been initialized, otherwise it can occasionally
   fail to route syslog messages properly on both IPv4 and IPv6 */
system_syslogd_start();

The system I've been testing on now is -

2.1-RELEASE (i386)
built on Wed Sep 11 18:16:44 EDT 2013
FreeBSD 8.3-RELEASE-p11

But the third install is on the latest version with the same issue.

[The Computer Guy]

If no-one has any suggestions what to do, anyone got any suggestions on how I could get pfSense to e-mail me when it boots? So that I know there's been a power cycle?

[pvoigt]

I have just stumbled over this thread - right before I intended to open a closely related new thread.

I can confirm some strange change of remote syslog behavior since I have moved to 2.1.x: When I cold boot or warm boot I can observe that all kernel messages are lost. All other messages are sent as expected. I am currently running 2.1.3.

In detail my pfsense machine shows in /var/log/system.log:

syslogd: sendto: Network is unreachable
syslogd: kernel boot file is /boot/kernel/kernel
kernel: Copyright (c) 1992-2012 The FreeBSD Project.
kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
kernel: The Regents of the University of California. All rights reserved.
kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
kernel: FreeBSD 8.3-RELEASE-p16 #0: Thu May  1 16:17:33 EDT 2014
...

while my remote syslog server shows:

syslogd: restart
syslogd: kernel boot file is /boot/kernel/kernel
kernel: Starting syslog...
kernel: done.
...

I conclude from the logs that for unknown reason my remote syslog server cannot be reached while kernel is loading and sending messages.

My syslog part of /cf/conf/config.xml is obviously not very special:

 <syslog><nentries>50</nentries>
        <remoteserver><my-syslog-server-ip< remoteserver=""><remoteserver2><remoteserver3><logall><enable><sourceip><ipproto>ipv4</ipproto></sourceip></enable></logall></remoteserver3></remoteserver2></my-syslog-server-ip<></remoteserver></syslog> 

The IP of my syslog server is an IPv4 address within the LAN. The LAN is a VLAN and my pfsense machine is connected to the syslog server by a VLAN aware L2 switch. I am losing kernel messages on both an ALIX.2D13 and a Soekris net6501.

So I am at least glad that I am just losing kernel messages only while your issue seems more serious.

From your mentioned comment in /etc/rc.bootup one may conclude that there a some issues left with remote syslogging. Does anybody else observe losing messages sent to remote syslog servers with pfsense 2.1.x?

Regards,
Peter

[The Computer Guy]

I did a bit more playing around with it, and I've noticed it's not ALL that disappear, it's been a few days, but it would appear that its mainly the messages, and filter logs that I'm not getting.

Are these what you mean by Kernel messages?

[pvoigt]

@The:

I did a bit more playing around with it, and I've noticed it's not ALL that disappear, it's been a few days, but it would appear that its mainly the messages, and filter logs that I'm not getting.

Are these what you mean by Kernel messages?

No, I mean all those lines containing "kernel: ". Please see my first code box in my last post. It is cleared from date and hostname information in the first two columns.

I  can even be more precise now after comparing the entries in /var/log/system.log on the pfSense box and in my remote syslog file: Only kernel messages emitted during the boot process get lost whereas subsequent kernel messages are correctly sent to the syslog server. I am thinking about opening a new thread on this specific topic.

Regards,
Peter

[cmb]

The logs generated before the system is up on the network will indeed fail to send as it can't reach the syslog server, and they're UDP, so they just disappear. That's always been the case.

[The Computer Guy]

That is different from my problem then sorry.

I'm not fussed that I'm losing logs before the system fires up, but I am concerned that I'm losing logs thereafter, I'm more concerned about traffic logs. These firewalls are used to control public traffic through captive portals, so it's important that I keep logs for my customers in case the network is used for illegal activities.

My Syslog server is setup to create a new folder every day, and splits the logs from the firewall into different firewalls. I can see when a firewall has been restarted as no new folders are created, and the logs last modified date is roughly the same time the firewall went off.

The only way to get it going again, is to untick "Send to remote server" and tick it again and click save. Folders and logfiles are then generated instantly on the log server.

[pvoigt]

@cmb:

The logs generated before the system is up on the network will indeed fail to send as it can't reach the syslog server, and they're UDP, so they just disappear. That's always been the case.

Hm, I cannot verify anymore because I have upgraded all machines but I remember pfSense 2.0.x sending kernel boot messages to a remote syslog server. Thanks for clarifying that my observation under pfSense 2.1.x is the expected behavior preventing me from making unnecessary noice in the forum  :)

Isn't there an easy way of caching kernel boot messages until the remote syslog server can be reached?

Regards,
Peter

[The Computer Guy]

Sorry to bump this back to the top, but I'm still having this issue, and it's also happening on a new APU 1.C board running the latest version of pfSense.

If I cannot resolve this issue, I would like to at least know when a pfSense box is rebooted, is there a way I can be notified of a reboot? Sendmail? What about having the pfSense box open a url on my website, which I can code to notify me that it's been opened etc etc?