Firebox LCD Driver for LCDProc

[g.jauch]

@stephenw10:

If you find a solution to that problem please tell us.  :)

It took me a while to find the "real" problem.

As I mentioned my problem occured only after a startup ( reboot, power on ), the LCD-Daemon is running but obviously no Client.

If you simply restart the LCDproc via /usr/local/etc/rc.d/lcdproc.sh after the boot everything is fine and stays fine!

Q: So whats wrong during the startup/system boot ???
A: Both scripts ( /usr/local/etc/rc.d/lcdproc.sh and /usr/local/pkg/lcdproc_client.php ) itself are fine!

The "real problem" is the OS… pfSense since version 2.1 has a "startup" bug!

I must take a deeper look into the the system rc-scripts/configuration.

The only thing i can tell right now is... that since pfSense 2.1 every script under /usr/local/etc/rc.d gets called twice ( with a pause of ~3 seconds ) during the system startup/boot

You can easily modify the lcdproc.sh to prevent a double start, actually there is already some code inside the "rc_start section", but this code seems not to work well!

>>> Here's a video showing the "problem/bug" <<<

stay tuned… i'll be back  8)

cu gunther

[stephenw10]

You'll find it gets started at least twice and that depending on what else you're loading and what speed your CPU is the resulting number of clients and servers can vary.
There is much in the lcdproc-dev thread, for example here: https://forum.pfsense.org/index.php?topic=44034.msg260465#msg260465

Steve

[g.jauch]

@stephenw10:

You'll find it gets started at least twice and that depending on what else you're loading and what speed your CPU is the resulting number of clients and servers can vary.
There is much in the lcdproc-dev thread, for example here: https://forum.pfsense.org/index.php?topic=44034.msg260465#msg260465

Steve

Hi steve,

I updated my previous posting… I forgot the video link!

This problem/bug has nothing to do with LCDproc !!!

I made a simple test script t.sh... and put it under /usr/local/etc/rc.d/
The script simply redirect it's output to a file under /tmp/debug_log.txt

Inside the script i only call the 'date' command and echo the $1 value !!!

Both VM's are fresh installed from the LiveCD... Quick Install, Standard Kernel, no changes!

Under 2.03 the script is processed/called 1 time ( which is correct ) under 2.1 ( and above ) it's processed/called 2 times during the startup !

Why is my script called twice... this doesn't make any sense to me  ???

cu gunther

[stephenw10]

Yes, interesting. I think in fact most of the investigative work done in the lcdproc-dev thread was before 2.1 was even in beta.
One thing to note is that the package system will restart all the packages if an interface goes down or up which can be a problem during boot. It should not do that IMHO. Another thing is that the lcdproc rc script issues a killall for the server and client so even if three clients have been started (which I've seen) they are all killed. Another thing is that the first instance of LCDd is 'un-killable' such that it doesn't get killed by the script and a new instance is started but fails because it can't attch to the parallel port.
Also of note is that we removed the loop script that continually restarted the server and client if it wasn't running. This seemed completely superfluous in 2.0 but since then a time limit on php processes has been introduced (I think) so that the php client will eventually die.

Perhaps the most interesting thing is that this is only a problem with the sdeclcd driver. Users with other LCD types run lcdproc-dev without any issues.

Steve

[g.jauch]

@stephenw10:

One thing to note is that the package system will restart all the packages if an interface goes down or up which can be a problem during boot.

Yes that causes the problem… I wonder how I've overlooked this error messages in system.log all the time :o

Apr 18 22:04:38 pfSense php: rc.newwanip: rc.newwanip: Informational is starting em0.
Apr 18 22:04:38 pfSense php: rc.newwanip: rc.newwanip: on (IP address: 192.168.211.130) (interface: wan) (real interface: em0).
Apr 18 22:04:41 pfSense check_reload_status: Updating all dyndns
Apr 18 22:04:45 pfSense php: rc.bootup: Creating rrd update script
Apr 18 22:04:45 pfSense syslogd: exiting on signal 15
Apr 18 22:04:45 pfSense syslogd: kernel boot file is /boot/kernel/kernel

>>> Apr 18 22:04:45 pfSense php: rc.start_packages: Restarting/Starting all packages.

Apr 18 22:04:46 pfSense login: login on ttyv0 as root
Apr 18 22:04:46 pfSense sshlockout[58995]: sshlockout/webConfigurator v3.0 starting up
Apr 18 22:04:46 pfSense php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Apr 18 22:04:46 pfSense php: rc.newwanip: Creating rrd update script
Apr 18 22:04:48 pfSense php: rc.newwanip: pfSense package system has detected an ip change 192.168.1.10 ->  192.168.211.130 ... Restarting packages.
Apr 18 22:04:48 pfSense check_reload_status: Starting packages
Apr 18 22:04:48 pfSense check_reload_status: Reloading filter

>>> Apr 18 22:04:51 pfSense php: rc.start_packages: Restarting/Starting all packages.

But if you use static IP's for WAN & LAN the restarts aren't triggered ! All is fine then.
The default is DHCP for WAN…

@stephenw10:

It should not do that IMHO.

Fully agree with that!

@stephenw10:

Another thing is that the lcdproc rc script issues a killall for the server and client so even if three clients have been started (which I've seen) they are all killed. Another thing is that the first instance of LCDd is 'un-killable' such that it doesn't get killed by the script and a new instance is started but fails because it can't attch to the parallel port.

Indeed… after the system is up there are 2 LCD's and one "unfinished/waiting/hanging" 'lcdproc.sh start' process running!

[2.1.2-RELEASE][root@pfSense.localdomain]/root(1): ps -auxww | grep -i lcd
root    6656  0.0  0.1  3644  1496  ??  IN    0:00.00 /bin/sh /usr/local/etc/rc.d/lcdproc.sh start
root    7886  0.0  0.1  3300  1136  ??  IN      0:00.00 /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobody
nobody  8224  0.0  0.1  3316  1360  ??  SNs     0:00.26 /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobody

If you kill the LCDd process running as root / PID 7886 in this case… the lcdproc.sh script proceeds and finaly connect to the "real" LCDd and all is good, display gets alive!

@stephenw10:

Also of note is that we removed the loop script that continually restarted the server and client if it wasn't running. This seemed completely superfluous in 2.0 but since then a time limit on php processes has been introduced (I think) so that the php client will eventually die.

I disliked the used rc_start code/logic inside the lcdproc.sh script since I've looked the first time into that file.
IMHO there shouldn't be a kill/termination of the "services" you are trying to start inside that procedure, if something went wrong with that kills you end up ( as you allready mentioned ) with multiple instances of running LCDd and lcdproc_client.php scripts.

Why not simply check inside the rc_start if there is already a LCDd & lcdproc_client.php running and react on that in a proper way.

• If no LCDd is running: Start it, if it's already running then ignore that 2.nd start request completely
• Same thing for the lcdproc_client.php, further… only start the lcdproc_client.php if there is a LCDd running, because without a LCDd the start of the php script is useless

Something like that should be used inside the rc_start segment of lcdproc.sh to be "save":

rc_start() {
if  [ ! `pgrep -anx LCDd` ]; then
	/usr/bin/nice -20 /usr/local/sbin/LCDd -c /usr/local/etc/LCDd.conf -u nobody
fi

if [ ! `pgrep -f lcdproc_client.php` ] -a [ `pgrep -anx LCDd` ];then
	/usr/bin/nice -20 /usr/local/bin/php -f /usr/local/pkg/lcdproc_client.php &
fi
}

@stephenw10:

Perhaps the most interesting thing is that this is only a problem with the sdeclcd driver. Users with other LCD types run lcdproc-dev without any issues.

Maybe they use static IP addresses! It should not be a driver problem.

cu gunther

[stephenw10]

@g.jauch:

It should not be a driver problem.

Indeed it should not yet it seems to be.
My only thoughts on that were that the sdeclcd driver somehow had a delay in initialising the hardware such that it is unable to start/stop in the required time when multiple restarts are called at boot. That might also explain why the driver seems to work better on the X-Core box which has a slower CPU. However no amount of adding or removing delays from the start script seemed to really solve the problem.

Interestingly using the Shellcmd package to start the client and server outside of the package system the commands only get run once. It might be worth looking at how shellcmd is arranged to run only once.

Steve

[g.jauch]

@stephenw10:

Interestingly using the Shellcmd package to start the client and server outside of the package system the commands only get run once. It might be worth looking at how shellcmd is arranged to run only once.

shellcmd add entries to /cf/conf/config.xml and does not use /usr/local/etc/rc.d/

cu gunther

[stephenw10]

Ah, yes of course.

Steve

[g.jauch]

Here's a new way to avoid the "restarts", multi-runs of /usr/local/etc/rc.d files at startup!

At least the way I did it ;)

Apr 20 21:05:23 pfSense php: rc.newwanip: pfSense package system has detected an ip change 192.168.178.45 ->  192.168.178.45 ... Restarting packages.

Strange… why is the restart triggered ??? there is no change in my WAN IP address at all !

So I looked inside the /etc/rc.newwanip script to see whats going on:

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($config['interfaces'][$interface]['ipaddr']))

If any of these 3 conditions are true then the restart_packages() gets triggered.

The first 2 conditions are fine… it's the 3rd who triggers

!is_ipaddrv4($config['interfaces'][$interface]['ipaddr'])

If WAN configuration is set to DHCP then

$config['interfaces'][$interface]['ipaddr']

$config is set to dhcp, and so the is_ipaddrv4("dhcp") returns false, because dhcp is clearly not an IPV4 address.

Till pfSense 2.0.3 this if-condition contained the first 2 conditions only!

So finally I looked inside the /etc/inc/util.inc which contains the implementation of this is_ipaddrv4() function

and added this code inside the function:

if ( $ipaddr == "dhcp" )
	return true;

Now everything works fine… and you can leave the /usr/local/etc/rc.d/lcdproc.sh script as it is.

cu Gunther

[ArkAngel]

Hello,

Thanks for doing all this debugging on the LCDproc-related problems (and globally all you did on the firebox-pfsense project). Your work is much appreciated!

So finally I looked inside the /etc/inc/util.inc which contains the implementation of this is_ipaddrv4() function

and added this code inside the function:

Code: [Select]

if ( $ipaddr == "dhcp" )
return true;

Now everything works fine… and you can leave the /usr/local/etc/rc.d/lcdproc.sh script as it is.

Tried implementing this as you said, inside the is_ipaddrv4() function, but I don't know what I'm doing wrong, when I rebooted the firewall apinger was shown as not running and this is what I had in the logfiles:

Apr 21 08:36:03 apinger: Starting Alarm Pinger, apinger(19845) 
Apr 21 08:36:03 apinger: No usable targets found, exiting 

Here's what the function looks like:

/* returns true if $ipaddr is a valid dotted IPv4 address */
function is_ipaddrv4($ipaddr) {
	if ($ipaddr == "dhcp")
		return true;

	if (!is_string($ipaddr) || empty($ipaddr))
		return false;

	$ip_long = ip2long($ipaddr);
	$ip_reverse = long2ip32($ip_long);

	if ($ipaddr == $ip_reverse)
		return true;
	else
		return false;
}

What I am doing wrong? If I remove the added lines and reboot, everything is back to normal.

Thanks!

ArkAngel

[stephenw10]

Hmm, this is tricky to my head around. More coffee perhaps! (more likely less  :P)

It seems odd that this check is for a valid IP address in the config. If it always failed on dhcp (and presumably, pppoe or any thing other than static) why does it eventually stop renewing the interfaces?
Simply adding that line seems dangerous, though I have no actual code to point to to back that up.

Steve

[g.jauch]

@ArkAngel:

What I am doing wrong? If I remove the added lines and reboot, everything is back to normal.

Thanks!

ArkAngel

Please answer me 2 questions:
1. What is the "Alarm Pinger"… is it a addon package ?
2. What version of LCDProc-Dev & pfSense do you use ?

I'll have a deeper look into it... tommorow ;)

@stephenw10:

Hmm, this is tricky to my head around. More coffee perhaps! (more likely less  :P)

It seems odd that this check is for a valid IP address in the config. If it always failed on dhcp (and presumably, pppoe or any thing other than static) why does it eventually stop renewing the interfaces?

• That is indeed a good question… why is this not turning into an endless loop ?
• I'll also take a look what pfSense 2.0.3 carries in the "$config['interfaces'][$interface]['ipaddr']" Array, I'm pretty sure it's the real IP address that you receive from your DHCP server instead of the word "dhcp".

I asume that after the restart… pfSense versions > 2.0.3 return also the real IP inside that "$config array"... otherwise you would get that mentioned "endless loop" !

@stephenw10:

Simply adding that line seems dangerous, though I have no actual code to point to to back that up.

yes it is !

Another approch could be to touch a file if "pfsense" does that restart and check inside the scripts under /usr/local/etc/rc.d/ if that file is there and if so simply exit and do nothing!

cu gunther

[ArkAngel]

Please answer me 2 questions:
1. What is the "Alarm Pinger"… is it a addon package ?
2. What version of LCDProc-Dev & pfSense do you use ?

I'll have a deeper look into it... tommorow ;)

1. Sorry I didn't noticed where this apinger came from. It is described as "Gateway Monitoring Daemon". The only packages I run is LCDProc-Dev and phpSysInfo.
2. LCDProc-Dev-0.5.6 pkg v. 0.9.7 (lightly modded, as per what I've found in the forum), pfSense 2.1.2-RELEASE (i386)

Fresh installation from a few days ago.

Thanks
ArkAngel

[g.jauch]

Hi,

1.st I should not post nore asume anything on a late evening !
2.nd I should read my previous postings again !

@ArkAngel:

Please answer me 2 questions:
1. What is the "Alarm Pinger"… is it a addon package ?
2. What version of LCDProc-Dev & pfSense do you use ?

I'll have a deeper look into it... tommorow ;)

1. Sorry I didn't noticed where this apinger came from. It is described as "Gateway Monitoring Daemon". The only packages I run is LCDProc-Dev and phpSysInfo.
2. LCDProc-Dev-0.5.6 pkg v. 0.9.7 (lightly modded, as per what I've found in the forum), pfSense 2.1.2-RELEASE (i386)

Fresh installation from a few days ago.

Thanks
ArkAngel

1. Ok, thanks for the info… I'm pretty new to pfSense!
2. that is the same setup I have

OK… here I go again!

@g.jauch:

Till pfSense 2.0.3 this if-condition contained the first 2 conditions only!

So finally I looked inside the /etc/inc/util.inc which contains the implementation of this is_ipaddrv4() function

and added this code inside the function:

if ( $ipaddr == "dhcp" )
	return true;

That was not the best idea… especially if you read the first sentence again!
stephenw10 also replied "It seems odd that this check is for a valid IP address in the config".
This check for a valid IPV4 address should be inside the function that is called after you hit the SAVE-Button inside the webconfigurator!

So leaf the /etc/inc/util.inc untouched… and instead of this if-statement inside the /etc/rc.newwanip

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($config['interfaces'][$interface]['ipaddr']))

get rid of the 3.rd / is_ipaddrv4 condition and try this:

if (!is_ipaddr($oldip) || $curwanip != $oldip )

That should be a better/saver solution, with less side-effects !

If your Alarm Pinger still won't work with that new code… then probably there is something wrong/incompatible with your mentioned other "slightly mods".

I'll take a deeper look into the rest this afternoon:

cu gunther

[g.jauch]

I think the coder of this function made a mistake inside the /etc/rc.newwanip

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($config['interfaces'][$interface]['ipaddr']))

Probably he wanted to check the $curwanip to be IPV4 conform ( that make more sense ).
So right now I use this code, everthing else untouched, all is fine the LCD "is working" after the startup and there shouldn't be any harmful side-effects if you use this code!

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($curwanip)

The script & the whole process itself is fine with the WAN settings! pfSense get all necessary informations from the DHCP and configures the WAN interface.

Apr 22 17:23:26 	php: rc.newwanip: rc.newwanip: Informational is starting sk0.
Apr 22 17:23:26 	php: rc.newwanip: rc.newwanip: on (IP address: 192.168.178.45) (interface: wan) (real interface: sk0).
Apr 22 17:23:26 	php: rc.newwanip: ,,1,dhcp, // thats my debug output of the if() false, false, true !
Apr 22 17:23:27 	php: rc.newwanip: ROUTING: setting default route to 192.168.178.1
Apr 22 17:23:28 	php: rc.bootup: ROUTING: setting default route to 192.168.178.1
Apr 22 17:23:29 	check_reload_status: Updating all dyndns
Apr 22 17:23:32 	php: rc.newwanip: Resyncing OpenVPN instances for interface WAN.
Apr 22 17:23:32 	php: rc.newwanip: Creating rrd update script
Apr 22 17:23:33 	php: rc.bootup: Creating rrd update script
Apr 22 17:23:33 	syslogd: exiting on signal 15
Apr 22 17:23:33 	syslogd: kernel boot file is /boot/kernel/kernel

Apr 22 17:23:34 	php: rc.start_packages: Restarting/Starting all packages.
Apr 22 17:23:34 	php: rc.newwanip: pfSense package system has detected an ip change 192.168.178.45 -> 192.168.178.45 ... Restarting packages.

Everything else inside the rc.newwanip script is fine so pfSense continues to start up… but because of the "dhcp" string the restart is triggered once! But it shouldn't do that... because everything is fine!

cu Gunther

[ArkAngel]

@g.jauch:

I think the coder of this function made a mistake inside the rc.newwanip

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($config['interfaces'][$interface]['ipaddr']))

Probably he wanted to check the $curwanip to be IPV4 conform ( that make more sense ).
So right now I use this code, everthing else untouched, all is fine and there shouldn't be any harmful side-effects!

if (!is_ipaddr($oldip) || $curwanip != $oldip || !is_ipaddrv4($curwanip)

The script & the whole process itself is fine with the WAN settings! pfSense get all necessary informations from the DHCP and configures the WAN interface.

Works fine for me! apinger is happily working and LCDProc-Dev seems to automatically work at startup, instead of having to restart it after each reboot!

Thanks a lot!

ArkAngel

[g.jauch]

@ArkAngel:

Works fine for me! apinger is happily working and LCDProc-Dev seems to automatically work at startup, instead of having to restart it after each reboot!

Thanks a lot!

ArkAngel

No problem… it's obvious that something is wrong in the code so I'm pretty sure that this "glitch" will be fixed in future pfSense versions.

LCDProc-Dev seems to be worst affected by the double start of all scripts under /usr/local/etc/rc.d

cu gunther

[stephenw10]

Interesting I'll have to give this a shot. I still expect the php client to be killed eventaually when it hits the time limit on php processes. That could be worked around by using the Service Watchdog package to monitor it.

Really there are many packages that have no need to be restarted due to an IP change. It seems to me there should be a configuration option to choose if your package is restarted or not. Obviously something for the devs.

Steve

[phil.davis]

Really there are many packages that have no need to be restarted due to an IP change. It seems to me there should be a configuration option to choose if your package is restarted or not. Obviously something for the devs.

Redmine issue raised: https://redmine.pfsense.org/issues/3623
Please read and comment there.
I have thought about this in the past also. In my first thoughts, it seems easiest to just pass an extra parameter in addition to "restart", to indicate the restart type/reason. Then each package can be modified to react differently to the parameter if it cares. All the existing packages that need to restart all the time anyway will not need any modification ("a good thing").

[g.jauch]

Of course it is always better if pfSense itself would inform the scripts under /usr/local/etc/rc.d for what reason they are called instead of the following "workaround".

Hi,

a simple approach to notice if a restart of packages occured could be realized like so:

1. The restart itself is initiated via a function inside /etc/inc/util.inc called send_event(). This function open a socket to a daemon and send "commands" to him.
In our case the command "service reload packages"

2. We can intercept/modify this send_event() function to see if a successfull restart of the packages has been triggered and touch an "event file" to indicate this.

3. Now we can check at the beginning of the scripts under /usr/local/etc/rc.d if the "event file" exists and act to it if needed.

4. At the end we need a "delete script" under /usr/local/etc/rc.d that runs after all other scripts and deletes our "event file" to reset our mechanism back to null

Let's start:

That is the original send_event() function of pfSense 2.1.2

function send_event($cmd) {
	global $g;

	if(!isset($g['event_address']))
		$g['event_address'] = "unix:///var/run/check_reload_status";

	$try = 0;
	while ($try < 3) {
		$fd = @fsockopen($g['event_address']);
		if ($fd) {
			fwrite($fd, $cmd);
			$resp = fread($fd, 4096);
			if ($resp != "OK\n")
				log_error("send_event: sent {$cmd} got {$resp}");
			fclose($fd);
			$try = 3;
		} else if (!is_process_running("check_reload_status"))
			mwexec_bg("/usr/bin/nice -n20 /usr/local/sbin/check_reload_status");
		$try++;
	}
}

here is the modified on:

function send_event($cmd) {
	global $g;

	if(!isset($g['event_address']))
		$g['event_address'] = "unix:///var/run/check_reload_status";

	$try = 0;
	while ($try < 3) {
		$fd = @fsockopen($g['event_address']);
		if ($fd) {
			fwrite($fd, $cmd);
			$resp = fread($fd, 4096);
			if ($resp != "OK\n")
			{
				log_error("send_event: sent {$cmd} got {$resp}");
			}
			else
			{
				if ( $cmd == "service reload packages" )
					mwexec_bg("/usr/bin/touch /tmp/service_reload_packages_running.txt");
			}
			fclose($fd);
			$try = 3;
		} else if (!is_process_running("check_reload_status"))
			mwexec_bg("/usr/bin/nice -n20 /usr/local/sbin/check_reload_status");
		$try++;
	}
}

The modification in detail:

if ($resp != "OK\n")
{
	log_error("send_event: sent {$cmd} got {$resp}");
}
else
{
	if ( $cmd == "service reload packages" )
	mwexec_bg("/usr/bin/touch /tmp/service_reload_packages_running.txt");
}

If the response from the daemon was "OK" ( that means that the transmitted command was successfully executed ) we check if that command was "service reload packages" and touch an "event file" under /tmp called service_reload_packages_running.txt

At the beginning of the scripts under /usr/local/etc/rc.d we can now simply check if this "event file" exist and exit/ignore the restart

if [ -f /tmp/service_reload_packages_running.txt ];then
    exit
fi

Our delete script simply deletes the "event file"

if [ -f /tmp/service_reload_packages_running.txt ];then
    rm /tmp/service_reload_packages_running.txt
fi

We call it zzz_delete.sh to make sure it's called at the end/lastly and put it under /usr/local/etc/rc.d

cu gunther