LCDProc 0.5.4-dev

m4f1050 · Jan 23, 2012, 3:33 AM

Here is the X700's System Log pertaining LCDproc 0.5.5 v0.8


Jan 22 22:26:33 LCDd: Connect from host 127.0.0.1:42443 on socket 13
Jan 22 22:26:31 LCDd: Connect from host 127.0.0.1:61742 on socket 12
Jan 22 22:26:29 LCDd: Listening for queries on 127.0.0.1:13666
Jan 22 22:26:29 LCDd: Using Configuration File: /usr/local/etc/LCDd.conf
Jan 22 22:26:29 LCDd: LCDd version 0.5.5 starting
Jan 22 22:26:26 php: lcdproc: Sync: End package sync
Jan 22 22:26:26 php: lcdproc: Sync: Begin package sync
Jan 22 22:26:26 php: lcdproc: Sync: End package sync
Jan 22 22:26:26 php: lcdproc: Sync: Begin package sync
Jan 22 22:26:26 php: : Restarting/Starting all packages.
Jan 22 22:26:21 php: lcdproc: Failed to connect to LCDd process Operation timed out (60)
Jan 22 22:26:20 check_reload_status: Starting packages
Jan 22 22:26:20 php: lcdproc: Failed to connect to LCDd process Operation timed out (60)

top -S for LCDd


PID.....USERNAME.THR.PRI.NICE.SIZE...RES....STATE.TIME.WCPU.COMMAND
27586.nobody......1.....74..r30...3368K.1488K.RUN.....0:00.0.00%.LCDd

tix · Jan 23, 2012, 2:57 PM

I turned LCDd on overnight and left my SSH sessions active running top -SH
This morning I have an unresponsive machine except the SSH sessions are still active. (Also, everyone was asleep so there was no usage from LAN to WAN.)

I killed the LCDd process after capturing the information below and load drops to less than .10 but I cannot restart it. If I use the webif it will generate the previously mentioned log entries and the same occurs starting it from the CLI.

top -SH

last pid: 31188;  load averages:  7.99,  7.75,  6.91        up 0+17:13:06  08:55:32
92 processes:  11 running, 68 sleeping, 1 zombie, 12 waiting
CPU: 18.9% user,  0.0% nice, 80.6% system,  0.5% interrupt,  0.0% idle
Mem: 62M Active, 15M Inact, 39M Wired, 34M Buf, 118M Free
Swap: 512M Total, 512M Free

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
34608 nobody    74  r30  3368K  1488K RUN     77:11 100.00% LCDd
   10 root     171 ki31     0K     8K RUN    797:50  0.00% idle
   11 root     -28    -     0K    96K WAIT     1:57  0.00% {swi5: +}
45412 root      44    0  6588K  4776K kqread   0:56  0.00% lighttpd
   11 root     -32    -     0K    96K WAIT     0:50  0.00% {swi4: clock}
    0 root     -16    0     0K    48K sched    0:44  0.00% {swapper}
34700 root      76    0 33092K 16244K RUN      0:31  0.00% php
45906 root      44    0  3712K  1960K RUN      0:31  0.00% top
32600 root      76    0 33092K 16136K RUN      0:31  0.00% php
40548 root      76   20  3656K  1492K piperd   0:14  0.00% sh
   13 root     -16    -     0K     8K -        0:13  0.00% yarrow
24126 root      44    0  4948K  2516K select   0:12  0.00% syslogd
19804 root      44    0  7992K  3604K select   0:11  0.00% sshd
   19 root      44    -     0K     8K syncer   0:07  0.00% syncer
35740 root      48    0 37188K 18800K piperd   0:07  0.00% php
21686 root      44    0  3316K   924K piperd   0:06  0.00% logger

System.log tail

 tail /var/log/system.log
Jan 23 02:35:21 pfsense dnsmasq[6805]: possible DNS-rebind attack detected: appsforbb.com
Jan 23 02:58:07 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 03:26:29 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 03:58:37 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 05:34:21 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 06:34:48 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 07:35:19 pfsense dnsmasq[6805]: read /etc/hosts - 31 addresses
Jan 23 07:59:59 pfsense dhclient: RENEW
Jan 23 07:59:59 pfsense dhclient: Creating resolv.conf

stephenw10 · Jan 23, 2012, 3:11 PM

Further confirmation of a problem. Thanks.
By the way pfSense uses circular logs so you have to use clog to tail them.
See: http://doc.pfsense.org/index.php/LOGS:Why_can%27t_I_view_view_log_files_with_cat/grep/etc%3F%28clog%29

Steve

m4f1050 · Jan 23, 2012, 6:39 PM

Well, today when I woke up and saw my Watchguard I noticed the "Thanks for using pfSense" and hung LCDd… I can restart it, but it would be nice for it not to crash. Will keep testing... All I did was add Interface (and on the dropdown kept WAN)

mdima · Jan 23, 2012, 7:41 PM

Hello everybody,
I just updated the package. The news are the following:

Fix the uptime screen
In the settings, "Enable LCDproc" becomes "Enable LCDproc at startup"
Added "After Install Info" message
Limited the client loop to three times. After three errors connecting to the LCDd service the client will end

I hope the 4th point will solve the problems for the Firebox boxes! Just wait for the 0.9 version of the package and update it…

Thanks to all for the feedbacks,
Michele

EDIT: With a feedback, please post the driver/port you're using, the values of the options of the service and the list of the screens active, thanks!!

power_matz · Jan 24, 2012, 11:04 AM

Hello,

can I leave the LED backlight on all the time? Is this implemented in this update?
Or will it come in a future version?

M.

stephenw10 · Jan 24, 2012, 11:55 AM

@power_matz: The backlight control is part of the driver not the package so this won't have changed.

@mdima: Awesome work! ;D I can't wait to test it.

Steve

fmertz · Jan 24, 2012, 3:06 PM

@power_matz:

can I leave the LED backlight on all the time? Is this implemented in this update?

We are currently focusing on fixing the issue of multiple instances of the server, as well as building the code for driving the LEDs. Once that is stable, we can revisit the user parameters.

tix · Jan 24, 2012, 4:23 PM

Well the update didn't help for me… :(


Mem: 44M Active, 15M Inact, 37M Wired, 28M Buf, 139M Free
Swap: 512M Total, 512M Free

48731 nobody    74  r30  3368K  1488K RUN     77:11 100.00% LCDd

Running clog of /var/log/system.log showed no entries related to the LCD processes.

However it DID continue to work properly this time although the screens changed about every 7 secs or so instead of 5 secs with the webif non-responsive and new SSH sessions not completing to connect. Existing SSH sessions continue just slowly as expected under the load.

For reference all my settings for the Firebox x700 are "default" with "2x20 columns" and the SDEC driver on /dev/lpt0. The only screens enabled are: Uptime, Load, States, Mbuf, & Interface Traffic (WAN).
GW monitoring is disabled.

EDIT: Forgot to add

like the new Uptime screen! Much cleaner.
in my set up, it seems to be failing after around 16-18 hours uptime and/or around 4-5am, not sure which yet, with no network activity occurring.

mdima · Jan 24, 2012, 4:36 PM

@tix:

For reference all my settings for the Firebox x700 are "default" with "2x20 columns" and the SDEC driver on /dev/lpt0. The only screens enabled are: Uptime, Load, States, Mbuf, & Interface Traffic (WAN).
GW monitoring is disabled.

Hi Tix,
thanks for the info! I am now running my secondary machine with the same screens in order to test the "client" part of the package. If there is something wrong in the client for this screens I will find out soon. Of course I can only test the client because I use a different driver… but if the test goes OK it means some problem in LCDd 0.5.5 or the driver.

@tix:

EDIT: Forgot to add

like the new Uptime screen! Much cleaner.

in my set up, it seems to be failing after around 16-18 hours uptime and/or around 4-5am, not sure which yet, with no network activity occurring.

Thanks for apreciating! I will see tomorrow how my secondary machine is doing…

Ciao,
Michele

m4f1050 · Jan 24, 2012, 4:41 PM

You think you can program more stuff on the LCDproc MENU?

Maybe stuff like webGUI restart, backup config, restore config, reboot system

Thanks! Doing an awesome job!

To all, I am using GWXepc for the Arm/Disarm LED to turn it green when pfSense comes up and red when it goes down.
What I did was edit the beep.sh and in "start" I added GWXepc -l green and in "stop" I added GWXepc -l red

:)

stephenw10 · Jan 24, 2012, 6:24 PM

I've now been running 0.54 versions of LCDd and sdeclcd for 24hrs and am still experiencing the same LCDd CPU usage lock out. So I think we can rule out going to 0.55 as a problem.
Interestingly I was able to observe it happening this afternoon and it ramps up slowly as if it's looping around creating steadily more and more processes until it hits 100% cpu.

One way to test this would be to compile the old driver against 0.55 and run that. I don't have a suitable compile environment setup at the moment though.

Steve

mdima · Jan 24, 2012, 6:32 PM

Steve,
how many instances of the client were running at that time? Do you have any log that evidence problems or concurrent clients running in the same time?

Thanks,
Michele

stephenw10 · Jan 24, 2012, 6:40 PM

Hmm, I'm not sure what you mean.
There were two distinct problems I experienced.
Firstly where multiple copies of the client ended up running. This happened immediately after either rebooting or restarting the service.
Secondly where LCDd ends up using 100% cpu. Just one client can be running when this happens.
I don't have much by way of logging. Can we increase the logging level in lcdd.conf?

Steve

mdima · Jan 24, 2012, 6:44 PM

@stephenw10:

Hmm, I'm not sure what you mean.
There were two distinct problems I experienced.
Firstly where multiple copies of the client ended up running. This happened immediately after either rebooting or restarting the service.

Ok, but thanks to the last change the "second client" was stopped after 2 more attempts to connect, I guess.

@stephenw10:

Secondly where LCDd ends up using 100% cpu. Just one client can be running when this happens.
I don't have much by way of logging. Can we increase the logging level in lcdd.conf?

Ok, that's awesome. At least we know that the issue is not related to multiple instances of the client running, which in any case should not give any problem (since LCDd is made to support different clients in the same time), but for sure it is not a "resoruce leak" because of the PHP client bothering the system…

Thanks,
Michele

stephenw10 · Jan 24, 2012, 7:00 PM

Yes I agree two distinct situations.
The first should have been solved by your recent update, thanks!
Though as you say LCDd can support multiple clients. Interesting though that this doesn't arise with any other driver even though it should not happen.

Steve

jpsb · Jan 24, 2012, 10:58 PM

@jpsb:

@mdima:

@jpsb:

I just use the HD44780 in the driver list, and by connecting the display, the system log shows "LCD2USB device found"
The display is a little slow, and works best by refresh frequency on 5 or 10 sec.

Hi JPSB,
mmhhh… seems strange... how many screens you have active? Did you have the same problem also with the previous version?

Thanks,
Michele

Hi Michele

Yes I had the same problem with last version.
I only have "Interface Traffic" and "Load" running.
But the new version with options to change contrast, etc. makes the display is running perfectly.
I have not had any luck getting the new driver "hd44780 Fast" to run.
But as I said, it runs perfectly with the other changes.
So once again, many thanks for a fantastic job.

See my youtube video of the display:
http://www.youtube.com/watch?v=moL-x1HpPew&feature=autoplay&list=ULmoL-x1HpPew&lf=mfu_in_order&playnext=46

Hi I having problem with the display running on the alix2d13 hardware.
After about 12 hours, the CPU load runs at 100%, and Pfsense freezer.
I have tried with multiple modules loaded but it makes no difference.

PFSense 2.0.1
alix2d13
4Gb CF-card
MiniPCI vpn1411 encryption accelerator
U204FB-A1 20x4 Display

mdima · Jan 25, 2012, 5:42 AM

What if we roll back to 0.5.4 (now the package works with 0.5.5)??

stephenw10 · Jan 25, 2012, 12:34 PM

I've been testing using 0.54 versions of LCDd and sdeclcd.so and there is no difference. As I type this I'm unable to access my box.
The fact that jpsb is experiencing a similar problem with a different driver is alarming.

I haven't tried going back to pfSense 2.0 yet. :-\

Still testing…

Steve

mdima · Jan 25, 2012, 1:11 PM

Steve,
can you try to use the LCDproc package (I mean not the "-dev" package) and see if the problem occurs?

I guess with the "LCDproc" package you didn't have any problem or you were not running it?

Thanks,
Michele