Is there a way to backrev to 2.2.5? 2.2.6 is no good for me… cp dies often

carzin

Ok, so I am glad other people are experiencing this. I forgot about my post. What is interesting is that when the portal dies, there is nothing in the log. It just stops logging. But everything else is functioning. I actually decided to go entirely crazy, and in a production environment, I upgraded to the 2.3 experimental track when the 2.2.6 was having to be rebooted MULTIPLE times in a day. it was easier for me to snapshot (on VM), the server, then upgrade and see. I had to add a 'zone' value in the login page, however.

I upgraded to whatever the daily build was last Thursday, and the server has been running without issue since then. I am going to declare victory if I make it through tomorrow.

I can only hope whatever bug is in 2.2.6 does not make it to the 2.3 track.

To answer some of the questions…

I run 4 pfsense boxes on VMware. I have given them 4 cores and 8 gigs of memory (plenty). When the service dies, if I person connects to our onboarding SSID, the DNS responder correctly gives the person the pfsense IP address for whatever DNS entry they are looking up, then the connection just spins. No page is launched. I can't tell you for sure what happens to pre-existing connections. We have 10 minute time outs, and by the time I discover the problem, the 10 minutes is up and captive portal status reports 0 users. I am not going to run this on physical hardware, because in our environment, it can be just too problematic. Since I run about 200 VMs just for my group, and nothing else has issues, I am going to say this is a pfSense issue, not a platform issue (I have about 60 servers on FreeBSD). This is not a rinky dink environment, and it is on a platform that cost millions. In the past versions, lightd would die periodically (maybe once a month), and it would be logged to splunk and I would get an alert and reboot the box. It didn't happen enough to really bother me (this is free, so what can you do?). But with 2.2.6, it was dying in a way that I got no messages in my log, and I would eventually get calls from the helpdesk. It was so bad, I was considering pulling out the entire service and doing something else.

users authenticate via an https portal which connects to a radius backend (off server) to authenticate.

This was the old trusty message I would look for to see if the captive portal died. The new 'dying' method doesn't do this:

network lighttpd 1 syslog IP_REMOVED Sun Jan 24 10:17:57 2016 splunkindex0p /var/log/SPOOL/networking-current/daemon-networking.log Jan 24 10:17:57 152.2.78.166 lighttpd[31698]: (mod_fastcgi.c.2912) connection was dropped after accept() (perhaps the fastcgi process died), write-offset: 8192 socket: unix:/tmp/php-fastcgi-cpzone.socket-5

Easter

Hi,
I was going to install the 2.2.5 version when I found this issue:
https://redmine.lighttpd.net/issues/2700
resolved with verion 1.4.39 https://www.lighttpd.net/2016/1/2/1.4.39/

The lighthttpd verion on pfSense 2.2.5 is 1.4.37 but on 2.2.6 I have lighttpd/1.4.38

Now I'm going to
pkg install lighttpd
trying with lighttpd/1.4.39

@carzin what is the lighttpd on your pfSense 2.3 ?

I Hope this will help.

@carzin:

Ok, so I am glad other people are experiencing this. I forgot about my post. What is interesting is that when the portal dies, there is nothing in the log. It just stops logging. But everything else is functioning. I actually decided to go entirely crazy, and in a production environment, I upgraded to the 2.3 experimental track when the 2.2.6 was having to be rebooted MULTIPLE times in a day. it was easier for me to snapshot (on VM), the server, then upgrade and see. I had to add a 'zone' value in the login page, however.

I upgraded to whatever the daily build was last Thursday, and the server has been running without issue since then. I am going to declare victory if I make it through tomorrow.

I can only hope whatever bug is in 2.2.6 does not make it to the 2.3 track.

To answer some of the questions…

I run 4 pfsense boxes on VMware. I have given them 4 cores and 8 gigs of memory (plenty). When the service dies, if I person connects to our onboarding SSID, the DNS responder correctly gives the person the pfsense IP address for whatever DNS entry they are looking up, then the connection just spins. No page is launched. I can't tell you for sure what happens to pre-existing connections. We have 10 minute time outs, and by the time I discover the problem, the 10 minutes is up and captive portal status reports 0 users. I am not going to run this on physical hardware, because in our environment, it can be just too problematic. Since I run about 200 VMs just for my group, and nothing else has issues, I am going to say this is a pfSense issue, not a platform issue (I have about 60 servers on FreeBSD). This is not a rinky dink environment, and it is on a platform that cost millions. In the past versions, lightd would die periodically (maybe once a month), and it would be logged to splunk and I would get an alert and reboot the box. It didn't happen enough to really bother me (this is free, so what can you do?). But with 2.2.6, it was dying in a way that I got no messages in my log, and I would eventually get calls from the helpdesk. It was so bad, I was considering pulling out the entire service and doing something else.

users authenticate via an https portal which connects to a radius backend (off server) to authenticate.

This was the old trusty message I would look for to see if the captive portal died. The new 'dying' method doesn't do this:

network lighttpd 1 syslog IP_REMOVED Sun Jan 24 10:17:57 2016 splunkindex0p /var/log/SPOOL/networking-current/daemon-networking.log Jan 24 10:17:57 152.2.78.166 lighttpd[31698]: (mod_fastcgi.c.2912) connection was dropped after accept() (perhaps the fastcgi process died), write-offset: 8192 socket: unix:/tmp/php-fastcgi-cpzone.socket-5

heper

lighttpd has been removed from 2.3

carzin

Easter: Thank you very much for figuring that out. On the 3 remaining the version 1.4.38 is installed.

I am glad this package is gone with 2.3. It has been the source of much grief for me.

Easter

Hi carzin,
after more than 20 hours since lighttpd-1.4.39_1 installed, there are no crashes and now the CP is handling more than 1500 users logged-in concurrently.
On Monday I can provide more data, anyway now I think the issue could be resolved this way.

The sequence I used are:

pkg upgrade

(just to install/upgrade pkg binaries)

pkg install lighttpd

on pfsense console menu choose: 11) Restart webConfigurator
(to restart lighttpd processes without to force log-out users)

@carzin:

Easter: Thank you very much for figuring that out. On the 3 remaining the version 1.4.38 is installed.

I am glad this package is gone with 2.3. It has been the source of much grief for me.

carzin

Thank you very much! I've done this on all my boxes. Time will tell. I'll just keep hoping that my box running 2.3 continues to do so without issue :)

Easter

Hi carzin,
my setup is working well uninterrupted since the lighttpd "upgrade"

I hope will be released an official update for pfSense CaptivePortal.

How to report this bug to packager?

@carzin:

Thank you very much! I've done this on all my boxes. Time will tell. I'll just keep hoping that my box running 2.3 continues to do so without issue :)

cmb

It's an issue in lighttpd itself, which we won't do anything about since lighttpd's removed from the next release entirely (because they kept putting out buggy releases, like this as one example of multiple ones in the past several months). So no need to report it, it's already fixed going forward.

carzin

Thanks, cmb. We have always had some issues with captive portal stability, seemingly linked to the lighttpd process. I usually get a call from the helpdesk once every month or two when the portal stops working and I restart the server. It went crazy with the latest release, to the point that I was thinking of abandoning pfsense. The beta 2.3 build has been super smooth in the nearly 2 weeks it has been up. When do you anticipate a proper release?

heper

@carzin:

When do you anticipate a proper release?

Its beta_x now. no clue if there is going to be a beta_y or z. then there will be atleast one or more release candidates.
i doubt they have set a fixed release date at this point. i'm hoping/guessing it'll be between april & june somewhere.

denis31

Hi,

Same issues for me (lighttpd core dump).
Sometimes several times per day or a few times per week.
Already authenticated users are OK but no captive portal page for newcomers.
I am also trying the lighttpd upgrade to 1.4.39_1… Time will says if it does the job.

What will be lighttpd replacement in PfSense 2.3 ?

DM

21hertz

Same problem here.

Happens a few times a week, seems random.
Services status still reports captiveportal as Up so the only chance to know if its down is if a client user is reporting or if you use external monitoring on the CP's http/https.

Gertjan

@denis31:

What will be lighttpd replacement in PfSense 2.3 ?

Beta section of the forum https://forum.pfsense.org/index.php?board=65.0 : third line ….

cmb

@21hertz:

Same problem here.

Happens a few times a week, seems random.
Services status still reports captiveportal as Up so the only chance to know if its down is if a client user is reporting or if you use external monitoring on the CP's http/https.

Try upgrading lighttpd as others have done successfully, 'pkg install lighttpd', then option 16 followed by option 11 at the console.

denis31

@21hertz:

Same problem here.

Happens a few times a week, seems random.
Services status still reports captiveportal as Up so the only chance to know if its down is if a client user is reporting or if you use external monitoring on the CP's http/https.

You can check this using CLI (ssh public key auth for automation):
ps -aux | grep http
root 13324 7.0 1.4 169520 115368 - S 2:52PM 57:12.67 /usr/local/sbin/lighttpd -f /var/etc/lighty-ssid_ups-CaptivePortal-SSL.co
root 11329 0.0 0.1 56884 7736 - S 2:52PM 0:40.39 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
root 13040 0.0 0.2 67120 17404 - S 2:52PM 3:42.73 /usr/local/sbin/lighttpd -f /var/etc/lighty-ssid_ups-CaptivePortal.conf

Or checking log file for Lighttpd 'Core dump' error.

Denis.

perjoh

Does anyone know if this problem i solved in 2.3 ?

Gertjan

Well ….. knowing that "lighttpd" isn't part of pfSEnse 2.2.3 anymore ....... ;)

2.2.3 works fine for me ....

Oops : I meant 2.3.

sebastiannielsen

Gertjan: The new version is named 2.3
The version you are referring to, is prior to 2.2.5 and 2.2.6 and still contains lightppd.

cmb

Yes it is solved in 2.3. lighttpd no longer exists, that's been switched to nginx which has no such problems.