Too many nginx errors that result to 502 Bad Gateway

[joshslaton]

Good day, I am problems regard the killing of php-fpm and restarting the web configurator a lot of times just for me to be able to access the webpage properly again. I am using version 2.3.4 as of the moment and the only thing set up there are FreeRADIUS(freeradius2), captiveportal and dhcp server. I cannot seem to find why those are occuring and in huge numbers. If there is other details you want to see, please post them here and ill get back to you asap. The image below is just one of the many errors that keeps on occuring on the system logs.

[doktornotor]

It's clearly your CP and Android client connectivity checks…

[joshslaton]

Thanks for the reply, is there a patch for work around for this ? I did my tests before on iOS and android regarding CPs. For iOS the CP automatically pops up right away after connecting to the SSID. For the android, it will have to notify you "Sign-in to the network" then clicking that would open the browser then redirect to portal. There were no pfsense-nginx errors before, but as I serve more and more people, thats when it happened.

[doktornotor]

@joshslaton:

There were no pfsense-nginx errors before, but as I serve more and more people, thats when it happened.

Well then it sounds like you need to bump the number of PHP-FPM processes quite a bit… There's no GUI for this, some patching required for /etc/rc.php_ini_setup. Play with this:

https://github.com/pfsense/pfsense/blob/RELENG_2_3_4/src/etc/rc.php_ini_setup#L310
https://github.com/pfsense/pfsense/blob/RELENG_2_3_4/src/etc/rc.php_ini_setup#L353

[joshslaton]

thanks doc, let me try that and observe for few hours, then i will get back to you.

[joshslaton]

Until what number can I increase those ?

[joshslaton]

Hi doktornotor, may I add another. Where can i find the database for this one. Its definitely not captiveportalwifi.db, I was going to try to extract all the mac addresses there for later use, if possible, without copy pasting from the  web GUI itself. Maybe its on a form of a *.db somewhere ?

[doktornotor]

Sorry, I have no idea about what database you are asking, plus it's just totally off-topic here. Start a new thread in the proper forum section.

P.S. And no, I don't have any magic numbers for number of processes/children etc., you need to play with those yourself to match your (unknown) number of users and usage patterns.

[Gertjan]

@joshslaton:

Hi doktornotor, may I add another. Where can i find the database for this one. Its definitely not captiveportalwifi.db, ….

At the same place as all the other settings : export your "config.xml" and you'll find them.
pfSense loads the MAC's in an "ipfw" table, as shown here.

[ashima]

I had similar issue…. I added this line

kern.ipc.somaxconn="4096"

in /boot/loader.conf and rebooted.... error is not repeating.

I hope this helps.

Ashima

[joshslaton]

Thanks for the suggestion guys, I will try that after I fix my pfsense. Right now, I am missing php-fpm.pid for some reason, im looking for solutions right now

[Gertjan]

@joshslaton:

…  Right now, I am missing php-fpm.pid for some reason, im looking for solutions right now

That's important info.
It should be here : /var/run (mine is - as is the socket).
This means php "dies" not very properly : a programmed kill should also wip de PID file. If PHP disappears for other reasons, the PID would persist and becomes a ghost file : the PID number in the PID file points to nothing anymore - the process isn't running.
There are no x hundreds reasons why PHP dies : it gets overload by requests and/or runs out of memory.

[joshslaton]

What I can do right now is try to recreate the problem again.

• Make a new gateway containing the same stuff. Captive portal, freeradius2.
• With the number of users I have, I am sure to get those warnings/errors from nginx saying connection refused going to some IP that they are trying to connect to while using the leases from the DHCP yet, unable to login in the portal, because there is an option like that on the clients side, that if you were able to access the captive portal, and if you try to cancel it, it will ask you if 1) Disconnect to the network, 2) Remain connected to the network, although it will not give you internet access.

While looking for config files that I can copy so I can give an "appropriate" number of workers, master process. I must did something at the 2 php-fpm.conf files located at:

• /usr/local/etc/php-fpm.conf
• /usr/local/lib/php-fpm.conf
  But unfortunately when I tried putting back the back up, the problem where it says failed to exec php-fpm fails.

After I restarted the whole gateway, it seems to not find the PID, which actually does not really exist because there was an error that says, failed to exec. As much as I want to test the gateway right now, I have hundreds of users. Maybe I will try it out after shift so I do not cause inconvinience to my users.@ashima:

I had similar issue…. I added this line

kern.ipc.somaxconn="4096"

in /boot/loader.conf and rebooted.... error is not repeating.

I hope this helps.

Ashima

I will try this later on, see if it helps

[seanr22a]

I'm seeing the same thing - 502 errors. I'm running three sites all of them worked perfect until 2.4.0-RC changed from bsd 11.0 to bsd 11.1 (a few days ago)
Strange thing is that it's only one site that have the problem. The problem site is the site with most users.